Bug 14141

Summary: devtool modify fails with submodules
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Ross Burton <ross.burton>
Component: devtools / tool chainAssignee: Julien Stephan <jstephan>
Status: RESOLVED FIXED QA Contact:
Severity: enhancement    
Priority: Medium+ CC: bluelightning, jon.mason, jstephan, meta.mr.watcher, meta.watcher, prabin.ca, randy.macleod, richard.purdie, sgw
Version: 3.4   
Target Milestone: 5.0 M1   
Hardware: x86   
OS: Multiple   
Whiteboard: backport 4.0.x
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Ross Burton 2020-12-03 14:09:27 UTC
If a recipe uses submodules and patches a file in a submodule, 'devtool modify' fails:

Exception: bb.process.ExecutionError: Execution of 'git add .; git -c user.name="OpenEmbedded" -c user.email="oe.patch@oe" commit -a -m "Committing changes from do_patch

%% ignore - from do_patch"' failed with exit code 1:
On branch devtool
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
  (commit or discard the untracked or modified content in submodules)
	modified:   edk2-platforms (modified content)

no changes added to commit (use "git add" and/or "git commit -a")


Inside S:

$ git status
On branch devtool
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
  (commit or discard the untracked or modified content in submodules)
	modified:   edk2-platforms (modified content)

Inside S/edk2-platforms:
$ git status
HEAD detached at d9d32e6c
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
	modified:   Platform/ARM/Drivers/CcixDxe/CcixEnumeration.c
	modified:   Platform/ARM/N1SdpPkg/Drivers/PlatformDxe/PlatformDxe.c
Comment 1 Randy MacLeod 2020-12-03 15:44:20 UTC
We should at least make devtool generate an error since it can't handle submodules.
Comment 2 Ross Burton 2020-12-08 16:28:00 UTC
So the initial root cause is bug 14146, which is part of patch.bbclass.
Comment 3 Randy MacLeod 2021-04-01 14:47:52 UTC
*** Bug 14333 has been marked as a duplicate of this bug. ***
Comment 4 Ross Burton 2022-04-28 15:04:21 UTC
Need to verify this is still an issue.
Comment 5 Ross Burton 2022-07-14 12:55:16 UTC
Passing to Saul as the new devtool maintainer.  We need to verify this is still an issue and write a test case though.
Comment 6 David Reyna 2023-06-22 09:41:37 UTC
I have a new intern to help resolve this issue.
Comment 7 Julien Stephan 2023-08-23 13:52:14 UTC
While trying to reproduce this issue I didn't get the error described here (so I guess it is not relevant anymore), but I got another one (see bellow) with the exact same use case: use devtool modify on a recipe that tries to patch files on submodules.

I reproduced this issue with at least two different recipes: vulkan-samples (poky) and hafnium (meta-arm).
I sent a patch to fix it: https://lists.openembedded.org/g/openembedded-core/message/186579 

while doing so, I also found another bug with vulkan-sample and fixed it (https://lists.openembedded.org/g/openembedded-core/message/186580)


ERROR: Error executing a python function in exec_func_python() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:patch_do_patch(d)
     0003:
File: '<..>/poky/meta/classes-global/patch.bbclass', lineno: 157, function: patch_do_patch
     0153:        except Exception as exc:
     0154:            bb.utils.remove(process_tmpdir, True)
     0155:            bb.fatal("Importing patch '%s' with striplevel '%s'\n%s" % (parm['patchname'], parm['striplevel'], repr(exc).replace("\\n", "\n")))
     0156:        try:
 *** 0157:            resolver.Resolve()
     0158:        except bb.BBHandledException as e:
     0159:            bb.utils.remove(process_tmpdir, True)
     0160:            bb.fatal("Applying patch '%s' on target directory '%s'\n%s" % (parm['patchname'], patchdir, repr(e).replace("\\n", "\n")))
     0161:
File: '<..>/poky/meta/lib/oe/patch.py', lineno: 769, function: Resolve
     0765:    def Resolve(self):
     0766:        olddir = os.path.abspath(os.curdir)
     0767:        os.chdir(self.patchset.dir)
     0768:        try:
 *** 0769:            self.patchset.Push()
     0770:        except Exception:
     0771:            import sys
     0772:            os.chdir(olddir)
     0773:            raise
File: '<..>/poky/meta/lib/oe/patch.py', lineno: 274, function: Push
     0270:            else:
     0271:                next = 0
     0272:
     0273:            bb.note("applying patch %s" % self.patches[next])
 *** 0274:            ret = self._applypatch(self.patches[next], force)
     0275:
     0276:            self._current = next
     0277:            return ret
     0278:
File: '<..>/poky/meta/lib/oe/patch.py', lineno: 556, function: _applypatch
     0552:        if os.path.lexists(hooks_dir_backup):
     0553:            raise Exception("Git hooks backup directory already exists: %s" % hooks_dir_backup)
     0554:        if os.path.lexists(hooks_dir):
     0555:            shutil.move(hooks_dir, hooks_dir_backup)
 *** 0556:        os.mkdir(hooks_dir)
     0557:        commithook = os.path.join(hooks_dir, 'commit-msg')
     0558:        applyhook = os.path.join(hooks_dir, 'applypatch-msg')
     0559:        with open(commithook, 'w') as f:
     0560:            # NOTE: the formatting here is significant; if you change it you'll also need to
Exception: NotADirectoryError: [Errno 20] Not a directory: '<..>/build/tmp/work/core2-64-poky-linux/vulkan-samples/git/devtooltmp-n87_zx1i/workdir/git/third_party/spdlog/.git/hooks'
Comment 8 Julien Stephan 2023-10-13 07:53:11 UTC
Hi all,
The fix was merged so closing this issue: https://git.yoctoproject.org/poky/commit/?id=060f285e58509fd1f38bc4eca990d4643e648c1a
Comment 9 Jon Mason 2023-10-13 15:49:38 UTC
devtool is still not working as I'm expecting.  For example, I'm working on hafnium in meta-arm.  After running it through devtool (see below), it removes the patches to the git submodules, and throws them in a single commit, and that commit is just modified SHAs.




$ devtool modify hafnium
...
$ devtool finish --force-patch-refresh hafnium meta-arm
NOTE: Starting bitbake server...
Loading cache: 100% |#############################################################################################################################################################################################################################################| Time: 0:00:00
Loaded 1949 entries from dependency cache.
Parsing recipes: 100% |###########################################################################################################################################################################################################################################| Time: 0:00:06
Parsing of 992 .bb files complete (991 cached, 1 parsed). 1949 targets, 206 skipped, 0 masked, 0 errors.
INFO: Updating patch 0001-arm-hafnium-fix-kernel-tool-linking.patch
INFO: Updating patch 0002-Fix-build-with-clang-15.patch
INFO: Adding new patch 0003-Committing-changes-from-do_patch.patch
INFO: Updating recipe hafnium_2.8.bb
INFO: Removing file /builder/meta-arm/build/../meta-arm/recipes-bsp/hafnium/hafnium/0001-Use-pkg-config-native-to-find-the-libssl-headers.patch
INFO: Removing file /builder/meta-arm/build/../meta-arm/recipes-bsp/hafnium/hafnium/0001-work-around-visibility-issue.patch
INFO: Cleaning sysroot for recipe hafnium...
INFO: Preserving source tree in /builder/meta-arm/build/workspace/attic/sources/hafnium.20231013154057
If you no longer need it then please delete it manually.
It is also possible to reuse it via devtool source tree argument.




(snip from git diff of hafnium_2.8.bb)

-           file://0001-Use-pkg-config-native-to-find-the-libssl-headers.patch;patchdir=third_party/linux \
-           file://0001-work-around-visibility-issue.patch;patchdir=third_party/dtc \
-          "
+           file://0003-Committing-changes-from-do_patch.patch \



$ cat meta-arm/recipes-bsp/hafnium/hafnium/0003-Committing-changes-from-do_patch.patch
From 0aa482233005e53ee1779140dead5a057b2855a4 Mon Sep 17 00:00:00 2001
From: OpenEmbedded <oe.patch@oe>
Date: Fri, 13 Oct 2023 15:39:09 +0000
Subject: [PATCH] Committing changes from do_patch

---
 third_party/dtc   | 2 +-
 third_party/linux | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/third_party/dtc b/third_party/dtc
index 36bdb76f..db81f757 160000
--- a/third_party/dtc
+++ b/third_party/dtc
@@ -1 +1 @@
-Subproject commit 36bdb76f62a80d0e1bb1c55aaff962311256ff6c
+Subproject commit db81f757ca14deb036feb4fae3ed040e72d3dc87
diff --git a/third_party/linux b/third_party/linux
index 157378f4..c3e6c1cf 160000
--- a/third_party/linux
+++ b/third_party/linux
@@ -1 +1 @@
-Subproject commit 157378f43faad830e4aa3815bde5fa9f9a9f5be6
+Subproject commit c3e6c1cf430dbfc51a8a5feca66e408b6b63e60e
Comment 10 Julien Stephan 2023-11-08 11:17:23 UTC
Hi Jon, all

I eventually managed to have a fix for that. It is on my dev branch here: https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/devtool-submodule-fix 

Can you try if this fix the issue you are facing? 

This is not yet ready for a proper submission, here is what is missing:
* devtool upgrade on gitsm recipes (ongoing right now)
* need to try with --initial-rev and --branch parameters
* .. maybe I am missing something else? 

I tried to do a lot of testing with various combination of parameters (like doing a finish on the same layer/ on another layer ... ) but I may be missing some corner cases. So any feedback would be much appreciated :)
Comment 11 Julien Stephan 2023-11-08 11:19:11 UTC
(In reply to Julien Stephan from comment #10)
> Hi Jon, all
> 
> I eventually managed to have a fix for that. It is on my dev branch here:
> https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/devtool-submodule-
> fix 
> 
> Can you try if this fix the issue you are facing? 
> 
> This is not yet ready for a proper submission, here is what is missing:
> * devtool upgrade on gitsm recipes (ongoing right now)
> * need to try with --initial-rev and --branch parameters
> * .. maybe I am missing something else? 
> 
> I tried to do a lot of testing with various combination of parameters (like
> doing a finish on the same layer/ on another layer ... ) but I may be
> missing some corner cases. So any feedback would be much appreciated :)

Also I added a first selftest for submodule, maybe I will add more for other scenario.

Also, pass a full devtool selftest locally so at least devtool should not be broken :)
Comment 12 Jon Mason 2023-11-08 20:07:13 UTC
This is working as expected for Hafnium.  Good job.

I'm seeing a different issue with edk2 (also present in meta-arm).  I'm seeing:
fatal: No url found for submodule path 'edk2-platforms' in .gitmodules


However, in the recipe the path is being specified.  See 
https://git.yoctoproject.org/meta-arm/tree/meta-arm/recipes-bsp/uefi/edk2-firmware.inc#n15

I believe there is a disconnect between the submodules that are expected to be present in the git tree, as opposed to the ones being specified in the recipe.  I believe the way edk2 is doing it is acceptable.  So, having this supported would be necessary.
Comment 13 Julien Stephan 2023-11-10 09:33:42 UTC
(In reply to Jon Mason from comment #12)
> This is working as expected for Hafnium.  Good job.
> 
> I'm seeing a different issue with edk2 (also present in meta-arm).  I'm
> seeing:
> fatal: No url found for submodule path 'edk2-platforms' in .gitmodules
> 
> 
> However, in the recipe the path is being specified.  See 
> https://git.yoctoproject.org/meta-arm/tree/meta-arm/recipes-bsp/uefi/edk2-
> firmware.inc#n15
> 
> I believe there is a disconnect between the submodules that are expected to
> be present in the git tree, as opposed to the ones being specified in the
> recipe.  I believe the way edk2 is doing it is acceptable.  So, having this
> supported would be necessary.

Hi Jon,

Thank you for testing it! 

I indeed reproduced the issue with edk2, but this is not only restricted to gitsm recipes. All recipes using SRC_URI such as: 

SRC_URI = " git://<primarysource> git://<someothersource>;destsuffix=git/<someothersource>"

will not work with devtool modify/finish (issue is already present in master). 

I will try to update my branch to fix this, and let you know.

Cheers
Julien
Comment 14 Julien Stephan 2023-11-22 12:33:53 UTC
Hi Jon, I just force pushed my branch on poky-contrib and sent a series to fix the submodue (including submodules declared into SRC_URI). 

Can you test it? 

Thank you
Cheers

Julien
Comment 16 Randy MacLeod 2025-02-14 16:46:35 UTC
bulk change to add Saul's non-WR address.
Comment 17 Randy MacLeod 2025-02-14 16:48:22 UTC
Bulk change: Remove Saul's old WR address.