If a recipe uses submodules and patches a file in a submodule, 'devtool modify' fails: Exception: bb.process.ExecutionError: Execution of 'git add .; git -c user.name="OpenEmbedded" -c user.email="oe.patch@oe" commit -a -m "Committing changes from do_patch %% ignore - from do_patch"' failed with exit code 1: On branch devtool Changes not staged for commit: (use "git add <file>..." to update what will be committed) (use "git restore <file>..." to discard changes in working directory) (commit or discard the untracked or modified content in submodules) modified: edk2-platforms (modified content) no changes added to commit (use "git add" and/or "git commit -a") Inside S: $ git status On branch devtool Changes not staged for commit: (use "git add <file>..." to update what will be committed) (use "git restore <file>..." to discard changes in working directory) (commit or discard the untracked or modified content in submodules) modified: edk2-platforms (modified content) Inside S/edk2-platforms: $ git status HEAD detached at d9d32e6c Changes not staged for commit: (use "git add <file>..." to update what will be committed) (use "git restore <file>..." to discard changes in working directory) modified: Platform/ARM/Drivers/CcixDxe/CcixEnumeration.c modified: Platform/ARM/N1SdpPkg/Drivers/PlatformDxe/PlatformDxe.c
We should at least make devtool generate an error since it can't handle submodules.
So the initial root cause is bug 14146, which is part of patch.bbclass.
*** Bug 14333 has been marked as a duplicate of this bug. ***
Need to verify this is still an issue.
Passing to Saul as the new devtool maintainer. We need to verify this is still an issue and write a test case though.
I have a new intern to help resolve this issue.
While trying to reproduce this issue I didn't get the error described here (so I guess it is not relevant anymore), but I got another one (see bellow) with the exact same use case: use devtool modify on a recipe that tries to patch files on submodules. I reproduced this issue with at least two different recipes: vulkan-samples (poky) and hafnium (meta-arm). I sent a patch to fix it: https://lists.openembedded.org/g/openembedded-core/message/186579 while doing so, I also found another bug with vulkan-sample and fixed it (https://lists.openembedded.org/g/openembedded-core/message/186580) ERROR: Error executing a python function in exec_func_python() autogenerated: The stack trace of python calls that resulted in this exception/failure was: File: 'exec_func_python() autogenerated', lineno: 2, function: <module> 0001: *** 0002:patch_do_patch(d) 0003: File: '<..>/poky/meta/classes-global/patch.bbclass', lineno: 157, function: patch_do_patch 0153: except Exception as exc: 0154: bb.utils.remove(process_tmpdir, True) 0155: bb.fatal("Importing patch '%s' with striplevel '%s'\n%s" % (parm['patchname'], parm['striplevel'], repr(exc).replace("\\n", "\n"))) 0156: try: *** 0157: resolver.Resolve() 0158: except bb.BBHandledException as e: 0159: bb.utils.remove(process_tmpdir, True) 0160: bb.fatal("Applying patch '%s' on target directory '%s'\n%s" % (parm['patchname'], patchdir, repr(e).replace("\\n", "\n"))) 0161: File: '<..>/poky/meta/lib/oe/patch.py', lineno: 769, function: Resolve 0765: def Resolve(self): 0766: olddir = os.path.abspath(os.curdir) 0767: os.chdir(self.patchset.dir) 0768: try: *** 0769: self.patchset.Push() 0770: except Exception: 0771: import sys 0772: os.chdir(olddir) 0773: raise File: '<..>/poky/meta/lib/oe/patch.py', lineno: 274, function: Push 0270: else: 0271: next = 0 0272: 0273: bb.note("applying patch %s" % self.patches[next]) *** 0274: ret = self._applypatch(self.patches[next], force) 0275: 0276: self._current = next 0277: return ret 0278: File: '<..>/poky/meta/lib/oe/patch.py', lineno: 556, function: _applypatch 0552: if os.path.lexists(hooks_dir_backup): 0553: raise Exception("Git hooks backup directory already exists: %s" % hooks_dir_backup) 0554: if os.path.lexists(hooks_dir): 0555: shutil.move(hooks_dir, hooks_dir_backup) *** 0556: os.mkdir(hooks_dir) 0557: commithook = os.path.join(hooks_dir, 'commit-msg') 0558: applyhook = os.path.join(hooks_dir, 'applypatch-msg') 0559: with open(commithook, 'w') as f: 0560: # NOTE: the formatting here is significant; if you change it you'll also need to Exception: NotADirectoryError: [Errno 20] Not a directory: '<..>/build/tmp/work/core2-64-poky-linux/vulkan-samples/git/devtooltmp-n87_zx1i/workdir/git/third_party/spdlog/.git/hooks'
Hi all, The fix was merged so closing this issue: https://git.yoctoproject.org/poky/commit/?id=060f285e58509fd1f38bc4eca990d4643e648c1a
devtool is still not working as I'm expecting. For example, I'm working on hafnium in meta-arm. After running it through devtool (see below), it removes the patches to the git submodules, and throws them in a single commit, and that commit is just modified SHAs. $ devtool modify hafnium ... $ devtool finish --force-patch-refresh hafnium meta-arm NOTE: Starting bitbake server... Loading cache: 100% |#############################################################################################################################################################################################################################################| Time: 0:00:00 Loaded 1949 entries from dependency cache. Parsing recipes: 100% |###########################################################################################################################################################################################################################################| Time: 0:00:06 Parsing of 992 .bb files complete (991 cached, 1 parsed). 1949 targets, 206 skipped, 0 masked, 0 errors. INFO: Updating patch 0001-arm-hafnium-fix-kernel-tool-linking.patch INFO: Updating patch 0002-Fix-build-with-clang-15.patch INFO: Adding new patch 0003-Committing-changes-from-do_patch.patch INFO: Updating recipe hafnium_2.8.bb INFO: Removing file /builder/meta-arm/build/../meta-arm/recipes-bsp/hafnium/hafnium/0001-Use-pkg-config-native-to-find-the-libssl-headers.patch INFO: Removing file /builder/meta-arm/build/../meta-arm/recipes-bsp/hafnium/hafnium/0001-work-around-visibility-issue.patch INFO: Cleaning sysroot for recipe hafnium... INFO: Preserving source tree in /builder/meta-arm/build/workspace/attic/sources/hafnium.20231013154057 If you no longer need it then please delete it manually. It is also possible to reuse it via devtool source tree argument. (snip from git diff of hafnium_2.8.bb) - file://0001-Use-pkg-config-native-to-find-the-libssl-headers.patch;patchdir=third_party/linux \ - file://0001-work-around-visibility-issue.patch;patchdir=third_party/dtc \ - " + file://0003-Committing-changes-from-do_patch.patch \ $ cat meta-arm/recipes-bsp/hafnium/hafnium/0003-Committing-changes-from-do_patch.patch From 0aa482233005e53ee1779140dead5a057b2855a4 Mon Sep 17 00:00:00 2001 From: OpenEmbedded <oe.patch@oe> Date: Fri, 13 Oct 2023 15:39:09 +0000 Subject: [PATCH] Committing changes from do_patch --- third_party/dtc | 2 +- third_party/linux | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/third_party/dtc b/third_party/dtc index 36bdb76f..db81f757 160000 --- a/third_party/dtc +++ b/third_party/dtc @@ -1 +1 @@ -Subproject commit 36bdb76f62a80d0e1bb1c55aaff962311256ff6c +Subproject commit db81f757ca14deb036feb4fae3ed040e72d3dc87 diff --git a/third_party/linux b/third_party/linux index 157378f4..c3e6c1cf 160000 --- a/third_party/linux +++ b/third_party/linux @@ -1 +1 @@ -Subproject commit 157378f43faad830e4aa3815bde5fa9f9a9f5be6 +Subproject commit c3e6c1cf430dbfc51a8a5feca66e408b6b63e60e
Hi Jon, all I eventually managed to have a fix for that. It is on my dev branch here: https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/devtool-submodule-fix Can you try if this fix the issue you are facing? This is not yet ready for a proper submission, here is what is missing: * devtool upgrade on gitsm recipes (ongoing right now) * need to try with --initial-rev and --branch parameters * .. maybe I am missing something else? I tried to do a lot of testing with various combination of parameters (like doing a finish on the same layer/ on another layer ... ) but I may be missing some corner cases. So any feedback would be much appreciated :)
(In reply to Julien Stephan from comment #10) > Hi Jon, all > > I eventually managed to have a fix for that. It is on my dev branch here: > https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/devtool-submodule- > fix > > Can you try if this fix the issue you are facing? > > This is not yet ready for a proper submission, here is what is missing: > * devtool upgrade on gitsm recipes (ongoing right now) > * need to try with --initial-rev and --branch parameters > * .. maybe I am missing something else? > > I tried to do a lot of testing with various combination of parameters (like > doing a finish on the same layer/ on another layer ... ) but I may be > missing some corner cases. So any feedback would be much appreciated :) Also I added a first selftest for submodule, maybe I will add more for other scenario. Also, pass a full devtool selftest locally so at least devtool should not be broken :)
This is working as expected for Hafnium. Good job. I'm seeing a different issue with edk2 (also present in meta-arm). I'm seeing: fatal: No url found for submodule path 'edk2-platforms' in .gitmodules However, in the recipe the path is being specified. See https://git.yoctoproject.org/meta-arm/tree/meta-arm/recipes-bsp/uefi/edk2-firmware.inc#n15 I believe there is a disconnect between the submodules that are expected to be present in the git tree, as opposed to the ones being specified in the recipe. I believe the way edk2 is doing it is acceptable. So, having this supported would be necessary.
(In reply to Jon Mason from comment #12) > This is working as expected for Hafnium. Good job. > > I'm seeing a different issue with edk2 (also present in meta-arm). I'm > seeing: > fatal: No url found for submodule path 'edk2-platforms' in .gitmodules > > > However, in the recipe the path is being specified. See > https://git.yoctoproject.org/meta-arm/tree/meta-arm/recipes-bsp/uefi/edk2- > firmware.inc#n15 > > I believe there is a disconnect between the submodules that are expected to > be present in the git tree, as opposed to the ones being specified in the > recipe. I believe the way edk2 is doing it is acceptable. So, having this > supported would be necessary. Hi Jon, Thank you for testing it! I indeed reproduced the issue with edk2, but this is not only restricted to gitsm recipes. All recipes using SRC_URI such as: SRC_URI = " git://<primarysource> git://<someothersource>;destsuffix=git/<someothersource>" will not work with devtool modify/finish (issue is already present in master). I will try to update my branch to fix this, and let you know. Cheers Julien
Hi Jon, I just force pushed my branch on poky-contrib and sent a series to fix the submodue (including submodules declared into SRC_URI). Can you test it? Thank you Cheers Julien
https://git.yoctoproject.org/poky/commit/?id=89f16624842a91e063dc9e6b98787d44c55e4900
bulk change to add Saul's non-WR address.
Bulk change: Remove Saul's old WR address.