<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!DOCTYPE bugzilla SYSTEM "https://bugzilla.yoctoproject.org/page.cgi?id=bugzilla.dtd">

<bugzilla version="5.0.6"
          urlbase="https://bugzilla.yoctoproject.org/"
          
          maintainer="it-coreprojects-helpdesk@linuxfoundation.org"
>

    <bug>
          <bug_id>14141</bug_id>
          
          <creation_ts>2020-12-03 14:09:27 +0000</creation_ts>
          <short_desc>devtool modify fails with submodules</short_desc>
          <delta_ts>2025-02-14 16:48:22 +0000</delta_ts>
          <reporter_accessible>1</reporter_accessible>
          <cclist_accessible>1</cclist_accessible>
          <classification_id>7</classification_id>
          <classification>Build System, Metadata &amp; Runtime</classification>
          <product>OE-Core</product>
          <component>devtools / tool chain</component>
          <version>3.4</version>
          <rep_platform>x86</rep_platform>
          <op_sys>Multiple</op_sys>
          <bug_status>RESOLVED</bug_status>
          <resolution>FIXED</resolution>
          
          
          <bug_file_loc></bug_file_loc>
          <status_whiteboard>backport 4.0.x</status_whiteboard>
          <keywords></keywords>
          <priority>Medium+</priority>
          <bug_severity>enhancement</bug_severity>
          <target_milestone>5.0 M1</target_milestone>
          
          
          <everconfirmed>1</everconfirmed>
          <reporter name="Ross Burton">ross.burton</reporter>
          <assigned_to name="Julien Stephan">jstephan</assigned_to>
          <cc>bluelightning</cc>
    
    <cc>jon.mason</cc>
    
    <cc>jstephan</cc>
    
    <cc>meta.mr.watcher</cc>
    
    <cc>meta.watcher</cc>
    
    <cc>prabin.ca</cc>
    
    <cc>randy.macleod</cc>
    
    <cc>richard.purdie</cc>
    
    <cc>sgw</cc>
          
          
          <cf_os>---</cf_os>
          <cf_regression_type>---</cf_regression_type>
          
          <cf_docchange>No (bug/feature does not impact docs)</cf_docchange>

      

      

      

          <comment_sort_order>oldest_to_newest</comment_sort_order>  
          <long_desc isprivate="0" >
    <commentid>88708</commentid>
    <comment_count>0</comment_count>
    <who name="Ross Burton">ross.burton</who>
    <bug_when>2020-12-03 14:09:27 +0000</bug_when>
    <thetext>If a recipe uses submodules and patches a file in a submodule, &apos;devtool modify&apos; fails:

Exception: bb.process.ExecutionError: Execution of &apos;git add .; git -c user.name=&quot;OpenEmbedded&quot; -c user.email=&quot;oe.patch@oe&quot; commit -a -m &quot;Committing changes from do_patch

%% ignore - from do_patch&quot;&apos; failed with exit code 1:
On branch devtool
Changes not staged for commit:
  (use &quot;git add &lt;file&gt;...&quot; to update what will be committed)
  (use &quot;git restore &lt;file&gt;...&quot; to discard changes in working directory)
  (commit or discard the untracked or modified content in submodules)
	modified:   edk2-platforms (modified content)

no changes added to commit (use &quot;git add&quot; and/or &quot;git commit -a&quot;)


Inside S:

$ git status
On branch devtool
Changes not staged for commit:
  (use &quot;git add &lt;file&gt;...&quot; to update what will be committed)
  (use &quot;git restore &lt;file&gt;...&quot; to discard changes in working directory)
  (commit or discard the untracked or modified content in submodules)
	modified:   edk2-platforms (modified content)

Inside S/edk2-platforms:
$ git status
HEAD detached at d9d32e6c
Changes not staged for commit:
  (use &quot;git add &lt;file&gt;...&quot; to update what will be committed)
  (use &quot;git restore &lt;file&gt;...&quot; to discard changes in working directory)
	modified:   Platform/ARM/Drivers/CcixDxe/CcixEnumeration.c
	modified:   Platform/ARM/N1SdpPkg/Drivers/PlatformDxe/PlatformDxe.c</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>88714</commentid>
    <comment_count>1</comment_count>
    <who name="Randy MacLeod">randy.macleod</who>
    <bug_when>2020-12-03 15:44:20 +0000</bug_when>
    <thetext>We should at least make devtool generate an error since it can&apos;t handle submodules.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>88733</commentid>
    <comment_count>2</comment_count>
    <who name="Ross Burton">ross.burton</who>
    <bug_when>2020-12-08 16:28:00 +0000</bug_when>
    <thetext>So the initial root cause is bug 14146, which is part of patch.bbclass.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>89847</commentid>
    <comment_count>3</comment_count>
    <who name="Randy MacLeod">randy.macleod</who>
    <bug_when>2021-04-01 14:47:52 +0000</bug_when>
    <thetext>*** Bug 14333 has been marked as a duplicate of this bug. ***</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>93206</commentid>
    <comment_count>4</comment_count>
    <who name="Ross Burton">ross.burton</who>
    <bug_when>2022-04-28 15:04:21 +0000</bug_when>
    <thetext>Need to verify this is still an issue.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>93666</commentid>
    <comment_count>5</comment_count>
    <who name="Ross Burton">ross.burton</who>
    <bug_when>2022-07-14 12:55:16 +0000</bug_when>
    <thetext>Passing to Saul as the new devtool maintainer.  We need to verify this is still an issue and write a test case though.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>95788</commentid>
    <comment_count>6</comment_count>
    <who name="David Reyna">david.reyna</who>
    <bug_when>2023-06-22 09:41:37 +0000</bug_when>
    <thetext>I have a new intern to help resolve this issue.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>96293</commentid>
    <comment_count>7</comment_count>
    <who name="Julien Stephan">jstephan</who>
    <bug_when>2023-08-23 13:52:14 +0000</bug_when>
    <thetext>While trying to reproduce this issue I didn&apos;t get the error described here (so I guess it is not relevant anymore), but I got another one (see bellow) with the exact same use case: use devtool modify on a recipe that tries to patch files on submodules.

I reproduced this issue with at least two different recipes: vulkan-samples (poky) and hafnium (meta-arm).
I sent a patch to fix it: https://lists.openembedded.org/g/openembedded-core/message/186579 

while doing so, I also found another bug with vulkan-sample and fixed it (https://lists.openembedded.org/g/openembedded-core/message/186580)


ERROR: Error executing a python function in exec_func_python() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: &apos;exec_func_python() autogenerated&apos;, lineno: 2, function: &lt;module&gt;
     0001:
 *** 0002:patch_do_patch(d)
     0003:
File: &apos;&lt;..&gt;/poky/meta/classes-global/patch.bbclass&apos;, lineno: 157, function: patch_do_patch
     0153:        except Exception as exc:
     0154:            bb.utils.remove(process_tmpdir, True)
     0155:            bb.fatal(&quot;Importing patch &apos;%s&apos; with striplevel &apos;%s&apos;\n%s&quot; % (parm[&apos;patchname&apos;], parm[&apos;striplevel&apos;], repr(exc).replace(&quot;\\n&quot;, &quot;\n&quot;)))
     0156:        try:
 *** 0157:            resolver.Resolve()
     0158:        except bb.BBHandledException as e:
     0159:            bb.utils.remove(process_tmpdir, True)
     0160:            bb.fatal(&quot;Applying patch &apos;%s&apos; on target directory &apos;%s&apos;\n%s&quot; % (parm[&apos;patchname&apos;], patchdir, repr(e).replace(&quot;\\n&quot;, &quot;\n&quot;)))
     0161:
File: &apos;&lt;..&gt;/poky/meta/lib/oe/patch.py&apos;, lineno: 769, function: Resolve
     0765:    def Resolve(self):
     0766:        olddir = os.path.abspath(os.curdir)
     0767:        os.chdir(self.patchset.dir)
     0768:        try:
 *** 0769:            self.patchset.Push()
     0770:        except Exception:
     0771:            import sys
     0772:            os.chdir(olddir)
     0773:            raise
File: &apos;&lt;..&gt;/poky/meta/lib/oe/patch.py&apos;, lineno: 274, function: Push
     0270:            else:
     0271:                next = 0
     0272:
     0273:            bb.note(&quot;applying patch %s&quot; % self.patches[next])
 *** 0274:            ret = self._applypatch(self.patches[next], force)
     0275:
     0276:            self._current = next
     0277:            return ret
     0278:
File: &apos;&lt;..&gt;/poky/meta/lib/oe/patch.py&apos;, lineno: 556, function: _applypatch
     0552:        if os.path.lexists(hooks_dir_backup):
     0553:            raise Exception(&quot;Git hooks backup directory already exists: %s&quot; % hooks_dir_backup)
     0554:        if os.path.lexists(hooks_dir):
     0555:            shutil.move(hooks_dir, hooks_dir_backup)
 *** 0556:        os.mkdir(hooks_dir)
     0557:        commithook = os.path.join(hooks_dir, &apos;commit-msg&apos;)
     0558:        applyhook = os.path.join(hooks_dir, &apos;applypatch-msg&apos;)
     0559:        with open(commithook, &apos;w&apos;) as f:
     0560:            # NOTE: the formatting here is significant; if you change it you&apos;ll also need to
Exception: NotADirectoryError: [Errno 20] Not a directory: &apos;&lt;..&gt;/build/tmp/work/core2-64-poky-linux/vulkan-samples/git/devtooltmp-n87_zx1i/workdir/git/third_party/spdlog/.git/hooks&apos;</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>96699</commentid>
    <comment_count>8</comment_count>
    <who name="Julien Stephan">jstephan</who>
    <bug_when>2023-10-13 07:53:11 +0000</bug_when>
    <thetext>Hi all,
The fix was merged so closing this issue: https://git.yoctoproject.org/poky/commit/?id=060f285e58509fd1f38bc4eca990d4643e648c1a</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>96705</commentid>
    <comment_count>9</comment_count>
    <who name="Jon Mason">jon.mason</who>
    <bug_when>2023-10-13 15:49:38 +0000</bug_when>
    <thetext>devtool is still not working as I&apos;m expecting.  For example, I&apos;m working on hafnium in meta-arm.  After running it through devtool (see below), it removes the patches to the git submodules, and throws them in a single commit, and that commit is just modified SHAs.




$ devtool modify hafnium
...
$ devtool finish --force-patch-refresh hafnium meta-arm
NOTE: Starting bitbake server...
Loading cache: 100% |#############################################################################################################################################################################################################################################| Time: 0:00:00
Loaded 1949 entries from dependency cache.
Parsing recipes: 100% |###########################################################################################################################################################################################################################################| Time: 0:00:06
Parsing of 992 .bb files complete (991 cached, 1 parsed). 1949 targets, 206 skipped, 0 masked, 0 errors.
INFO: Updating patch 0001-arm-hafnium-fix-kernel-tool-linking.patch
INFO: Updating patch 0002-Fix-build-with-clang-15.patch
INFO: Adding new patch 0003-Committing-changes-from-do_patch.patch
INFO: Updating recipe hafnium_2.8.bb
INFO: Removing file /builder/meta-arm/build/../meta-arm/recipes-bsp/hafnium/hafnium/0001-Use-pkg-config-native-to-find-the-libssl-headers.patch
INFO: Removing file /builder/meta-arm/build/../meta-arm/recipes-bsp/hafnium/hafnium/0001-work-around-visibility-issue.patch
INFO: Cleaning sysroot for recipe hafnium...
INFO: Preserving source tree in /builder/meta-arm/build/workspace/attic/sources/hafnium.20231013154057
If you no longer need it then please delete it manually.
It is also possible to reuse it via devtool source tree argument.




(snip from git diff of hafnium_2.8.bb)

-           file://0001-Use-pkg-config-native-to-find-the-libssl-headers.patch;patchdir=third_party/linux \
-           file://0001-work-around-visibility-issue.patch;patchdir=third_party/dtc \
-          &quot;
+           file://0003-Committing-changes-from-do_patch.patch \



$ cat meta-arm/recipes-bsp/hafnium/hafnium/0003-Committing-changes-from-do_patch.patch
From 0aa482233005e53ee1779140dead5a057b2855a4 Mon Sep 17 00:00:00 2001
From: OpenEmbedded &lt;oe.patch@oe&gt;
Date: Fri, 13 Oct 2023 15:39:09 +0000
Subject: [PATCH] Committing changes from do_patch

---
 third_party/dtc   | 2 +-
 third_party/linux | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/third_party/dtc b/third_party/dtc
index 36bdb76f..db81f757 160000
--- a/third_party/dtc
+++ b/third_party/dtc
@@ -1 +1 @@
-Subproject commit 36bdb76f62a80d0e1bb1c55aaff962311256ff6c
+Subproject commit db81f757ca14deb036feb4fae3ed040e72d3dc87
diff --git a/third_party/linux b/third_party/linux
index 157378f4..c3e6c1cf 160000
--- a/third_party/linux
+++ b/third_party/linux
@@ -1 +1 @@
-Subproject commit 157378f43faad830e4aa3815bde5fa9f9a9f5be6
+Subproject commit c3e6c1cf430dbfc51a8a5feca66e408b6b63e60e</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>97221</commentid>
    <comment_count>10</comment_count>
    <who name="Julien Stephan">jstephan</who>
    <bug_when>2023-11-08 11:17:23 +0000</bug_when>
    <thetext>Hi Jon, all

I eventually managed to have a fix for that. It is on my dev branch here: https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/devtool-submodule-fix 

Can you try if this fix the issue you are facing? 

This is not yet ready for a proper submission, here is what is missing:
* devtool upgrade on gitsm recipes (ongoing right now)
* need to try with --initial-rev and --branch parameters
* .. maybe I am missing something else? 

I tried to do a lot of testing with various combination of parameters (like doing a finish on the same layer/ on another layer ... ) but I may be missing some corner cases. So any feedback would be much appreciated :)</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>97222</commentid>
    <comment_count>11</comment_count>
    <who name="Julien Stephan">jstephan</who>
    <bug_when>2023-11-08 11:19:11 +0000</bug_when>
    <thetext>(In reply to Julien Stephan from comment #10)
&gt; Hi Jon, all
&gt; 
&gt; I eventually managed to have a fix for that. It is on my dev branch here:
&gt; https://git.yoctoproject.org/poky-contrib/log/?h=jstephan/devtool-submodule-
&gt; fix 
&gt; 
&gt; Can you try if this fix the issue you are facing? 
&gt; 
&gt; This is not yet ready for a proper submission, here is what is missing:
&gt; * devtool upgrade on gitsm recipes (ongoing right now)
&gt; * need to try with --initial-rev and --branch parameters
&gt; * .. maybe I am missing something else? 
&gt; 
&gt; I tried to do a lot of testing with various combination of parameters (like
&gt; doing a finish on the same layer/ on another layer ... ) but I may be
&gt; missing some corner cases. So any feedback would be much appreciated :)

Also I added a first selftest for submodule, maybe I will add more for other scenario.

Also, pass a full devtool selftest locally so at least devtool should not be broken :)</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>97224</commentid>
    <comment_count>12</comment_count>
    <who name="Jon Mason">jon.mason</who>
    <bug_when>2023-11-08 20:07:13 +0000</bug_when>
    <thetext>This is working as expected for Hafnium.  Good job.

I&apos;m seeing a different issue with edk2 (also present in meta-arm).  I&apos;m seeing:
fatal: No url found for submodule path &apos;edk2-platforms&apos; in .gitmodules


However, in the recipe the path is being specified.  See 
https://git.yoctoproject.org/meta-arm/tree/meta-arm/recipes-bsp/uefi/edk2-firmware.inc#n15

I believe there is a disconnect between the submodules that are expected to be present in the git tree, as opposed to the ones being specified in the recipe.  I believe the way edk2 is doing it is acceptable.  So, having this supported would be necessary.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>97242</commentid>
    <comment_count>13</comment_count>
    <who name="Julien Stephan">jstephan</who>
    <bug_when>2023-11-10 09:33:42 +0000</bug_when>
    <thetext>(In reply to Jon Mason from comment #12)
&gt; This is working as expected for Hafnium.  Good job.
&gt; 
&gt; I&apos;m seeing a different issue with edk2 (also present in meta-arm).  I&apos;m
&gt; seeing:
&gt; fatal: No url found for submodule path &apos;edk2-platforms&apos; in .gitmodules
&gt; 
&gt; 
&gt; However, in the recipe the path is being specified.  See 
&gt; https://git.yoctoproject.org/meta-arm/tree/meta-arm/recipes-bsp/uefi/edk2-
&gt; firmware.inc#n15
&gt; 
&gt; I believe there is a disconnect between the submodules that are expected to
&gt; be present in the git tree, as opposed to the ones being specified in the
&gt; recipe.  I believe the way edk2 is doing it is acceptable.  So, having this
&gt; supported would be necessary.

Hi Jon,

Thank you for testing it! 

I indeed reproduced the issue with edk2, but this is not only restricted to gitsm recipes. All recipes using SRC_URI such as: 

SRC_URI = &quot; git://&lt;primarysource&gt; git://&lt;someothersource&gt;;destsuffix=git/&lt;someothersource&gt;&quot;

will not work with devtool modify/finish (issue is already present in master). 

I will try to update my branch to fix this, and let you know.

Cheers
Julien</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>97301</commentid>
    <comment_count>14</comment_count>
    <who name="Julien Stephan">jstephan</who>
    <bug_when>2023-11-22 12:33:53 +0000</bug_when>
    <thetext>Hi Jon, I just force pushed my branch on poky-contrib and sent a series to fix the submodue (including submodules declared into SRC_URI). 

Can you test it? 

Thank you
Cheers

Julien</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>97553</commentid>
    <comment_count>15</comment_count>
    <who name="Richard Purdie">richard.purdie</who>
    <bug_when>2023-12-13 14:59:21 +0000</bug_when>
    <thetext>https://git.yoctoproject.org/poky/commit/?id=89f16624842a91e063dc9e6b98787d44c55e4900</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>101011</commentid>
    <comment_count>16</comment_count>
    <who name="Randy MacLeod">randy.macleod</who>
    <bug_when>2025-02-14 16:46:35 +0000</bug_when>
    <thetext>bulk change to add Saul&apos;s non-WR address.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>101065</commentid>
    <comment_count>17</comment_count>
    <who name="Randy MacLeod">randy.macleod</who>
    <bug_when>2025-02-14 16:48:22 +0000</bug_when>
    <thetext>Bulk change: Remove Saul&apos;s old WR address.</thetext>
  </long_desc>
      
      

    </bug>

</bugzilla>