Bug 14333

Hi All,

I had updated oe-core to latest commit recently (as per today it is pointed to 2d903126e8bbece3a5171c3488c3deae1f0aa3ee). I have a weird observation with this recent commit, especially how submodule helper is used on workspace created by devtool. 

We have SRC_URI as follows on our edk2 recipes 

SRC_URI = "\
           gitsm://<url>;name=edk2;destsuffix=edk2;branchmaster \
           git://<url>;name=edk2-platforms;destsuffix=edk2/edk2-platforms;branch=master \
 "
our intention to fetch edk2-platform inside edk2 workspace. so that, we had set destsuffix as shown above. And we are getting required workspace (even with devtool also) successfully. 

Here edk2-platform is not a submodule of edk2. Once the workspace is created with "devtool modify <recipe_name>" command (edk2-platforms reside inside edk2 workspace),  "git submodule status" on edk2 workspace returns error as follows

"fatal: no submodule mapping found in .gitmodules for path 'edk2-platforms"

This error should be ignored since edk2-platforms is not a submodule of edk2. But Yocto build system does not have the intelligence to ignore this.

When we build this edk2 recipe with recent poky commits, we are getting following error

"CalledProcessError: Command '['git', 'submodule', 'status']' returned non-zero exit status 128" 

on bisecting over the oe-core commits, I had found commit: 42c86799ce265101b821030525ec568d34670430 causing the problem. I have reverted this commit as shown below and able to bypass above error.

diff --git a/meta/classes/externalsrc.bbclass b/meta/classes/externalsrc.bbclass
index 54cc7edbae..c7b2bf2f49 100644
--- a/meta/classes/externalsrc.bbclass
+++ b/meta/classes/externalsrc.bbclass
@@ -217,16 +217,14 @@ def srctree_hash_files(d, srcdir=None):
             env['GIT_INDEX_FILE'] = tmp_index.name
             subprocess.check_output(['git', 'add', '-A', '.'], cwd=s_dir, env=env)
             git_sha1 = subprocess.check_output(['git', 'write-tree'], cwd=s_dir, env=env).decode("utf-8")
-            submodule_helper = subprocess.check_output(['git', 'submodule', 'status'], cwd=s_dir, env=env).decode("utf-8")
+            submodule_helper = subprocess.check_output(['git', 'submodule--helper', 'list'], cwd=s_dir, env=env).decode("utf-8")
             for line in submodule_helper.splitlines():
-                module_relpath = line.split()[1]
-                if not module_relpath.split('/')[0] == '..':
-                    module_dir = os.path.join(s_dir, module_relpath)
-                    proc = subprocess.Popen(['git', 'add', '-A', '.'], cwd=module_dir, env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
-                    proc.communicate()
-                    proc = subprocess.Popen(['git', 'write-tree'], cwd=module_dir, env=env, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)
-                    stdout, _ = proc.communicate()
-                    git_sha1 += stdout.decode("utf-8")
+                module_dir = os.path.join(s_dir, line.rsplit(maxsplit=1)[1])
+                proc = subprocess.Popen(['git', 'add', '-A', '.'], cwd=module_dir, env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+                proc.communicate()
+                proc = subprocess.Popen(['git', 'write-tree'], cwd=module_dir, env=env, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL)
+                stdout, _ = proc.communicate()
+                git_sha1 += stdout.decode("utf-8")
             sha1 = hashlib.sha1(git_sha1.encode("utf-8")).hexdigest()
         with open(oe_hash_file, 'w') as fobj:
             fobj.write(sha1)

Either this commit (42c86799ce265101b821030525ec568d34670430) should be removed from oe-core or need a proper way of handling submodule helper on workspace created by devtool.

steps to reproduce the issues.

1. create a recipe with SRC_URI to checkout multiple workspace in recursive way (Refer SRC_URI I had mentioned above)
2. source enviroment
3. devtool modify <recipe_name>
4. cd workspace/sources/<recipe_name>/
5. git submodule status
   this shows error "fatal: no submodule mapping found in .gitmodules for path <pkg_name2>
6. cd -
4. bitbake <recipe_name>
   this gives error "CalledProcessError: Command '['git', 'submodule', 'status']' returned non-zero exit status 128"