14325 2021-03-25 05:47:10 +0000 externalsrc bbclass expansion error when the repo has a submodule and the externalsrc is in a subfolder 2021-03-29 22:56:10 +0000 1 1 7 Build System, Metadata & Runtime OE-Core core 3.1.6 x86 Multiple RESOLVED FIXED Medium normal 3.1.7 1 Ahmed.Hossam steve meta.mr.watcher meta.watcher randy.macleod --- --- No (bug/feature does not impact docs) oldest_to_newest 89734 0 Ahmed.Hossam 2021-03-25 05:47:10 +0000 The externalsrc bbclass tries to track changes to submodules but assumes that the external source directory is at the root of the repo, assuming a code repo that looks like this: . └── repo ├── myapp1 ├── myapp2 └── myapp3 └── submodule This line http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/classes/externalsrc.bbclass?h=dunfell#n221 tries to append the submodule directory (myapp3/submodule) to the source directory s_dir (repo/myapp1) resulting in an unexisting path (repo/myapp1/myapp3/submodule), it should instead append it to the git directory git_dir to get the right path (repo/myapp3/submodule). perhaps it should also ignore it in case the submodule is not under the app directory s_dir? Error resulting from current bug: ERROR: ExpansionError during parsing /path/to/poky/../meta-mylayer/recipes-myapp1/myapp1.bb Traceback (most recent call last): File "Var <do_compile[file-checksums]>", line 1, in <module> File "/path/to/poky/meta/classes/externalsrc.bbclass", line 222, in srctree_hash_files(d=<bb.data_smart.DataSmart object at 0x7f3144edb438>, srcdir=None): module_dir = os.path.join(s_dir, line.rsplit(maxsplit=1)[1]) > proc = subprocess.Popen(['git', 'add', '-A', '.'], cwd=module_dir, env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) proc.communicate() File "/usr/lib/python3.6/subprocess.py", line 729, in Popen.__init__(args=['git', 'add', '-A', '.'], bufsize=-1, executable=None, stdin=None, stdout=-3, stderr=-3, preexec_fn=None, close_fds=True, shell=False, cwd='/path/to/poky/../meta-mylayer/recipes-myrecipe/../../my-code-repo/myapp1/myapp3/submodule', env={'MACHINE': 'qemux86', 'USER': 'user', 'PWD': '/path/to/my-code-repo', 'HOME': '/home/user', 'BBPATH': '/path/to/build', 'BB_ENV_EXTRAWHITE': 'ALL_PROXY BBPATH_EXTRA BB_LOGCONFIG BB_NO_NETWORK BB_NUMBER_THREADS BB_SETSCENE_ENFORCE BB_SRCREV_POLICY DISTRO FTPS_PROXY FTP_PROXY GIT_PROXY_COMMAND HTTPS_PROXY HTTP_PROXY MACHINE NO_PROXY PARALLEL_MAKE SCREENDIR SDKMACHINE SOCKS5_PASSWD SOCKS5_USER SSH_AGENT_PID SSH_AUTH_SOCK STAMPS_DIR TCLIBC TCMODE all_proxy ftp_proxy ftps_proxy http_proxy https_proxy no_proxy MACHINE', 'TERM': 'xterm-256color', 'SHELL': '/bin/bash', 'LOGNAME': 'user', 'PATH': '/path/to/poky/scripts:/path/to/poky/bitbake/bin:/path/to/user/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin', 'LC_ALL': 'en_US.UTF-8', 'GIT_INDEX_FILE': '/tmp/oe-devtool-indexbm0ky5kz'}, universal_newlines=False, startupinfo=None, creationflags=0, restore_signals=True, start_new_session=False, pass_fds=(), encoding=None, errors=None): errread, errwrite, > restore_signals, start_new_session) except: File "/usr/lib/python3.6/subprocess.py", line 1364, in Popen._execute_child(args=['git', 'add', '-A', '.'], executable=b'git', preexec_fn=None, close_fds=True, pass_fds=(), cwd='/path/to/poky/../meta-mylayer/recipes-myrecipe/../../my-code-repo/myapp1/myapp3/submodule', env={'MACHINE': 'qemux86', 'USER': 'user', 'PWD': '/path/to/my-code-repo', 'HOME': '/home/user', 'BBPATH': '/path/to/build', 'BB_ENV_EXTRAWHITE': 'ALL_PROXY BBPATH_EXTRA BB_LOGCONFIG BB_NO_NETWORK BB_NUMBER_THREADS BB_SETSCENE_ENFORCE BB_SRCREV_POLICY DISTRO FTPS_PROXY FTP_PROXY GIT_PROXY_COMMAND HTTPS_PROXY HTTP_PROXY MACHINE NO_PROXY PARALLEL_MAKE SCREENDIR SDKMACHINE SOCKS5_PASSWD SOCKS5_USER SSH_AGENT_PID SSH_AUTH_SOCK STAMPS_DIR TCLIBC TCMODE all_proxy ftp_proxy ftps_proxy http_proxy https_proxy no_proxy MACHINE', 'TERM': 'xterm-256color', 'SHELL': '/bin/bash', 'LOGNAME': 'user', 'PATH': '/path/to/poky/scripts:/path/to/poky/bitbake/bin:/path/to/user/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin', 'LC_ALL': 'en_US.UTF-8', 'GIT_INDEX_FILE': '/tmp/oe-devtool-indexbm0ky5kz'}, startupinfo=None, creationflags=0, shell=False, p2cread=-1, p2cwrite=-1, c2pread=-1, c2pwrite=31, errread=-1, errwrite=31, restore_signals=True, start_new_session=False): err_msg += ': ' + repr(err_filename) > raise child_exception_type(errno_num, err_msg, err_filename) raise child_exception_type(err_msg) bb.data_smart.ExpansionError: Failure expanding variable do_compile[file-checksums], expression was ${@srctree_hash_files(d)} which triggered exception FileNotFoundError: [Errno 2] No such file or directory: '/path/to/poky/../meta-mylayer/recipes-myrecipe/../../my-code-repo/myapp1/myapp3/submodule': '/path/to/poky/../meta-mylayer/recipes-myrecipe/../../my-code-repo/myapp1/myapp3/submodule' 89760 1 randy.macleod 2021-03-25 15:11:45 +0000 Does this happen on the master branch as well ? Ahmed, it would be good if you could test that and let Steve know. 89789 2 Ahmed.Hossam 2021-03-26 11:28:44 +0000 I tested master today and the issue doesn't happen, it was fixed by the commit: http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/meta/classes/externalsrc.bbclass?h=42c86799ce265101b821030525ec568d34670430 89794 3 steve 2021-03-26 14:26:15 +0000 The patch that fixes this has been backported from master and is in this week's set of patches out for review: https://lists.openembedded.org/g/openembedded-core/message/149885 It should be merged within the next few days. 89823 4 steve 2021-03-29 22:56:10 +0000 Fix is now merged: https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?h=dunfell&id=97d92eb0e87b0cca026335b8ae74f1c1dc469d8b