I have tested in scarthgap 5.0.6, but I think it is also probably happening in master. I debugged this a bit, but it looks like more debugging is needed. The recipe takes 50 minutes to compile in my real project which uses a big git repository, but it should take 1 minute normally and it is wasting a lot of build resources doing unnecessary work: Reproduce with: time bitbake -c compile -f python3-setuptools-scm-native The issue is in build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/setuptools-scm-8.0.4/src/setuptools_scm/_file_finders/git.py, there is a call to "git archive --prefix /home/user/dev/poky HEAD" which creates a tarball of the poky repository, and passes this via stdout to another function. This is not a huge issue in poky itself, however when poky is just a submodule of a big repository, this introduces huge slowdowns (e.g. this is creating a 120GB tarball and a 50 minutes slowdown in my real project). The intent of this call is to create a tarball of the package getting built (python3-setuptools-scm-native), however since this is a pypi tarball there is no git directory, and the top-level git directory wrongly gets detected as the poky repository, in my case "/home/ecordonnier/dev/poky". See my debug logs with a call-stack: poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4$ rg "foobarr" . ERROR setuptools_scm._file_finders.git foobarr git_find_files on toplevel: /home/ecordonnier/dev/poky path: ERROR setuptools_scm._file_finders.git foobarr run git archive on /home/ecordonnier/dev/poky ERROR setuptools_scm._file_finders.git foobarr cmd is ['git', 'archive', '--prefix', '/home/ecordonnier/dev/poky/', 'HEAD'] Call stack printed inside ./setuptools-scm-8.0.4/src/setuptools_scm/_file_finders/git.py using: import tracebackdef _git_ls_files_and_dirs(toplevel: str) -> tuple[set[str], set[str]]: # use git archive instead of git ls-file to honor # export-ignore git attribute log.error("foobarr run git archive on %s" % toplevel) for line in traceback.format_stack(): print("%s" % line.strip()) File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/recipe-sysroot-native/usr/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 353, in <module> main() File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/recipe-sysroot-native/usr/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 335, in main json_out['return_val'] = hook(**hook_input['kwargs']) File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/recipe-sysroot-native/usr/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel return hook(config_settings) File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/recipe-sysroot-native/usr/lib/python3.12/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel return self._get_build_requires(config_settings, requirements=['wheel']) File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/recipe-sysroot-native/usr/lib/python3.12/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires self.run_setup() File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/recipe-sysroot-native/usr/lib/python3.12/site-packages/setuptools/build_meta.py", line 311, in run_setup exec(code, locals()) File "<string>", line 1, in <module> File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/recipe-sysroot-native/usr/lib/python3.12/site-packages/setuptools/__init__.py", line 103, in setup return distutils.core.setup(**attrs) File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/recipe-sysroot-native/usr/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 185, in setup return run_commands(dist) File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/recipe-sysroot-native/usr/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 201, in run_commands dist.run_commands() File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/recipe-sysroot-native/usr/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands self.run_command(cmd) File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/recipe-sysroot-native/usr/lib/python3.12/site-packages/setuptools/dist.py", line 963, in run_command super().run_command(command) File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/recipe-sysroot-native/usr/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 988, in run_command cmd_obj.run() File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/recipe-sysroot-native/usr/lib/python3.12/site-packages/setuptools/command/egg_info.py", line 321, in run self.find_sources() File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/recipe-sysroot-native/usr/lib/python3.12/site-packages/setuptools/command/egg_info.py", line 329, in find_sources mm.run() File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/recipe-sysroot-native/usr/lib/python3.12/site-packages/setuptools/command/egg_info.py", line 550, in run self.add_defaults() File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/recipe-sysroot-native/usr/lib/python3.12/site-packages/setuptools/command/egg_info.py", line 591, in add_defaults rcfiles = list(walk_revctrl()) File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/recipe-sysroot-native/usr/lib/python3.12/site-packages/setuptools/command/sdist.py", line 16, in walk_revctrl yield from ep.load()(dirname) File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/setuptools-scm-8.0.4/src/setuptools_scm/_file_finders/__init__.py", line 103, in find_files res: list[str] = command(path) File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/setuptools-scm-8.0.4/src/setuptools_scm/_file_finders/git.py", line 103, in git_find_files git_files, git_dirs = _git_ls_files_and_dirs(toplevel) File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/setuptools-scm-8.0.4/src/setuptools_scm/_file_finders/git.py", line 74, in _git_ls_files_and_dirs for line in traceback.format_stack(): This seems to be a generic bug in meta/classes-recipe/python_pep517.bbclass
I've double-checked and I can also reproduce on master: ``` ecordonnier@lj8k2dq3:~/dev/poky/build$ bitbake -c compile -f python3-setuptools-scm-native ecordonnier@lj8k2dq3:~/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.1.0$ rg "foobarr" . ./temp/log.do_compile.193884 21:ERROR setuptools_scm._file_finders.git foobarr: cmd is ['git', 'archive', '--prefix', '/home/ecordonnier/dev/poky/', 'HEAD'] 56:ERROR setuptools_scm._file_finders.git foobarr: cmd is ['git', 'archive', '--prefix', '/home/ecordonnier/dev/poky/', 'HEAD'] ```
Created attachment 5097 [details] Working compile log Pristine master? I can't replicate. I've attached my complete log.do_compile for python3-setuptools-scm-native to this bug, can you compare them and see if you get any clues?
Yes, it is reproducible with pristine master from today. The logs you sent look as expected. The logs I copied in the ticket were logs I added to demonstrate the issue. The reason you did not notice an issue is that your build directory is inside poky, which is only 48MB, and therefore "git archive --prefix poky-path HEAD" only takes about 2 seconds. However with my setup (build directory inside a big repository with git-lfs files where "git archive my-repo-path HEAD" creates a 120GB tarball), this takes about 50 minutes. ~/dev/poky$ time git archive --prefix ~/dev/poky/ HEAD > out real 0m0.201s user 0m0.166s sys 0m0.035s ~/dev/poky$ ls -lh out -rw-rw-r-- 1 ecordonnier ecordonnier 48M Feb 12 23:06 out Full reproduction steps: ``` cat test.sh #!/bin/bash git clone -b eco/python3-setuptools-scm-issue https://github.com/Ecordonnier/poky.git test-poky cd test-poky source oe-init-build-env bitbake -c compile -f python3-setuptools-scm-native && cat tmp/work/x86_64-linux/python3-setuptools-scm-native/8.1.0/temp/log.do_compile | grep -i foobarr ./test.sh ... ... Summary: There were 10 WARNING messages. ERROR setuptools_scm._file_finders.git foobarr running command with wrong toplevel: ['git', 'archive', '--prefix', '/home/asteba/dev/tmp/test-poky/', 'HEAD'] ERROR setuptools_scm._file_finders.git foobarr running command with wrong toplevel: ['git', 'archive', '--prefix', '/home/asteba/dev/tmp/test-poky/', 'HEAD'] ``` The --prefix argument is supposed to be the git repository containing only python3-setuptools-scm, not the full poky repository, but due to the lack of .git directory inside the python3-setuptools-scm tarball, the logic of git.py finds a wrong toplevel git directory. I asked for help upstream: https://github.com/pypa/setuptools-scm/issues/1103
Looks like an upstream bug so I'm glad you've filed that and hopefully they get some traction. Now there _is_ functionality in git that is intended to stop this sort of thing, specifically GIT_CEILING_DIRECTORIES which puts a cap on how far up the directory tree to hunt for a git repository. We set that by default to WORKDIR: $ pwd /home/rosbur01/Yocto/poky/meta/recipes-devtools/python $ git rev-parse --show-toplevel /home/rosbur01/Yocto/poky $ GIT_CEILING_DIRECTORIES=/home/rosbur01/Yocto/poky/meta git rev-parse --show-toplevel fatal: not a git repository (or any of the parent directories): .git So that definitely works. setuptools-scm for unknowable reasons doesn't just use --show-toplevel (maybe it wants to support old releases without that?) but the underlying function it uses still has the same behaviour: $ git rev-parse --show-prefix meta/recipes-devtools/python/ $ GIT_CEILING_DIRECTORIES=/home/rosbur01/Yocto/poky/meta git rev-parse --show-prefix fatal: not a git repository (or any of the parent directories): .git So the question is why doesn't this work? Maybe it purges the environment? The variable was added to git in 1.6.0 back in 2008, so I presume you don't have an incredibly old git release.
Oh, right. setuptools-scm explicitly strips any variable starting with GIT_ from the environment. <facepalm>
I'm using Ubuntu 24.04 with git 2.43.0 ;-) Thanks for the help. I wasn't aware of the existence of GIT_CEILING_DIRECTORIES
Etienne: can you try adding "GIT_CEILING_DIRECTORIES" to this line in _run_cmd.py:no_git_env(): or k in ("GIT_EXEC_PATH", "GIT_SSH", "GIT_SSH_COMMAND") That _should_ fix the problem.
It seems to work, nice. I can send a patch to the yocto mailing-list with your sign-off as well as to setuptools-scm via github.
Great, thanks!
The fix is merged to master as https://github.com/yoctoproject/poky/commit/d55cd3fc39b638b4b765b07c484e58880996febf