Bug 15740

Summary: python3-setuptools-scm-native creates a git archive of the entire poky repository
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Etienne Cordonnier <ecordonnier>
Component: devtools / tool chainAssignee: Ross Burton <ross.burton>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium+ CC: ecordonnier, meta.mr.watcher, meta.watcher, randy.macleod, ross.burton
Version: 5.2   
Target Milestone: 5.2 M3   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
Working compile log none

Description Etienne Cordonnier 2025-02-12 16:21:05 UTC
I have tested in scarthgap 5.0.6, but I think it is also probably happening in master.

I debugged this a bit, but it looks like more debugging is needed. The recipe takes 50 minutes to compile in my real project which uses a big git repository, but it should take 1 minute normally and it is wasting a lot of build resources doing unnecessary work:

Reproduce with:

time bitbake -c compile -f python3-setuptools-scm-native


The issue is in build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/setuptools-scm-8.0.4/src/setuptools_scm/_file_finders/git.py, there is a call to "git archive --prefix /home/user/dev/poky HEAD" which creates a tarball of the poky repository, and passes this via stdout to another function. This is not a huge issue in poky itself, however when poky is just a submodule of a big repository, this introduces huge slowdowns (e.g. this is creating a 120GB tarball and a 50 minutes slowdown in my real project).

The intent of this call is to create a tarball of the package getting built (python3-setuptools-scm-native), however since this is a pypi tarball there is no git directory, and the top-level git directory wrongly gets detected as the poky repository, in my case "/home/ecordonnier/dev/poky".

See my debug logs with a call-stack:

poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4$ rg "foobarr" .
ERROR setuptools_scm._file_finders.git foobarr git_find_files on toplevel: /home/ecordonnier/dev/poky path: 
ERROR setuptools_scm._file_finders.git foobarr run git archive on /home/ecordonnier/dev/poky
ERROR setuptools_scm._file_finders.git foobarr cmd is ['git', 'archive', '--prefix', '/home/ecordonnier/dev/poky/', 'HEAD']


Call stack printed inside ./setuptools-scm-8.0.4/src/setuptools_scm/_file_finders/git.py using:
import tracebackdef _git_ls_files_and_dirs(toplevel: str) -> tuple[set[str], set[str]]:
    # use git archive instead of git ls-file to honor
    # export-ignore git attribute    log.error("foobarr run git archive on %s" % toplevel)
    for line in traceback.format_stack():
        print("%s" % line.strip())
 

File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/recipe-sysroot-native/usr/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
    main()
File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/recipe-sysroot-native/usr/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 335, in main
    json_out['return_val'] = hook(**hook_input['kwargs'])
File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/recipe-sysroot-native/usr/lib/python3.12/site-packages/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
    return hook(config_settings)
File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/recipe-sysroot-native/usr/lib/python3.12/site-packages/setuptools/build_meta.py", line 325, in get_requires_for_build_wheel
    return self._get_build_requires(config_settings, requirements=['wheel'])
File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/recipe-sysroot-native/usr/lib/python3.12/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
    self.run_setup()
File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/recipe-sysroot-native/usr/lib/python3.12/site-packages/setuptools/build_meta.py", line 311, in run_setup
    exec(code, locals())
File "<string>", line 1, in <module>
File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/recipe-sysroot-native/usr/lib/python3.12/site-packages/setuptools/__init__.py", line 103, in setup
    return distutils.core.setup(**attrs)
File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/recipe-sysroot-native/usr/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 185, in setup
    return run_commands(dist)
File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/recipe-sysroot-native/usr/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
    dist.run_commands()
File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/recipe-sysroot-native/usr/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
    self.run_command(cmd)
File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/recipe-sysroot-native/usr/lib/python3.12/site-packages/setuptools/dist.py", line 963, in run_command
    super().run_command(command)
File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/recipe-sysroot-native/usr/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
    cmd_obj.run()
File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/recipe-sysroot-native/usr/lib/python3.12/site-packages/setuptools/command/egg_info.py", line 321, in run
    self.find_sources()
File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/recipe-sysroot-native/usr/lib/python3.12/site-packages/setuptools/command/egg_info.py", line 329, in find_sources
    mm.run()
File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/recipe-sysroot-native/usr/lib/python3.12/site-packages/setuptools/command/egg_info.py", line 550, in run
    self.add_defaults()
File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/recipe-sysroot-native/usr/lib/python3.12/site-packages/setuptools/command/egg_info.py", line 591, in add_defaults
    rcfiles = list(walk_revctrl())
File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/recipe-sysroot-native/usr/lib/python3.12/site-packages/setuptools/command/sdist.py", line 16, in walk_revctrl
    yield from ep.load()(dirname)
File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/setuptools-scm-8.0.4/src/setuptools_scm/_file_finders/__init__.py", line 103, in find_files
    res: list[str] = command(path)
File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/setuptools-scm-8.0.4/src/setuptools_scm/_file_finders/git.py", line 103, in git_find_files
    git_files, git_dirs = _git_ls_files_and_dirs(toplevel)
File "/home/ecordonnier/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.0.4/setuptools-scm-8.0.4/src/setuptools_scm/_file_finders/git.py", line 74, in _git_ls_files_and_dirs
    for line in traceback.format_stack(): 

This seems to be a generic bug in meta/classes-recipe/python_pep517.bbclass
Comment 1 Etienne Cordonnier 2025-02-12 16:55:27 UTC
I've double-checked and I can also reproduce on master:

```
ecordonnier@lj8k2dq3:~/dev/poky/build$ bitbake -c compile -f python3-setuptools-scm-native

ecordonnier@lj8k2dq3:~/dev/poky/build/tmp/work/x86_64-linux/python3-setuptools-scm-native/8.1.0$ rg "foobarr" .
./temp/log.do_compile.193884
21:ERROR setuptools_scm._file_finders.git foobarr: cmd is ['git', 'archive', '--prefix', '/home/ecordonnier/dev/poky/', 'HEAD']
56:ERROR setuptools_scm._file_finders.git foobarr: cmd is ['git', 'archive', '--prefix', '/home/ecordonnier/dev/poky/', 'HEAD']
```
Comment 2 Ross Burton 2025-02-12 18:10:02 UTC
Created attachment 5097 [details]
Working compile log

Pristine master? I can't replicate.  I've attached my complete log.do_compile for python3-setuptools-scm-native to this bug, can you compare them and see if you get any clues?
Comment 3 Etienne Cordonnier 2025-02-12 22:28:20 UTC
Yes, it is reproducible with pristine master from today.

The logs you sent look as expected. The logs I copied in the ticket were logs I added to demonstrate the issue. The reason you did not notice an issue is that your build directory is inside poky, which is only 48MB, and therefore "git archive --prefix poky-path HEAD" only takes about 2 seconds. However with my setup (build directory inside a big repository with git-lfs files where "git archive my-repo-path HEAD" creates a 120GB tarball), this takes about 50 minutes.

~/dev/poky$ time git archive --prefix ~/dev/poky/ HEAD > out

real	0m0.201s
user	0m0.166s
sys	0m0.035s
~/dev/poky$ ls -lh out 
-rw-rw-r-- 1 ecordonnier ecordonnier 48M Feb 12 23:06 out



Full reproduction steps:
```
cat test.sh 
#!/bin/bash

git clone -b eco/python3-setuptools-scm-issue https://github.com/Ecordonnier/poky.git test-poky
cd test-poky
source oe-init-build-env
bitbake -c compile -f python3-setuptools-scm-native && cat tmp/work/x86_64-linux/python3-setuptools-scm-native/8.1.0/temp/log.do_compile | grep -i foobarr


./test.sh
...
...
Summary: There were 10 WARNING messages.
ERROR setuptools_scm._file_finders.git foobarr running command with wrong toplevel: ['git', 'archive', '--prefix', '/home/asteba/dev/tmp/test-poky/', 'HEAD']
ERROR setuptools_scm._file_finders.git foobarr running command with wrong toplevel: ['git', 'archive', '--prefix', '/home/asteba/dev/tmp/test-poky/', 'HEAD']


```

The --prefix argument is supposed to be the git repository containing only python3-setuptools-scm, not the full poky repository, but due to the lack of .git directory inside the python3-setuptools-scm tarball, the logic of git.py finds a wrong toplevel git directory.

I asked for help upstream: https://github.com/pypa/setuptools-scm/issues/1103
Comment 4 Ross Burton 2025-02-13 14:17:18 UTC
Looks like an upstream bug so I'm glad you've filed that and hopefully they get some traction.

Now there _is_ functionality in git that is intended to stop this sort of thing, specifically GIT_CEILING_DIRECTORIES which puts a cap on how far up the directory tree to hunt for a git repository.  We set that by default to WORKDIR:

$ pwd
/home/rosbur01/Yocto/poky/meta/recipes-devtools/python

$ git rev-parse --show-toplevel
/home/rosbur01/Yocto/poky

$ GIT_CEILING_DIRECTORIES=/home/rosbur01/Yocto/poky/meta git rev-parse --show-toplevel
fatal: not a git repository (or any of the parent directories): .git

So that definitely works.  setuptools-scm for unknowable reasons doesn't just use --show-toplevel (maybe it wants to support old releases without that?) but the underlying function it uses still has the same behaviour:

$ git rev-parse --show-prefix
meta/recipes-devtools/python/

$ GIT_CEILING_DIRECTORIES=/home/rosbur01/Yocto/poky/meta git rev-parse --show-prefix
fatal: not a git repository (or any of the parent directories): .git

So the question is why doesn't this work?  Maybe it purges the environment? The variable was added to git in 1.6.0 back in 2008, so I presume you don't have an incredibly old git release.
Comment 5 Ross Burton 2025-02-13 14:20:27 UTC
Oh, right.

setuptools-scm explicitly strips any variable starting with GIT_ from the environment.

<facepalm>
Comment 6 Etienne Cordonnier 2025-02-13 14:21:44 UTC
I'm using Ubuntu 24.04 with git 2.43.0 ;-)

Thanks for the help. I wasn't aware of the existence of GIT_CEILING_DIRECTORIES
Comment 7 Ross Burton 2025-02-13 14:23:42 UTC
Etienne: can you try adding "GIT_CEILING_DIRECTORIES" to this line in _run_cmd.py:no_git_env():

or k in ("GIT_EXEC_PATH", "GIT_SSH", "GIT_SSH_COMMAND")

That _should_ fix the problem.
Comment 8 Etienne Cordonnier 2025-02-13 14:36:31 UTC
It seems to work, nice. I can send a patch to the yocto mailing-list with your sign-off as well as to setuptools-scm via github.
Comment 9 Ross Burton 2025-02-13 15:07:44 UTC
Great, thanks!
Comment 10 Etienne Cordonnier 2025-02-18 15:23:30 UTC
The fix is merged to master as https://github.com/yoctoproject/poky/commit/d55cd3fc39b638b4b765b07c484e58880996febf