14156 2020-12-18 10:55:20 +0000 fetch/gitsm: submodules are checked out as mirrors with altered origin information 2025-07-18 08:40:36 +0000 1 1 7 Build System, Metadata & Runtime BitBake bitbake unspecified x86 Multiple NEW Medium normal 5.99 1 bojan.milosevic.work unassigned akuster bojan.milosevic.work niqingliang2003 poky.bs.watcher poky.watcher --- --- Don't know oldest_to_newest 88795 0 bojan.milosevic.work 2020-12-18 10:55:20 +0000 The gitsm fetcher handling git repos with submodules does not work as expected, since the submodules are fetched as mirrored. When using gitsm in a recipe, the git repo and submodules will be correctly fetched at the requested revision. The main git is ok and will correctly show the current commit and all the branches on origin and it will allow for further interaction (fetch, checkout commits/branches, commit, push). A submodule git will be checked out as a mirror, hence it will show only local branches up to the one fetched, it will not have correct references for origin. It will not allow for further interaction with the git repo: no fetch, no pulling/checking out remote branches and no push of new commits. This is a major issue for our workflow, since we use several projects with submodules (including in the kernel on some projects) and we need to work on them both within a yocto build and from outside yocto. This behavior was not present on sumo, but I found it on zeus. I've performed some tests as reported below. I'm currently comparing sumo and zeus because they are included in the vendor's BSP releases we are working on. To test/reproduce the behavior I used the git-submodule-test recipe and its git repo which contains recursive submodules (the recursive part is not of interest here, the behavior is the same with a single submodule). between tests I've deleted the build dir to have a clean build each time. Below you can find the resulting git config in different cases: 1) manually cloning the repo -> OK 2) using poky master -> FAIL 3) using poky zeus -> FAIL 4) using poky sumo -> OK my knowledge of git is limited, but I've identified the differences as that in cases (2) and (3) the main git has the submodules not set as active and the submodule git config has remote.origin.fetch=+refs/*:refs/* remote.origin.mirror=true I also tested copying the gitsm.py file from sumo to zeus and it works fine, so I'm using that as a workaround at the moment. ########################################################################## Locally cloned git repo $ git clone https://git.yoctoproject.org/git/git-submodule-test $ cd git-submodule-test $ git submodule update --init --recursive $ git config --list http.sslcainfo=/etc/ssl/certs/ca-certificates.crt color.ui=auto core.repositoryformatversion=0 core.filemode=true core.bare=false core.logallrefupdates=true remote.origin.url=https://git.yoctoproject.org/git/git-submodule-test remote.origin.fetch=+refs/heads/*:refs/remotes/origin/* branch.master.remote=origin branch.master.merge=refs/heads/master submodule.bitbake.active=true submodule.bitbake.url=git://git.openembedded.org/bitbake submodule.bitbake-gitsm-test1.active=true submodule.bitbake-gitsm-test1.url=git://git.yoctoproject.org/bitbake-gitsm-test1 $ cd bitbake $ git config --list http.sslcainfo=/etc/ssl/certs/ca-certificates.crt color.ui=auto core.repositoryformatversion=0 core.filemode=true core.bare=false core.logallrefupdates=true core.worktree=../../../bitbake remote.origin.url=git://git.openembedded.org/bitbake remote.origin.fetch=+refs/heads/*:refs/remotes/origin/* branch.master.remote=origin branch.master.merge=refs/heads/master ########################################################################## Yocto master Commit: 38b45cd73eb360ce6dbe441ea497044acd27d777 $ git clone git://git.yoctoproject.org/poky poky_test $ cd poky_test $ . oe-init-build-env >> add layer meta-selftest and re-run init script $ bitbake git-submodule-test $ cd tmp/work/core2-64-poky-linux/git-submodule-test/1.0-r0/git $ git config --list http.sslcainfo=/etc/ssl/certs/ca-certificates.crt color.ui=auto core.repositoryformatversion=0 core.filemode=true core.bare=false core.logallrefupdates=true remote.origin.url=git://git.yoctoproject.org/git-submodule-test remote.origin.fetch=+refs/heads/*:refs/remotes/origin/* branch.master.remote=origin branch.master.merge=refs/heads/master submodule.bitbake.url=/media/hddata/GitLab/tmp/external/poky_test/build/downloads/git2/git.openembedded.org.bitbake submodule.bitbake-gitsm-test1.url=/media/hddata/GitLab/tmp/external/poky_test/build/downloads/git2/git.yoctoproject.org.bitbake-gitsm-test1 $ cd bitbake $ git config --list http.sslcainfo=/etc/ssl/certs/ca-certificates.crt color.ui=auto core.repositoryformatversion=0 core.filemode=true core.bare=false core.worktree=../../../bitbake remote.origin.url=git://git.openembedded.org/bitbake remote.origin.fetch=+refs/*:refs/* remote.origin.mirror=true ########################################################################## Yocto zeus Commit: d88d62c20d7d8da85f02edb170dae0280624ad7e $ git clone git://git.yoctoproject.org/poky poky_test $ cd poky_test $ . oe-init-build-env >> add recipe git-submodule-test taken from yocto master >> add layer meta-selftest and re-run init script $ bitbake git-submodule-test $ cd tmp/work/core2-64-poky-linux/git-submodule-test/1.0-r0/git $ git config --list http.sslcainfo=/etc/ssl/certs/ca-certificates.crt color.ui=auto core.repositoryformatversion=0 core.filemode=true core.bare=false core.logallrefupdates=true remote.origin.url=git://git.yoctoproject.org/git-submodule-test remote.origin.fetch=+refs/heads/*:refs/remotes/origin/* branch.master.remote=origin branch.master.merge=refs/heads/master submodule.bitbake.url=/media/hddata/GitLab/tmp/external/poky_test/build/downloads/git2/git.openembedded.org.bitbake submodule.bitbake-gitsm-test1.url=/media/hddata/GitLab/tmp/external/poky_test/build/downloads/git2/git.yoctoproject.org.bitbake-gitsm-test1 $ cd bitbake $ git config --list http.sslcainfo=/etc/ssl/certs/ca-certificates.crt color.ui=auto core.repositoryformatversion=0 core.filemode=true core.bare=false core.worktree=../../../bitbake remote.origin.url=git://git.openembedded.org/bitbake remote.origin.fetch=+refs/*:refs/* remote.origin.mirror=true ########################################################################## Yocto sumo Commit: b39f4146de84d7b36861859ec669d9c8e2ca77c6 $ git clone git://git.yoctoproject.org/poky poky_test $ cd poky_test $ . oe-init-build-env >> add layer meta-selftest $ bitbake git-submodule-test $ cd tmp/work/core2-64-poky-linux/git-submodule-test/1.0-r0/git $ git config --list http.sslcainfo=/etc/ssl/certs/ca-certificates.crt color.ui=auto core.repositoryformatversion=0 core.filemode=true core.bare=false core.logallrefupdates=true remote.origin.url=git://git.yoctoproject.org/git-submodule-test remote.origin.fetch=+refs/heads/*:refs/remotes/origin/* branch.master.remote=origin branch.master.merge=refs/heads/master submodule.bitbake.active=true submodule.bitbake.url=git://git.openembedded.org/bitbake submodule.bitbake-gitsm-test1.active=true submodule.bitbake-gitsm-test1.url=git://git.yoctoproject.org/bitbake-gitsm-test1 $ cd bitbake $ git config --list http.sslcainfo=/etc/ssl/certs/ca-certificates.crt color.ui=auto core.repositoryformatversion=0 core.filemode=true core.bare=false core.logallrefupdates=true core.worktree=../../../bitbake remote.origin.url=git://git.openembedded.org/bitbake remote.origin.fetch=+refs/heads/*:refs/remotes/origin/* branch.master.remote=origin branch.master.merge=refs/heads/master 88813 1 akuster 2020-12-24 18:39:38 +0000 bojan, Thank you for all the detail. 88814 2 akuster 2020-12-24 18:40:00 +0000 triaging 95076 3 richard.purdie 2023-03-07 13:53:44 +0000 Just to summarise, the issue as reported is that the repository origin information isn't the same as a direct clone of the repository. Due to the way the fetcher mirrors and checks out data for speed, this isn't really surprising. The question is whether we could/should be updating and reconfiguring the checkout to match a standard clone and I don't have an answer on that. It isn't something we've guaranteed to do so not a bug as such. 97133 4 randy.macleod 2023-10-30 15:37:29 +0000 Build move to 5.0 -- ../Randy 98837 5 randy.macleod 2024-04-25 01:13:55 +0000 Bulk move of 5.0 medium importance issues to 5.99. Move to 5.1 if you want to actively work on an issue. 98869 6 randy.macleod 2024-04-25 01:23:12 +0000 Move to 5.99 since no one is actively working on this issue. Move to 5.x if you start on a task, please. 102485 7 niqingliang2003 2025-07-18 08:40:36 +0000 main repo of gitsm can't work on local git mirror, PREMIRROR setting: git://.*/.* git:///mnt/datum/repositories/HOST/PATH;protocol=file \ gitsm://.*/.* git:///mnt/datum/repositories/HOST/PATH;protocol=file \ expected is: git:///mnt/datum/repositories/github.com/libjxl/libjxl.git;protocol=file but poky want: git:///mnt/datum/repositories/github.com/libjxl/git2_github.com.libjxl.libjxl.git.tar.gz;protocol=file DEBUG: For url ['gitsm', 'github.com', '/libjxl/libjxl.git', '', '', OrderedDict({'protocol': 'https', 'nobranch': '1'})] comparing ['git', '.*', '/.*', '', '', OrderedDict()] to ['git', '', '/mnt/datum/repositories/HOST/PATH', '', '', OrderedDict({'protocol': 'file'})] DEBUG: For url ['gitsm', 'github.com', '/libjxl/libjxl.git', '', '', OrderedDict({'protocol': 'https', 'nobranch': '1'})] comparing ['gitsm', '.*', '/.*', '', '', OrderedDict()] to ['git', '', '/mnt/datum/repositories/HOST/PATH', '', '', OrderedDict({'protocol': 'file'})] DEBUG: For url gitsm://github.com/libjxl/libjxl.git;protocol=https;nobranch=1 returning git:///mnt/datum/repositories/github.com/libjxl/git2_github.com.libjxl.libjxl.git.tar.gz;protocol=file DEBUG: Mirror fetch failure for url git:///mnt/datum/repositories/github.com/libjxl/git2_github.com.libjxl.libjxl.git.tar.gz;protocol=file (original url: gitsm://github.com/libjxl/libjxl.git;protocol=https;nobranch=1) DEBUG: URL: 'git:///mnt/datum/repositories/github.com/libjxl/git2_github.com.libjxl.libjxl.git.tar.gz;protocol=file' has invalid parameters. The url does not set any branch parameter or set nobranch=1. DEBUG: For url ['gitsm', 'github.com', '/libjxl/libjxl.git', '', '', OrderedDict({'protocol': 'https', 'nobranch': '1'})] comparing ['https?', '.*', '/.*', '', '', OrderedDict()] to ['file', '', '/mnt/datum/repositories/misc', '', '', OrderedDict()] DEBUG: For url ['gitsm', 'github.com', '/libjxl/libjxl.git', '', '', OrderedDict({'protocol': 'https', 'nobranch': '1'})] comparing ['git', 'sourceware.org', '/git/glibc.git', '', '', OrderedDict()] to ['https', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', OrderedDict()] DEBUG: For url ['gitsm', 'github.com', '/libjxl/libjxl.git', '', '', OrderedDict({'protocol': 'https', 'nobranch': '1'})] comparing ['git', 'sourceware.org', '/git/binutils-gdb.git', '', '', OrderedDict()] to ['https', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', OrderedDict()] DEBUG: Trying Upstream ERROR: Bitbake Fetcher Error: NetworkAccess('gitsm://github.com/libjxl/libjxl.git;protocol=https;nobranch=1', 'LANG=C git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all -c clone.defaultRemoteName=origin clone --bare --mirror https://github.com/libjxl/libjxl.git /mnt/datum/downloads/git2/github.com.libjxl.libjxl.git --progress') DEBUG: Python function base_do_fetch finished