The gitsm fetcher handling git repos with submodules does not work as expected, since the submodules are fetched as mirrored. When using gitsm in a recipe, the git repo and submodules will be correctly fetched at the requested revision. The main git is ok and will correctly show the current commit and all the branches on origin and it will allow for further interaction (fetch, checkout commits/branches, commit, push). A submodule git will be checked out as a mirror, hence it will show only local branches up to the one fetched, it will not have correct references for origin. It will not allow for further interaction with the git repo: no fetch, no pulling/checking out remote branches and no push of new commits. This is a major issue for our workflow, since we use several projects with submodules (including in the kernel on some projects) and we need to work on them both within a yocto build and from outside yocto. This behavior was not present on sumo, but I found it on zeus. I've performed some tests as reported below. I'm currently comparing sumo and zeus because they are included in the vendor's BSP releases we are working on. To test/reproduce the behavior I used the git-submodule-test recipe and its git repo which contains recursive submodules (the recursive part is not of interest here, the behavior is the same with a single submodule). between tests I've deleted the build dir to have a clean build each time. Below you can find the resulting git config in different cases: 1) manually cloning the repo -> OK 2) using poky master -> FAIL 3) using poky zeus -> FAIL 4) using poky sumo -> OK my knowledge of git is limited, but I've identified the differences as that in cases (2) and (3) the main git has the submodules not set as active and the submodule git config has remote.origin.fetch=+refs/*:refs/* remote.origin.mirror=true I also tested copying the gitsm.py file from sumo to zeus and it works fine, so I'm using that as a workaround at the moment. ########################################################################## Locally cloned git repo $ git clone https://git.yoctoproject.org/git/git-submodule-test $ cd git-submodule-test $ git submodule update --init --recursive $ git config --list http.sslcainfo=/etc/ssl/certs/ca-certificates.crt color.ui=auto core.repositoryformatversion=0 core.filemode=true core.bare=false core.logallrefupdates=true remote.origin.url=https://git.yoctoproject.org/git/git-submodule-test remote.origin.fetch=+refs/heads/*:refs/remotes/origin/* branch.master.remote=origin branch.master.merge=refs/heads/master submodule.bitbake.active=true submodule.bitbake.url=git://git.openembedded.org/bitbake submodule.bitbake-gitsm-test1.active=true submodule.bitbake-gitsm-test1.url=git://git.yoctoproject.org/bitbake-gitsm-test1 $ cd bitbake $ git config --list http.sslcainfo=/etc/ssl/certs/ca-certificates.crt color.ui=auto core.repositoryformatversion=0 core.filemode=true core.bare=false core.logallrefupdates=true core.worktree=../../../bitbake remote.origin.url=git://git.openembedded.org/bitbake remote.origin.fetch=+refs/heads/*:refs/remotes/origin/* branch.master.remote=origin branch.master.merge=refs/heads/master ########################################################################## Yocto master Commit: 38b45cd73eb360ce6dbe441ea497044acd27d777 $ git clone git://git.yoctoproject.org/poky poky_test $ cd poky_test $ . oe-init-build-env >> add layer meta-selftest and re-run init script $ bitbake git-submodule-test $ cd tmp/work/core2-64-poky-linux/git-submodule-test/1.0-r0/git $ git config --list http.sslcainfo=/etc/ssl/certs/ca-certificates.crt color.ui=auto core.repositoryformatversion=0 core.filemode=true core.bare=false core.logallrefupdates=true remote.origin.url=git://git.yoctoproject.org/git-submodule-test remote.origin.fetch=+refs/heads/*:refs/remotes/origin/* branch.master.remote=origin branch.master.merge=refs/heads/master submodule.bitbake.url=/media/hddata/GitLab/tmp/external/poky_test/build/downloads/git2/git.openembedded.org.bitbake submodule.bitbake-gitsm-test1.url=/media/hddata/GitLab/tmp/external/poky_test/build/downloads/git2/git.yoctoproject.org.bitbake-gitsm-test1 $ cd bitbake $ git config --list http.sslcainfo=/etc/ssl/certs/ca-certificates.crt color.ui=auto core.repositoryformatversion=0 core.filemode=true core.bare=false core.worktree=../../../bitbake remote.origin.url=git://git.openembedded.org/bitbake remote.origin.fetch=+refs/*:refs/* remote.origin.mirror=true ########################################################################## Yocto zeus Commit: d88d62c20d7d8da85f02edb170dae0280624ad7e $ git clone git://git.yoctoproject.org/poky poky_test $ cd poky_test $ . oe-init-build-env >> add recipe git-submodule-test taken from yocto master >> add layer meta-selftest and re-run init script $ bitbake git-submodule-test $ cd tmp/work/core2-64-poky-linux/git-submodule-test/1.0-r0/git $ git config --list http.sslcainfo=/etc/ssl/certs/ca-certificates.crt color.ui=auto core.repositoryformatversion=0 core.filemode=true core.bare=false core.logallrefupdates=true remote.origin.url=git://git.yoctoproject.org/git-submodule-test remote.origin.fetch=+refs/heads/*:refs/remotes/origin/* branch.master.remote=origin branch.master.merge=refs/heads/master submodule.bitbake.url=/media/hddata/GitLab/tmp/external/poky_test/build/downloads/git2/git.openembedded.org.bitbake submodule.bitbake-gitsm-test1.url=/media/hddata/GitLab/tmp/external/poky_test/build/downloads/git2/git.yoctoproject.org.bitbake-gitsm-test1 $ cd bitbake $ git config --list http.sslcainfo=/etc/ssl/certs/ca-certificates.crt color.ui=auto core.repositoryformatversion=0 core.filemode=true core.bare=false core.worktree=../../../bitbake remote.origin.url=git://git.openembedded.org/bitbake remote.origin.fetch=+refs/*:refs/* remote.origin.mirror=true ########################################################################## Yocto sumo Commit: b39f4146de84d7b36861859ec669d9c8e2ca77c6 $ git clone git://git.yoctoproject.org/poky poky_test $ cd poky_test $ . oe-init-build-env >> add layer meta-selftest $ bitbake git-submodule-test $ cd tmp/work/core2-64-poky-linux/git-submodule-test/1.0-r0/git $ git config --list http.sslcainfo=/etc/ssl/certs/ca-certificates.crt color.ui=auto core.repositoryformatversion=0 core.filemode=true core.bare=false core.logallrefupdates=true remote.origin.url=git://git.yoctoproject.org/git-submodule-test remote.origin.fetch=+refs/heads/*:refs/remotes/origin/* branch.master.remote=origin branch.master.merge=refs/heads/master submodule.bitbake.active=true submodule.bitbake.url=git://git.openembedded.org/bitbake submodule.bitbake-gitsm-test1.active=true submodule.bitbake-gitsm-test1.url=git://git.yoctoproject.org/bitbake-gitsm-test1 $ cd bitbake $ git config --list http.sslcainfo=/etc/ssl/certs/ca-certificates.crt color.ui=auto core.repositoryformatversion=0 core.filemode=true core.bare=false core.logallrefupdates=true core.worktree=../../../bitbake remote.origin.url=git://git.openembedded.org/bitbake remote.origin.fetch=+refs/heads/*:refs/remotes/origin/* branch.master.remote=origin branch.master.merge=refs/heads/master
bojan, Thank you for all the detail.
triaging
Just to summarise, the issue as reported is that the repository origin information isn't the same as a direct clone of the repository. Due to the way the fetcher mirrors and checks out data for speed, this isn't really surprising. The question is whether we could/should be updating and reconfiguring the checkout to match a standard clone and I don't have an answer on that. It isn't something we've guaranteed to do so not a bug as such.
Build move to 5.0 -- ../Randy
Bulk move of 5.0 medium importance issues to 5.99. Move to 5.1 if you want to actively work on an issue.
Move to 5.99 since no one is actively working on this issue. Move to 5.x if you start on a task, please.
main repo of gitsm can't work on local git mirror, PREMIRROR setting: git://.*/.* git:///mnt/datum/repositories/HOST/PATH;protocol=file \ gitsm://.*/.* git:///mnt/datum/repositories/HOST/PATH;protocol=file \ expected is: git:///mnt/datum/repositories/github.com/libjxl/libjxl.git;protocol=file but poky want: git:///mnt/datum/repositories/github.com/libjxl/git2_github.com.libjxl.libjxl.git.tar.gz;protocol=file DEBUG: For url ['gitsm', 'github.com', '/libjxl/libjxl.git', '', '', OrderedDict({'protocol': 'https', 'nobranch': '1'})] comparing ['git', '.*', '/.*', '', '', OrderedDict()] to ['git', '', '/mnt/datum/repositories/HOST/PATH', '', '', OrderedDict({'protocol': 'file'})] DEBUG: For url ['gitsm', 'github.com', '/libjxl/libjxl.git', '', '', OrderedDict({'protocol': 'https', 'nobranch': '1'})] comparing ['gitsm', '.*', '/.*', '', '', OrderedDict()] to ['git', '', '/mnt/datum/repositories/HOST/PATH', '', '', OrderedDict({'protocol': 'file'})] DEBUG: For url gitsm://github.com/libjxl/libjxl.git;protocol=https;nobranch=1 returning git:///mnt/datum/repositories/github.com/libjxl/git2_github.com.libjxl.libjxl.git.tar.gz;protocol=file DEBUG: Mirror fetch failure for url git:///mnt/datum/repositories/github.com/libjxl/git2_github.com.libjxl.libjxl.git.tar.gz;protocol=file (original url: gitsm://github.com/libjxl/libjxl.git;protocol=https;nobranch=1) DEBUG: URL: 'git:///mnt/datum/repositories/github.com/libjxl/git2_github.com.libjxl.libjxl.git.tar.gz;protocol=file' has invalid parameters. The url does not set any branch parameter or set nobranch=1. DEBUG: For url ['gitsm', 'github.com', '/libjxl/libjxl.git', '', '', OrderedDict({'protocol': 'https', 'nobranch': '1'})] comparing ['https?', '.*', '/.*', '', '', OrderedDict()] to ['file', '', '/mnt/datum/repositories/misc', '', '', OrderedDict()] DEBUG: For url ['gitsm', 'github.com', '/libjxl/libjxl.git', '', '', OrderedDict({'protocol': 'https', 'nobranch': '1'})] comparing ['git', 'sourceware.org', '/git/glibc.git', '', '', OrderedDict()] to ['https', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', OrderedDict()] DEBUG: For url ['gitsm', 'github.com', '/libjxl/libjxl.git', '', '', OrderedDict({'protocol': 'https', 'nobranch': '1'})] comparing ['git', 'sourceware.org', '/git/binutils-gdb.git', '', '', OrderedDict()] to ['https', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', OrderedDict()] DEBUG: Trying Upstream ERROR: Bitbake Fetcher Error: NetworkAccess('gitsm://github.com/libjxl/libjxl.git;protocol=https;nobranch=1', 'LANG=C git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all -c clone.defaultRemoteName=origin clone --bare --mirror https://github.com/libjxl/libjxl.git /mnt/datum/downloads/git2/github.com.libjxl.libjxl.git --progress') DEBUG: Python function base_do_fetch finished