Bug 14156

Summary:	fetch/gitsm: submodules are checked out as mirrors with altered origin information
Product:	[Build System, Metadata & Runtime] BitBake	Reporter:	bojan <bojan.milosevic.work>
Component:	bitbake	Assignee:	Unassigned <unassigned>
Status:	NEW ---	QA Contact:
Severity:	normal
Priority:	Medium	CC:	akuster, bojan.milosevic.work, niqingliang2003, poky.bs.watcher, poky.watcher
Version:	unspecified
Target Milestone:	5.99
Hardware:	x86
OS:	Multiple
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	Don't know

Description bojan 2020-12-18 10:55:20 UTC

The gitsm fetcher handling git repos with submodules does not work as expected, since the submodules are fetched as mirrored. 

When using gitsm in a recipe, the git repo and submodules will be correctly fetched at the requested revision.

The main git is ok and will correctly show the current commit and all the branches on origin and it will allow for further interaction (fetch, checkout commits/branches, commit, push).

A submodule git will be checked out as a mirror, hence it will show only local branches up to the one fetched, it will not have correct references for origin. It will not allow for further interaction with the git repo: no fetch, no pulling/checking out remote branches and no push of new commits.

This is a major issue for our workflow, since we use several projects with submodules (including in the kernel on some projects) and we need to work on them both within a yocto build and from outside yocto.

This behavior was not present on sumo, but I found it on zeus. I've performed some tests as reported below. I'm currently comparing sumo and zeus because they are included in the vendor's BSP releases we are working on.

To test/reproduce the behavior I used the git-submodule-test recipe and its git repo which contains recursive submodules (the recursive part is not of interest here, the behavior is the same with a single submodule). between tests I've deleted the build dir to have a clean build each time.

Below you can find the resulting git config in different cases:
1) manually cloning the repo -> OK
2) using poky master -> FAIL
3) using poky zeus -> FAIL
4) using poky sumo -> OK

my knowledge of git is limited, but I've identified the differences as that in cases (2) and (3) the main git has the submodules not set as active and the submodule git config has 
remote.origin.fetch=+refs/*:refs/*
remote.origin.mirror=true

I also tested copying the gitsm.py file from sumo to zeus and it works fine, so I'm using that as a workaround at the moment.

##########################################################################
Locally cloned git repo

$ git clone https://git.yoctoproject.org/git/git-submodule-test
$ cd git-submodule-test
$ git submodule update --init --recursive
$ git config --list
http.sslcainfo=/etc/ssl/certs/ca-certificates.crt
color.ui=auto
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
remote.origin.url=https://git.yoctoproject.org/git/git-submodule-test
remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
branch.master.remote=origin
branch.master.merge=refs/heads/master
submodule.bitbake.active=true
submodule.bitbake.url=git://git.openembedded.org/bitbake
submodule.bitbake-gitsm-test1.active=true
submodule.bitbake-gitsm-test1.url=git://git.yoctoproject.org/bitbake-gitsm-test1

$ cd bitbake
$ git config --list
http.sslcainfo=/etc/ssl/certs/ca-certificates.crt
color.ui=auto
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
core.worktree=../../../bitbake
remote.origin.url=git://git.openembedded.org/bitbake
remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
branch.master.remote=origin
branch.master.merge=refs/heads/master


##########################################################################
Yocto master
Commit: 38b45cd73eb360ce6dbe441ea497044acd27d777

$ git clone git://git.yoctoproject.org/poky poky_test
$ cd poky_test
$ . oe-init-build-env
>> add layer meta-selftest and re-run init script
$ bitbake git-submodule-test
$ cd tmp/work/core2-64-poky-linux/git-submodule-test/1.0-r0/git
$ git config --list
http.sslcainfo=/etc/ssl/certs/ca-certificates.crt
color.ui=auto
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
remote.origin.url=git://git.yoctoproject.org/git-submodule-test
remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
branch.master.remote=origin
branch.master.merge=refs/heads/master
submodule.bitbake.url=/media/hddata/GitLab/tmp/external/poky_test/build/downloads/git2/git.openembedded.org.bitbake
submodule.bitbake-gitsm-test1.url=/media/hddata/GitLab/tmp/external/poky_test/build/downloads/git2/git.yoctoproject.org.bitbake-gitsm-test1

$ cd bitbake
$ git config --list
http.sslcainfo=/etc/ssl/certs/ca-certificates.crt
color.ui=auto
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.worktree=../../../bitbake
remote.origin.url=git://git.openembedded.org/bitbake
remote.origin.fetch=+refs/*:refs/*
remote.origin.mirror=true


##########################################################################
Yocto zeus
Commit: d88d62c20d7d8da85f02edb170dae0280624ad7e

$ git clone git://git.yoctoproject.org/poky poky_test
$ cd poky_test
$ . oe-init-build-env
>> add recipe git-submodule-test taken from yocto master
>> add layer meta-selftest and re-run init script
$ bitbake git-submodule-test
$ cd tmp/work/core2-64-poky-linux/git-submodule-test/1.0-r0/git
$ git config --list
http.sslcainfo=/etc/ssl/certs/ca-certificates.crt
color.ui=auto
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
remote.origin.url=git://git.yoctoproject.org/git-submodule-test
remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
branch.master.remote=origin
branch.master.merge=refs/heads/master
submodule.bitbake.url=/media/hddata/GitLab/tmp/external/poky_test/build/downloads/git2/git.openembedded.org.bitbake
submodule.bitbake-gitsm-test1.url=/media/hddata/GitLab/tmp/external/poky_test/build/downloads/git2/git.yoctoproject.org.bitbake-gitsm-test1

$ cd bitbake
$ git config --list
http.sslcainfo=/etc/ssl/certs/ca-certificates.crt
color.ui=auto
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.worktree=../../../bitbake
remote.origin.url=git://git.openembedded.org/bitbake
remote.origin.fetch=+refs/*:refs/*
remote.origin.mirror=true


##########################################################################
Yocto sumo
Commit: b39f4146de84d7b36861859ec669d9c8e2ca77c6

$ git clone git://git.yoctoproject.org/poky poky_test
$ cd poky_test
$ . oe-init-build-env
>> add layer meta-selftest
$ bitbake git-submodule-test
$ cd tmp/work/core2-64-poky-linux/git-submodule-test/1.0-r0/git
$ git config --list
http.sslcainfo=/etc/ssl/certs/ca-certificates.crt
color.ui=auto
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
remote.origin.url=git://git.yoctoproject.org/git-submodule-test
remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
branch.master.remote=origin
branch.master.merge=refs/heads/master
submodule.bitbake.active=true
submodule.bitbake.url=git://git.openembedded.org/bitbake
submodule.bitbake-gitsm-test1.active=true
submodule.bitbake-gitsm-test1.url=git://git.yoctoproject.org/bitbake-gitsm-test1

$ cd bitbake
$ git config --list
http.sslcainfo=/etc/ssl/certs/ca-certificates.crt
color.ui=auto
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
core.worktree=../../../bitbake
remote.origin.url=git://git.openembedded.org/bitbake
remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
branch.master.remote=origin
branch.master.merge=refs/heads/master

Comment 1 Armin Kuster 2020-12-24 18:39:38 UTC

bojan,

Thank you for all the detail.

Comment 2 Armin Kuster 2020-12-24 18:40:00 UTC

triaging

Comment 3 Richard Purdie 2023-03-07 13:53:44 UTC

Just to summarise, the issue as reported is that the repository origin information isn't the same as a direct clone of the repository. Due to the way the fetcher mirrors and checks out data for speed, this isn't really surprising.

The question is whether we could/should be updating and reconfiguring the checkout to match a standard clone and I don't have an answer on that. It isn't something we've guaranteed to do so not a bug as such.

Comment 4 Randy MacLeod 2023-10-30 15:37:29 UTC

Build move to 5.0 -- ../Randy

Comment 5 Randy MacLeod 2024-04-25 01:13:55 UTC

Bulk move of 5.0 medium importance issues to 5.99.
Move to 5.1 if you want to actively work on an issue.

Comment 6 Randy MacLeod 2024-04-25 01:23:12 UTC

Move to 5.99 since no one is actively working on this issue.
Move to 5.x if you start on a task, please.

Comment 7 niqingliang2003 2025-07-18 08:40:36 UTC

main repo of gitsm can't work on local git mirror, 

PREMIRROR setting:
git://.*/.*     git:///mnt/datum/repositories/HOST/PATH;protocol=file \
gitsm://.*/.*     git:///mnt/datum/repositories/HOST/PATH;protocol=file \

expected is:
git:///mnt/datum/repositories/github.com/libjxl/libjxl.git;protocol=file

but poky want:
git:///mnt/datum/repositories/github.com/libjxl/git2_github.com.libjxl.libjxl.git.tar.gz;protocol=file

DEBUG: For url ['gitsm', 'github.com', '/libjxl/libjxl.git', '', '', OrderedDict({'protocol': 'https', 'nobranch': '1'})] comparing ['git', '.*', '/.*', '', '', OrderedDict()] to ['git', '', '/mnt/datum/repositories/HOST/PATH', '', '', OrderedDict({'protocol': 'file'})]
DEBUG: For url ['gitsm', 'github.com', '/libjxl/libjxl.git', '', '', OrderedDict({'protocol': 'https', 'nobranch': '1'})] comparing ['gitsm', '.*', '/.*', '', '', OrderedDict()] to ['git', '', '/mnt/datum/repositories/HOST/PATH', '', '', OrderedDict({'protocol': 'file'})]
DEBUG: For url gitsm://github.com/libjxl/libjxl.git;protocol=https;nobranch=1 returning git:///mnt/datum/repositories/github.com/libjxl/git2_github.com.libjxl.libjxl.git.tar.gz;protocol=file
DEBUG: Mirror fetch failure for url git:///mnt/datum/repositories/github.com/libjxl/git2_github.com.libjxl.libjxl.git.tar.gz;protocol=file (original url: gitsm://github.com/libjxl/libjxl.git;protocol=https;nobranch=1)
DEBUG: URL: 'git:///mnt/datum/repositories/github.com/libjxl/git2_github.com.libjxl.libjxl.git.tar.gz;protocol=file' has invalid parameters. The url does not set any branch parameter or set nobranch=1.
DEBUG: For url ['gitsm', 'github.com', '/libjxl/libjxl.git', '', '', OrderedDict({'protocol': 'https', 'nobranch': '1'})] comparing ['https?', '.*', '/.*', '', '', OrderedDict()] to ['file', '', '/mnt/datum/repositories/misc', '', '', OrderedDict()]
DEBUG: For url ['gitsm', 'github.com', '/libjxl/libjxl.git', '', '', OrderedDict({'protocol': 'https', 'nobranch': '1'})] comparing ['git', 'sourceware.org', '/git/glibc.git', '', '', OrderedDict()] to ['https', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', OrderedDict()]
DEBUG: For url ['gitsm', 'github.com', '/libjxl/libjxl.git', '', '', OrderedDict({'protocol': 'https', 'nobranch': '1'})] comparing ['git', 'sourceware.org', '/git/binutils-gdb.git', '', '', OrderedDict()] to ['https', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', OrderedDict()]
DEBUG: Trying Upstream
ERROR: Bitbake Fetcher Error: NetworkAccess('gitsm://github.com/libjxl/libjxl.git;protocol=https;nobranch=1', 'LANG=C git -c gc.autoDetach=false -c core.pager=cat -c safe.bareRepository=all -c clone.defaultRemoteName=origin clone --bare --mirror https://github.com/libjxl/libjxl.git /mnt/datum/downloads/git2/github.com.libjxl.libjxl.git --progress')
DEBUG: Python function base_do_fetch finished