Bug 13665

Summary: gitsm fetcher and PREMIRROR does not use git to clone repositories
Product: [Build System, Metadata & Runtime] BitBake Reporter: Konrad Scherer <konrad.scherer>
Component: bitbakeAssignee: Pavel Zhukov <pavel>
Status: RESOLVED NOTABUG QA Contact:
Severity: normal    
Priority: Medium CC: liezhi.yang, poky.bs.watcher, poky.watcher, randy.macleod, richard.purdie
Version: 3.1   
Target Milestone: 4.1   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Don't know

Description Konrad Scherer 2019-12-02 16:58:15 UTC 
I recently found a strange problem involving the gitsm fetcher and our PREMIRRORS and I was able to reproduce it using poky.

> git clone git://git.yoctoproject.org/poky poky1
> cd poky1
> . oe-init-build-env
> bitbake ovmf

The ovmf recipe has git submodules inside git submodules. This step works fine.

I then exposed the downloads directory over http (http://<host>/downloads) and the downloads/git2 directory over http with git-http-backend enabled (http://<host>/git).

I noticed that there are shallow clone tarballs in downloads that match the git repos in downloads/git2. For example:

downloads/git2/boringssl.googlesource.com.boringssl/
downloads/git2_boringssl.googlesource.com.boringssl.tar.gz

Since these are duplicates of the git repos, I deleted the tar.gz files

> rm -f downloads/git2_*

<new shell>

> git clone git://git.yoctoproject.org/poky poky2
> cd poky2
> . oe-init-build-env
> cat >> conf/local.conf <<EOF
WRS_MIRROR_HOST = "<host>"
BB_ALLOWED_NETWORKS = "${WRS_MIRROR_HOST}"

BB_NO_NETWORK = '0'
BB_FETCH_PREMIRRORONLY = '1'

PREMIRRORS_append = " \
     .*://.*/.* http://${WRS_MIRROR_HOST}/downloads/ \n \
     git://.*/.* git://${WRS_MIRROR_HOST}/git/MIRRORNAME;protocol=http \n \
     gitsm://.*/.* git://${WRS_MIRROR_HOST}/git/MIRRORNAME;protocol=http \n \
"
CONNECTIVITY_CHECK_URIS = ""
EOF
> bitbake ovmf

This fails with:

ERROR: ovmf-edk2-stable201905-r0 do_unpack: gitsm: submodule unpack failed: UnpackError Unpack failure for URL: 'gitsm://github.com/openssl/openssl;protocol=https;name=CryptoPkg/Library/OpensslLib/openssl;subpath=CryptoPkg/Library/OpensslLib/openssl;bareclone=1;nobranch=1'. No up to date source found: clone directory not available or not up to date: /ala-lpggp22/kscherer/gitsm/poky2/build/downloads/git2/github.com.openssl.openssl; shallow clone not enabled

If I leave the tarballs in the PREMIRROR the build succeeds. The other recipes that use git are fetched properly. The logs don't have any warnings before the error and I verified that the objects are indeed present in the repo.

Are my PREMIRROR settings correct? Is this a bug in the gitsm fetcher?
Comment 1 Randy MacLeod 2020-01-10 15:00:52 UTC 
Mark, Do you think you'll work on this in the next few weeks?
Comment 2 Pavel Zhukov 2022-05-28 13:43:22 UTC 
It looks like a feature rather than a bug.

BB_FETCH_PREMIRRORONLY functionality is kind of broken in the current bitbake (see https://bugzilla.yoctoproject.org/show_bug.cgi?id=13353 https://bugzilla.yoctoproject.org/show_bug.cgi?id=9061 https://bugzilla.yoctoproject.org/show_bug.cgi?id=13233). Trying to run git clone (and ls-remote) from upstream if premirror has been specified is a actual bug while erroring out if BB_FETCH_PREMIRRORONLY has been specified but mirror doesn't contain revision is a proper behaviour to not break reproducible builds.
Comment 3 Pavel Zhukov 2022-06-09 07:27:17 UTC 
Fixes for mentioned bugs are merged and behaviour should be consistent now.
If BB_FETCH_PREMIRRORONLY = '1' has been specified and revision not in the PREMIRROR the build will fail (with NetworkAccess error). 
Fetchers will not try upstream for git fetcher anymore. 

https://git.openembedded.org/bitbake/commit/?id=b47ecab3e3aad5c5c376ec023aa82a51aa0f3b86