| Summary: | git shallow can't work with premirror | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] BitBake | Reporter: | niqingliang2003 <niqingliang2003> | ||||||
| Component: | bitbake | Assignee: | Robert Yang <liezhi.yang> | ||||||
| Status: | RESOLVED FIXED | QA Contact: | |||||||
| Severity: | normal | ||||||||
| Priority: | Medium+ | CC: | liezhi.yang, niqingliang2003, poky.bs.watcher, poky.watcher, randy.macleod, richard.purdie, stefan-koch | ||||||
| Version: | 6.0 | ||||||||
| Target Milestone: | 5.3 M2 | ||||||||
| Hardware: | x86 | ||||||||
| OS: | Multiple | ||||||||
| Whiteboard: | |||||||||
| OS type for building Yocto: | --- | Type of Regression: | --- | ||||||
| Verified: | Documentation change: | Don't know | |||||||
| Attachments: |
|
||||||||
So, My thought: If the bitbake expect the shallow file whose name is official path, then should generate shallow file with official path name, but not mirror path name. If the bitbake expect bare repo, should keep bare repo, but not link it to a tar file. The root cause maybe the premirrors which is expected a tar, but indeed a bare repo. If the bitbake want a tar when git protocol is file, should refuse a local bare repo when fetching. anyway, the xxx.git located in git2 should not link to a shallow Robert, this seems like something you would know about. Can you own the bug and fix it? Yes, it works well for us, I will look into it. Hello, what about the status? any wordaround? Please try the patch here: https://lists.openembedded.org/g/bitbake-devel/message/16371 I guess that it can fix your issue. I checked the commit, which is committed on Jun 22, 2024, maybe I had a time travel? when it will be pushed into main branch? The patch has been rejected, because bitbake doesn't want to support create tarballs from local mirrors. You can find the info from the email threads. this patch still not work: the basename of ud.path dropped the sourceware.org: (please see the git shallow file name) gitsrcname = '%s' % os.path.basename(ud.path) and I think the xxx.git located in git2 should be a bare repo. lrwxrwxrwx 1 ll ll 78 May 12 16:44 ../../downloads/git2/sourceware.org.git.glibc.git -> /mnt/datum/downloads/gitshallow_glibc.git_0557361-1_release.2.17.master.tar.gz -rw-r--r-- 1 ll ll 6 May 12 16:44 ../../downloads/git2/sourceware.org.git.glibc.git.done -rw-r--r-- 1 ll ll 24414206 May 12 16:44 ../../downloads/gitshallow_glibc.git_0557361-1_release.2.17.master.tar.gz -rw-r--r-- 1 ll ll 0 May 12 16:44 ../../downloads/gitshallow_glibc.git_0557361-1_release.2.17.master.tar.gz.done If bitbake doesn't want to support create tarballs from local mirrors, why the bitbake generated tarball from local mirror on my env? maybe I missed some configuration? The mirror in your repo is: /mnt/datum/repositories/sourceware.org/git/glibc.git/ I think its mirrorname should be: sourceware.org.git.glibc.git Not glibc.git. Just like the ones here: https://downloads.yoctoproject.org/mirror/sources/git2/ (In reply to niqingliang2003 from comment #10) > If bitbake doesn't want to support create tarballs from local mirrors, > why the bitbake generated tarball from local mirror on my env? It can generate that, but the taqrball doesn't work, you can reply to the email thread to see whether there is a chance to make it get merged (If the patch works for you). > > maybe I missed some configuration? the shallow file name is just reserved glibc.git: os.path.basename(ud.path) the xxx.git located in git2 has sourceware, but linked to the gitshallow. the unpacking still can't find both bare repo or shallow with sourceware. I replied the thread at: https://lists.openembedded.org/g/bitbake-devel/topic/106812685?p=Created,,,20,1,0,0 If the bitbake don't want to generate shallow for local mirror, maybe we should set shallow_skip_fast as True if protocol is "file": - ud.shallow_skip_fast = False + ud.shallow_skip_fast = (ud.proto == "file") "bitbake doesn't want to support create tarballs from local mirrors" so only local mirror? how to define "local mirror"? I don't know if it is equal to 'proto=file'. maybe you mean "all mirror"? Created attachment 5115 [details]
test git shallow for local mirror
test git shallow for local mirror
add the script for test git shallow for local mirror. add test script to reproduce the bug Created attachment 5116 [details]
test git shallow for local mirror (fix download directory)
* Have reproduced the issue with the second script: test git shallow for local mirror (fix download directory) (1.15 KB, application/x-shellscript) 2025-05-16 01:21 UTC, niqingliang2003 * Robert's mentioned patch did not solve the issue, since ud.path contains the PREMIRROR prefix path: https://lists.openembedded.org/g/bitbake-devel/message/16371 * When adding within `fetch2/__init__.py`, `def try_mirror_url(fetch, origud, ud, ld, check = False)` + if ud.shallow: + ensure_symlink(ud.fullshallow, origud.fullshallow) before `ensure_symlink(ud.localpath, origud.localpath)` * This would solve the issue * Then these lines are within `fetch2/git.py` are not needed: - ud.shallow_skip_fast = False + ud.shallow_skip_fast = (ud.proto == "file") 🖒 Have submitted a patch that adds an additional symlink to the mailinglist: https://lists.openembedded.org/g/bitbake-devel/topic/patch_0_1_fetch2_ensure_a/113226201 This should solve this bug. I think the next step here is to see if we can add this to bitbake-selftest (in lib/bb/tests/fetch.py) and if the patch can be tweaked as I suggested on the mailing list... 1. Have already prepared a selftest, but not submitted yet 2. I'll start now to figure out how to use that tweak Fixed with 6e1434d9 commit on bitbake's master branch. Fix merged as above, thanks Stefan |
latest bitbake updated the git shallow logic. and the glibc recipe who enabled shallow can't work on my environment. I'm using PREMIRRORS_prepend, redirect the git repo of glibc to my local repo: git://.*/.* git:///mnt/datum/repositories/HOST/PATH;protocol=file the local path of glibc repo is a bare git repo, but not tarball. (/mnt/datum/repositories/sourceware.org/git/glibc.git/) the bitbake will create a gitshallow of local repo, the filename of it contain local path. and bitbake will create a symlink in downloads/git2/xxx.git which is linked to the shallow tarball, but, indeed xxx.git located in git2 should be a bare repository. so when unpacking, bitbake can't find the source code of glibc: 1. can't find valid bare repo in git2 (which is a symlink to tar.gz) 2. can't find valid shallow (whose name should be official path) error info: glibc-2.17-r0 do_unpack: Bitbake Fetcher Error: UnpackError('No up to date source found: clone directory not available or not up to date: /mnt/datum/downloads/git2/sourceware.org.git.glibc.git; shallow clone not available: /mnt/datum/downloads/gitshallow_sourceware.org.git.glibc.git_0557361-1_release.2.17.master.tar.gz', 'git://sourceware.org/git/glibc.git;protocol=https;branch=release/2.17/master;name=glibc') the file system structure is: lrwxrwxrwx 1 ll ll 120 May 7 17:04 /mnt/datum/downloads/git2/sourceware.org.git.glibc.git -> /mnt/datum/downloads/gitshallow_mnt.datum.repositories.sourceware.org.git.glibc.git_0557361-1_release.2.17.master.tar.gz -rw-r--r-- 1 ll ll 6 May 7 17:04 /mnt/datum/downloads/git2/sourceware.org.git.glibc.git.done -rw-r--r-- 1 ll ll 0 May 7 17:09 /mnt/datum/downloads/git2/sourceware.org.git.glibc.git.lock -rw-r--r-- 1 ll ll 24413461 May 7 17:04 /mnt/datum/downloads/gitshallow_mnt.datum.repositories.sourceware.org.git.glibc.git_0557361-1_release.2.17.master.tar.gz -rw-r--r-- 1 ll ll 0 May 7 17:04 /mnt/datum/downloads/gitshallow_mnt.datum.repositories.sourceware.org.git.glibc.git_0557361-1_release.2.17.master.tar.gz.done