15856 2025-05-08 00:20:43 +0000 git shallow can't work with premirror 2025-06-04 14:35:59 +0000 1 1 7 Build System, Metadata & Runtime BitBake bitbake 6.0 x86 Multiple RESOLVED FIXED Medium+ normal 5.3 M2 1 niqingliang2003 liezhi.yang liezhi.yang niqingliang2003 poky.bs.watcher poky.watcher randy.macleod richard.purdie stefan-koch --- --- Don't know oldest_to_newest 101901 0 niqingliang2003 2025-05-08 00:20:43 +0000 latest bitbake updated the git shallow logic. and the glibc recipe who enabled shallow can't work on my environment. I'm using PREMIRRORS_prepend, redirect the git repo of glibc to my local repo: git://.*/.* git:///mnt/datum/repositories/HOST/PATH;protocol=file the local path of glibc repo is a bare git repo, but not tarball. (/mnt/datum/repositories/sourceware.org/git/glibc.git/) the bitbake will create a gitshallow of local repo, the filename of it contain local path. and bitbake will create a symlink in downloads/git2/xxx.git which is linked to the shallow tarball, but, indeed xxx.git located in git2 should be a bare repository. so when unpacking, bitbake can't find the source code of glibc: 1. can't find valid bare repo in git2 (which is a symlink to tar.gz) 2. can't find valid shallow (whose name should be official path) error info: glibc-2.17-r0 do_unpack: Bitbake Fetcher Error: UnpackError('No up to date source found: clone directory not available or not up to date: /mnt/datum/downloads/git2/sourceware.org.git.glibc.git; shallow clone not available: /mnt/datum/downloads/gitshallow_sourceware.org.git.glibc.git_0557361-1_release.2.17.master.tar.gz', 'git://sourceware.org/git/glibc.git;protocol=https;branch=release/2.17/master;name=glibc') the file system structure is: lrwxrwxrwx 1 ll ll 120 May 7 17:04 /mnt/datum/downloads/git2/sourceware.org.git.glibc.git -> /mnt/datum/downloads/gitshallow_mnt.datum.repositories.sourceware.org.git.glibc.git_0557361-1_release.2.17.master.tar.gz -rw-r--r-- 1 ll ll 6 May 7 17:04 /mnt/datum/downloads/git2/sourceware.org.git.glibc.git.done -rw-r--r-- 1 ll ll 0 May 7 17:09 /mnt/datum/downloads/git2/sourceware.org.git.glibc.git.lock -rw-r--r-- 1 ll ll 24413461 May 7 17:04 /mnt/datum/downloads/gitshallow_mnt.datum.repositories.sourceware.org.git.glibc.git_0557361-1_release.2.17.master.tar.gz -rw-r--r-- 1 ll ll 0 May 7 17:04 /mnt/datum/downloads/gitshallow_mnt.datum.repositories.sourceware.org.git.glibc.git_0557361-1_release.2.17.master.tar.gz.done 101902 1 niqingliang2003 2025-05-08 00:28:37 +0000 So, My thought: If the bitbake expect the shallow file whose name is official path, then should generate shallow file with official path name, but not mirror path name. If the bitbake expect bare repo, should keep bare repo, but not link it to a tar file. The root cause maybe the premirrors which is expected a tar, but indeed a bare repo. If the bitbake want a tar when git protocol is file, should refuse a local bare repo when fetching. 101903 2 niqingliang2003 2025-05-08 00:42:48 +0000 anyway, the xxx.git located in git2 should not link to a shallow 101910 3 randy.macleod 2025-05-08 14:50:11 +0000 Robert, this seems like something you would know about. Can you own the bug and fix it? 101911 4 liezhi.yang 2025-05-08 14:52:34 +0000 Yes, it works well for us, I will look into it. 101925 5 niqingliang2003 2025-05-12 02:22:47 +0000 Hello, what about the status? any wordaround? 101926 6 liezhi.yang 2025-05-12 06:47:45 +0000 Please try the patch here: https://lists.openembedded.org/g/bitbake-devel/message/16371 I guess that it can fix your issue. 101927 7 niqingliang2003 2025-05-12 07:21:31 +0000 I checked the commit, which is committed on Jun 22, 2024, maybe I had a time travel? when it will be pushed into main branch? 101928 8 liezhi.yang 2025-05-12 07:29:42 +0000 The patch has been rejected, because bitbake doesn't want to support create tarballs from local mirrors. You can find the info from the email threads. 101929 9 niqingliang2003 2025-05-12 08:48:43 +0000 this patch still not work: the basename of ud.path dropped the sourceware.org: (please see the git shallow file name) gitsrcname = '%s' % os.path.basename(ud.path) and I think the xxx.git located in git2 should be a bare repo. lrwxrwxrwx 1 ll ll 78 May 12 16:44 ../../downloads/git2/sourceware.org.git.glibc.git -> /mnt/datum/downloads/gitshallow_glibc.git_0557361-1_release.2.17.master.tar.gz -rw-r--r-- 1 ll ll 6 May 12 16:44 ../../downloads/git2/sourceware.org.git.glibc.git.done -rw-r--r-- 1 ll ll 24414206 May 12 16:44 ../../downloads/gitshallow_glibc.git_0557361-1_release.2.17.master.tar.gz -rw-r--r-- 1 ll ll 0 May 12 16:44 ../../downloads/gitshallow_glibc.git_0557361-1_release.2.17.master.tar.gz.done 101930 10 niqingliang2003 2025-05-12 08:50:42 +0000 If bitbake doesn't want to support create tarballs from local mirrors, why the bitbake generated tarball from local mirror on my env? maybe I missed some configuration? 101931 11 liezhi.yang 2025-05-12 08:56:39 +0000 The mirror in your repo is: /mnt/datum/repositories/sourceware.org/git/glibc.git/ I think its mirrorname should be: sourceware.org.git.glibc.git Not glibc.git. Just like the ones here: https://downloads.yoctoproject.org/mirror/sources/git2/ 101932 12 liezhi.yang 2025-05-12 08:58:12 +0000 (In reply to niqingliang2003 from comment #10) > If bitbake doesn't want to support create tarballs from local mirrors, > why the bitbake generated tarball from local mirror on my env? It can generate that, but the taqrball doesn't work, you can reply to the email thread to see whether there is a chance to make it get merged (If the patch works for you). > > maybe I missed some configuration? 101933 13 niqingliang2003 2025-05-12 09:11:41 +0000 the shallow file name is just reserved glibc.git: os.path.basename(ud.path) the xxx.git located in git2 has sourceware, but linked to the gitshallow. the unpacking still can't find both bare repo or shallow with sourceware. 101934 14 niqingliang2003 2025-05-12 09:27:13 +0000 I replied the thread at: https://lists.openembedded.org/g/bitbake-devel/topic/106812685?p=Created,,,20,1,0,0 101935 15 niqingliang2003 2025-05-12 09:28:44 +0000 If the bitbake don't want to generate shallow for local mirror, maybe we should set shallow_skip_fast as True if protocol is "file": - ud.shallow_skip_fast = False + ud.shallow_skip_fast = (ud.proto == "file") 101936 16 niqingliang2003 2025-05-12 09:38:20 +0000 "bitbake doesn't want to support create tarballs from local mirrors" so only local mirror? how to define "local mirror"? I don't know if it is equal to 'proto=file'. maybe you mean "all mirror"? 101970 17 5115 niqingliang2003 2025-05-16 01:12:48 +0000 Created attachment 5115 test git shallow for local mirror test git shallow for local mirror 101971 18 niqingliang2003 2025-05-16 01:13:19 +0000 add the script for test git shallow for local mirror. 101972 19 niqingliang2003 2025-05-16 01:19:06 +0000 add test script to reproduce the bug 101973 20 5116 niqingliang2003 2025-05-16 01:21:57 +0000 Created attachment 5116 test git shallow for local mirror (fix download directory) 101985 21 stefan-koch 2025-05-16 17:11:29 +0000 * Have reproduced the issue with the second script: test git shallow for local mirror (fix download directory) (1.15 KB, application/x-shellscript) 2025-05-16 01:21 UTC, niqingliang2003 * Robert's mentioned patch did not solve the issue, since ud.path contains the PREMIRROR prefix path: https://lists.openembedded.org/g/bitbake-devel/message/16371 * When adding within `fetch2/__init__.py`, `def try_mirror_url(fetch, origud, ud, ld, check = False)` + if ud.shallow: + ensure_symlink(ud.fullshallow, origud.fullshallow) before `ensure_symlink(ud.localpath, origud.localpath)` * This would solve the issue * Then these lines are within `fetch2/git.py` are not needed: - ud.shallow_skip_fast = False + ud.shallow_skip_fast = (ud.proto == "file") 101988 22 niqingliang2003 2025-05-19 05:02:50 +0000 🖒 102015 23 stefan-koch 2025-05-21 09:16:33 +0000 Have submitted a patch that adds an additional symlink to the mailinglist: https://lists.openembedded.org/g/bitbake-devel/topic/patch_0_1_fetch2_ensure_a/113226201 This should solve this bug. 102069 24 richard.purdie 2025-05-26 13:05:36 +0000 I think the next step here is to see if we can add this to bitbake-selftest (in lib/bb/tests/fetch.py) and if the patch can be tweaked as I suggested on the mailing list... 102070 25 stefan-koch 2025-05-26 13:58:35 +0000 1. Have already prepared a selftest, but not submitted yet 2. I'll start now to figure out how to use that tweak 102087 26 stefan-koch 2025-05-28 14:27:41 +0000 Fixed with 6e1434d9 commit on bitbake's master branch. 102135 27 richard.purdie 2025-06-04 14:35:59 +0000 Fix merged as above, thanks Stefan 5115 2025-05-16 01:12:48 +0000 2025-05-16 01:12:48 +0000 test git shallow for local mirror setupbuilddir application/x-shellscript 1150 niqingliang2003 IyEvdXNyL2Jpbi9lbnYgYmFzaAoKc2V0IC1lCgojIyMjIHByZXBhcmUgbmVlZGVkIHJlcG8gKGZv ciB0ZXN0IGxvY2FsIG1pcnJvcikKI1BSRU1JUlJPUj0iL21udC9kYXR1bS9yZXBvc2l0b3JpZXMv IgpQUkVNSVJST1I9Imh0dHBzOi8vIgpnaXQgY2xvbmUgLS1kZXB0aCAxICR7UFJFTUlSUk9SfWdp dGh1Yi5jb20veW9jdG9wcm9qZWN0L3Bva3kuZ2l0IHBva3kKZ2l0IGNsb25lIC0tbWlycm9yICR7 UFJFTUlSUk9SfXNvdXJjZXdhcmUub3JnL2dpdC9nbGliYy5naXQgc291cmNld2FyZS5vcmcvZ2l0 L2dsaWJjLmdpdAp1bnNldCBQUkVNSVJST1IKCiMjIyMgcHJlcGFyZSBkaXJlY3RvcnkKU0VMRl9E SVI9ImByZWFkbGluayAtZmUgJHtCQVNIX1NPVVJDRX0gfCB4YXJncyBkaXJuYW1lYCIKUE9LWURJ Uj0iYHJlYWRsaW5rIC1mZSAke1NFTEZfRElSfS9wb2t5L2AiCkJVSUxEX0RJUj0iJHtTRUxGX0RJ Un0vYnVpbGQiCmlmICEgWyAtZCAke0JVSUxEX0RJUn0gXTsgdGhlbgoJbWtkaXIgLXAgJHtCVUlM RF9ESVJ9CmZpCgojIyMjIHNldCBlbnZpcm9ubWVudApPRV9FTlZfU0NSSVBUPSIke1BPS1lESVJ9 L29lLWluaXQtYnVpbGQtZW52IgpCSVRCQUtFX0JJTl9ESVI9IiR7UE9LWURJUn0vYml0YmFrZS9i aW4iClRFTVBMQVRFQ09ORj0iJHtQT0tZRElSfS9tZXRhLXBva3kvY29uZi90ZW1wbGF0ZXMvZGVm YXVsdCIKc291cmNlICR7T0VfRU5WX1NDUklQVH0gJHtCVUlMRF9ESVJ9CnVuc2V0IFRFTVBMQVRF Q09ORgoKIyMjIyBleGVjdXRlIGJpdGJha2UgdG8gZ2VuZXJhdGUgY29uZiBmaWxlcwpiaXRiYWtl IC0tdmVyc2lvbgoKIyMjIyB1cGRhdGUgbG9jYWwuY29uZgpMT0NBTENPTkY9IiR7QlVJTERfRElS fS9jb25mL2xvY2FsLmNvbmYiCmVjaG8gIgpCQl9HRU5FUkFURV9NSVJST1JfVEFSQkFMTFMgPSBc IjBcIgpCQl9OT19ORVRXT1JLID0gXCIxXCIKQkJfRkVUQ0hfUFJFTUlSUk9ST05MWSA9IFwiMVwi ClBSRU1JUlJPUlM6cHJlcGVuZCA9IFwiIFxcCmdpdDovLy4qLy4qICAgICBnaXQ6Ly8ke1NFTEZf RElSfS9IT1NUL1BBVEg7cHJvdG9jb2w9ZmlsZSBcXApcIgoiID4+ICR7TE9DQUxDT05GfQoKIyMj IyBub3cgdW5wYWNrIGdsaWJjIGZvciB0ZXN0aW5nIGdpdCBzaGFsbG93CmJpdGJha2UgZ2xpYmMg LWMgdW5wYWNrCg== 5116 2025-05-16 01:21:57 +0000 2025-05-16 01:21:57 +0000 test git shallow for local mirror (fix download directory) setupbuilddir2 application/x-shellscript 1174 niqingliang2003 IyEvdXNyL2Jpbi9lbnYgYmFzaAoKc2V0IC1lClNFTEZfRElSPSJgcmVhZGxpbmsgLWZlICR7QkFT SF9TT1VSQ0V9IHwgeGFyZ3MgZGlybmFtZWAiCgojIyMjIHByZXBhcmUgbmVlZGVkIHJlcG8gKGZv ciB0ZXN0IGxvY2FsIG1pcnJvcikKI1BSRU1JUlJPUj0iL21udC9kYXR1bS9yZXBvc2l0b3JpZXMv IgpQUkVNSVJST1I9Imh0dHBzOi8vIgpnaXQgY2xvbmUgLS1kZXB0aCAxICR7UFJFTUlSUk9SfWdp dGh1Yi5jb20veW9jdG9wcm9qZWN0L3Bva3kuZ2l0ICR7U0VMRl9ESVJ9L3Bva3kKZ2l0IGNsb25l IC0tbWlycm9yICR7UFJFTUlSUk9SfXNvdXJjZXdhcmUub3JnL2dpdC9nbGliYy5naXQgJHtTRUxG X0RJUn0vc291cmNld2FyZS5vcmcvZ2l0L2dsaWJjLmdpdAp1bnNldCBQUkVNSVJST1IKCiMjIyMg cHJlcGFyZSBkaXJlY3RvcnkKUE9LWURJUj0iYHJlYWRsaW5rIC1mZSAke1NFTEZfRElSfS9wb2t5 L2AiCkJVSUxEX0RJUj0iJHtTRUxGX0RJUn0vYnVpbGQiCmlmICEgWyAtZCAke0JVSUxEX0RJUn0g XTsgdGhlbgoJbWtkaXIgLXAgJHtCVUlMRF9ESVJ9CmZpCgojIyMjIHNldCBlbnZpcm9ubWVudApP RV9FTlZfU0NSSVBUPSIke1BPS1lESVJ9L29lLWluaXQtYnVpbGQtZW52IgpCSVRCQUtFX0JJTl9E SVI9IiR7UE9LWURJUn0vYml0YmFrZS9iaW4iClRFTVBMQVRFQ09ORj0iJHtQT0tZRElSfS9tZXRh LXBva3kvY29uZi90ZW1wbGF0ZXMvZGVmYXVsdCIKc291cmNlICR7T0VfRU5WX1NDUklQVH0gJHtC VUlMRF9ESVJ9CnVuc2V0IFRFTVBMQVRFQ09ORgoKIyMjIyBleGVjdXRlIGJpdGJha2UgdG8gZ2Vu ZXJhdGUgY29uZiBmaWxlcwpiaXRiYWtlIC0tdmVyc2lvbgoKIyMjIyB1cGRhdGUgbG9jYWwuY29u ZgpMT0NBTENPTkY9IiR7QlVJTERfRElSfS9jb25mL2xvY2FsLmNvbmYiCmVjaG8gIgpCQl9HRU5F UkFURV9NSVJST1JfVEFSQkFMTFMgPSBcIjBcIgpCQl9OT19ORVRXT1JLID0gXCIxXCIKQkJfRkVU Q0hfUFJFTUlSUk9ST05MWSA9IFwiMVwiClBSRU1JUlJPUlM6cHJlcGVuZCA9IFwiIFxcCmdpdDov Ly4qLy4qICAgICBnaXQ6Ly8ke1NFTEZfRElSfS9IT1NUL1BBVEg7cHJvdG9jb2w9ZmlsZSBcXApc IgoiID4+ICR7TE9DQUxDT05GfQoKIyMjIyBub3cgdW5wYWNrIGdsaWJjIGZvciB0ZXN0aW5nIGdp dCBzaGFsbG93CmJpdGJha2UgZ2xpYmMgLWMgdW5wYWNrCg==