10699 2016-11-21 21:49:20 +0000 Ensure consistency in repository URLs 2026-08-20 15:19:12 +0000 1 1 6 Yocto Project Subprojects Layer Index Layer Index unspecified All Multiple NEW Medium enhancement 6.99 1 bluelightning tim.orling adityags2004 akhilathota499 akuster brian.avery Falital randy.macleod tim.orling --- --- No (bug/feature does not impact docs) oldest_to_newest 68471 0 bluelightning 2016-11-21 21:49:20 +0000 Two layers can fetch from the same repository, but be using different URLs to fetch it e.g. git://github.com/project/repo and https://github.com/project/repo. We should have a mechanism to avoid this happening because it not only wastes fetching time and space, it may also cause problems for scripts making use of the data. Not all git servers have a standardised mapping between the different protocol URLs, so we'll probably need to have a table (possibly in the database) that we can populate with mappings for known URLs. The mapping would be done when the entry is submitted. 92969 1 tim.orling 2022-04-10 19:27:58 +0000 NOTE: git://github.com/ URLs are no longer allowed by GitHub. 101535 2 Falital 2025-03-30 12:41:28 +0000 My idea to fix this would be allow the fetcher to remap url based on a variable: Example: BB_URL_REMAP = "git://github.com/openembedded/bitbake git://git.openembedded.org/bitbake" Will change all url starting with "git://github.com/openembedded/bitbake" to "git://git.openembedded.org/bitbake" A variable called BB_ALLOW_URL_REMAP can be used to disable this feature if needed. 101883 3 Falital 2025-05-05 19:00:22 +0000 I added a commit to support this see https://lists.openembedded.org/g/bitbake-devel/topic/patch_bitbake_fetch_add/112636148 102003 4 Falital 2025-05-20 17:14:16 +0000 It seems I misunderstood this so I will move it back to new. 104539 5 tim.orling 2026-03-05 16:27:44 +0000 Bumping to 6.99 105365 6 adityags2004 2026-05-07 05:29:07 +0000 I would like to work on this bug. 105486 7 adityags2004 2026-05-19 07:04:32 +0000 Looking at the discussions it looks like a patch is created and this bug is fixed. can I know Is this bug fixed, or needs a patch to be created? 105489 8 Falital 2026-05-19 11:44:12 +0000 Hi Aditya, See https://lists.yoctoproject.org/g/docs/topic/bitbake_devel_patch/112640014 The patch is not valid. 105490 9 adityags2004 2026-05-19 11:51:39 +0000 Hi Sawas, I haven't started working on this bug since I am a newcomer I am searching for newcomer bugs to fix so that I get deeper understanding in yocto things. I haven't submitted any patch yet on this bug, my question is since I saw this comment-on-comment history "I added a commit to support this see https://lists.openembedded.org/g/bitbake-devel/topic/patch_bitbake_fetch_add/112636148" I need to know if this bug is fixed or needs to be worked on. Thanks, Aditya GS 105491 10 Falital 2026-05-19 12:04:18 +0000 Hi Aditya GS, I am not working on this, as far as I know this is not fixed. Regards, Savvas 105502 11 adityags2004 2026-05-20 10:33:56 +0000 I will try to fix this, The issue was reproduced by fetching the same repository using two different URL formats: git://github.com/git/git https://github.com/git/git/archive/master.tar.gz BitBake treats these URLs as different sources and performs separate fetch operations, resulting in duplicate downloads: downloads/git2/github.com.git.git downloads/master.tar.gz This shows that BitBake does not normalize equivalent repository URLs, leading to inconsistent caching behavior and redundant fetch operations. 105516 12 adityags2004 2026-05-21 11:02:52 +0000 Reproduced the issue and verified that the same repository can be fetched multiple times when accessed via different URL forms. Observed that BitBake treats git:// URLs and HTTP tarball URLs using different fetchers and storage mechanisms, resulting in separate downloads: git repositories stored under downloads/git2/ tarball downloads stored directly under downloads/ Attempted a normalization approach in FetchData, but this does not resolve the issue since different fetch backends are used. This suggests that a more fundamental change may be required, such as normalizing repository identity at a higher level or improving fetch key generation logic. 105517 13 adityags2004 2026-05-21 11:04:58 +0000 At this point, I am moving on from this bug as it appears to require a more in-depth design change beyond a simple fix. Thanks, Aditya GS 105519 14 adityags2004 2026-05-21 11:27:16 +0000 I also attempted an approach based on canonicalizing the repository URL (e.g., deriving a common identity such as github.com/git/git), with the intention of treating equivalent URLs as the same source. However, this did not resolve the issue, since BitBake continues to use the full URL (including protocol and parameters) for fetch key generation and cache handling. As a result, different URL forms are still treated as distinct sources and result in separate downloads. This suggests that canonicalization at the FetchData level alone is insufficient, and that resolving the issue would require changes at a higher level, such as in fetch key computation or DL_DIR handling. 105520 15 5219 adityags2004 2026-05-21 11:34:42 +0000 Created attachment 5219 Tried canonical approach Submitting an exploratory patch introducing canonical repository identification (canonical_id) in FetchData. Testing shows this approach does not eliminate duplicate fetches, as BitBake still uses full URLs for fetch key generation. Sharing for discussion and feedback on whether canonicalization should be handled at a higher level. 105763 16 tim.orling 2026-06-06 19:48:48 +0000 I don't think we want to change the bitbake fetcher itself. The fix for this should be related to/part of: #16272 #16240 105768 17 adityags2004 2026-06-08 05:39:11 +0000 Hi Tim, Thanks for the clarification. Based on the discussion and your feedback, it makes sense that this issue is better handled at a higher level rather than modifying the BitBake fetcher directly. Since this overlaps with broader design aspects covered in #16272 and #16240, I will step back from this bug for now and focus on other issues more suitable for my current experience level. Thanks for the guidance. Regards, Aditya 5219 2026-05-21 11:34:42 +0000 2026-05-21 11:34:42 +0000 Tried canonical approach 0001-fetch2-add-canonical-repository-identity-for-URL-nor.patch text/plain 2191 adityags2004 RnJvbSBkMDRkMDc3YTY4NjRlZmI0YmU4MWRlMmNmZTk4ODE5ODFlMWY3N2Q4IE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBBZGl0eWEgRyBTIDxhZGl0eWFnczIwMDRAZ21haWwuY29tPgpE YXRlOiBUaHUsIDIxIE1heSAyMDI2IDE3OjAwOjE1ICswNTMwClN1YmplY3Q6IFtQQVRDSF0gZmV0 Y2gyOiBhZGQgY2Fub25pY2FsIHJlcG9zaXRvcnkgaWRlbnRpdHkgZm9yIFVSTAogbm9ybWFsaXph dGlvbiAoZXhwZXJpbWVudGFsKQoKW1lPQ1RPICMxMDY5OV0KClRoaXMgY2hhbmdlIGludHJvZHVj ZXMgYSBiYXNpYyBjYW5vbmljYWxpemF0aW9uIG1lY2hhbmlzbSBmb3IgR2l0CnJlcG9zaXRvcnkg VVJMcyBieSBkZXJpdmluZyBhIG5vcm1hbGl6ZWQgcmVwb3NpdG9yeSBpZGVudGlmaWVyCihlLmcu LCBnaXRodWIuY29tL2dpdC9naXQpIGFuZCBzdG9yaW5nIGl0IGFzIGBjYW5vbmljYWxfaWRgIGlu CkZldGNoRGF0YS4KClRoZSBnb2FsIGlzIHRvIGV4cGxvcmUgdGhlIHBvc3NpYmlsaXR5IG9mIHRy ZWF0aW5nIGVxdWl2YWxlbnQKcmVwb3NpdG9yeSBVUkxzIChlLmcuLCBkaWZmZXJlbnQgcHJvdG9j b2xzIG9yIHBhcmFtZXRlciB2YXJpYXRpb25zKQphcyB0aGUgc2FtZSBzb3VyY2UuCgpIb3dldmVy LCB0aGlzIGFwcHJvYWNoIGFsb25lIGRvZXMgbm90IHJlc29sdmUgdGhlIGlzc3VlLCBzaW5jZSBC aXRCYWtlCmNvbnRpbnVlcyB0byB1c2UgdGhlIGZ1bGwgVVJMIHN0cmluZyBmb3IgZmV0Y2gga2V5 IGdlbmVyYXRpb24gYW5kCmNhY2hlIGhhbmRsaW5nLCByZXN1bHRpbmcgaW4gZHVwbGljYXRlIGZl dGNoZXMgd2hlbiBVUkxzIGRpZmZlci4KClRoaXMgcGF0Y2ggaXMgcHJvdmlkZWQgYXMgYW4gZXhw bG9yYXRvcnkgc3RlcCB0byBoaWdobGlnaHQgdGhhdApjYW5vbmljYWxpemF0aW9uIGF0IHRoZSBG ZXRjaERhdGEgbGV2ZWwgbWF5IG5vdCBiZSBzdWZmaWNpZW50LAphbmQgdGhhdCBhIGhpZ2hlci1s ZXZlbCBjaGFuZ2UgKGUuZy4gZmV0Y2gga2V5IGdlbmVyYXRpb24gb3IgRExfRElSCmhhbmRsaW5n KSBtYXkgYmUgcmVxdWlyZWQuCgpTaWduZWQtb2ZmLWJ5OiBBZGl0eWEgR1MgPGFkaXR5YWdzMjAw NEBnbWFpbC5jb20+Ci0tLQogbGliL2JiL2ZldGNoMi9fX2luaXRfXy5weSB8IDEwICsrKysrKysr KysKIDEgZmlsZSBjaGFuZ2VkLCAxMCBpbnNlcnRpb25zKCspCgpkaWZmIC0tZ2l0IGEvbGliL2Ji L2ZldGNoMi9fX2luaXRfXy5weSBiL2xpYi9iYi9mZXRjaDIvX19pbml0X18ucHkKaW5kZXggZjdk NWRmZTlhLi40YzFlNWRiNWMgMTAwNjQ0Ci0tLSBhL2xpYi9iYi9mZXRjaDIvX19pbml0X18ucHkK KysrIGIvbGliL2JiL2ZldGNoMi9fX2luaXRfXy5weQpAQCAtMjcsNiArMjcsMTQgQEAgaW1wb3J0 IGJiLnV0aWxzCiBpbXBvcnQgYmIuY2hlY2tzdW0KIGltcG9ydCBiYi5wcm9jZXNzCiBpbXBvcnQg YmIuZXZlbnQKK2RlZiBjYW5vbmljYWxfcmVwb19pZCh1cmwpOgorICAgICMgb25seSBvcGVyYXRl IG9uIGdpdCB1cmxzCisgICAgaWYgdXJsLnN0YXJ0c3dpdGgoImdpdDovL2dpdGh1Yi5jb20iKToK KyAgICAgICAgIyBzdHJpcCBwcm90b2NvbCBhbmQgcGFyYW1ldGVycworICAgICAgICBiYXNlID0g dXJsLnNwbGl0KCI7IilbMF0KKyAgICAgICAgYmFzZSA9IGJhc2UucmVwbGFjZSgiZ2l0Oi8vIiwg IiIpCisgICAgICAgIHJldHVybiBiYXNlCisgICAgcmV0dXJuIHVybAogCiBfX3ZlcnNpb25fXyA9 ICIyIgogX2NoZWNrc3VtX2NhY2hlID0gYmIuY2hlY2tzdW0uRmlsZUNoZWNrc3VtQ2FjaGUoKQpA QCAtMTMyNCw2ICsxMzMyLDggQEAgY2xhc3MgRmV0Y2hEYXRhKG9iamVjdCk6CiAgICAgICAgIChz ZWxmLnR5cGUsIHNlbGYuaG9zdCwgc2VsZi5wYXRoLCBzZWxmLnVzZXIsIHNlbGYucHN3ZCwgc2Vs Zi5wYXJtKSA9IGRlY29kZXVybChkLmV4cGFuZCh1cmwpKQogICAgICAgICBzZWxmLmRhdGUgPSBz ZWxmLmdldFNSQ0RhdGUoZCkKICAgICAgICAgc2VsZi51cmwgPSB1cmwKKyAgICAgICAgc2VsZi5j YW5vbmljYWxfaWQgPSBjYW5vbmljYWxfcmVwb19pZCh1cmwpCisKICAgICAgICAgaWYgbm90IHNl bGYudXNlciBhbmQgInVzZXIiIGluIHNlbGYucGFybToKICAgICAgICAgICAgIHNlbGYudXNlciA9 IHNlbGYucGFybVsidXNlciJdCiAgICAgICAgIGlmIG5vdCBzZWxmLnBzd2QgYW5kICJwc3dkIiBp biBzZWxmLnBhcm06Ci0tIAoyLjM0LjEKCg==