Two layers can fetch from the same repository, but be using different URLs to fetch it e.g. git://github.com/project/repo and https://github.com/project/repo. We should have a mechanism to avoid this happening because it not only wastes fetching time and space, it may also cause problems for scripts making use of the data. Not all git servers have a standardised mapping between the different protocol URLs, so we'll probably need to have a table (possibly in the database) that we can populate with mappings for known URLs. The mapping would be done when the entry is submitted.
NOTE: git://github.com/ URLs are no longer allowed by GitHub.
My idea to fix this would be allow the fetcher to remap url based on a variable: Example: BB_URL_REMAP = "git://github.com/openembedded/bitbake git://git.openembedded.org/bitbake" Will change all url starting with "git://github.com/openembedded/bitbake" to "git://git.openembedded.org/bitbake" A variable called BB_ALLOW_URL_REMAP can be used to disable this feature if needed.
I added a commit to support this see https://lists.openembedded.org/g/bitbake-devel/topic/patch_bitbake_fetch_add/112636148
It seems I misunderstood this so I will move it back to new.
Bumping to 6.99
I would like to work on this bug.
Looking at the discussions it looks like a patch is created and this bug is fixed. can I know Is this bug fixed, or needs a patch to be created?
Hi Aditya, See https://lists.yoctoproject.org/g/docs/topic/bitbake_devel_patch/112640014 The patch is not valid.
Hi Sawas, I haven't started working on this bug since I am a newcomer I am searching for newcomer bugs to fix so that I get deeper understanding in yocto things. I haven't submitted any patch yet on this bug, my question is since I saw this comment-on-comment history "I added a commit to support this see https://lists.openembedded.org/g/bitbake-devel/topic/patch_bitbake_fetch_add/112636148" I need to know if this bug is fixed or needs to be worked on. Thanks, Aditya GS
Hi Aditya GS, I am not working on this, as far as I know this is not fixed. Regards, Savvas
I will try to fix this, The issue was reproduced by fetching the same repository using two different URL formats: git://github.com/git/git https://github.com/git/git/archive/master.tar.gz BitBake treats these URLs as different sources and performs separate fetch operations, resulting in duplicate downloads: downloads/git2/github.com.git.git downloads/master.tar.gz This shows that BitBake does not normalize equivalent repository URLs, leading to inconsistent caching behavior and redundant fetch operations.
Reproduced the issue and verified that the same repository can be fetched multiple times when accessed via different URL forms. Observed that BitBake treats git:// URLs and HTTP tarball URLs using different fetchers and storage mechanisms, resulting in separate downloads: git repositories stored under downloads/git2/ tarball downloads stored directly under downloads/ Attempted a normalization approach in FetchData, but this does not resolve the issue since different fetch backends are used. This suggests that a more fundamental change may be required, such as normalizing repository identity at a higher level or improving fetch key generation logic.
At this point, I am moving on from this bug as it appears to require a more in-depth design change beyond a simple fix. Thanks, Aditya GS
I also attempted an approach based on canonicalizing the repository URL (e.g., deriving a common identity such as github.com/git/git), with the intention of treating equivalent URLs as the same source. However, this did not resolve the issue, since BitBake continues to use the full URL (including protocol and parameters) for fetch key generation and cache handling. As a result, different URL forms are still treated as distinct sources and result in separate downloads. This suggests that canonicalization at the FetchData level alone is insufficient, and that resolving the issue would require changes at a higher level, such as in fetch key computation or DL_DIR handling.
Created attachment 5219 [details] Tried canonical approach Submitting an exploratory patch introducing canonical repository identification (canonical_id) in FetchData. Testing shows this approach does not eliminate duplicate fetches, as BitBake still uses full URLs for fetch key generation. Sharing for discussion and feedback on whether canonicalization should be handled at a higher level.
I don't think we want to change the bitbake fetcher itself. The fix for this should be related to/part of: #16272 #16240
Hi Tim, Thanks for the clarification. Based on the discussion and your feedback, it makes sense that this issue is better handled at a higher level rather than modifying the BitBake fetcher directly. Since this overlaps with broader design aspects covered in #16272 and #16240, I will step back from this bug for now and focus on other issues more suitable for my current experience level. Thanks for the guidance. Regards, Aditya