Bug 10699

NOTE: git://github.com/ URLs are no longer allowed by GitHub.

My idea to fix this would be allow the fetcher to remap url based on a variable:

Example:

BB_URL_REMAP = "git://github.com/openembedded/bitbake git://git.openembedded.org/bitbake"

Will change all url starting with "git://github.com/openembedded/bitbake" to "git://git.openembedded.org/bitbake"

A variable called BB_ALLOW_URL_REMAP can be used to disable this feature if needed.

I added a commit to support this see https://lists.openembedded.org/g/bitbake-devel/topic/patch_bitbake_fetch_add/112636148

It seems I misunderstood this so I will move it back to new.

Bumping to 6.99

I would like to work on this bug.

Looking at the discussions it looks like a patch is created and this bug is fixed. can I know Is this bug fixed, or needs a patch to be created?

Hi Aditya, 

See https://lists.yoctoproject.org/g/docs/topic/bitbake_devel_patch/112640014
The patch is not valid.

Hi Sawas,
I haven't started working on this bug since I am a newcomer I am searching for newcomer bugs to fix so that I get deeper understanding in yocto things.
I haven't submitted any patch yet on this bug, my question is since I saw this comment-on-comment history 
"I added a commit to support this see https://lists.openembedded.org/g/bitbake-devel/topic/patch_bitbake_fetch_add/112636148"
I need to know if this bug is fixed or needs to be worked on.

Thanks,
Aditya GS

Hi Aditya GS,

I am not working on this, as far as I know this is not fixed.

Regards,
Savvas

I will try to fix this,
The issue was reproduced by fetching the same repository using
two different URL formats:

    git://github.com/git/git
    https://github.com/git/git/archive/master.tar.gz

BitBake treats these URLs as different sources and performs
separate fetch operations, resulting in duplicate downloads:

    downloads/git2/github.com.git.git
    downloads/master.tar.gz

This shows that BitBake does not normalize equivalent repository
URLs, leading to inconsistent caching behavior and redundant
fetch operations.

Reproduced the issue and verified that the same repository can be fetched
multiple times when accessed via different URL forms.

Observed that BitBake treats git:// URLs and HTTP tarball URLs using
different fetchers and storage mechanisms, resulting in separate downloads:

    git repositories stored under downloads/git2/
    tarball downloads stored directly under downloads/

Attempted a normalization approach in FetchData, but this does not resolve
the issue since different fetch backends are used.

This suggests that a more fundamental change may be required, such as
normalizing repository identity at a higher level or improving fetch key
generation logic.

At this point, I am moving on from this bug as it appears to require a
more in-depth design change beyond a simple fix.

Thanks,
Aditya GS

I also attempted an approach based on canonicalizing the repository URL
(e.g., deriving a common identity such as github.com/git/git), with the
intention of treating equivalent URLs as the same source.

However, this did not resolve the issue, since BitBake continues to use
the full URL (including protocol and parameters) for fetch key generation
and cache handling. As a result, different URL forms are still treated as
distinct sources and result in separate downloads.

This suggests that canonicalization at the FetchData level alone is
insufficient, and that resolving the issue would require changes at a
higher level, such as in fetch key computation or DL_DIR handling.

Created attachment 5219 [details]
Tried canonical approach

Submitting an exploratory patch introducing canonical repository
identification (canonical_id) in FetchData.

Testing shows this approach does not eliminate duplicate fetches,
as BitBake still uses full URLs for fetch key generation.

Sharing for discussion and feedback on whether canonicalization
should be handled at a higher level.

I don't think we want to change the bitbake fetcher itself.

The fix for this should be related to/part of:

#16272
#16240

Hi Tim,

Thanks for the clarification.

Based on the discussion and your feedback, it makes sense that this
issue is better handled at a higher level rather than modifying the
BitBake fetcher directly.

Since this overlaps with broader design aspects covered in #16272 and
#16240, I will step back from this bug for now and focus on other
issues more suitable for my current experience level.

Thanks for the guidance.

Regards,
Aditya