Bug 14015 - URL Arguments in MIRROR/PREMIRROR get encoded
Summary: URL Arguments in MIRROR/PREMIRROR get encoded
Status: RESOLVED OBSOLETE
Alias: None
Product: BitBake
Classification: Build System, Metadata & Runtime
Component: bitbake (show other bugs)
Version: 3.1.2
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 5.99
Assignee: Jahn
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2020-08-19 05:48 UTC by Jahn
Modified: 2024-02-08 16:09 UTC (History)
6 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Don't know


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jahn 2020-08-19 05:48:07 UTC
After successfully putting a PREMIRROR for our builds on a blob storage,
I wanted to keep that storage completely private, so adding I thought the following would work:

SAS_TOKEN = "?sv=2019-12-12&ss=bf&srt=co&sp=rdl&spr=https&sig=abcdef_TEST"

INHERIT += "own-mirrors"
SOURCE_MIRROR_URL = "https://somewhere.blob.core.windows.net/our-mirror/downloads/BASENAME${SAS_TOKEN}"

BB_FETCH_PREMIRRORONLY = "1"


Well it kind of does work, but the arguments of the SAS_TOKEN get encoded and the fetch via wget will fail.

Expected behavior: SAS_TOKEN will not be encoded / = have a possibility to add URL parts that will not be encoded.


FYI: I also asked a question about this on stackoverflow
https://stackoverflow.com/questions/63486550/yocto-premirror-source-mirror-url-with-url-arguments-sas-token-possible
Comment 1 Randy MacLeod 2023-07-26 21:22:01 UTC
Bulk move of 64 bugs to 4.3 M3 after a quick review.
If a bug is actually fixed, please add a commit link and resolve it.
-- Randy for YP bug team.
Comment 2 Trevor Gamblin 2023-09-14 16:36:54 UTC
Moved to M4.
Comment 3 Randy MacLeod 2023-10-30 15:29:09 UTC
Bulk move to 5.0 M1. -- Randy
Comment 4 Mark 2023-12-03 00:54:16 UTC
Jahn, do you know of a publicly available example that I could use to attempt to reproduce and review this issue?

It appears that the code does map with your assertion that the URL is encoded. dodwnload()->try_mirrors()->build_mirroruris()->uri_replace() which does a decode to complete the find and replace of the URL ends with a call to encodeurl() which encodes just the "path" component of the URL with a call to urllib.parse.quote(path).

The interesting bit is that wget will work just fine with an encoded URL, per the manual "You can encode unsafe characters in a URL as ‘%xy’, xy being the hexadecimal representation of the character’s ASCII value. Some common unsafe characters include ‘%’ (quoted as ‘%25’), ‘:’ (quoted as ‘%3A’), and ‘@’ (quoted as ‘%40’). Refer to RFC1738 for a comprehensive list of unsafe characters." - https://www.gnu.org/software/wget/manual/wget.html which is why I ask about a reproducer.
Comment 5 Randy MacLeod 2023-12-21 20:38:42 UTC
Jahn, are you able to share a reproducer as Mark requested?
Comment 6 Randy MacLeod 2024-02-01 16:06:38 UTC
Moving to 5.99 (near term release but not 5.0) since we don't have a reproducer.
Comment 7 Randy MacLeod 2024-02-08 16:09:04 UTC
No repsponse for a few years so closing. We'd consider the bug again if we can get a reproducer.