Bug 14015

Summary: URL Arguments in MIRROR/PREMIRROR get encoded
Product: [Build System, Metadata & Runtime] BitBake Reporter: Jahn <yoctoproject.bugzilla>
Component: bitbakeAssignee: Jahn <yoctoproject.bugzilla>
Status: RESOLVED OBSOLETE QA Contact:
Severity: normal    
Priority: Medium+ CC: mark.asselstine, poky.bs.watcher, poky.watcher, randy.macleod, tgamblin, Vikkram.Ravi
Version: 3.1.2   
Target Milestone: 5.99   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Don't know

Description Jahn 2020-08-19 05:48:07 UTC 
After successfully putting a PREMIRROR for our builds on a blob storage,
I wanted to keep that storage completely private, so adding I thought the following would work:

SAS_TOKEN = "?sv=2019-12-12&ss=bf&srt=co&sp=rdl&spr=https&sig=abcdef_TEST"

INHERIT += "own-mirrors"
SOURCE_MIRROR_URL = "https://somewhere.blob.core.windows.net/our-mirror/downloads/BASENAME${SAS_TOKEN}"

BB_FETCH_PREMIRRORONLY = "1"


Well it kind of does work, but the arguments of the SAS_TOKEN get encoded and the fetch via wget will fail.

Expected behavior: SAS_TOKEN will not be encoded / = have a possibility to add URL parts that will not be encoded.


FYI: I also asked a question about this on stackoverflow
https://stackoverflow.com/questions/63486550/yocto-premirror-source-mirror-url-with-url-arguments-sas-token-possible
Comment 1 Randy MacLeod 2023-07-26 21:22:01 UTC 
Bulk move of 64 bugs to 4.3 M3 after a quick review.
If a bug is actually fixed, please add a commit link and resolve it.
-- Randy for YP bug team.
Comment 2 Trevor Gamblin 2023-09-14 16:36:54 UTC 
Moved to M4.
Comment 3 Randy MacLeod 2023-10-30 15:29:09 UTC 
Bulk move to 5.0 M1. -- Randy
Comment 4 Mark 2023-12-03 00:54:16 UTC 
Jahn, do you know of a publicly available example that I could use to attempt to reproduce and review this issue?

It appears that the code does map with your assertion that the URL is encoded. dodwnload()->try_mirrors()->build_mirroruris()->uri_replace() which does a decode to complete the find and replace of the URL ends with a call to encodeurl() which encodes just the "path" component of the URL with a call to urllib.parse.quote(path).

The interesting bit is that wget will work just fine with an encoded URL, per the manual "You can encode unsafe characters in a URL as ‘%xy’, xy being the hexadecimal representation of the character’s ASCII value. Some common unsafe characters include ‘%’ (quoted as ‘%25’), ‘:’ (quoted as ‘%3A’), and ‘@’ (quoted as ‘%40’). Refer to RFC1738 for a comprehensive list of unsafe characters." - https://www.gnu.org/software/wget/manual/wget.html which is why I ask about a reproducer.
Comment 5 Randy MacLeod 2023-12-21 20:38:42 UTC 
Jahn, are you able to share a reproducer as Mark requested?
Comment 6 Randy MacLeod 2024-02-01 16:06:38 UTC 
Moving to 5.99 (near term release but not 5.0) since we don't have a reproducer.
Comment 7 Randy MacLeod 2024-02-08 16:09:04 UTC 
No repsponse for a few years so closing. We'd consider the bug again if we can get a reproducer.