14476 2021-07-07 08:34:15 +0000 Fetchers are not able to use passwords with '=' sign 2022-10-21 12:37:11 +0000 1 1 7 Build System, Metadata & Runtime BitBake bitbake unspecified x86 x86_64 RESOLVED WORKSFORME Medium normal Future 1 pe unassigned mark.asselstine poky.bs.watcher poky.watcher randy.macleod --- --- Don't know oldest_to_newest 90869 0 pe 2021-07-07 08:34:15 +0000 In line 402 of fetcher2/__init__.py (Version 3.0) is a split("="). If you use a "=" in your password the fetcher will fail with: ERROR: /home/user/devel/my_project/my_layer/meta-my_layer-winplus/recipes-prebuild-images/fpga/fpga-mainboard.bb: Error executing a python function in <code>: The stack trace of python calls that resulted in this exception/failure was: File: '<code>', lineno: 3, function: <module> 0001:__anon_22__home_user_devel_my_project_sources_poky_meta_conf_machine_include_arm_feature_arm_thumb_inc(d) 0002:__anon_25__home_user_devel_my_project_sources_poky_meta_classes_patch_bbclass(d) *** 0003:__anon_688__home_user_devel_my_project_sources_poky_meta_classes_base_bbclass(d) 0004:__anon_64__home_user_devel_my_project_sources_meta_freescale_classes_fsl_dynamic_packagearch_bbclass(d) 0005:__anon_1186__home_user_devel_my_project_sources_poky_meta_classes_insane_bbclass(d) 0006:__anon_251__home_user_devel_my_project_sources_poky_meta_classes_package_bbclass(d) 0007:__anon_718__home_user_devel_my_project_sources_poky_meta_classes_package_rpm_bbclass(d) File: '/home/user/devel/my_project/sources/poky/meta/classes/base.bbclass', lineno: 585, function: __anon_688__home_user_devel_my_project_sources_poky_meta_classes_base_bbclass 0581: needsrcrev = False 0582: srcuri = d.getVar('SRC_URI') 0583: for uri in srcuri.split(): 0584: (scheme, _ , path) = bb.fetch.decodeurl(uri)[:3] *** 0585: 0586: # HTTP/FTP use the wget fetcher 0587: if scheme in ("http", "https", "ftp"): 0588: d.appendVarFlag('do_fetch', 'depends', ' wget-native:do_populate_sysroot') 0589: File: '/home/user/devel/my_project/sources/poky/bitbake/lib/bb/fetch2/__init__.py', lineno: 402, function: decodeurl 0398: for s in parm.split(';'): 0399: if s: 0400: if not '=' in s: 0401: raise MalformedUrl(url, "The URL: '%s' is invalid: parameter %s does not specify a value (missing '=')" % (url, s)) *** 0402: s1, s2 = s.split('=') 0403: p[s1] = s2 0404: 0405: return type, host, urllib.parse.unquote(path), user, pswd, p 0406: Exception: ValueError: too many values to unpack (expected 2) ERROR: Failed to parse recipe: /home/user/devel/my_project/my_layer/meta-my_layer-winplus/recipes-prebuild-images/fpga/fpga-mainboard.bb 90886 1 randy.macleod 2021-07-08 15:02:05 +0000 We like to fix this but no one is available. please send a patch if you are able to, see: https://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded 90898 2 pe 2021-07-08 20:41:50 +0000 Hi, i started to fix it. I will need some time to find a possible solution. Because everything with = and ; will be handled as an additional parameter. If i find a solution i will let you know. 90947 3 randy.macleod 2021-07-15 17:01:53 +0000 Thanks Patrick. Note that if you have a preliminary patch that you want to discuss, it's likely that people will respond to it if you post it with an RFC prefix in the subject line on the email list. 90948 4 pe 2021-07-15 18:06:38 +0000 The idea is to support for "url quoted strings" in parameters. Therefore i would create a new parameter called quote_params. If this one is set to true the fetcher unquotes every parameter. To make it a bit more comfortable i would wrap around quote to make it available via ${@...fetch.quote(...)}. Maybe it is also possible to add a MYVARIABLE_quoted to make it availabe magically? I really would like to come up with an easy idea. But i think its not really possible here, because user and password are handled via a modified URL, which is great until you want/need = or ? or ; in your paramaters. 94186 5 mark.asselstine 2022-10-17 01:03:12 +0000 Patrick, can you share the problematic SRC_URI line please? The username and password should never be making it into the parameter processing based on the re. https://git.yoctoproject.org/poky/tree/bitbake/lib/bb/fetch2/__init__.py#n355 m = re.compile('(?P<type>[^:]*)://((?P<user>[^/;]+)@)?(?P<location>[^;]+)(;(?P<parm>.*))?').match(url) NOTE that everything after the '://' and before the '@' will be taken as the username and password, only ';' and '/' are special characters, which at first glance appear to be something we can remove, but I would have to do some research first. The '=' character is fine to have in a password. If you copy the decodeurl() function as it is today, or from summer 2021 it handles '=' just fine. So again please provide your SRC_URI as there might be another detail which is causing your issue. Thanks. 94226 6 mark.asselstine 2022-10-21 02:26:19 +0000 I have submitted a patch which adds a new test to qualify decodeurl() on passwords which contain the '=' character. I believe this issue was caused by passing the password as a parameter instead of doing something like "git://user:pass@somewhere.com". 94227 7 pe 2022-10-21 06:33:27 +0000 Hi, Yes this is true. During that time the Company was using gitlab and you cannot change the password of the access tokens. It had the = very often. So it was caused while using an old version ? 94228 8 mark.asselstine 2022-10-21 12:37:11 +0000 Patrick, I am not sure which version you were using at the time of your bug report but the logic for extracting the user:password has been the same for many years and is solid. The only special characters are '/' and ';' which can't be in the password (unfortunately there is no way to make these not special). I did run across this Stackoverflow post which is incorrect https://stackoverflow.com/questions/25508382/bitbake-yocto-git-fetch-uri-authentication If I find my Stackoverflow login I will respond to the post to prevent others from using this approach. The link to my review is https://lore.kernel.org/bitbake-devel/20221021022018.2454713-1-mark.asselstine@windriver.com/T/#u Thanks for our bug report and if you still have an issue please make updates.