I am trying to put a sstate mirror on a company artifactory server. The sstate data was successfully uploaded with the jfrog cli. In order to make it simple I do not use a hash equivalence server, so: BB_SIGNATURE_HANDLER = "OEBasicHash" BB_HASHSERVE = "" The uploaded data itself works, when I have the same data that was uploaded to the artifactory locally and use SSTATE_MIRRORS ?= "\ file://.* file:///home/user/shared-sstate-cache/cache/sstate-cache/PATH" => most tasks are taken from the cache, so the caching works. Now to the problem: The project in the company's artifactory needs authentication for reading, too. So I use: SSTATE_MIRRORS ?= "\ file://.* https://username:password@artifactory.company.com:443/artifactory/project/cache/sstate-cache/PATH;downloadfilename=PATH" Direct download works, for example: wget https://artifactory.company.com.com:443/artifactory/project/cache/sstate-cache/universal/6b/6b/sstate%3Asqlite3-native%3Ax86_64-linux%3A3.38.5%3Ar0%3Ax86_64%3A10%3A6b6bf56a7e449822374854b8c076f90db2e1d706fe9976813b2cd9b2c6703bad_deploy_source_date_epoch.tar.zst --user=username --password=password => downloads the file correctly But in the Yocto build it does NOT work. The sstate mirror cache is silently ignored Using bitbake -DDD gives some info: DEBUG: checkstatus() urlopen failed: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'username:password@artifactory.company.com'. (_ssl.c:1007)> DEBUG: SState: Unsuccessful fetch test for file://universal/31/fe/sstate:lzop-native:x86_64- ....... There is also a Traceback: Traceback (most recent call last): File "/home/user/test/yocto-build/sources/oe-core/meta/classes/sstate.bbclass", line 999, in checkstatus fetcher.checkstatus() File "/home/user/test/yocto-build/sources/bitbake/lib/bb/fetch2/__init__.py", line 1841, in checkstatus raise FetchError("URL %s doesn't work" % u, u) Of course there is no certificate for 'username:password@artifactory.company.com', but there is a valid certificate for 'artifactory.company.com'. So this seems to be a bug. Somehow user and password are extracted correctly, but user/password are not "subtracted" for this checkstatus() call. Please also note that: SSTATE_MIRRORS ?= "\ file://.* https://artifactory.company.com:443/artifactory/project/cache/sstate-cache/PATH;user=username:password;downloadfilename=PATH" does not work at all. Here username:password is not taken into account, when doing so, I just get (for example): DEBUG: checkstatus: trying again DEBUG: checkstatus: trying again DEBUG: checkstatus: trying again DEBUG: checkstatus: trying again DEBUG: checkstatus() urlopen failed: HTTP Error 401: Unauthorized DEBUG: SState: Unsuccessful fetch test for file://d1/f1/sstate:libgcc- .... There is no example in the documentation about injecting username and password into SSTATE_MIRRORS. One last comment: username:password seems to work fine, when using https in a SOURCE_MIRROR_URL, I used the same company-artifactory server.
Ross has some ideas on how to proceed. Peter, Are you able to work with him to develop a patch?
Thank you the quick reaction. I checked the bitbake codebase in the meantime, and I believe that this problem was already fixed in commit: commit 88350002d45e0aa85ecd5356da2c8d71e450641e Author: Kasper Revsbech <kasper.revsbech.ext@siemensgamesa.com> Date: Tue Nov 29 14:22:46 2022 +0100 fetch2/wget: handle username/password in uri We are using yocto - kirkstone, and bitbake does not include this commit there. I cherry-picked this commit to bitbake and am currently testing.
The shared sstate mirror works now as expected for me using SSTATE_MIRRORS ?= "\ file://.* https://username:password@artifactory.company.com:443/artifactory/project/cache/sstate-cache/PATH;downloadfilename=PATH" Please update the documentation, I mean put an example with user / password for SSTATE_MIRRORS and SOURCE_MIRROR_URL. Please merge commit 88350002d45e0aa85ecd5356da2c8d71e450641e into the Kirkstone branch. Thank you in advance.
Would you be able to post the backport patch? Backport requests need to be done via patches on the list, just prefix the subject with [kirkstone] so the maintainer sees it.
I am confused. What exactly shall I do? I read: Check if the fix is already present in the master branch: This will result in the most straightforward path into the stable branch for the fix. 1. If the fix is present in the master branch — submit a backport request by email: You should send an email to the relevant stable branch maintainer and the mailing list with details of the bug or CVE to be fixed, the commit hash on the master branch that fixes the issue and the stable branches which you would like this fix to be backported to. So I shall just send an email to bitbake-devel@lists.openembedded.org ? What shall I write? The commit is already in the master, and it is mergeable. No conflicts So why shall I write a patch? It is commit 88350002d45e0aa85ecd5356da2c8d71e450641e made by Kasper Revsbach
We don't take backport requests (which are patches which have already been merged into master) via bugzilla. Checkout the right branch of bitbake, cherry-pick the commit, git send-email the patch.
There are other solutions to don't exposing the user and password on the url. Placing them on the "~/.wgetrc" on the build machine or container that runs bitbake will provide the same. The following links provide more information about that: https://www.gnu.org/software/wget/manual/html_node/Wgetrc-Commands.html https://www.gnu.org/software/wget/manual/html_node/Sample-Wgetrc.html
A colleague pointed me at this issue. I am currently experimenting with sstate cache mirrors over https, with Basic Auth and had good results with a syntax like this: https://cache.example.com/sstate-cache-kirkstone/PATH;user=cache-user;pswd=super-secret-password;downloadfilename=PATH What unlocked me was the explicit passing of 'user' and 'pswd' parameters (after reading the code).
Thank you! I will make more tests, but this form works now for me, too.
Michael, we need to document this as an example, it seems.
I am using now a better solution instead of modifying local.conf: In the home folder (of the user, that starts the build) one can put a file .netrc The linux permission of the file needs to be 400 or 600. Contents of .netrc is like machine <https-server> login <user> password <token>
I mean the comapny's https-server still needs to be in the local.conf, but without user and password, which is a lot nicer.
Bulk move of 5.1 M4 to 5.2 M2 as approved by AlexB.
Hi, I've sent a patch that documents this: https://lists.yoctoproject.org/g/docs/message/6024 Peter, I would appreciate if you could test the instructions there and if they work, add your Tested-by[1] in reply to my email above. [1]: https://www.kernel.org/doc/html/latest/process/submitting-patches.html#using-reported-by-tested-by-reviewed-by-suggested-by-and-fixes Moving this to In progress review. Antonin
Patch merged on master: https://git.yoctoproject.org/yocto-docs/commit/?id=4c303cce0627e0f2ce52e72597541e220585aaed Marking this bug as Resolved.