Bug 13157

Summary: Bitbake fails to fetch from a premirror that doesn't support https
Product: [Build System, Metadata & Runtime] BitBake Reporter: Maxime Roussin-Bélanger <maxime.roussinbelanger>
Component: bitbakeAssignee: Maxime Roussin-Bélanger <maxime.roussinbelanger>
Status: RESOLVED OBSOLETE QA Contact:
Severity: normal    
Priority: Medium CC: poky.bs.watcher, poky.watcher, randy.macleod
Version: 2.7   
Target Milestone: 2.7   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Maxime Roussin-Bélanger 2019-01-29 16:52:16 UTC
Preconditions/Environment
-------------------------
A PREMIRROR that doesn't support https

Triggering Action/Cause
-----------------------
bitbake a recipe that uses https protocol and fetches from github.

Expectation
-----------
Try to redirect to http and download the archive? Skip the PREMIRROR and download from upstream...

Actual Result
-------------
wget fails when trying to fetch from the premirror because https is reserved for a login page.

wget gets redirected to the login page, download the HTML, tries to untar the html page, then obviously fails to untar it.

I was trying to build tools inside meta-clang layer.

URL transformed to HTTPS due to an HSTS policy is a warning given by wget.


Reproducibility
---------------
5/5
Comment 1 Maxime Roussin-Bélanger 2019-01-30 18:52:23 UTC
A real-life example of the problem

http://errors.yoctoproject.org/Errors/Details/220286/
Comment 2 Richard Purdie 2019-01-31 16:22:15 UTC
I'm not sure this is a bitbake problem but a misconfigured server.

The logs show a fetch of http://bitbucket.org/eigen/eigen/get/3.3.7.tar.bz2 which redirected to https://bitbucket.org/eigen/eigen/get/3.3.7.tar.bz2. The fact that this redirected url is invalid is an issue with the hosting of those files.

I tried this locally and it worked so perhaps you could retest please? Perhaps the server issue was fixed?
Comment 3 Maxime Roussin-Bélanger 2019-01-31 17:03:57 UTC
The redirect URL isn't invalid, you can see that the file was downloaded without problem:

2019-01-30 00:25:44 (3.51 MB/s) - ‘TOPDIR/../downloads/libeigen-3.3.7.tar.bz2’ saved [1665168/1665168]

But then for some reason, the fetcher can't find it?

How can that happen?!
"
The fetch command returned success for url http://bitbucket.org/eigen/eigen/get/3.3.7.tar.bz2 but TOPDIR/../downloads/libeigen-3.3.7.tar.bz2 doesn't exist?!
"

I was not able to reproduce the error ever on local. It only did it on the remote build machine that yocto is using. When I sent the patch I made sure that it worked on local first.

The only error I was able to create was when I trying to build a recipe from the meta-clang layer that had the protocol set to https. But my company PREMIRROR that had the cache tarball doesn't support HTTPS. But you can download the tarball in HTTP no problem. HTTPS on the PREMIRROR is reserved for a login page.

That's probably another issue though.
Comment 4 Maxime Roussin-Bélanger 2019-01-31 19:02:53 UTC
To get around the redirection of HTTP -> HTTPS, I had to modify the FETCHCMD_wget with

FETCHCMD_wget = "/usr/bin/env wget -t 2 -T 30 --passive-ftp --no-check-certificate --no-hsts"

I added the "--no-hsts" part.
Comment 5 Maxime Roussin-Bélanger 2019-01-31 19:03:34 UTC
To get around the redirection of HTTP -> HTTPS, I had to modify the FETCHCMD_wget with

FETCHCMD_wget = "/usr/bin/env wget -t 2 -T 30 --passive-ftp --no-check-certificate --no-hsts"

I added the "--no-hsts" part.
Comment 6 Stephen K Jolley 2019-02-02 04:03:30 UTC
Richard,

It appears that Maxime has answered your question. I am moving it to Accepted unless you have other questions.
Comment 7 Randy MacLeod 2019-05-30 15:24:16 UTC
It appears that his was a problem with the local network configuration.
If it is not, please open a new more clear bug describing the issue.