Bug 13157 - Bitbake fails to fetch from a premirror that doesn't support https
Summary: Bitbake fails to fetch from a premirror that doesn't support https
Status: RESOLVED OBSOLETE
Alias: None
Product: BitBake
Classification: Build System, Metadata & Runtime
Component: bitbake (show other bugs)
Version: 2.7
Hardware: x86 Multiple
: Medium normal
Target Milestone: 2.7
Assignee: Maxime Roussin-Bélanger
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2019-01-29 16:52 UTC by Maxime Roussin-Bélanger
Modified: 2019-05-30 15:24 UTC (History)
3 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Maxime Roussin-Bélanger 2019-01-29 16:52:16 UTC
Preconditions/Environment
-------------------------
A PREMIRROR that doesn't support https

Triggering Action/Cause
-----------------------
bitbake a recipe that uses https protocol and fetches from github.

Expectation
-----------
Try to redirect to http and download the archive? Skip the PREMIRROR and download from upstream...

Actual Result
-------------
wget fails when trying to fetch from the premirror because https is reserved for a login page.

wget gets redirected to the login page, download the HTML, tries to untar the html page, then obviously fails to untar it.

I was trying to build tools inside meta-clang layer.

URL transformed to HTTPS due to an HSTS policy is a warning given by wget.


Reproducibility
---------------
5/5
Comment 1 Maxime Roussin-Bélanger 2019-01-30 18:52:23 UTC
A real-life example of the problem

http://errors.yoctoproject.org/Errors/Details/220286/
Comment 2 Richard Purdie 2019-01-31 16:22:15 UTC
I'm not sure this is a bitbake problem but a misconfigured server.

The logs show a fetch of http://bitbucket.org/eigen/eigen/get/3.3.7.tar.bz2 which redirected to https://bitbucket.org/eigen/eigen/get/3.3.7.tar.bz2. The fact that this redirected url is invalid is an issue with the hosting of those files.

I tried this locally and it worked so perhaps you could retest please? Perhaps the server issue was fixed?
Comment 3 Maxime Roussin-Bélanger 2019-01-31 17:03:57 UTC
The redirect URL isn't invalid, you can see that the file was downloaded without problem:

2019-01-30 00:25:44 (3.51 MB/s) - ‘TOPDIR/../downloads/libeigen-3.3.7.tar.bz2’ saved [1665168/1665168]

But then for some reason, the fetcher can't find it?

How can that happen?!
"
The fetch command returned success for url http://bitbucket.org/eigen/eigen/get/3.3.7.tar.bz2 but TOPDIR/../downloads/libeigen-3.3.7.tar.bz2 doesn't exist?!
"

I was not able to reproduce the error ever on local. It only did it on the remote build machine that yocto is using. When I sent the patch I made sure that it worked on local first.

The only error I was able to create was when I trying to build a recipe from the meta-clang layer that had the protocol set to https. But my company PREMIRROR that had the cache tarball doesn't support HTTPS. But you can download the tarball in HTTP no problem. HTTPS on the PREMIRROR is reserved for a login page.

That's probably another issue though.
Comment 4 Maxime Roussin-Bélanger 2019-01-31 19:02:53 UTC
To get around the redirection of HTTP -> HTTPS, I had to modify the FETCHCMD_wget with

FETCHCMD_wget = "/usr/bin/env wget -t 2 -T 30 --passive-ftp --no-check-certificate --no-hsts"

I added the "--no-hsts" part.
Comment 5 Maxime Roussin-Bélanger 2019-01-31 19:03:34 UTC
To get around the redirection of HTTP -> HTTPS, I had to modify the FETCHCMD_wget with

FETCHCMD_wget = "/usr/bin/env wget -t 2 -T 30 --passive-ftp --no-check-certificate --no-hsts"

I added the "--no-hsts" part.
Comment 6 Stephen K Jolley 2019-02-02 04:03:30 UTC
Richard,

It appears that Maxime has answered your question. I am moving it to Accepted unless you have other questions.
Comment 7 Randy MacLeod 2019-05-30 15:24:16 UTC
It appears that his was a problem with the local network configuration.
If it is not, please open a new more clear bug describing the issue.