Bug 13990 - pypi bbclass no longer processes downloads from https://files.pythonhosted.org/
Summary: pypi bbclass no longer processes downloads from https://files.pythonhosted.org/
Status: CLOSED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: oe-core other (show other bugs)
Version: 2.7.5
Hardware: Other Multiple
: Medium+ critical
Target Milestone: 3.2 M3
Assignee: akuster
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2020-07-23 14:57 UTC by Frank Earl
Modified: 2020-08-21 16:46 UTC (History)
2 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Frank Earl 2020-07-23 14:57:15 UTC
Overview:

PyPI has changed the way you download content (Over two years ago, in fact...) so that if you use the current bbclass to download a new version of a recipe or add a new recipe leveraging something from out of PyPI, you will get a 404 error on the download.  This is due to a CDN hashing algorithm that is completely different from what is used in the recipe for the URI generation.


Steps to Reproduce: 

Attempt to download a version not in the current global downloads cache or add a new packaging recipe for a new PyPI package.


Actual Results:

404 error for fetcher download attempt.


Expected Results:

Download of the archive file with sources to be set up with setuptools.


Builds affected:

ANYTHING using pypi.bbclass at this time as the old method will no longer work.  Needs to have fixes applied to multiple versions.  Happened with me on Warrior and Zeus with a Tegra and a RaspberryPI 4 target.


Root Cause:

The following block has the issue-


def pypi_src_uri(d):
    package = d.getVar('PYPI_PACKAGE')
    package_ext = d.getVar('PYPI_PACKAGE_EXT')
    pv = d.getVar('PV')
    return 'https://files.pythonhosted.org/packages/source/%s/%s/%s-%s.%s' % (package[0], package, package, pv, package_ext)


The URI used in the return is no longer valid.  They use a "simple" interface or similar which provides a series of parsable links, checksums, etc. that pip and similar tools use to do their download operations against.


A line-item link dump is as follows:


<!DOCTYPE html>
<html>
  <head>
    <title>Links for PyGTK</title>
  </head>
  <body>
    <h1>Links for PyGTK</h1>
    <a href="https://files.pythonhosted.org/packages/7c/18/fa4f2de77500dd62a314fd845ff6e903ac2ce551164cb421c5750969f799/pygtk-2.24.0.tar.bz2#sha256=cd1c1ea265bd63ff669e92a2d3c2a88eb26bcd9e5363e0f82c896e649f206912">pygtk-2.24.0.tar.bz2</a><br/>
    <a href="https://files.pythonhosted.org/packages/e3/f0/8b842afbfef3803d01e54951a6b06118a06a4192d0f7ca53151a9d97f25e/pygtk-2.24.0.win32-py2.6.msi#sha256=f7919aa9fe2db45a4f176763fccc4bd7ae1198c6c77698f8469bda106ccdf15d">pygtk-2.24.0.win32-py2.6.msi</a><br/>
    <a href="https://files.pythonhosted.org/packages/f3/0c/b733dca9201c550935e3e1246ed3657e47bd7e9378ae2154d942d4f6b500/pygtk-2.24.0.win32-py2.7.msi#sha256=b3c6a3f1484acb3dce19e99cf8321fe292846e969fda71289b5bfee0c0cfc30d">pygtk-2.24.0.win32-py2.7.msi</a><br/>
    <a href="https://files.pythonhosted.org/packages/85/52/3d9bb924bd2c9bdb8afd9b7994cd09160fad5948bb2eca18fd7ffa12cfdc/pygtk-2.24.0.win32-py2.6.exe#sha256=16336e79f9a7913e5b2d1cf50120896495aac8892be2d352f660b905205c48db">pygtk-2.24.0.win32-py2.6.exe</a><br/>
    </body>
</html>
<!--SERIAL 265829-->


This is fairly straightforward to parse, but I was time constrained so I did the following hack in a new .bbclass:


def pypi_src_uri(d):
    # Bring in requests...  We're going to need it to get the actual URI
    # for the package...
    import os
    package = d.getVar('PYPI_PACKAGE')
    package_ext = d.getVar('PYPI_PACKAGE_EXT')
    pv = d.getVar('PV')
    # NASTY, UGLY **HACK** ALERT!! - Use a response fetch from wget to get the official response from
    # PyPI's simple interface and then crack it apart from there to get the actual URL...
    cmd = os.popen('wget -q -O - https://pypi.org/simple/%s/ | awk \'BEGIN{FS="[\\"#]"} /%s-%s.%s/ {print $2}\'' % (package, package, pv, package_ext))
    actual_uri = cmd.read()
    cmd.close()
    # Check for a blind spot.  We're grabbing a redirector URI.  This means the app server won't hand us anything
    # useful for a result that we can pick off if version or name is wrong.  So flag this and DIE hard...
    if not actual_uri :
        bb.fatal("pypi-new : Unable to find %s-%s.%s on PyPI." % (package, pv, package_ext))
    return actual_uri
Comment 3 akuster 2020-08-21 16:45:45 UTC
in zeus 3.0.4.
Comment 4 akuster 2020-08-21 16:46:00 UTC
closing