<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!DOCTYPE bugzilla SYSTEM "https://bugzilla.yoctoproject.org/page.cgi?id=bugzilla.dtd">

<bugzilla version="5.0.6"
          urlbase="https://bugzilla.yoctoproject.org/"
          
          maintainer="it-coreprojects-helpdesk@linuxfoundation.org"
>

    <bug>
          <bug_id>13990</bug_id>
          
          <creation_ts>2020-07-23 14:57:15 +0000</creation_ts>
          <short_desc>pypi bbclass no longer processes downloads from https://files.pythonhosted.org/</short_desc>
          <delta_ts>2020-08-21 16:46:00 +0000</delta_ts>
          <reporter_accessible>1</reporter_accessible>
          <cclist_accessible>1</cclist_accessible>
          <classification_id>7</classification_id>
          <classification>Build System, Metadata &amp; Runtime</classification>
          <product>OE-Core</product>
          <component>oe-core other</component>
          <version>2.7.5</version>
          <rep_platform>Other</rep_platform>
          <op_sys>Multiple</op_sys>
          <bug_status>CLOSED</bug_status>
          <resolution>FIXED</resolution>
          
          
          <bug_file_loc></bug_file_loc>
          <status_whiteboard></status_whiteboard>
          <keywords></keywords>
          <priority>Medium+</priority>
          <bug_severity>critical</bug_severity>
          <target_milestone>3.2 M3</target_milestone>
          
          
          <everconfirmed>1</everconfirmed>
          <reporter name="Frank Earl">linusti</reporter>
          <assigned_to name="akuster">akuster808</assigned_to>
          <cc>akuster808</cc>
    
    <cc>randy.macleod</cc>
          
          
          <cf_os>---</cf_os>
          <cf_regression_type>---</cf_regression_type>
          
          <cf_docchange>No (bug/feature does not impact docs)</cf_docchange>

      

      

      

          <comment_sort_order>oldest_to_newest</comment_sort_order>  
          <long_desc isprivate="0" >
    <commentid>87831</commentid>
    <comment_count>0</comment_count>
    <who name="Frank Earl">linusti</who>
    <bug_when>2020-07-23 14:57:15 +0000</bug_when>
    <thetext>Overview:

PyPI has changed the way you download content (Over two years ago, in fact...) so that if you use the current bbclass to download a new version of a recipe or add a new recipe leveraging something from out of PyPI, you will get a 404 error on the download.  This is due to a CDN hashing algorithm that is completely different from what is used in the recipe for the URI generation.


Steps to Reproduce: 

Attempt to download a version not in the current global downloads cache or add a new packaging recipe for a new PyPI package.


Actual Results:

404 error for fetcher download attempt.


Expected Results:

Download of the archive file with sources to be set up with setuptools.


Builds affected:

ANYTHING using pypi.bbclass at this time as the old method will no longer work.  Needs to have fixes applied to multiple versions.  Happened with me on Warrior and Zeus with a Tegra and a RaspberryPI 4 target.


Root Cause:

The following block has the issue-


def pypi_src_uri(d):
    package = d.getVar(&apos;PYPI_PACKAGE&apos;)
    package_ext = d.getVar(&apos;PYPI_PACKAGE_EXT&apos;)
    pv = d.getVar(&apos;PV&apos;)
    return &apos;https://files.pythonhosted.org/packages/source/%s/%s/%s-%s.%s&apos; % (package[0], package, package, pv, package_ext)


The URI used in the return is no longer valid.  They use a &quot;simple&quot; interface or similar which provides a series of parsable links, checksums, etc. that pip and similar tools use to do their download operations against.


A line-item link dump is as follows:


&lt;!DOCTYPE html&gt;
&lt;html&gt;
  &lt;head&gt;
    &lt;title&gt;Links for PyGTK&lt;/title&gt;
  &lt;/head&gt;
  &lt;body&gt;
    &lt;h1&gt;Links for PyGTK&lt;/h1&gt;
    &lt;a href=&quot;https://files.pythonhosted.org/packages/7c/18/fa4f2de77500dd62a314fd845ff6e903ac2ce551164cb421c5750969f799/pygtk-2.24.0.tar.bz2#sha256=cd1c1ea265bd63ff669e92a2d3c2a88eb26bcd9e5363e0f82c896e649f206912&quot;&gt;pygtk-2.24.0.tar.bz2&lt;/a&gt;&lt;br/&gt;
    &lt;a href=&quot;https://files.pythonhosted.org/packages/e3/f0/8b842afbfef3803d01e54951a6b06118a06a4192d0f7ca53151a9d97f25e/pygtk-2.24.0.win32-py2.6.msi#sha256=f7919aa9fe2db45a4f176763fccc4bd7ae1198c6c77698f8469bda106ccdf15d&quot;&gt;pygtk-2.24.0.win32-py2.6.msi&lt;/a&gt;&lt;br/&gt;
    &lt;a href=&quot;https://files.pythonhosted.org/packages/f3/0c/b733dca9201c550935e3e1246ed3657e47bd7e9378ae2154d942d4f6b500/pygtk-2.24.0.win32-py2.7.msi#sha256=b3c6a3f1484acb3dce19e99cf8321fe292846e969fda71289b5bfee0c0cfc30d&quot;&gt;pygtk-2.24.0.win32-py2.7.msi&lt;/a&gt;&lt;br/&gt;
    &lt;a href=&quot;https://files.pythonhosted.org/packages/85/52/3d9bb924bd2c9bdb8afd9b7994cd09160fad5948bb2eca18fd7ffa12cfdc/pygtk-2.24.0.win32-py2.6.exe#sha256=16336e79f9a7913e5b2d1cf50120896495aac8892be2d352f660b905205c48db&quot;&gt;pygtk-2.24.0.win32-py2.6.exe&lt;/a&gt;&lt;br/&gt;
    &lt;/body&gt;
&lt;/html&gt;
&lt;!--SERIAL 265829--&gt;


This is fairly straightforward to parse, but I was time constrained so I did the following hack in a new .bbclass:


def pypi_src_uri(d):
    # Bring in requests...  We&apos;re going to need it to get the actual URI
    # for the package...
    import os
    package = d.getVar(&apos;PYPI_PACKAGE&apos;)
    package_ext = d.getVar(&apos;PYPI_PACKAGE_EXT&apos;)
    pv = d.getVar(&apos;PV&apos;)
    # NASTY, UGLY **HACK** ALERT!! - Use a response fetch from wget to get the official response from
    # PyPI&apos;s simple interface and then crack it apart from there to get the actual URL...
    cmd = os.popen(&apos;wget -q -O - https://pypi.org/simple/%s/ | awk \&apos;BEGIN{FS=&quot;[\\&quot;#]&quot;} /%s-%s.%s/ {print $2}\&apos;&apos; % (package, package, pv, package_ext))
    actual_uri = cmd.read()
    cmd.close()
    # Check for a blind spot.  We&apos;re grabbing a redirector URI.  This means the app server won&apos;t hand us anything
    # useful for a result that we can pick off if version or name is wrong.  So flag this and DIE hard...
    if not actual_uri :
        bb.fatal(&quot;pypi-new : Unable to find %s-%s.%s on PyPI.&quot; % (package, pv, package_ext))
    return actual_uri</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>87858</commentid>
    <comment_count>1</comment_count>
    <who name="akuster">akuster808</who>
    <bug_when>2020-07-30 08:21:13 +0000</bug_when>
    <thetext>Zeus could use with:

https://git.openembedded.org/openembedded-core/commit/meta/classes/pypi.bbclass?h=dunfell&amp;id=0603f6d9f2abfa67b99b1bc39228f6aa16a0370d

https://git.openembedded.org/openembedded-core/commit/meta/classes/pypi.bbclass?h=dunfell&amp;id=e5f3f961242d888f3f786af8f793bf1d247fdff0

will have to send patches to list.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>87900</commentid>
    <comment_count>2</comment_count>
    <who name="akuster">akuster808</who>
    <bug_when>2020-08-08 07:52:48 +0000</bug_when>
    <thetext>patches sent to list

https://patchwork.openembedded.org/patch/175156/
https://patchwork.openembedded.org/patch/175157/</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>88012</commentid>
    <comment_count>3</comment_count>
    <who name="akuster">akuster808</who>
    <bug_when>2020-08-21 16:45:45 +0000</bug_when>
    <thetext>in zeus 3.0.4.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>88013</commentid>
    <comment_count>4</comment_count>
    <who name="akuster">akuster808</who>
    <bug_when>2020-08-21 16:46:00 +0000</bug_when>
    <thetext>closing</thetext>
  </long_desc>
      
      

    </bug>

</bugzilla>