Bug 12552

Summary: checkuri fails in master
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Ross Burton <ross.burton>
Component: coreAssignee: Maxin B. John <maxin.john>
Status: RESOLVED WORKSFORME QA Contact:
Severity: normal    
Priority: Medium+ CC: meta.mr.watcher, meta.watcher
Version: 2.5   
Target Milestone: 2.5 M4   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Ross Burton 2018-02-15 15:50:38 UTC
Summary: 5 tasks failed:
  /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-checkuri/build/meta/recipes-extended/pax/pax_3.4.bb:do_checkuri
  /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-checkuri/build/meta/recipes-devtools/zisofs-tools/zisofs-tools-native_1.0.8.bb:do_checkuri
  /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-checkuri/build/meta/recipes-extended/lsb/lsbinitscripts_9.72.bb:do_checkuri
  /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-checkuri/build/meta/recipes-extended/man/man_1.6g.bb:do_checkuri
  /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-checkuri/build/meta/recipes-kernel/latencytop/latencytop_0.5.bb:do_checkuri


All failing in the fetcher:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:do_checkuri(d)
     0003:
File: '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-checkuri/build/meta/classes/utility-tasks.bbclass', lineno: 48, function: do_checkuri
     0044:        return
     0045:
     0046:    try:
     0047:        fetcher = bb.fetch2.Fetch(src_uri, d)
 *** 0048:        fetcher.checkstatus()
     0049:    except bb.fetch2.BBFetchException as e:
     0050:        bb.fatal(str(e))
     0051:}
     0052:
File: '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-checkuri/build/bitbake/lib/bb/fetch2/__init__.py', lineno: 1739, function: checkstatus
     1735:            mirrors = mirror_from_string(self.d.getVar('PREMIRRORS'))
     1736:            ret = try_mirrors(self, self.d, ud, mirrors, True)
     1737:            if not ret:
     1738:                # Next try checking from the original uri, u
 *** 1739:                ret = m.checkstatus(self, ud, self.d)
     1740:                if not ret:
     1741:                    # Finally, try checking uri, u, from MIRRORS
     1742:                    mirrors = mirror_from_string(self.d.getVar('MIRRORS'))
     1743:                    ret = try_mirrors(self, self.d, ud, mirrors, True)
File: '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-checkuri/build/bitbake/lib/bb/fetch2/wget.py', lineno: 336, function: checkstatus
     0332:                add_basic_auth("%s:%s" % (login, password), r)
     0333:            except (TypeError, ImportError, IOError, netrc.NetrcParseError):
     0334:                 pass
     0335:
 *** 0336:            with opener.open(r) as response:
     0337:                pass
     0338:        except urllib.error.URLError as e:
     0339:            if try_again:
     0340:                logger.debug(2, "checkstatus: trying again")
File: '/usr/lib/python3.6/urllib/request.py', lineno: 532, function: open
     0528:        # post-process response
     0529:        meth_name = protocol+"_response"
     0530:        for processor in self.process_response.get(protocol, []):
     0531:            meth = getattr(processor, meth_name)
 *** 0532:            response = meth(req, response)
     0533:
     0534:        return response
     0535:
     0536:    def _open(self, req, data=None):
File: '/usr/lib/python3.6/urllib/request.py', lineno: 642, function: http_response
     0638:        # According to RFC 2616, "2xx" code indicates that the client's
     0639:        # request was successfully received, understood, and accepted.
     0640:        if not (200 <= code < 300):
     0641:            response = self.parent.error(
 *** 0642:                'http', request, response, code, msg, hdrs)
     0643:
     0644:        return response
     0645:
     0646:    https_response = http_response
File: '/usr/lib/python3.6/urllib/request.py', lineno: 564, function: error
     0560:            dict = self.handle_error
     0561:            meth_name = proto + '_error'
     0562:            http_err = 0
     0563:        args = (dict, proto, meth_name) + args
 *** 0564:        result = self._call_chain(*args)
     0565:        if result:
     0566:            return result
     0567:
     0568:        if http_err:
File: '/usr/lib/python3.6/urllib/request.py', lineno: 504, function: _call_chain
     0500:        # could.  Otherwise, they return the response.
     0501:        handlers = chain.get(kind, ())
     0502:        for handler in handlers:
     0503:            func = getattr(handler, meth_name)
 *** 0504:            result = func(*args)
     0505:            if result is not None:
     0506:                return result
     0507:
     0508:    def open(self, fullurl, data=None, timeout=socket._GLOBAL_DEFAULT_TIMEOUT):
File: '/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-checkuri/build/bitbake/lib/bb/fetch2/wget.py', lineno: 279, function: http_error_405
     0275:                                  if k.lower() not in ("content-length", "content-type"))
     0276:                return self.parent.open(urllib.request.Request(req.get_full_url(),
     0277:                                                        headers=newheaders,
     0278:                                                        origin_req_host=req.origin_req_host,
 *** 0279:                                                        unverifiable=True))
     0280:
     0281:            """
     0282:            Some servers (e.g. GitHub archives, hosted on Amazon S3) return 403
     0283:            Forbidden when they actually mean 405 Method Not Allowed.
File: '/usr/lib/python3.6/urllib/request.py', lineno: 532, function: open
     0528:        # post-process response
     0529:        meth_name = protocol+"_response"
     0530:        for processor in self.process_response.get(protocol, []):
     0531:            meth = getattr(processor, meth_name)
 *** 0532:            response = meth(req, response)
     0533:
     0534:        return response
     0535:
     0536:    def _open(self, req, data=None):
File: '/usr/lib/python3.6/urllib/request.py', lineno: 642, function: http_response
     0638:        # According to RFC 2616, "2xx" code indicates that the client's
     0639:        # request was successfully received, understood, and accepted.
     0640:        if not (200 <= code < 300):
     0641:            response = self.parent.error(
 *** 0642:                'http', request, response, code, msg, hdrs)
     0643:
     0644:        return response
     0645:
     0646:    https_response = http_response
[ ... ]
Comment 1 Maxin B. John 2018-03-09 04:51:43 UTC
The checkuril log looks a bit confusing:

-------
zisofs-tools was removed on Feb 12:

commit d594496190c0a4009a7690adbf54e80a95f36642
Author: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Date:   Mon Feb 12 13:05:16 2018 +0200

    image-live.bbclass: drop support for compressed ISO images
    
    While modern Linux kernels still support it, the userspace tools
    haven't been updated in over a decade. Also, squashfs provides
    both better performance, and better compression ratio:
    
    https://elinux.org/Squash_Fs_Comparisons
    
    (From OE-Core rev: 9443981d3934b366e39404719486d2b34d8a9d73)
    
    Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
    Signed-off-by: Ross Burton <ross.burton@intel.com>
    Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
-------

pax was removed on Feb 12

commit 1dd146d5acd6bb4c0d65045071e2fd27590b8df4
Author: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Date:   Mon Feb 12 13:05:15 2018 +0200

    pax: remove the recipe
    
    The situation with pax (vs. tar and cpio) is perfectly described here:
    https://xkcd.com/927/
    
    The only reason pax is still around, kind of, is because both POSIX and LSB
    mandate it. Outside of those documents, it's not used by anyone.
    
    Meanwhile, the upstream URI we've been taking it from went down, and rather
    than seek an alternative source, I just went ahead and removed the recipe.
    
    (From OE-Core rev: a62a7e11274c4965e122ed5bf01024cae8437af0)
    
    Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
    Signed-off-by: Ross Burton <ross.burton@intel.com>
    Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

-----

latencytop was removed on March 1

commit bcbc7bbc4fb967d8d4ae6333f71b73491a80b94e
Author: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Date:   Thu Mar 1 16:00:41 2018 +0200

    latencytop: remove recipe
    
    Last commit and release were in 2009; website is down; it's a dead project.
    
    (From OE-Core rev: 36aae56e7f86a4d5ce93e4528e7dcc42f60c705e)
    
    Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
    Signed-off-by: Ross Burton <ross.burton@intel.com>
    Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

------

man was replaced with man-db on Feb 12

commit d9046f8fd88108b006bf0615d56a826552b986a2
Author: Alexander Kanavin <alexander.kanavin@linux.intel.com>
Date:   Mon Feb 12 13:05:18 2018 +0200

    man-db: replace the outdated man implementation
    
    Until now oe-core has been using a well obsolete implementation of man.
    
    Man-db on the other hand is used by all modern Linux distros, is actively
    maintained, has a standard build system, and does not require 30 patches
    to build.
    
    (From OE-Core rev: 69e549a841a1681f927667f5017a1cd669ac9914)
    
    Signed-off-by: Alexander Kanavin <alexander.kanavin@linux.intel.com>
    Signed-off-by: Ross Burton <ross.burton@intel.com>
    Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

------
Comment 2 Maxin B. John 2018-03-09 05:35:26 UTC
In the case of sbinitscripts, 

"pkgs.fedoraproject.org" redirects to "https://src.fedoraproject.org". It will be good to update SRC_URI to reflect it.
Comment 3 Maxin B. John 2018-03-09 06:08:09 UTC
Alex has already sent a fix/upgrade for lsbinitscripts.
http://lists.openembedded.org/pipermail/openembedded-core/2018-March/148046.html