Bug 15993

Created attachment 5145 [details]
AB monocypher mirror test log.do_fetch (provided by Mathieu)

Added logs transmitted by Matthieu to compare

Interesting:
* In AB logs, PREMIRRORS list is really short (2 elements) where locally it is a lot longer
* In AB logs, no PREMIRRORS matches but does locally (that is where the fetch succeed)
* IN AB logs, no MIRROR matches (but that is because the recipe overwrite MIRRORS instead of extending it.)

The 2 PREMIRRORS lines comes from 
mirrors.bbclass
 83 PREMIRRORS += "git://sourceware.org/git/glibc.git https://downloads.yoctoproject.org/mirror/sources/ \
 84               git://sourceware.org/git/binutils-gdb.git https://downloads.yoctoproject.org/mirror/sources/"

but in poky/meta/lib/oeqa/selftest/cases/buildoptions.py, test is defined with:
@OETestTag("yocto-mirrors")
class SourceMirroring(OESelftestTestCase):
    # Can we download everything from the Yocto Sources Mirror over http only
    def test_yocto_source_mirror(self):
        self.write_config("""
BB_ALLOWED_NETWORKS = "downloads.yoctoproject.org"
MIRRORS = ""
DL_DIR = "${TMPDIR}/test_downloads"
STAMPS_DIR = "${TMPDIR}/test_stamps"
SSTATE_DIR = "${TMPDIR}/test_sstate-cache"
PREMIRRORS = "\\
    bzr://.*/.*   http://downloads.yoctoproject.org/mirror/sources/ \\n \\
    cvs://.*/.*   http://downloads.yoctoproject.org/mirror/sources/ \\n \\
    git://.*/.*   http://downloads.yoctoproject.org/mirror/sources/ \\n \\
    gitsm://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \\n \\
    hg://.*/.*    http://downloads.yoctoproject.org/mirror/sources/ \\n \\
    osc://.*/.*   http://downloads.yoctoproject.org/mirror/sources/ \\n \\
    p4://.*/.*    http://downloads.yoctoproject.org/mirror/sources/ \\n \\
    svn://.*/.*   http://downloads.yoctoproject.org/mirror/sources/ \\n \\
    ftp://.*/.*      http://downloads.yoctoproject.org/mirror/sources/ \\n \\
    http://.*/.*     http://downloads.yoctoproject.org/mirror/sources/ \\n \\
    https://.*/.*    http://downloads.yoctoproject.org/mirror/sources/ \\n"
""")

Maybe with a full builddir I could understand why PREMIRROR/MIRROR is not what I expect?
For example: ubuntu2204-vk-1:/srv/pokybuild/yocto-worker/meta-oe-mirror/build/ (or any other failed meta-oe-mirror build)

> "that is because the recipe overwrite MIRRORS instead of extending it"

Patch sent to fix this: https://lists.openembedded.org/g/openembedded-devel/topic/meta_oe_patch_monocypher/115681529
But, IMHO, the underlying problem remain, AB's PREMIRROR/MIRROR are not what I expect for this test.

Patch sent: https://lists.openembedded.org/g/openembedded-core/topic/patch/115698256
> Currently, the definitions of MIRRORS and PREMIRRORS made by
> test_yocto_source_mirror() are overwritten when run on the autobuilder:
> * MIRRORS = "" is extended by mirrors.bbclass to its usual value.
> * PREMIRRORS = "* dl.yp.org" is overwritten by the autobuilder.conf
>   config fragment to ""
> 
> Simplified variable history:
>   # $MIRRORS [4 operations]
>   #   set .../contribution-yocto/builds/build-qemux86-64-st/conf/selftest.inc:3
>   #     ""
>   #   append .../contribution-yocto/builds/build-qemux86-64-st/../../layers/poky/meta/classes-global/mirrors.bbclass:57
>   #     "${DEBIAN_MIRROR}	..."
>   MIRRORS=" http://deb.debian.org/debian/pool ..."
> 
>   # $PREMIRRORS [4 operations]
>   #   set .../contribution-yocto/builds/build-qemux86-64-st/conf/selftest.inc:18
>   #     "    bzr://.*/.*   http://downloads.yoctoproject.org/mirror/sources/ \n ..."
>   #   set .../contribution-yocto/builds/build-qemux86-64-st/../../layers/poky/meta/conf/fragments/yocto-autobuilder/autobuilder.conf:6
>   #     ""
>   #   append .../contribution-yocto/builds/build-qemux86-64-st/../../layers/poky/meta/classes-global/mirrors.bbclass:84
>   #     "git://sourceware.org/git/glibc.git https://downloads.yoctoproject.org/mirror/sources/  ..."
>   PREMIRRORS=" git://sourceware.org/git/glibc.git https://downloads.yoctoproject.org/mirror/sources/  ..."
> 
> Despite this unexpected configuration, the test passes because after
> failing to find a PREMIRROR (empty), failing to download the normal
> SRC_URI (because of BB_ALLOWED_NETWORKS), the fetcher tries the MIRRORS
> which have by default download.yoctoproject.org. For example, on a
> failed meta-oe-mirror test[0], we can see a lot of warnings:
> "stdio: WARNING: ... do_fetch: Failed to fetch URL https://... attempting MIRRORS if available"
> 
> By using the ":forcevariable" override, test_yocto_source_mirror() makes
> sure the correct value is set for MIRRORS and PREMIRRORS (whatever is
> the configuration).
> 
> Simplified variable history (after the fix):
>   # $MIRRORS [4 operations]
>   #   append .../contribution-yocto/builds/build-qemux86-64-st/../../layers/poky/meta/classes-global/mirrors.bbclass:57
>   #     "${DEBIAN_MIRROR}	..."
>   #   append .../contribution-yocto/builds/build-qemux86-64-st/../../layers/poky/meta/classes-global/mirrors.bbclass:68
>   #     "git://salsa.debian.org/.* ..."
>   #   override[forcevariable]:set .../contribution-yocto/builds/build-qemux86-64-st/conf/selftest.inc:3
>   #     ""
>   MIRRORS=""
> 
>   # $PREMIRRORS [4 operations]
>   #   set .../contribution-yocto/builds/build-qemux86-64-st/../../layers/poky/meta/conf/fragments/yocto-autobuilder/autobuilder.conf:6
>   #     ""
>   #   append .../contribution-yocto/builds/build-qemux86-64-st/../../layers/poky/meta/classes-global/mirrors.bbclass:84
>   #     "git://sourceware.org/git/glibc.git ..."
>   #   override[forcevariable]:set .../contribution-yocto/builds/build-qemux86-64-st/conf/selftest.inc:18
>   #     "    bzr://.*/.*   http://downloads.yoctoproject.org/mirror/sources/ ..."
>   # pre-expansion value:
>   #   "    bzr://.*/.*   http://downloads.yoctoproject.org/mirror/sources/ ..."
>   PREMIRRORS="    bzr://.*/.*   http://downloads.yoctoproject.org/mirror/sources/ ..."
> 
> [0]: https://autobuilder.yoctoproject.org/valkyrie/#/builders/82/builds/470/steps/17/logs/warnings

Fixed in
monocypher: prepend MIRRORS instead of overwriting
https://git.openembedded.org/meta-openembedded/commit/?id=b9471f4967c3d8ca828907c2fa41ea86b57efb97
oeqa/selftest/buildoptions: Fix test_yocto_source_mirror MIRRORS & PREMIRRORS definitions
https://git.openembedded.org/openembedded-core/commit/?id=ab810d1239d4db0bc0f23db31c70cc9c6e59357e