Created attachment 5144 [details] local monocypher mirror test log.do_fetch On AB, the meta-oe-mirror builder[0] repeatably fails on the monocypher recipe: stdio: ERROR: monocypher-4.0.2-r0 do_fetch: The URL: 'https://monocypher.org/download/monocypher-4.0.2.tar.gz' is not trusted and cannot be used stdio: ERROR: monocypher-4.0.2-r0 do_fetch: Bitbake Fetcher Error: FetchError('Unable to fetch URL from any source.', 'https://monocypher.org/download/monocypher-4.0.2.tar.gz') stdio: ERROR: Logfile of failure stored in: /srv/pokybuild/yocto-worker/meta-oe-mirror/build/build-st/tmp/work/x86-64-v3-poky-linux/monocypher/4.0.2/temp/log.do_fetch.1492172 stdio: ERROR: Task (/srv/pokybuild/yocto-worker/meta-oe-mirror/build/meta-openembedded/meta-oe/recipes-crypto/monocypher/monocypher_4.0.2.bb:do_fetch) failed with exit code '1' But, locally the test is OK. The archive mirrored on downloads.yoctoproject.org is present and correct. Archive downloaded from mirrors $ wget https://downloads.yoctoproject.org/mirror/sources/monocypher-4.0.2.tar.gz $ sha256sum monocypher-4.0.2.tar.gz bf275d4c53ff94af6cdc723a4e002e9f080f4d1436c86c76bb37870b34807f1d7b32331d8ff8a1aeb369e946f3769021e03e63efac25b82efc5abf54dc084714 monocypher-4.0.2.tar.gz (matches the sha512sum from the recipe) Archive downloaded from upstream $ wget https://monocypher.org/download/monocypher-4.0.2.tar.gz $ sha512sum monocypher-4.0.2.tar.gz bf275d4c53ff94af6cdc723a4e002e9f080f4d1436c86c76bb37870b34807f1d7b32331d8ff8a1aeb369e946f3769021e03e63efac25b82efc5abf54dc084714 monocypher-4.0.2.tar.gz (same sha512sum) [0]: https://autobuilder.yoctoproject.org/valkyrie/#/builders/82
Created attachment 5145 [details] AB monocypher mirror test log.do_fetch (provided by Mathieu) Added logs transmitted by Matthieu to compare
Interesting: * In AB logs, PREMIRRORS list is really short (2 elements) where locally it is a lot longer * In AB logs, no PREMIRRORS matches but does locally (that is where the fetch succeed) * IN AB logs, no MIRROR matches (but that is because the recipe overwrite MIRRORS instead of extending it.) The 2 PREMIRRORS lines comes from mirrors.bbclass 83 PREMIRRORS += "git://sourceware.org/git/glibc.git https://downloads.yoctoproject.org/mirror/sources/ \ 84 git://sourceware.org/git/binutils-gdb.git https://downloads.yoctoproject.org/mirror/sources/" but in poky/meta/lib/oeqa/selftest/cases/buildoptions.py, test is defined with: @OETestTag("yocto-mirrors") class SourceMirroring(OESelftestTestCase): # Can we download everything from the Yocto Sources Mirror over http only def test_yocto_source_mirror(self): self.write_config(""" BB_ALLOWED_NETWORKS = "downloads.yoctoproject.org" MIRRORS = "" DL_DIR = "${TMPDIR}/test_downloads" STAMPS_DIR = "${TMPDIR}/test_stamps" SSTATE_DIR = "${TMPDIR}/test_sstate-cache" PREMIRRORS = "\\ bzr://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \\n \\ cvs://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \\n \\ git://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \\n \\ gitsm://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \\n \\ hg://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \\n \\ osc://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \\n \\ p4://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \\n \\ svn://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \\n \\ ftp://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \\n \\ http://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \\n \\ https://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \\n" """)
Maybe with a full builddir I could understand why PREMIRROR/MIRROR is not what I expect? For example: ubuntu2204-vk-1:/srv/pokybuild/yocto-worker/meta-oe-mirror/build/ (or any other failed meta-oe-mirror build)
> "that is because the recipe overwrite MIRRORS instead of extending it" Patch sent to fix this: https://lists.openembedded.org/g/openembedded-devel/topic/meta_oe_patch_monocypher/115681529 But, IMHO, the underlying problem remain, AB's PREMIRROR/MIRROR are not what I expect for this test.
Patch sent: https://lists.openembedded.org/g/openembedded-core/topic/patch/115698256 > Currently, the definitions of MIRRORS and PREMIRRORS made by > test_yocto_source_mirror() are overwritten when run on the autobuilder: > * MIRRORS = "" is extended by mirrors.bbclass to its usual value. > * PREMIRRORS = "* dl.yp.org" is overwritten by the autobuilder.conf > config fragment to "" > > Simplified variable history: > # $MIRRORS [4 operations] > # set .../contribution-yocto/builds/build-qemux86-64-st/conf/selftest.inc:3 > # "" > # append .../contribution-yocto/builds/build-qemux86-64-st/../../layers/poky/meta/classes-global/mirrors.bbclass:57 > # "${DEBIAN_MIRROR} ..." > MIRRORS=" http://deb.debian.org/debian/pool ..." > > # $PREMIRRORS [4 operations] > # set .../contribution-yocto/builds/build-qemux86-64-st/conf/selftest.inc:18 > # " bzr://.*/.* http://downloads.yoctoproject.org/mirror/sources/ \n ..." > # set .../contribution-yocto/builds/build-qemux86-64-st/../../layers/poky/meta/conf/fragments/yocto-autobuilder/autobuilder.conf:6 > # "" > # append .../contribution-yocto/builds/build-qemux86-64-st/../../layers/poky/meta/classes-global/mirrors.bbclass:84 > # "git://sourceware.org/git/glibc.git https://downloads.yoctoproject.org/mirror/sources/ ..." > PREMIRRORS=" git://sourceware.org/git/glibc.git https://downloads.yoctoproject.org/mirror/sources/ ..." > > Despite this unexpected configuration, the test passes because after > failing to find a PREMIRROR (empty), failing to download the normal > SRC_URI (because of BB_ALLOWED_NETWORKS), the fetcher tries the MIRRORS > which have by default download.yoctoproject.org. For example, on a > failed meta-oe-mirror test[0], we can see a lot of warnings: > "stdio: WARNING: ... do_fetch: Failed to fetch URL https://... attempting MIRRORS if available" > > By using the ":forcevariable" override, test_yocto_source_mirror() makes > sure the correct value is set for MIRRORS and PREMIRRORS (whatever is > the configuration). > > Simplified variable history (after the fix): > # $MIRRORS [4 operations] > # append .../contribution-yocto/builds/build-qemux86-64-st/../../layers/poky/meta/classes-global/mirrors.bbclass:57 > # "${DEBIAN_MIRROR} ..." > # append .../contribution-yocto/builds/build-qemux86-64-st/../../layers/poky/meta/classes-global/mirrors.bbclass:68 > # "git://salsa.debian.org/.* ..." > # override[forcevariable]:set .../contribution-yocto/builds/build-qemux86-64-st/conf/selftest.inc:3 > # "" > MIRRORS="" > > # $PREMIRRORS [4 operations] > # set .../contribution-yocto/builds/build-qemux86-64-st/../../layers/poky/meta/conf/fragments/yocto-autobuilder/autobuilder.conf:6 > # "" > # append .../contribution-yocto/builds/build-qemux86-64-st/../../layers/poky/meta/classes-global/mirrors.bbclass:84 > # "git://sourceware.org/git/glibc.git ..." > # override[forcevariable]:set .../contribution-yocto/builds/build-qemux86-64-st/conf/selftest.inc:18 > # " bzr://.*/.* http://downloads.yoctoproject.org/mirror/sources/ ..." > # pre-expansion value: > # " bzr://.*/.* http://downloads.yoctoproject.org/mirror/sources/ ..." > PREMIRRORS=" bzr://.*/.* http://downloads.yoctoproject.org/mirror/sources/ ..." > > [0]: https://autobuilder.yoctoproject.org/valkyrie/#/builders/82/builds/470/steps/17/logs/warnings
Fixed in monocypher: prepend MIRRORS instead of overwriting https://git.openembedded.org/meta-openembedded/commit/?id=b9471f4967c3d8ca828907c2fa41ea86b57efb97 oeqa/selftest/buildoptions: Fix test_yocto_source_mirror MIRRORS & PREMIRRORS definitions https://git.openembedded.org/openembedded-core/commit/?id=ab810d1239d4db0bc0f23db31c70cc9c6e59357e