Bug 14690 - setscene errors accessing YP sstate servers from multiple threads
Summary: setscene errors accessing YP sstate servers from multiple threads
Status: RESOLVED FIXED
Alias: None
Product: Infrastructure General
Classification: Infrastructure
Component: infrastructure-general (show other bugs)
Version: 4.0
Hardware: x86 Multiple
: High normal
Target Milestone: 4.0 M3
Assignee: Michael Halstead
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2022-01-14 19:16 UTC by Michael Opdenacker
Modified: 2022-02-24 16:11 UTC (History)
3 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Michael Opdenacker 2022-01-14 19:16:01 UTC 
I'm building the latest Poky ("core-image-minimal" plus a few extra
packages), on Ubuntu 20.04.

I'm always getting the below setscene errors after upgrading my Poky
sources:

... <more similar errors before>
WARNING: libffi-native-3.4.2-r0 do_populate_sysroot_setscene: Failed to
fetch URL
file://universal/c8/be/sstate:libffi-native:x86_64-linux:3.4.2:r0:x86_64:7:c8be7b784ce8ebbf6d897367bc8e96f6718040858d61f9a4f6aa5325821b0ce5_populate_sysroot.tar.zst.siginfo;downloadfilename=universal/c8/be/sstate:libffi-native:x86_64-linux:3.4.2:r0:x86_64:7:c8be7b784ce8ebbf6d897367bc8e96f6718040858d61f9a4f6aa5325821b0ce5_populate_sysroot.tar.zst.siginfo,
attempting MIRRORS if available
ERROR: libffi-native-3.4.2-r0 do_populate_sysroot_setscene: Fetcher
failure: Unable to find file
file://universal/c8/be/sstate:libffi-native:x86_64-linux:3.4.2:r0:x86_64:7:c8be7b784ce8ebbf6d897367bc8e96f6718040858d61f9a4f6aa5325821b0ce5_populate_sysroot.tar.zst.siginfo;downloadfilename=universal/c8/be/sstate:libffi-native:x86_64-linux:3.4.2:r0:x86_64:7:c8be7b784ce8ebbf6d897367bc8e96f6718040858d61f9a4f6aa5325821b0ce5_populate_sysroot.tar.zst.siginfo
anywhere. The paths that were searched were:
    /media/mike/ssd/yocto/poky/build/sstate-cache
    /media/mike/ssd/yocto/poky/build/sstate-cache
WARNING: libpcre2-native-10.39-r0 do_populate_sysroot_setscene: Failed
to fetch URL
file://universal/bf/03/sstate:libpcre2-native:x86_64-linux:10.39:r0:x86_64:7:bf035286ff06470377fe9c9298ba116222c9d557f2fa44d418beb919aa185db9_populate_sysroot.tar.zst.siginfo;downloadfilename=universal/bf/03/sstate:libpcre2-native:x86_64-linux:10.39:r0:x86_64:7:bf035286ff06470377fe9c9298ba116222c9d557f2fa44d418beb919aa185db9_populate_sysroot.tar.zst.siginfo,
attempting MIRRORS if available
ERROR: libpcre2-native-10.39-r0 do_populate_sysroot_setscene: Fetcher
failure: Unable to find file
file://universal/bf/03/sstate:libpcre2-native:x86_64-linux:10.39:r0:x86_64:7:bf035286ff06470377fe9c9298ba116222c9d557f2fa44d418beb919aa185db9_populate_sysroot.tar.zst.siginfo;downloadfilename=universal/bf/03/sstate:libpcre2-native:x86_64-linux:10.39:r0:x86_64:7:bf035286ff06470377fe9c9298ba116222c9d557f2fa44d418beb919aa185db9_populate_sysroot.tar.zst.siginfo
anywhere. The paths that were searched were:
    /media/mike/ssd/yocto/poky/build/sstate-cache
    /media/mike/ssd/yocto/poky/build/sstate-cache
ERROR: libpcre2-native-10.39-r0 do_populate_sysroot_setscene: No
suitable staging package found
ERROR: Logfile of failure stored in:
/media/mike/ssd/yocto/poky/build/tmp/work/x86_64-linux/libpcre2-native/10.39-r0/temp/log.do_populate_sysroot_setscene.315121
WARNING: Setscene task
(virtual:native:/media/mike/ssd/yocto/poky/meta/recipes-support/libpcre/libpcre2_10.39.bb:do_populate_sysroot_setscene)
failed with exit code '1' - real task will be run instead
Currently  4 running tasks (1838 of 1838/467 of 4630)  10%
|################

The errors are not critical but paint my terminal with scary red messages.
Here are my settings:

BB_HASHSERVE = "auto"
BB_SIGNATURE_HANDLER = "OEEquivHash"
SSTATE_MIRRORS ?= "file://.* http://sstate.yoctoproject.org/dev/PATH;downloadfilename=PATH"
BB_HASHSERVE_UPSTREAM = "typhoon.yocto.io:8687"

The errors go away after setting:
BB_NUMBER_THREADS="1" (my system normally has 4 threads)
Thanks to Jose Quaresma for the workaround!

There may be an issue on the infrastructure, not support simultaneous requests from the same client...

I'll be happy to make further tests if needed.
Comment 1 Jose Quaresma 2022-01-19 23:30:11 UTC 
I have tested the state mirror locally with my own http mirror and it works perfectly. There are no problems with the concurrent connections.

I'm more sure that the root cause is some bandwidth/connection limitation on the http YP mirror.
Comment 2 Michael Opdenacker 2022-01-26 10:00:28 UTC 
The setscene issues are now warnings (better!) but there are fetcher errors now, that don't seem to be critical either:

...
WARNING: gpgme-native-1.16.0-r0 do_populate_sysroot_setscene: No sstate archive obtainable, will run full task instead.
WARNING: Logfile for failed setscene task is /media/mike/ssd/yocto/poky/build/tmp/work/x86_64-linux/gpgme-native/1.16.0-r0/temp/log.do_populate_sysroot_setscene.863220
WARNING: Setscene task (virtual:native:/media/mike/ssd/yocto/poky/meta/recipes-support/gpgme/gpgme_1.16.0.bb:do_populate_sysroot_setscene) failed with exit code '1' - real task will be run instead
WARNING: docbook-xml-dtd4-native-4.5-r0 do_populate_sysroot_setscene: No sstate archive obtainable, will run full task instead.
WARNING: Logfile for failed setscene task is /media/mike/ssd/yocto/poky/build/tmp/work/x86_64-linux/docbook-xml-dtd4-native/4.5-r0/temp/log.do_populate_sysroot_setscene.863223
WARNING: Setscene task (virtual:native:/media/mike/ssd/yocto/poky/meta/recipes-devtools/docbook-xml/docbook-xml-dtd4_4.5.bb:do_populate_sysroot_setscene) failed with exit code '1' - real task will be run instead
WARNING: itstool-native-2.0.7-r0 do_populate_sysroot_setscene: No sstate archive obtainable, will run full task instead.
WARNING: Logfile for failed setscene task is /media/mike/ssd/yocto/poky/build/tmp/work/x86_64-linux/itstool-native/2.0.7-r0/temp/log.do_populate_sysroot_setscene.863224
WARNING: Setscene task (virtual:native:/media/mike/ssd/yocto/poky/meta/recipes-support/itstool/itstool_2.0.7.bb:do_populate_sysroot_setscene) failed with exit code '1' - real task will be run instead
WARNING: libxau-native-1_1.0.9-r0 do_populate_sysroot_setscene: No sstate archive obtainable, will run full task instead.
WARNING: Logfile for failed setscene task is /media/mike/ssd/yocto/poky/build/tmp/work/x86_64-linux/libxau-native/1_1.0.9-r0/temp/log.do_populate_sysroot_setscene.863229
WARNING: Setscene task (virtual:native:/media/mike/ssd/yocto/poky/meta/recipes-graphics/xorg-lib/libxau_1.0.9.bb:do_populate_sysroot_setscene) failed with exit code '1' - real task will be run instead
WARNING: libxml2-native-2.9.12-r0 do_populate_sysroot_setscene: Failed to fetch URL file://universal/a6/72/sstate:libxml2-native:x86_64-linux:2.9.12:r0:x86_64:7:a67256090f630006c779e3ae43fde80224e33887736a79c9cc7f071ffa935325_populate_sysroot.tar.zst;downloadfilename=universal/a6/72/sstate:libxml2-native:x86_64-linux:2.9.12:r0:x86_64:7:a67256090f630006c779e3ae43fde80224e33887736a79c9cc7f071ffa935325_populate_sysroot.tar.zst, attempting MIRRORS if available
ERROR: libxml2-native-2.9.12-r0 do_populate_sysroot_setscene: Fetcher failure: Unable to find file file://universal/a6/72/sstate:libxml2-native:x86_64-linux:2.9.12:r0:x86_64:7:a67256090f630006c779e3ae43fde80224e33887736a79c9cc7f071ffa935325_populate_sysroot.tar.zst;downloadfilename=universal/a6/72/sstate:libxml2-native:x86_64-linux:2.9.12:r0:x86_64:7:a67256090f630006c779e3ae43fde80224e33887736a79c9cc7f071ffa935325_populate_sysroot.tar.zst anywhere. The paths that were searched were:
    /media/mike/ssd/yocto/poky/build/sstate-cache
    /media/mike/ssd/yocto/poky/build/sstate-cache
...
Comment 3 Randy MacLeod 2022-02-24 16:11:46 UTC 
Michael Halstead changed the setup of the sstate server.