Bug 15289 - [scarthgap] AB-INT: sstatetests.SStatePrintdiff.test_gcc_runtime_vs_gcc_source failure
Summary: [scarthgap] AB-INT: sstatetests.SStatePrintdiff.test_gcc_runtime_vs_gcc_sourc...
Status: RESOLVED FIXED
Alias: None
Product: Functional (self) Testing
Classification: QA/Testing
Component: oe-selftest (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 5.0.22
Assignee: Yoann Congal
QA Contact:
URL:
Whiteboard: AB-INT
Depends on:
Blocks:
 
Reported: 2023-11-20 22:11 UTC by Alexandre Belloni
Modified: 2026-11-12 16:00 UTC (History)
7 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Alexandre Belloni 2023-11-20 22:11:58 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2459/steps/14/logs/stdio

oe-selftest-armhost ubuntu2204-arm-1


This has been discussed in this thread:
https://lore.kernel.org/all/20231018193932e6cb99ef@mail.local/
Comment 1 Alexandre Belloni 2023-11-20 22:32:07 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2452/steps/15/logs/stdio

oe-selftest-armhost ubuntu1804-arm-1
Comment 2 Alexander Kanavin 2023-11-24 08:04:43 UTC
There have been fixes from RP, but the latest fail indicates it is not enough. I've reformatted the error for easier reading:

AssertionError: "Task gcc-source-13.2.0:do_preconfigure couldn't be used from the cache because:" not found in 'NOTE: Reconnecting to bitbake server...
Loading cache...done.
Loaded 1917 entries from dependency cache.
Parsing recipes...done.
Parsing of 968 .bb files complete (967 cached, 1 parsed). 1917 targets, 66 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION           = "2.6.0"
BUILD_SYS            = "aarch64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "aarch64-poky-linux"
MACHINE              = "qemuarm64"
DISTRO               = "poky"
DISTRO_VERSION       = "4.3+snapshot-3e1a7f27ae52a4c4cc736429d08931c51dcf8d60"
TUNE_FEATURES        = "aarch64 armv8a crc cortexa57"
TARGET_FPU           = ""
meta                 
meta-poky            
meta-yocto-bsp       = "abelloni/master-next:3e1a7f27ae52a4c4cc736429d08931c51dcf8d60"
meta-selftest        = "master:0ee8c132bec7504658696f385e7cdbbace7330b3"

Initialising tasks...done.
The differences between the current build and any cached tasks start at the following tasks:
virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-core/util-linux/util-linux-libuuid_2.39.2.bb:do_configure
virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-extended/zstd/zstd_1.5.5.bb:do_configure
virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-core/ncurses/ncurses_6.4.bb:do_configure
virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-support/libcap-ng/libcap-ng_0.8.3.bb:do_configure
virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-devtools/pkgconfig/pkgconfig_git.bb:do_configure
/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-devtools/libtool/libtool-native_2.4.7.bb:do_configure
virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-support/sqlite/sqlite3_3.43.2.bb:do_configure
virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-gnome/gtk-doc/gtk-doc_1.33.2.bb:do_configure
virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-support/gdbm/gdbm_1.23.bb:do_configure
virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-support/libcap/libcap_2.69.bb:do_configure
virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-support/attr/acl_2.3.1.bb:do_configure
virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-support/gmp/gmp_6.3.0.bb:do_configure
virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-support/libgpg-error/libgpg-error_1.47.bb:do_configure
virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-extended/gperf/gperf_3.1.bb:do_configure
virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-core/readline/readline_8.2.bb:do_configure
virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-devtools/flex/flex_2.6.4.bb:do_configure
virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-devtools/autoconf-archive/autoconf-archive_2023.02.20.bb:do_configure
virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-devtools/perl-cross/perlcross_1.5.bb:do_configure
virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-core/zlib/zlib_1.3.bb:do_configure
virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-support/attr/attr_2.5.1.bb:do_configure
virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-support/libpcre/libpcre2_10.42.bb:do_configure
virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-devtools/make/make_4.4.1.bb:do_configure
virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-devtools/automake/automake_1.16.
6045b
  Hash for task dependency zstd-native:do_patch changed from f279503e4d330fbee555cd860be8a1a7bd30616dbe7d233ca37155e4cc29e490 to ece0002a5c9482ec0b93bfc3394b9fd012297522de059788d5ffe6ce82f19b20
      basehash changed from 13a61b22ae8f12485c4a38579cdeafe331f6ad6c43993df6dcc3b82f106e3bad to 89f58e67bc4659b234008a66bbbc490fff85e11dd306acfa06bf642dba6bd4d0
      Variable ERROR_QA value changed:
@@ -1,3 +1,2 @@
 dev-so debug-deps dev-deps debug-files arch pkgconfig la             perms dep-cmp pkgvarcheck perm-config perm-line perm-link             split-strip packages-list pkgv-undefined var-undefined             version-going-backwards expanded-d invalid-chars             license-checksum dev-elf file-rdeps configure-unsafe             configure-gettext perllocalpod shebang-size             already-stripped installed-vs-shipped ldflags compile-host-path             install-host-path pn-overrides unknown-configure-option             useless-rpaths rpaths staticdev empty-dirs             patch-fuzz             ${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', ' usrmerge', '', d)} patch-status
 DISTRO_FEATURES{usrmerge} = Unset
-_remove of packages-list

Task ncurses-native:do_configure couldn't be used from the cache because:
  We need hash 78d0aae4714a6d64f770cf7675e315b63ce791776aefb1d5ad8d99c392630889, most recent matching task was 4011fecea3f8074a5a2da8f17e2e33cd5259fdd85e9aaa75694251bd21abb378
  Hash for task dependency ncurses-native:do_patch changed from c432f2c3617202891c56bb24f7dd6fb16f45bdb65a7973688732aefeacde93cb to 928fda744c0d9aa7c535bb2c886a9f376fb1f39067d097941e54662d82b3e454
      basehash changed from b48fde1ec634424e098454778e7931936afa4d58122c2c9cb84748bb55a626ed to d44e1f0a1871bb013ad69f4a6ab325dae506d36ba9998b976ef84d3116a7c8f0
      Variable ERROR_QA value changed:
@@ -1,3 +1,2 @@
 dev-so debug-deps dev-deps debug-files arch pkgconfig la             perms dep-cmp pkgvarcheck perm-config perm-line perm-link             split-strip packages-list pkgv-undefined var-undefined             version-going-backwards expanded-d invalid-chars             license-checksum dev-elf file-rdeps configure-unsafe             configure-gettext perllocalpod shebang-size             already-stripped installed-vs-shipped ldflags compile-host-path             install-host-path pn-overrides unknown-configure-option             useless-rpaths rpaths staticdev empty-dirs             patch-fuzz             ${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', ' usrmerge', '', d)} patch-status
 DISTRO_FEATURES{usrmerge} = Unset
-_remove of packages-list

Task libcap-ng-native:do_configure couldn't be used from the cache because:
  We need hash f76f24d30d7e29732fb2c941b1509cdc72110434f8158fe3114f4af3426c9d7f, most recent matching task was f151f850357fc67f9a2b356f42542942726808c8f5526a2f4e5da2712c9ce209
  Hash for task dependency libcap-ng-native:do_patch changed from 7a632fb6de5f089e0e975cee6411939a5a5230726cccc18eb90c728efaf5d7cd to 628fefbb5f40d1032eacb5bfe45247b896b15baa0ea3dd254303148ab77a25f6
      basehash changed from 042405e4118583324f220a859644ccb2cc5764c0c65a47d05e885004eb30c1a9 to d4e1c98dbd47c2ed96d4b1d1e3907b9063256880e102ca0243ab25ff415ca45d
      Variable ERROR_QA value changed:
@@ -1,3 +1,2 @@
 dev-so debug-deps dev-deps debug-files arch pkgconfig la             perms dep-cmp pkgvarcheck perm-config perm-line perm-link             split-strip packages-list pkgv-undefined var-undefined             version-going-backwards expanded-d invalid-chars             license-checksum dev-elf file-rdeps configure-unsafe             configure-gettext perllocalpod shebang-size             already-stripped installed-vs-shipped ldflags compile-host-path             install-host-path pn-overrides unknown-configure-option             useless-rpaths rpaths staticdev empty-dirs             patch-fuzz             ${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', ' usrmerge', '', d)} patch-status
 DISTRO_FEATURES{usrmerge} = Unset
-_remove of packages-list

Task pkgconfig-na
taticdev empty-dirs             patch-fuzz             ${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', ' usrmerge', '', d)} patch-status
 DISTRO_FEATURES{usrmerge} = Unset
-_remove of packages-list

Task gtk-doc-native:do_configure couldn't be used from the cache because:
  We need hash 078a314de2e6a16abd4c56e7165a5aa481729a122906bf12244c1d4778f42f47, most recent matching task was 6f9d3a87d154e89f24ad5017f36debd8b8cf51968aed4bfb697434db68421f00
  Hash for task dependency gtk-doc-native:do_prepare_recipe_sysroot changed from 85ef48432d75e95a03ac7a546721ec7b66ff8f99b54d5cfbf607eb52575d6fe8 to 20663adb53f3f250b619769eef76ffa34ef8d5b28fa717aac89a007f0163b76d
      runtaskdeps changed:
['autoconf-native:do_populate_sysroot automake-native:do_populate_sysroot gtk-doc-native:do_fetch libtool-native:do_populate_sysroot', -libxslt-native:do_populate_sysroot, 'pkgconfig-native:do_populate_sysroot', -python3-pygments-native:do_populate_sysroot python3-six-native:do_populate_sysroot xmlto-native:do_populate_sysroot]
      Number of task dependencies changed
      Dependency on task libxslt-native:do_populate_sysroot was removed with hash eef25fc35c5ba643dd249636971b23297a349ecbad91d334da73530c614dd764
      Dependency on task python3-pygments-native:do_populate_sysroot was removed with hash c7a9972945a1581c9be54a1d46c1930a3183d50630363d83e4c553567e5a153c
      Dependency on task python3-six-native:do_populate_sysroot was removed with hash 4f92da447b69c5d1f632e75ca543d6b6d1cd57c6a04e3239aa784b3ae809f5c6
      Dependency on task xmlto-native:do_populate_sysroot was removed with hash 3cb19240b8be124409c2673b3efeadd62e760c6b68b7548163a5230c841d061e

Task gdbm-native:do_configure couldn't be used from the cache because:
  We need hash 9da4a4005dae91d82693b5a4a7bd3aa9a25aeb54979f63df884cd3d722a2294a, most recent matching task was 54813d40c91dce89a9e3acf1bb4cd14ebbe390d3c82b97b85108c81dea861bab
  Hash for task dependency gdbm-native:do_patch changed from 6082cff62895abdf77fee41de6021331ba78639f17bfbf41e5cbd552c483f548 to 03e06f7058f907b748c39c7672db1dd7495003c5b52519cf2a93284a280d1283
      basehash changed from 17b75ae3b32eeab59dd9ed118e0ee534afdc83c93304e1c6e82957658e8a7e00 to 0a8fa0a1876c9a10e2d2642e0369281b0faf378d7bfe3b71b103a876ec199dea
      Variable ERROR_QA value changed:
@@ -1,3 +1,2 @@
 dev-so debug-deps dev-deps debug-files arch pkgconfig la             perms dep-cmp pkgvarcheck perm-config perm-line perm-link             split-strip packages-list pkgv-undefined var-undefined             version-going-backwards expanded-d invalid-chars             license-checksum dev-elf file-rdeps configure-unsafe             configure-gettext perllocalpod shebang-size             already-stripped installed-vs-shipped ldflags compile-host-path             install-host-path pn-overrides unknown-configure-option             useless-rpaths rpaths staticdev empty-dirs             patch-fuzz             ${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', ' usrmerge', '', d)} patch-status
 DISTRO_FEATURES{usrmerge} = Unset
-_remove of packages-list

Task libcap-native:do_configure couldn't be used from the cache because:
  We need hash 49438a9bf8589873fbe356f2107465faffd7f7e607ac9ee3454376c864e83958, most recent matching task was 85e769dcdb5756451234421be32f2e59d176eadc205c3538014edbc133471e74
  Hash for task dependency libcap-native:do_patch changed from 008fdc3b75d5abd597483e2b93e67167eaa434d25daac5d7662a6727f4414357 to 65a9120bb5bbd9ff1d5965c51d73a9fc9cd98d3ce784624957aed6749f012013
      Hash for task dependency libcap-native:do_unpack changed from d123579241602cec26366a1dd8ce1db08fefcc57b5f2dcca4fe41ca26fd3b4c8 to c070d559fa2fe2ea9e51c46be16daf1f7e9b6f897391f7bf797388a41a887e71
          Hash for task dependency xz-native:do_populate_sysroot changed from e764559dd41b88e01eee48c33b36c90aea8930ceab6500904539b1b494a803c8 to 5c53f4cadbe8e5f2a23858836dd351c86ab0ab3e9fc452326bda1442a36e31b1
              Hash for task dependency xz-native:do_install changed from 2d2dd12bc988cf8ef3146a2ffe74d6
n-going-backwards expanded-d invalid-chars             license-checksum dev-elf file-rdeps configure-unsafe             configure-gettext perllocalpod shebang-size             already-stripped installed-vs-shipped ldflags compile-host-path             install-host-path pn-overrides unknown-configure-option             useless-rpaths rpaths staticdev empty-dirs             patch-fuzz             ${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', ' usrmerge', '', d)} patch-status
 DISTRO_FEATURES{usrmerge} = Unset
-_remove of packages-list

Task libgpg-error-native:do_configure couldn't be used from the cache because:
  We need hash e1754254d14a519466980ab86174e0aa5bbe5b9c1309caec74a585695b30bede, most recent matching task was 434951e68c5c317dd9b6475bc5270a280bf0774fdc2c0b5eaba06dc116db6cac
  Hash for task dependency libgpg-error-native:do_patch changed from 176aec682ebf41f7c1013033ac1d93953116afb0b6fd891d4b3afcdbb3371502 to 8881f3c5dc29cd08930fddd8ec7c18b3c3a50234439f0fb50e5182972b22299e
      basehash changed from 151adb611155af57040cb6b10851498e8c0325cab4825fcbc543a1c66bc9d317 to 4ee092ecb9d189bf1d8d1fed2d5510be2ee0fc9f5c9ffa88ab018c707ca5f3bf
      Variable ERROR_QA value changed:
@@ -1,3 +1,2 @@
 dev-so debug-deps dev-deps debug-files arch pkgconfig la             perms dep-cmp pkgvarcheck perm-config perm-line perm-link             split-strip packages-list pkgv-undefined var-undefined             version-going-backwards expanded-d invalid-chars             license-checksum dev-elf file-rdeps configure-unsafe             configure-gettext perllocalpod shebang-size             already-stripped installed-vs-shipped ldflags compile-host-path             install-host-path pn-overrides unknown-configure-option             useless-rpaths rpaths staticdev empty-dirs             patch-fuzz             ${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', ' usrmerge', '', d)} patch-status
 DISTRO_FEATURES{usrmerge} = Unset
-_remove of packages-list

Task gperf-native:do_configure couldn't be used from the cache because:
  We need hash e25771ffabe31e201e27d7c21a83cd452c9cee2da63bc0035591c8d1b00732fb, most recent matching task was 09d89dcac23107a528871095e0f87263bd1d32fb2e3ca906d67786b6f9f60b98
  Hash for task dependency gperf-native:do_patch changed from 19d53b78b17d69a0d2fbafaf8a8afb378db36ef7a467ad1e8422dfc059fac59e to 363c3035f8f9c94cdfec63eb7e5d811048df35ab49505636d67dd87a56769a2a
      basehash changed from f483e8d4baa023be2542f9c11270fdc7c20843e83469024d948cbe74f7f99f9b to 22d4e3c3d3eb46619453d69c66ca0930da99b29e7cb719cae08bf38ccb6902da
      Variable ERROR_QA value changed:
@@ -1,3 +1,2 @@
 dev-so debug-deps dev-deps debug-files arch pkgconfig la             perms dep-cmp pkgvarcheck perm-config perm-line perm-link             split-strip packages-list pkgv-undefined var-undefined             version-going-backwards expanded-d invalid-chars             license-checksum dev-elf file-rdeps configure-unsafe             configure-gettext perllocalpod shebang-size             already-stripped installed-vs-shipped ldflags compile-host-path             install-host-path pn-overrides unknown-configure-option             useless-rpaths rpaths staticdev empty-dirs             patch-fuzz             ${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', ' usrmerge', '', d)} patch-status
 DISTRO_FEATURES{usrmerge} = Unset
-_remove of packages-list

Task readline-native:do_configure couldn't be used from the cache because:
  We need hash bc5676bb88a6216f621882fdef5062de1f4ef61056a49f8d1ee0db95a938bd3a, most recent matching task was 0f3d4ea1ab2e107a43ba7421e08816741d9dcde302427b0a4b0ced16c128c701
  Hash for task dependency readline-native:do_patch changed from 8f0da205ba857e16ad591a76f95c6c7be49143b5d844918094a407d24799ca16 to ca990408ca2b662c7d9da6275e18aa9fbee3dbb5a1dac5494182378a06716c25
      basehash changed from 2e9189669cbd26df89666b6e9f69cc946cff7087326145b4e8f622a0929f263f to 3e16529ad29b7bd5370082b0082e9a846ddaac092b6228d876c677f1c7e8316c
      Variable ERROR_QA value chan
71c46fe24bb838d76b07fc0e6bd18ccb639dddc2170f0ba341f84e952f21cd3
                              Variable ERROR_QA value changed:
@@ -1,3 +1,2 @@
 dev-so debug-deps dev-deps debug-files arch pkgconfig la             perms dep-cmp pkgvarcheck perm-config perm-line perm-link             split-strip packages-list pkgv-undefined var-undefined             version-going-backwards expanded-d invalid-chars             license-checksum dev-elf file-rdeps configure-unsafe             configure-gettext perllocalpod shebang-size             already-stripped installed-vs-shipped ldflags compile-host-path             install-host-path pn-overrides unknown-configure-option             useless-rpaths rpaths staticdev empty-dirs             patch-fuzz             ${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', ' usrmerge', '', d)} patch-status
 DISTRO_FEATURES{usrmerge} = Unset
-_remove of packages-list

Task perlcross-native:do_configure couldn't be used from the cache because:
  We need hash 104f0df03d7025ee14b6d8eefb43efa96b808c20515767dfb11098a9f1431308, most recent matching task was 0a6eb9e851d7bd6afebf07c7147717b376b1b9829dad234113c5212a32fa0cbb
  Hash for task dependency perlcross-native:do_patch changed from c00f9d4be2cb2fa9a243eb9e7da210e9977c622baf2d0125b46cc8b27fbc4ecc to 8f7f22bad5602dc34e74df1acd31cb66cef7803aeff7e74ba2fb5e113c2546cb
      basehash changed from c06164bcd426cffaa0f5695ec0dd4bbb828cc664fe4c452e2d259bd9c2ae49d0 to 12a0738565e530095604080e4535d084aa642167065e4ff913e9f25c52391768
      Variable ERROR_QA value changed:
@@ -1,3 +1,2 @@
 dev-so debug-deps dev-deps debug-files arch pkgconfig la             perms dep-cmp pkgvarcheck perm-config perm-line perm-link             split-strip packages-list pkgv-undefined var-undefined             version-going-backwards expanded-d invalid-chars             license-checksum dev-elf file-rdeps configure-unsafe             configure-gettext perllocalpod shebang-size             already-stripped installed-vs-shipped ldflags compile-host-path             install-host-path pn-overrides unknown-configure-option             useless-rpaths rpaths staticdev empty-dirs             patch-fuzz             ${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', ' usrmerge', '', d)} patch-status
 DISTRO_FEATURES{usrmerge} = Unset
-_remove of version-going-backwards

Task zlib-native:do_configure couldn't be used from the cache because:
  We need hash b1c2545159fa3059512fafdb9dec3f9b495b37d32c4a7c7067d36bacb9695cc8, most recent matching task was 370dd44db29173bfcc937360e59ca24e107b2071ffd3aa1f25673185200299e2
  Hash for task dependency zlib-native:do_patch changed from 8f06c6e4aad9fe8b538bc706cab6dfe55bedf5eecbebf8e36f91fb577a983d50 to 6b97a1b7148a7af10d79e51936d0753daf0a11f00f44b1a8584eaaba208b3dde
      basehash changed from 08666a7f72d14327e54800cee5c1a937bb5fa756a4c80f4030d255a641bee918 to 12635de44e1e4c122428b45b165612eeb29345c2ae721708b175218280e34e36
      Variable ERROR_QA value changed:
@@ -1,3 +1,2 @@
 dev-so debug-deps dev-deps debug-files arch pkgconfig la             perms dep-cmp pkgvarcheck perm-config perm-line perm-link             split-strip packages-list pkgv-undefined var-undefined             version-going-backwards expanded-d invalid-chars             license-checksum dev-elf file-rdeps configure-unsafe             configure-gettext perllocalpod shebang-size             already-stripped installed-vs-shipped ldflags compile-host-path             install-host-path pn-overrides unknown-configure-option             useless-rpaths rpaths staticdev empty-dirs             patch-fuzz             ${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', ' usrmerge', '', d)} patch-status
 DISTRO_FEATURES{usrmerge} = Unset
-_remove of packages-list

Task attr-native:do_configure couldn't be used from the cache because:
  We need hash 6e0c9c80f8bd85ed3f764ba1404f02bb4cf7537b90ced791a9116424231afb72, most recent matching task was 60fdadeb2a6a58e1ac8fc408f396d8b8fa67c264a0fba4e477c2671c5d5deb38
  Hash for task dependency attr-native
ause:
  We need hash 94ecf7aa5f620a11e20f1c196a24fe732d02617358d53fe9819ccbb62b21b06c, most recent matching task was 1273bb9ebb9c61e86a28e02605631ec0d7834450a22101ee9e406d7f105141d1
  Hash for task dependency automake-native:do_patch changed from d4af84800e05769a8827418006b47cbb185b8071f3c9d3dcf40eb058f72bee9b to 4218facf6a5587608321545e8ef9b2fc0f0242e78f5df3f540be68323783d04f
      basehash changed from 8b754c828881371c229c04d5461c0779947bc331fbafb284ad8f619fec14f636 to 646b486eacaa8cc692b6df84945ca29424b5dbaad5af71b3f9895312e7ac3adc
      Variable ERROR_QA value changed:
@@ -1,3 +1,2 @@
 dev-so debug-deps dev-deps debug-files arch pkgconfig la             perms dep-cmp pkgvarcheck perm-config perm-line perm-link             split-strip packages-list pkgv-undefined var-undefined             version-going-backwards expanded-d invalid-chars             license-checksum dev-elf file-rdeps configure-unsafe             configure-gettext perllocalpod shebang-size             already-stripped installed-vs-shipped ldflags compile-host-path             install-host-path pn-overrides unknown-configure-option             useless-rpaths rpaths staticdev empty-dirs             patch-fuzz             ${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', ' usrmerge', '', d)} patch-status
 DISTRO_FEATURES{usrmerge} = Unset
-_remove of packages-list

Task libffi-native:do_configure couldn't be used from the cache because:
  We need hash 3dff6d02052e6c15ea58f7c5bb16908906f57c52afd0298f9f403455b894b3b9, most recent matching task was 479e6514c56e5caba9d079033fa9f8fca2923988e018cfe62e3476c1807ef068
  Hash for task dependency libffi-native:do_patch changed from 43b3ea9f929322e8f6d7468e09197d3677d921ad29870da5cf93f4b371c687cd to 5a553999c15003046b6f9b77e0bf3ce4de932a809cc92973032b5b35798c0f88
      basehash changed from f82550e8dd18c6ac7f43b790dce5ab4662c114b9929547adc1019a02b8e1dd4f to 28f8d1d03c5c77a92c0040d6fb2b8ddc3fe50616f80c40c05d49606710b5e6b0
      Variable ERROR_QA value changed:
@@ -1,3 +1,2 @@
 dev-so debug-deps dev-deps debug-files arch pkgconfig la             perms dep-cmp pkgvarcheck perm-config perm-line perm-link             split-strip packages-list pkgv-undefined var-undefined             version-going-backwards expanded-d invalid-chars             license-checksum dev-elf file-rdeps configure-unsafe             configure-gettext perllocalpod shebang-size             already-stripped installed-vs-shipped ldflags compile-host-path             install-host-path pn-overrides unknown-configure-option             useless-rpaths rpaths staticdev empty-dirs             patch-fuzz             ${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', ' usrmerge', '', d)} patch-status
 DISTRO_FEATURES{usrmerge} = Unset
-_remove of packages-list

Task libtirpc-native:do_configure couldn't be used from the cache because:
  We need hash fab0edbebdd084882c35cac3f879d4c45635eb77f080705dfcbb974e0f8306f4, most recent matching task was b8a37c9d5f846d69c37691565995e31202ce00583c7ebd6d7b6886f8e7291388
  Hash for task dependency libtirpc-native:do_patch changed from 3861750145c4d86ef2f46657b98dd45ab55e5ce58ade225256d6b037231decc9 to 96dae4f2173234f1be2950a6ca64030c6aae17781fb3374d74b3b9b0559ac664
      basehash changed from 0d4d2ff4a4890c6b23cd79c10fbdf8f3a074734fc70e35fe8154e1886f13274e to 75ef90114c32119657bab4a9308c4bdde699e29cb982021c10e40daa05dc85b6
      Variable ERROR_QA value changed:
@@ -1,3 +1,2 @@
 dev-so debug-deps dev-deps debug-files arch pkgconfig la             perms dep-cmp pkgvarcheck perm-config perm-line perm-link             split-strip packages-list pkgv-undefined var-undefined             version-going-backwards expanded-d invalid-chars             license-checksum dev-elf file-rdeps configure-unsafe             configure-gettext perllocalpod shebang-size             already-stripped installed-vs-shipped ldflags compile-host-path             install-host-path pn-overrides unknown-configure-option             useless-rpaths rpaths staticdev empty-dirs             patch-fuzz             
ed installed-vs-shipped ldflags compile-host-path             install-host-path pn-overrides unknown-configure-option             useless-rpaths rpaths staticdev empty-dirs             patch-fuzz             ${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', ' usrmerge', '', d)} patch-status
 DISTRO_FEATURES{usrmerge} = Unset
-_remove of packages-list

Task xz-native:do_configure couldn't be used from the cache because:
  We need hash f211b9b2324a3f5c9a94c3f36d6c8ffbc8b65df50a0b4bb4701b465beb56f80a, most recent matching task was f6e625f3bd9f597319c25e8350d0ecc095e9c885c9024ccbbab39f0384b60e01
  Hash for task dependency xz-native:do_patch changed from b810caadc9f84bf9e800ca30a0a3705ee3786b40aba854af819bd56c08b2b64c to 270fbbd60d0314139660719020be7df86e9003b8fd45dc6d400b9892fe12e598
      basehash changed from da78befedaa4f36420bfba595689761146c015ed481eb04161763485086ee2af to 071c46fe24bb838d76b07fc0e6bd18ccb639dddc2170f0ba341f84e952f21cd3
      Variable ERROR_QA value changed:
@@ -1,3 +1,2 @@
 dev-so debug-deps dev-deps debug-files arch pkgconfig la             perms dep-cmp pkgvarcheck perm-config perm-line perm-link             split-strip packages-list pkgv-undefined var-undefined             version-going-backwards expanded-d invalid-chars             license-checksum dev-elf file-rdeps configure-unsafe             configure-gettext perllocalpod shebang-size             already-stripped installed-vs-shipped ldflags compile-host-path             install-host-path pn-overrides unknown-configure-option             useless-rpaths rpaths staticdev empty-dirs             patch-fuzz             ${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', ' usrmerge', '', d)} patch-status
 DISTRO_FEATURES{usrmerge} = Unset
-_remove of packages-list

Task re2c-native:do_configure couldn't be used from the cache because:
  We need hash 86f0f629b65a1c16f52b61db260741502a24cb3f9a6599bbe67645f61d8ecf83, most recent matching task was c2c878351f0ea64cb9e63fe8211cabea6794f6fb28209d0041f3c0f7dddc0f3d
  Hash for task dependency re2c-native:do_patch changed from c5effdd3236c619756826eadb061390acad2c6feaea620835c5c5aaa73d2ddb8 to ae98092be6e6e04fef6f222ef5ba6b6eae0b47b669ba86d0603cbb59855bf812
      Hash for task dependency re2c-native:do_unpack changed from c621e7d591ba2682c5ce0e1cb30a713b79434a7a34b1c08ff1c36df1d3ffb09f to 65fbe52b95276a572d31fe67b85433f100a555e1ab3ff23cef03c4c0855500b0
          Hash for task dependency xz-native:do_populate_sysroot changed from e764559dd41b88e01eee48c33b36c90aea8930ceab6500904539b1b494a803c8 to 5c53f4cadbe8e5f2a23858836dd351c86ab0ab3e9fc452326bda1442a36e31b1
              Hash for task dependency xz-native:do_install changed from 2d2dd12bc988cf8ef3146a2ffe74d69e7c8b72dcc9e9d9dc256286472a246878 to 9d7ca2c40d670ff0b681268c6e8260e8a32a0d1e5c0879cd8185fceff256a1db
                  Hash for task dependency xz-native:do_compile changed from 0a9af4d2649e83137df728f142926a072e4bf1228821668d9fc6ef5b2d9a3594 to 31d9b2bcadda3808cc6b9c0508e3de8e5198a8b9593092987122e329204d30aa
                      Hash for task dependency xz-native:do_configure changed from f6e625f3bd9f597319c25e8350d0ecc095e9c885c9024ccbbab39f0384b60e01 to f211b9b2324a3f5c9a94c3f36d6c8ffbc8b65df50a0b4bb4701b465beb56f80a
                          Hash for task dependency xz-native:do_patch changed from b810caadc9f84bf9e800ca30a0a3705ee3786b40aba854af819bd56c08b2b64c to 270fbbd60d0314139660719020be7df86e9003b8fd45dc6d400b9892fe12e598
                              basehash changed from da78befedaa4f36420bfba595689761146c015ed481eb04161763485086ee2af to 071c46fe24bb838d76b07fc0e6bd18ccb639dddc2170f0ba341f84e952f21cd3
                              Variable ERROR_QA value changed:
@@ -1,3 +1,2 @@
 dev-so debug-deps dev-deps debug-files arch pkgconfig la             perms dep-cmp pkgvarcheck perm-config perm-line perm-link             split-strip packages-list pkgv-undefined var-undefined             version-going-backwards expanded-d invalid-chars             license-checksum dev-elf file-rdeps configure-unsafe
            configure-gettext perllocalpod shebang-size             already-stripped installed-vs-shipped ldflags compile-host-path             install-host-path pn-overrides unknown-configure-option             useless-rpaths rpaths staticdev empty-dirs             patch-fuzz             ${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', ' usrmerge', '', d)} patch-status
 DISTRO_FEATURES{usrmerge} = Unset
-_remove of packages-list

Task libunistring-native:do_configure couldn't be used from the cache because:
  We need hash bb6e322b56036c9fb9fb73de4eb798978fd7b7b57a8b9fc8206a6cc68c63d033, most recent matching task was d6b3aa0b1b4d1254c2897fdc0fbe9b36f8e4ae71a033af9f5780f6f53d8443bb
  Hash for task dependency libunistring-native:do_patch changed from 934f86f42f4b58946d165bc9ecebac3dfb48b717f79addf83eafe2cc974e0253 to 4b3c8a27e4ff01ffa8d2e6374d3ff08ff6da241a0dc57941ea0293159630e110
      basehash changed from d92c82fc710cbcb569e222aa9d0df8f4e0742c1b676a451e5cad446c69f6ec73 to 9d51a58bf08755590ca2eb7bce15379e665b651bf762fd96ccf40c148b3ac00d
      Variable ERROR_QA value changed:
@@ -1,3 +1,2 @@
 dev-so debug-deps dev-deps debug-files arch pkgconfig la             perms dep-cmp pkgvarcheck perm-config perm-line perm-link             split-strip packages-list pkgv-undefined var-undefined             version-going-backwards expanded-d invalid-chars             license-checksum dev-elf file-rdeps configure-unsafe             configure-gettext perllocalpod shebang-size             already-stripped installed-vs-shipped ldflags compile-host-path             install-host-path pn-overrides unknown-configure-option             useless-rpaths rpaths staticdev empty-dirs             patch-fuzz             ${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', ' usrmerge', '', d)} patch-status
 DISTRO_FEATURES{usrmerge} = Unset
-_remove of packages-list'


There has also been another error of the same kind here:

https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2473/steps/14/logs/stdio

NOTE: Reconnecting to bitbake server...
Loading cache...done.
Loaded 1917 entries from dependency cache.
Parsing recipes...done.
Parsing of 968 .bb files complete (967 cached, 1 parsed). 1917 targets, 66 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION           = "2.6.0"
BUILD_SYS            = "aarch64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "aarch64-poky-linux"
MACHINE              = "qemuarm64"
DISTRO               = "poky"
DISTRO_VERSION       = "4.3+snapshot-5ff688fe29bd644e28a226b655b3fdcce8a8ebb8"
TUNE_FEATURES        = "aarch64 armv8a crc cortexa57"
TARGET_FPU           = ""
meta                 
meta-poky            
meta-yocto-bsp       = "master:5ff688fe29bd644e28a226b655b3fdcce8a8ebb8"
meta-selftest        = "master:6a4088497af794ffebc01a1ba6eee23e7f5ae1f9"

Initialising tasks...done.
The differences between the current build and any cached tasks start at the following tasks:
virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-devtools/perl-cross/perlcross_1.5.bb:do_configure
/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy-native.bb:do_configure
virtual:native:/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-devtools/gnu-config/gnu-config_git.bb:do_configure
/home/pokybuild/yocto-worker/oe-selftest-armhost/build/meta/recipes-core/gettext/gettext-minimal-native_0.22.bb:do_configure
NOTE: Writing task signature files

Task perlcross-native:do_configure couldn't be used from the cache because:
  We need hash 39a7761cef3251a43c51c91a83930a29612974f0b7dd91135c681acee999472f, most recent matching task was 3d5d31a1c184b53c827a4ca5b7c1399460e76c2267b38faeb6432d595f1446d5
  Task dependencies changed from:
['AR', 'AS', 'B', 'BUILD_AR', 'BUILD_AS', 'BUILD_AS_ARCH', 'BUILD_CC', 'BUILD_CCLD', 'BUILD_CC_ARCH', 'BUILD_CFLAGS', 'BUILD_CPP', 'BUILD_CPPFLAGS', 'BUILD_CXX', 'BUILD_CXXFLAGS', 'BUILD_FC', 'BUILD_LD', 'BUILD_LDFLAGS', 'BUILD_LD_ARCH', 'BUILD_NM', 'BUILD_OPTIMIZATION', 'BUILD_PREFIX', 'BUILD_RANLIB', 'BUILD_STRIP', 'CC', 'CCACHE_BASEDIR', 'CCACHE_COMPILERCHECK', 'CCACHE_CONFIGPATH', 'CCACHE_TEMPDIR', 'CCLD', 'CFLAGS', 'CONFIG_SITE', 'CPP', 'CPPFLAGS', 'CXX', 'CXXFLAGS', 'DEBUG_BUILD', 'DEPENDS', 'DISABLE_STATIC', 'ERROR_QA', 'EXTRA_NATIVE_PKGCONFIG_PATH', 'EXTRA_OECONF', 'FC', 'HOST_PREFIX', 'INSANE_SKIP', 'LC_ALL', 'LD', 'LDFLAGS', 'MAKE', 'MLPREFIX', 'NATIVE_PACKAGE_PATH_SUFFIX', 'NM', 'OBJCOPY', 'OBJDUMP', 'P', 'PACKAGECONFIG', 'PERL_HASH_SEED', 'PKG_CONFIG_DIR', 'PKG_CONFIG_DISABLE_UNINSTALLED', 'PKG_CONFIG_LIBDIR', 'PKG_CONFIG_PATH', 'PKG_CONFIG_SYSROOT_DIR', 'PN', 'PSEUDO_DISABLED', 'PV', 'PYTHONHASHSEED', 'QA_ERRORS_FOUND', 'QA_LOGFILE', 'RANLIB', 'READELF', 'RECIPE_SYSROOT_NATIVE', 'S', 'STAGING_BASE_LIBDIR_NATIVE', 'STAGING_BINDIR_NATIVE', 'STAGING_DATADIR', 'STAGING_DATADIR_NATIVE', 'STAGING_DIR_NATIVE', 'STAGING_ETCDIR_NATIVE', 'STAGING_INCDIR_NATIVE', 'STAGING_LIBDIR_NATIVE', 'STAGING_SBINDIR_NATIVE', 'STRINGS', 'STRIP', 'TZ', 'UNKNOWN_CONFIGURE_OPT_IGNORE', 'base_bindir', 'base_libdir', 'base_libdir_native', 'base_prefix', 'base_sbindir', 'bindir', 'bindir_native', 'datadir', 'datadir_native', 'do_qa_configure', 'docdir', 'exec_prefix', 'includedir', 'includedir_native', 'infodir', 'libdir', 'libdir_native', 'libexecdir', 'localstatedir', 'lt_cv_sys_lib_dlsearch_path_spec', 'mandir', 'nonarch_base_libdir', 'nonarch_libdir', 'oe.qa.exit_if_errors', 'oe.qa.exit_with_message_if_errors', 'oe.qa.handle_error', 'oe.qa.write_error', 'oe.types.boolean', 'oe.utils.vartrue', 'prefix', 'prefix_native', 'root_prefix', 'sbindir', 'sbindir_n

 d)} patch-status
 DISTRO_FEATURES{usrmerge} = Unset
-_remove of packages-list

Task gnu-config-native:do_configure couldn't be used from the cache because:
  We need hash fe9ccd7132f7e8aa33587f807b210792da2c8ab7c363472f459629502ec71c9d, most recent matching task was 492e2b0ac161069f340036aad9b35c6a17d33d26a5b0a9cb37105210f0a78c9f
  Hash for task dependency gnu-config-native:do_patch changed from 116e7c2189ff17af9b84ffcbdf2d84fbf1e27ceb9d0458b4093d75ee71c9b0e6 to 6fd64d81a794dc3b634631208cb7f48435cad4ca19718658d3df6222b8e0ee9a
      basehash changed from d30c1e4c8a67155b18b9d4ecbcce251fee20b5b9af0aa5ea0c33a9e342e73179 to 36d0617c68a33050bb71c8be346f7b9cb4a9239145d3722917a1bd4f67593473
      Variable ERROR_QA value changed:
@@ -1,3 +1,2 @@
 dev-so debug-deps dev-deps debug-files arch pkgconfig la             perms dep-cmp pkgvarcheck perm-config perm-line perm-link             split-strip packages-list pkgv-undefined var-undefined             version-going-backwards expanded-d invalid-chars             license-checksum dev-elf file-rdeps configure-unsafe             configure-gettext perllocalpod shebang-size             already-stripped installed-vs-shipped ldflags compile-host-path             install-host-path pn-overrides unknown-configure-option             useless-rpaths rpaths staticdev empty-dirs             patch-fuzz             ${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', ' usrmerge', '', d)} patch-status
 DISTRO_FEATURES{usrmerge} = Unset
-_remove of packages-list

Task gettext-minimal-native:do_configure couldn't be used from the cache because:
  We need hash d0eb1315f496261293f8bff065118acb2d6ca3c5a85debba0b428a7665216bdd, most recent matching task was 2cfb66c3f0f5c55d141c057f7d711a6967d8073d378f3f65979100c93eebbfc5
  Hash for task dependency gettext-minimal-native:do_patch changed from f198fb256cbdb26c59b36c14592b03bc1f2404b334709feb9599bdaaa7cbac60 to e39e49d97c891245e141bc37012da820c76319e6f3f8ffa040a9a6b691a7ae0a
      basehash changed from 19ea81645b7e8bfc809584bf4e66390cdba1809d3868d6b088fe823f15af5b78 to 3ba1ad58d4c916569b0bf5014b03b5e23c6ca6be18f16573ed7c65ded0a2e1e0
      Variable ERROR_QA value changed:
@@ -1,3 +1,2 @@
 dev-so debug-deps dev-deps debug-files arch pkgconfig la             perms dep-cmp pkgvarcheck perm-config perm-line perm-link             split-strip packages-list pkgv-undefined var-undefined             version-going-backwards expanded-d invalid-chars             license-checksum dev-elf file-rdeps configure-unsafe             configure-gettext perllocalpod shebang-size             already-stripped installed-vs-shipped ldflags compile-host-path             install-host-path pn-overrides unknown-configure-option             useless-rpaths rpaths staticdev empty-dirs             patch-fuzz             ${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', ' usrmerge', '', d)} patch-status
 DISTRO_FEATURES{usrmerge} = Unset
-_remove of packages-list
Comment 3 Alexander Kanavin 2023-11-24 08:08:53 UTC
So the test is comparing with signatures from an unrelated selftest which tweaks ERROR_QA.

It seems that part of the chain of siginfo files is missing leading it to find earlier differences. This is strange, because the test takes great care to ensure all siginfo files are present, invoking bitbake with --runall.

One of the failing tests build dir have been stashed on ubuntu1804-arm-1 worker, but I'm not sure what to look for in there. Ideas?
Comment 4 Alexander Kanavin 2023-11-24 08:31:06 UTC
Here's another one:
https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2481/steps/15/logs/stdio

The fail is identical to the second fail above (four base native recipes in do_configure).

It's peculiar that so far it's been observed only on arm. Another cross-builder issue?
Comment 5 Alexander Kanavin 2023-11-25 21:34:18 UTC
Here's a snippet from another recent fail:

Task gnu-config-native:do_configure couldn\'t be used from the cache because:
We need hash fe9ccd7132f7e8aa33587f807b210792da2c8ab7c363472f459629502ec71c9d, most recent matching task was 492e2b0ac161069f340036aad9b35c6a17d33d26a5b0a9cb37105210f0a78c9f

I went and looked up these hashes in the AB cache:

/srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/49/2e/sstate:gnu-config-native:aarch64-linux:20230216+git:r0:aarch64:11:492e2b0ac161069f340036aad9b35c6a17d33d26a5b0a9cb37105210f0a78c9f_configure.tar.zst.siginfo

/srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/fe/9c/sstate:gnu-config-native:x86_64-linux:20230216+git:r0:x86_64:11:fe9ccd7132f7e8aa33587f807b210792da2c8ab7c363472f459629502ec71c9d_configure.tar.zst.siginfo

So the needed siginfo file does exist in the cache, but only for the x86 host.

Why isn't the test creating the arm item? I guess we need to add extra diagnostics to the step in the test where that is supposed to be happening, so that we can see what is actually being written into the cache:

        bitbake("--runall build --runall deploy_source_date_epoch {}".format(target))
Comment 6 Alexander Kanavin 2023-11-27 18:22:25 UTC
After poking at this a bit more, this seems to be about absent siginfo files in sstate, when they should not be absent. 

So either they haven't been created to begin with, or they've been deleted by an over-zealous cleanup script, or their timestamps aren't updated properly to avoid deletion. I even found a few instances where siginfo is absent, but corresponding sstate archive is still present.

Which brings the next question: where is that script, what (or who) runs it, and are there logs? I'd like to check for some absent items from the logs, whether they've been created to begin with, and when and how they were deleted. This would rule out some possibilities
Comment 7 Alexander Kanavin 2023-11-28 07:48:24 UTC
Michael, can you point me to sstate management code (and logs if they exist) please?
Comment 8 Richard Purdie 2023-11-28 09:48:19 UTC
The "script" amounts to a find command for files unused for more than ~15 days which deletes them if I recall correctly. The command is run by something like cron directly on the BSD based NAS for efficiency since walking the NFS mount is much slower. As such, there isn't a big availability of commands, the NAS is a minimal environment. I think Michael can access deletion logs but an idea of the pattern file you're looking for may be helpful as the logs are huge and stored offline.
Comment 9 Alexander Kanavin 2023-11-28 09:56:44 UTC
*fe9ccd7132f7e8aa33587f807b210792da2c8ab7c363472f459629502ec71c9d* would be interesting.
Comment 12 Alexandre Belloni 2023-11-29 23:14:18 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2466/steps/14/logs/stdio
oe-selftest-armhost ubuntu2004-arm-1
Comment 14 Michael Halstead 2023-11-30 00:28:34 UTC
I restored the deletion logs going back to June 2023 and didn't find anything matching fe9ccd7132f7e8aa33587f807b210792da2c8ab7c363472f459629502ec71c9d.

Logs are available at http://autobuilder.yocto.io/pub/sstate-deletion-logs/


I pushed the scripts that clean up to https://git.yoctoproject.org/yocto-autobuilder-helper/tree/scripts/sstate-clean.sh and https://git.yoctoproject.org/yocto-autobuilder-helper/tree/scripts/sstate-clean.py.

Please let me know if more information is needed.
Comment 15 Alexander Kanavin 2023-11-30 10:49:39 UTC
Thanks Michael, I also did a couple of checks in this logs, and I think we can rule out erroneous sstate deletions.
Comment 16 Alexander Kanavin 2023-11-30 15:37:47 UTC
I've gotten slightly further with this. The failure has been reproduced here:

(buildbot-venv) pokybuild@ubuntu2204-arm-1:~/akanavin-test/poky/build-st$ bitbake -DDD -S printdiff perlcross-native 

The relevant output follows, I will comment on it in a separate comment:

DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/16/2b/sstate:perlcross-native:aarch64-linux:1.5:r0:aarch64:11:162b184a07a2e5b747dda20f9bae6157c203b2db4a604915aeb38f03f2c4a0bc_collect_spdx_deps.tar.zst.siginfo
DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/32/ee/sstate:perlcross-native::1.5:r0::11:32eefab6b4a5176ba175a1d81bfa05d47856db8c8630f4247091ea26cf8f0c72_populate_lic.tar.zst.siginfo
DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/af/8e/sstate:perlcross-native::1.5:r0::11:af8e7a357227c742961a2c6885162c469d2c23ce3e57f87ed1ec066451c2cc75_patch.tar.zst.siginfo
DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/34/89/sstate:quilt-native::0.67:r0::11:348958020048228e9ea49bcb672ceaa0f258cbe2b978b461ed342d7160591d0d_patch.tar.zst.siginfo
DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/a7/83/sstate:perlcross-native:aarch64-linux:1.5:r0:aarch64:11:a78370ba49a43af61a41ca6573390e50c73019e3e032bf23c58780a1cf38a763_create_spdx.tar.zst.siginfo
DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/09/32/sstate:quilt-native::0.67:r0::11:0932efe8f151d16c33906875b1ef6eb07fe53565fffd912c007f0960cc39f4fb_fetch.tar.zst.siginfo
DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/58/3d/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:583daa7235b948d1d26e3ef84f80617bafbf744e366a1dd5ab604911438ce316_install.tar.zst.siginfo
DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/0f/23/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:0f23c20f13d05794c0d799f2710d028442ab41da2638464f40f67a33288a3ed2_configure.tar.zst.siginfo
DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/07/fe/sstate:perlcross-native:aarch64-linux:1.5:r0:aarch64:11:07fe0a996fa0908e711eb8b6599b6148ebb94f8a6fde3692ce8341f93eaed21a_populate_sysroot.tar.zst.siginfo
DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/11/6f/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:116f8d4762cd111492fffed036d13d4bcca35e9bafc743e5fce5432ff64b2201_deploy_source_date_epoch.tar.zst.siginfo
DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/47/61/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:4761a8d1c5192678b297b66a8b1f9d677c67e166b6cb7bd325f9f1dd841a7350_recipe_qa.tar.zst.siginfo
DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/7f/de/sstate:perlcross-native:aarch64-linux:1.5:r0:aarch64:11:7fdea467f0350e1e76cab40ac87dd1262f625fec9b3fb24ccd424d64439606f1_deploy_source_date_epoch.tar.zst.siginfo
DEBUG: SState: Looked for but didn't find file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/87/9d/sstate:perlcross-native:aarch64-linux:1.5:r0:aarch64:11:879d474b6d396285f3ffeb621efe34ae409cf36659de7f509e05f4fbdf1ac40b_compile.tar.zst.siginfo
DEBUG: SState: Looked for but didn't find file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/39/a7/sstate:perlcross-native:aarch64-linux:1.5:r0:aarch64:11:39a7761cef3251a43c51c91a83930a29612974f0b7dd91135c681acee999472f_configure.tar.zst.siginfo
DEBUG: SState: Looked for but didn't find file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/95/b4/sstate:perlcross-native:aarch64-linux:1.5:r0:aarch64:11:95b41b9c95faf601e4a64e2df2550d7daead6d669877d5ec6241145926d5ffe9_install.tar.zst.siginfo
DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/66/5d/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:665d8b47bbb50e88d56d3831b09c5dd14a224db7ad8f957d9c206d8617d74ad0_prepare_recipe_sysroot.tar.zst.siginfo
DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/f0/c8/sstate:perlcross-native:aarch64-linux:1.5:r0:aarch64:11:f0c8f2e61d7e5caf7c7b989d1bb019591e9383ed62c648913f83ce14bddb1de2_prepare_recipe_sysroot.tar.zst.siginfo
DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/83/e8/sstate:quilt-native::0.67:r0::11:83e8cd1ab086bcc015657f1b7f933d974dd72ce277a9ff8028eb4b87e5d1f5d1_unpack.tar.zst.siginfo
DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/4b/50/sstate:perlcross-native:aarch64-linux:1.5:r0:aarch64:11:4b50bf02af2854a6807f0394b7b6ddba6eff35e700f596b16ecd87353591e8ed_recipe_qa.tar.zst.siginfo
DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/f2/57/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:f25773f9c2ae02d1de6c037b6f4161c79b3ba37ce5add86cf087158ba26c52d9_populate_sysroot.tar.zst.siginfo
DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/86/61/sstate:quilt-native:aarch64-linux:0.67:r0:aarch64:11:866142e7b9a3676d76f8abe0900de3170c4e163314d483115585d29d3c2f389c_compile.tar.zst.siginfo
DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/07/08/sstate:perlcross-native:aarch64-linux:1.5:r0:aarch64:11:070895b1acd74f90df600a291f1fc57835eeef9d808b11743abc9e76babf6b05_create_runtime_spdx.tar.zst.siginfo
DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/67/10/sstate:perlcross-native::1.5:r0::11:67103e808d7652d7efd94fd9571be4be709de7667a296095ef336ce02ca3c597_unpack.tar.zst.siginfo
DEBUG: SState: Found valid sstate file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/40/dc/sstate:perlcross-native::1.5:r0::11:40dcf7a07efd1ae173287ad3ce9c2267199788b32d1de25c638b090792e8f068_fetch.tar.zst.siginfo
The differences between the current build and any cached tasks start at the following tasks:
virtual:native:/home/pokybuild/akanavin-test/poky/meta/recipes-devtools/perl-cross/perlcross_1.5.bb:do_configure
NOTE: Writing task signature files

Task perlcross-native:do_configure couldn't be used from the cache because:
  We need hash 39a7761cef3251a43c51c91a83930a29612974f0b7dd91135c681acee999472f, most recent matching task was 0371dad648fd4b7a302c615201550e36d72c26762b58fa341544facd5b333893
  Hash for task dependency perlcross-native:do_deploy_source_date_epoch changed from 32d6c3f886ffa23f02ced8386ecf64362b8d39aac1e9504082b2bd0b6191e73a to 7fdea467f0350e1e76cab40ac87dd1262f625fec9b3fb24ccd424d64439606f1
      Hash for task dependency perlcross-native:do_patch changed from 19f6da1f965d9fa02d8cf58455955f5bec56caf5bfd6f715848cbb14f78f3a55 to af8e7a357227c742961a2c6885162c469d2c23ce3e57f87ed1ec066451c2cc75
          Hash for task dependency perlcross-native:do_unpack changed from dd10eee95078eceb20fd649f2ad2467c34d788dac3fb37d868740541a29a8f01 to 67103e808d7652d7efd94fd9571be4be709de7667a296095ef336ce02ca3c597
              Hash for task dependency perlcross-native:do_fetch changed from ab71d583ae6e87080e54604da78aa71f4888fd357642a379aa0056cc4bbabc80 to 40dcf7a07efd1ae173287ad3ce9c2267199788b32d1de25c638b090792e8f068
                  runtaskdeps changed:
                  perlcross-native:do_recipe_qa with hash 7fb85cf7c85c19871644d6f4ce796d33710dc53f18ebd0fd837bafc4e55919a2
 changed to
perlcross-native:do_recipe_qa with hash 4b50bf02af2854a6807f0394b7b6ddba6eff35e700f596b16ecd87353591e8ed
                  Hash for task dependency perlcross-native:do_recipe_qa changed from 7fb85cf7c85c19871644d6f4ce796d33710dc53f18ebd0fd837bafc4e55919a2 to 4b50bf02af2854a6807f0394b7b6ddba6eff35e700f596b16ecd87353591e8ed
Comment 17 Alexander Kanavin 2023-11-30 15:48:28 UTC
First, note the three signatures of perlcross-native that do not exist:

DEBUG: SState: Looked for but didn't find file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/87/9d/sstate:perlcross-native:aarch64-linux:1.5:r0:aarch64:11:879d474b6d396285f3ffeb621efe34ae409cf36659de7f509e05f4fbdf1ac40b_compile.tar.zst.siginfo
DEBUG: SState: Looked for but didn't find file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/39/a7/sstate:perlcross-native:aarch64-linux:1.5:r0:aarch64:11:39a7761cef3251a43c51c91a83930a29612974f0b7dd91135c681acee999472f_configure.tar.zst.siginfo
DEBUG: SState: Looked for but didn't find file /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/95/b4/sstate:perlcross-native:aarch64-linux:1.5:r0:aarch64:11:95b41b9c95faf601e4a64e2df2550d7daead6d669877d5ec6241145926d5ffe9_install.tar.zst.siginfo


I have checked from the sstate deletion logs provided by Michael, and none of the three are there. Which means that they have never been created to begin with. So how was the (existing) populate_sysroot task fulfilled?

buildbot-venv) pokybuild@ubuntu2204-arm-1:~/akanavin-test/poky/build-st$ bitbake-dumpsig /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/07/fe/sstate:perlcross-native:aarch64-linux:1.5:r0:aarch64:11:07fe0a996fa0908e711eb8b6599b6148ebb94f8a6fde3692ce8341f93eaed21a_populate_sysroot.tar.zst.siginfo

...

Hash for dependent task perlcross-native:do_install is d569f99eeb167bd4a1ce91ad8c636311c79c506c2bc008b2c24f267740f06e30
Computed base hash is aa90daf400307b73d2130794fba4a9ae0c2f8c890c275537886b664abfaa9d06 and from file aa90daf400307b73d2130794fba4a9ae0c2f8c890c275537886b664abfaa9d06
Computed task hash is 8bfcf3a0c3523e290f6e883e9abedbc2db07df80b1cfe4a0cdfadf75221f90c3


Note that both the task hash and the do_install hash used to compute it do not match the output from printdiff. Also, the do_install hash printed by above dimpsing command does exist in sstate.

What this means (to me), is that the hash equivalency server had provided a shortcut to this object and the intermediate tasks never ran.
Comment 18 Richard Purdie 2023-11-30 15:54:36 UTC
I guess what could happen is that the x86 build ran and provided a hash equivalence link back to an existing sstate object. I don't understand how the intermediate siginfo hashes wouldn't be there if the end result was generated though.
Comment 19 Alexander Kanavin 2023-11-30 16:15:42 UTC
The siginfo files for do_install referred to by populate_sysroot are present for both arm and x86:

$ ls /srv/autobuilder/autobuilder.yocto.io/pub/sstate/universal/d5/69/
...
sstate:perlcross-native:aarch64-linux:1.5:r0:aarch64:11:d569f99eeb167bd4a1ce91ad8c636311c79c506c2bc008b2c24f267740f06e30_install.tar.zst.siginfo
sstate:perlcross-native:x86_64-linux:1.5:r0:x86_64:11:d569f99eeb167bd4a1ce91ad8c636311c79c506c2bc008b2c24f267740f06e30_install.tar.zst.siginfo


What I think had happened is that, first, quilt-native was built with input hash X and output A, and perlcross-native was build on top of that with input hash Y + A and output B.

Then quilt-native was rebuilt with input hash Z and same output A, thus perlcross-native would still have input hash Y + A, and so there was no need to rebuild it, configure/compile/install were skipped and B was taken straight from the cache.
Comment 20 Alexander Kanavin 2023-11-30 16:23:25 UTC
To set the record straight on what was deemed equivalent to what, we could ask a few questions from the hash equivalence server? Joshua, how can this be done from command line?
Comment 21 Alexander Kanavin 2023-11-30 17:58:09 UTC
Joshua pointed out that quilt-native isn't included in signature calculations, and so isn't a factor.

I've retrieved these records for the hashes that perlcross-native populate_sysroot signature file refers to:

(buildbot-venv) pokybuild@ubuntu2204-arm-1:~/akanavin-test/poky/build-st$ bitbake-hashclient --address hashserv.yocto.io:8686 get oe.sstatesig.OEOuthashBasic 07fe0a996fa0908e711eb8b6599b6148ebb94f8a6fde3692ce8341f93eaed21a
{
    "PN": null,
    "PR": null,
    "PV": null,
    "created": "2023-09-05 19:05:24.101343",
    "id": 88892422,
    "method": "oe.sstatesig.OEOuthashBasic",
    "outhash": "2a04361fa2e36e1be1449054131f9613cc52a083d14dc1ebfe5b2f42951780ef",
    "outhash_siginfo": null,
    "owner": null,
    "task": null,
    "taskhash": "07fe0a996fa0908e711eb8b6599b6148ebb94f8a6fde3692ce8341f93eaed21a",
    "unihash": "07fe0a996fa0908e711eb8b6599b6148ebb94f8a6fde3692ce8341f93eaed21a"
}

(buildbot-venv) pokybuild@ubuntu2204-arm-1:~/akanavin-test/poky/build-st$ bitbake-hashclient --address hashserv.yocto.io:8686 get oe.sstatesig.OEOuthashBasic 8bfcf3a0c3523e290f6e883e9abedbc2db07df80b1cfe4a0cdfadf75221f90c3
{
    "PN": null,
    "PR": null,
    "PV": null,
    "created": "2023-11-06 19:01:00.096764",
    "id": 105987851,
    "method": "oe.sstatesig.OEOuthashBasic",
    "outhash": "2a04361fa2e36e1be1449054131f9613cc52a083d14dc1ebfe5b2f42951780ef",
    "outhash_siginfo": null,
    "owner": null,
    "task": null,
    "taskhash": "8bfcf3a0c3523e290f6e883e9abedbc2db07df80b1cfe4a0cdfadf75221f90c3",
    "unihash": "07fe0a996fa0908e711eb8b6599b6148ebb94f8a6fde3692ce8341f93eaed21a"
}

8bfcf3a0c3523e290f6e883e9abedbc2db07df80b1cfe4a0cdfadf75221f90c3 is not in the cache and was never deleted.

Ideas? Can you try to interpret this, because I'm not sure how to read it?
Comment 22 Joshua Watt 2023-11-30 18:36:54 UTC
8bfcf3a0c3523e290f6e883e9abedbc2db07df80b1cfe4a0cdfadf75221f90c3 won't exist in sstate, on purpose. 

What happens is that 07fe0a996fa0908e711eb8b6599b6148ebb94f8a6fde3692ce8341f93eaed21a is the first taskhash reported with output hash 2a04361fa2e36e1be1449054131f9613cc52a083d14dc1ebfe5b2f42951780ef.

When it comes time to build 8bfcf3a0c3523e290f6e883e9abedbc2db07df80b1cfe4a0cdfadf75221f90c3, the first query to the server returns the same value for the unihash, because the hash server isn't aware of this particular hash. However, after the task runs it's seen that the output hash is 2a04361fa2e36e1be1449054131f9613cc52a083d14dc1ebfe5b2f42951780ef, so the server reports back that the unihash should be change to 07fe0a996fa0908e711eb8b6599b6148ebb94f8a6fde3692ce8341f93eaed21a, and the sstate file thus will have that in it's file name instead. Thus 8bfcf3a0c3523e290f6e883e9abedbc2db07df80b1cfe4a0cdfadf75221f90c3 won't exist in sstate.
Comment 23 Alexander Kanavin 2023-11-30 21:25:33 UTC
After poking at this a bit more the best theory I have is this was triggered by changes to recipe_qa, perhaps this commit:

commit d90fb73db5b07a1f407d59e698c5114d342e2322
Author: Richard Purdie <richard.purdie@linuxfoundation.org>
Date:   Mon Nov 6 16:31:10 2023 +0000

    insane: Move unpack tests to do_recipe_qa


This changes the recipe_qa task hash, but doesn't change its sstate output (because recipe_qa writes nothing to sstate). So all recipe_qa hashes are mapped to the same unihash:

(buildbot-venv) pokybuild@ubuntu2204-arm-1:~/akanavin-test/poky/build-st$ bitbake-hashclient --address hashserv.yocto.io:8686 get oe.sstatesig.OEOuthashBasic 4b50bf02af2854a6807f0394b7b6ddba6eff35e700f596b16ecd87353591e8ed
{
    "PN": null,
    "PR": null,
    "PV": null,
    "created": "2023-08-31 12:40:29.314344",
    "id": 87224043,
    "method": "oe.sstatesig.OEOuthashBasic",
    "outhash": "4a6aa7f1049dba3192bbf82da9c1166e66c1d11f21313c3906be38220fa8a4d4",
    "outhash_siginfo": null,
    "owner": null,
    "task": null,
    "taskhash": "4b50bf02af2854a6807f0394b7b6ddba6eff35e700f596b16ecd87353591e8ed",
    "unihash": "4b50bf02af2854a6807f0394b7b6ddba6eff35e700f596b16ecd87353591e8ed"
}


(buildbot-venv) pokybuild@ubuntu2204-arm-1:~/akanavin-test/poky/build-st$ bitbake-hashclient --address hashserv.yocto.io:8686 get oe.sstatesig.OEOuthashBasic 7fb85cf7c85c19871644d6f4ce796d33710dc53f18ebd0fd837bafc4e55919a2
{
    "PN": null,
    "PR": null,
    "PV": null,
    "created": "2023-11-06 18:58:58.720689",
    "id": 105987334,
    "method": "oe.sstatesig.OEOuthashBasic",
    "outhash": "4a6aa7f1049dba3192bbf82da9c1166e66c1d11f21313c3906be38220fa8a4d4",
    "outhash_siginfo": null,
    "owner": null,
    "task": null,
    "taskhash": "7fb85cf7c85c19871644d6f4ce796d33710dc53f18ebd0fd837bafc4e55919a2",
    "unihash": "4b50bf02af2854a6807f0394b7b6ddba6eff35e700f596b16ecd87353591e8ed"
}

And then hash equivalency provided a shortcut straight to setscene populate_sysroot, which means configure/compile/install never happened. What I still don't understand is the timings: troubles started some two weeks later than this commit.

I'd like to get to an isolated reproducer for this, because even if this is fixed in printdiff code, it would serve as a test case.
Comment 24 Alexandre Belloni 2023-11-30 23:57:43 UTC
This would actually check out, because this failure happened on the 6th:

https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/2379/steps/15/logs/stdio
Comment 25 Joshua Watt 2023-12-01 15:09:33 UTC
Ok, this kinda make sense then. I guess the question is, should do_recipe_qa even be subject to hash equivalence? More generally should any task that produces no sstate output be subject to hash equivalence?
Comment 26 Alexander Kanavin 2023-12-01 15:17:42 UTC
Sadly I have to take the recipe_qa idea back. The commit that changed do_recipe_qa also changed do_unpack, so it would trigger a rebuild.

I'm going to look at cross-host builds, it's a bit of a mystery to me how sstate with an arch-independent native tool like perlcross-native manages to jump from one build host (x86) to another (arm) but RP says it happens, and that's where the trouble could originate.
Comment 27 Joshua Watt 2023-12-01 15:22:44 UTC
Actually, I just noticed that do_image_qa doesn't have SSTATE_SKIP_CREATION set like do_image_qa. Skipping the creation of the (empty) sstate object will also skip hash equivalence, so I _think_ this might be the fix for this problem?

https://lists.openembedded.org/g/openembedded-core/message/191671
Comment 28 Alexander Kanavin 2023-12-01 16:32:51 UTC
So regardless of whether we get to the bottom of this particular chain of events, printdiff needs to look into siginfo files and follow the references contained in them (backwards towards the more elemental tasks), instead of just trying to look up 'straight' task hashes. That way it would follow hashequiv indirections properly, hopefully.
Comment 29 Alexander Kanavin 2023-12-04 10:31:07 UTC
After thinking about this over the weekend, I realized something:

Why is printdiff complaining about some intermediate missing signatures (configure/compile/install), if the final object (e.g. populate_sysroot) is present and available? 

Running the actual build is not going to care that those signature are missing, it will shortcut directly to populate_sysroot. And so printdiff should follow the same logic: don't traverse the whole dependency tree and report everything missing; if a given sstate object can be fulfilled, then stop there.

Maybe it doesn't matter how we got into that situation; with hash equivalency it's bound to happen again.
Comment 31 Richard Purdie 2023-12-04 23:04:22 UTC
Imagine we're building perlcross-native on two different builders, one is x86_64 (X) and one is aarch64 (A).

Note that the only dependency of perlcross-native is quilt-native.

Also keep in mind that native tasks have the same taskhash regardless of which architecture they're run on so that the target data can be swapped between different native architectures. The difference is only in the sstate filename which does encode that.

On A, we build perlcross-native and have:

quilt-native builds, then,
perlcross-native:do_compile with hash H1
perlcross-native:do_install with hash H2
perlcross-native:do_populate_sysroot with hash H3

In parallel with this, on X, we also build perlcross-native but quilt-native builds first. The quilt-native build finds an equivalence for quilt-native:do_populate_sysroot. For whatever reason, no equivalence was found for the A build. Note that native builds are not "reproducible" in binary form as we relocate them to avoid host paths.

Imagine that the hash for perlcross-native:do_populate_sysroot is reported from X before the one from A but the outhash is the same (quite likely if you look at the perlcross recipe).

This means the tasks in the graph on X rehash using the new hash but the build on A does not. This means the sstate object is written out with the new hash for both X and A.

If you then run the same build on A, quilt-native will rebuild since it is missing with the new hash. The intermediate tasks with H1 and H2 are now orphaned but nothing will actually rebuild them since the right sstate objects are written/present.

I think that explains why this happens. I think Alexander may have an approach on how we can try and fix things.
Comment 32 Alexander Kanavin 2023-12-06 09:27:36 UTC
Patches:
https://git.yoctoproject.org/poky-contrib/log/?h=akanavin/fix-printdiff
(I didn't write proper commit messages yet)

Just started a-full with them:
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/6294
Comment 33 Alexander Kanavin 2023-12-07 19:57:58 UTC
The patchset has been tested on the ab, I just would like to confirm that it doesn't regress selftest times, and then I'll submit to mailing list.
Comment 35 Randy MacLeod 2024-01-04 15:38:10 UTC
Bump to M2, patches in master-next.
Comment 37 Yoann Congal 2026-07-06 15:50:14 UTC
oe-selftest-fedora alma8-vk-2 scarthgap completed at 2026-07-06 15:13:28+00:00
https://autobuilder.yoctoproject.org/valkyrie/?#/builders/48/builds/4057
Comment 38 Yoann Congal 2026-07-08 17:59:36 UTC
oe-selftest-debian debian11-vk-1 scarthgap completed at 2026-07-08 17:44:24+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/4250

I have the patches for this bug merged on scarthgap:
> https://patchwork.yoctoproject.org/project/oe-core/list/?series=20066
1-3/6 merged (like master)

> https://git.yoctoproject.org/poky/commit/?id=6e6bb93fda590ddb9e27eb8886a6a67645e35c82
merged like master
Comment 39 Yoann Congal 2026-07-08 18:01:34 UTC
Reopening
Comment 40 Yoann Congal 2026-07-08 18:07:01 UTC
oe-selftest-fedora rocky8-vk-1 scarthgap completed at 2026-07-08 18:02:55+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/4075

From the same a-full run as Comment 37
Comment 41 Yoann Congal 2026-07-08 22:08:21 UTC
Rebuilds of above failure also failed:
oe-selftest-fedora alma8-vk-1 scarthgap completed at 2026-07-08 19:41:50+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/4077

oe-selftest-debian debian12-vk-2 scarthgap completed at 2026-07-08 19:33:11+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/4252

But my local build succeded:

 $ oe-selftest -vr sstatetests.SStatePrintdiff.test_gcc_runtime_vs_gcc_source


2026-07-08 23:37:48,228 - oe-selftest - INFO - test_gcc_runtime_vs_gcc_source (sstatetests.SStatePrintdiff.test_gcc_runtime_vs_gcc_source)
2026-07-09 00:00:24,251 - oe-selftest - INFO -  ... ok
2026-07-09 00:00:34,253 - oe-selftest - INFO - ----------------------------------------------------------------------
2026-07-09 00:00:34,254 - oe-selftest - INFO - Ran 1 test in 1366.160s
2026-07-09 00:00:34,254 - oe-selftest - INFO - OK
2026-07-09 00:00:36,632 - oe-selftest - INFO - RESULTS:
2026-07-09 00:00:36,632 - oe-selftest - INFO - RESULTS - sstatetests.SStatePrintdiff.test_gcc_runtime_vs_gcc_source: PASSED (1356.02s)
Comment 42 Yoann Congal 2026-07-09 07:15:49 UTC
All previous scarthgap builds were on my scarthgap-nut branch, to check that is is not something I added with my -nut series, I've tried a origin/scarthgap build, it also failed:
oe-selftest-debian debian12-vk-2 scarthgap completed at 2026-07-08 23:47:08+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/4253

That, plus my local oe-selftest passing on my -nut branch, I conclude that it is not related to the new patches but seem related to AB state (its hashserv, sstate, ...)
Comment 43 Randy MacLeod 2026-07-09 14:41:15 UTC
RP: Don't trust the diff output for <reasons>.
Comment 44 Yoann Congal 2026-07-10 14:42:30 UTC
Two autobuilder runs of the same test (target oe-selftest-debian, scarthgap), differing only in sstate/hashequiv config:

  PASS - isolated (local empty SSTATE_DIR + BB_HASHSERVE = 'auto'):
    https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/4265

  FAIL - shared (SSTATE_DIR = .../pub/sstate + BB_HASHSERVE = wss://hashserv-eu.yoctoproject.org/ws):
    https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/4266



(In reply to Randy MacLeod from comment #43)
> RP: Don't trust the diff output for <reasons>.
IIUC <reasons> = "because of hash equivalence, tasks with widly different input (task code, variable, ...) can be comparable and are compared but this is legitimate."
Comment 45 Yoann Congal 2026-07-18 08:23:55 UTC
oe-selftest-armhost ubuntu2404-vk-arm2 scarthgap completed at 2026-07-17 13:13:09+00:00
https://autobuilder.yoctoproject.org/valkyrie/?#/builders/23/builds/4352
oe-selftest-debian ubuntu2204-vk-4 scarthgap completed at 2026-07-17 12:26:26+00:00
https://autobuilder.yoctoproject.org/valkyrie/?#/builders/35/builds/4298
oe-selftest-fedora stream9-vk-1 scarthgap completed at 2026-07-17 15:55:02+00:00
https://autobuilder.yoctoproject.org/valkyrie/?#/builders/48/builds/4118
Comment 46 Yoann Congal 2026-07-23 10:50:29 UTC
I can reproduce this locally but using AB provided sstate/hashserve:
==== local.conf ====
SANITY_TESTED_DISTROS = ""

BB_HASHSERVE_DB_DIR = "${SSTATE_DIR}"
SSTATE_MIRRORS += " file://.*  http://sstate.yoctoproject.org/all/PATH;downloadfilename=PATH"
BB_HASHSERVE = 'auto'
BB_HASHSERVE_UPSTREAM = 'wss://hashserv.yoctoproject.org/ws'
BB_SIGNATURE_HANDLER = "OEEquivHash"
========

=> $ oe-selftest -Kvr sstatetests.SStatePrintdiff.test_gcc_runtime_vs_gcc_source

2026-07-21 00:00:24,099 - oe-selftest - INFO - ======================================================================
2026-07-21 00:00:24,099 - oe-selftest - INFO - FAIL: test_gcc_runtime_vs_gcc_source (sstatetests.SStatePrintdiff.test_gcc_runtime_vs_gcc_source)
2026-07-21 00:00:24,099 - oe-selftest - INFO - ----------------------------------------------------------------------
2026-07-21 00:00:24,099 - oe-selftest - INFO - Traceback (most recent call last):
  File "/home/yocon/Documents/projets/yocto/maint_lts/scarthgap/poky_bz15289/meta/lib/oeqa/selftest/cases/sstatetests.py", line 884, in test_gcc_runtime_vs_gcc_source
            self.run_test_printdiff_changerecipe("gcc-runtime", "gcc-source", "-c do_preconfigure {}".format(gcc_source_pn),
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    """
    ^^^
    ...<3 lines>...
    """,
    ^^^^
    expected_sametmp_output, expected_difftmp_output)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yocon/Documents/projets/yocto/maint_lts/scarthgap/poky_bz15289/meta/lib/oeqa/selftest/cases/sstatetests.py", line 827, in run_test_printdiff_changerecipe
    self.assertIn(item, result_sametmp.output, msg = "Item {} not found in output:\n{}".format(item, result_sametmp.output))
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: "Task gcc-source-13.4.0:do_preconfigure couldn't be used from the cache because:" not found in:
Loading cache...done.
Loaded 1964 entries from dependency cache.
Parsing recipes...done.
Parsing of 989 .bb files complete (988 cached, 1 parsed). 1964 targets, 52 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION           = "2.8.1"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "x86_64-poky-linux"
MACHINE              = "qemux86-64"
DISTRO               = "poky"
DISTRO_VERSION       = "5.0.19"
TUNE_FEATURES        = "m64 core2"
TARGET_FPU           = ""
meta                 
meta-poky            
meta-yocto-bsp       = "poky_bz15289:6ccffb86e86b4d41f61dc0c06c2b732c72676537"
meta-selftest        = "main:00662e2978c8d2ff95d37e75a4cfc1c1bc2d1b8b"

Initialising tasks...Sstate summary: Wanted 22 Local 2 Mirrors 1 Missed 19 Current 249 (13% match, 92% complete)
Removing 9 stale sstate objects for arch core2-64...done.
Removing 1 stale sstate objects for arch x86_64...done.
Removing 3 stale sstate objects for arch qemux86_64...done.
Checking sstate mirror object availability...done.
The differences between the current build and any cached tasks start at the following tasks:
virtual:native:/home/yocon/Documents/projets/yocto/maint_lts/scarthgap/poky_bz15289/meta/recipes-extended/xz/xz_5.4.7.bb:do_deploy_source_date_epoch
/home/yocon/Documents/projets/yocto/maint_lts/scarthgap/poky_bz15289/meta/recipes-devtools/libtool/libtool-native_2.4.7.bb:do_deploy_source_date_epoch
/home/yocon/Documents/projets/yocto/maint_lts/scarthgap/poky_bz15289/meta/recipes-core/gettext/gettext-minimal-native_0.22.5.bb:do_deploy_source_date_epoch
virtual:native:/home/yocon/Documents/projets/yocto/maint_lts/scarthgap/poky_bz15289/meta/recipes-devtools/autoconf/autoconf_2.72e.bb:do_deploy_source_date_epoch
/home/yocon/Documents/projets/yocto/maint_lts/scarthgap/poky_bz15289/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy-native.bb:do_deploy_source_date_epoch
/home/yocon/Documents/projets/yocto/maint_lts/scarthgap/poky_bz15289/meta/recipes-devtools/m4/m4-native_1.4.19.bb:do_deploy_source_date_epoch
/home/yocon/Documents/projets/yocto/maint_lts/scarthgap/poky_bz15289/meta/recipes-devtools/quilt/quilt-native_0.67.bb:do_deploy_source_date_epoch
virtual:native:/home/yocon/Documents/projets/yocto/maint_lts/scarthgap/poky_bz15289/meta/recipes-devtools/automake/automake_1.16.5.bb:do_deploy_source_date_epoch
NOTE: Writing task signature files
NOTE: Tasks Summary: Attempted 0 tasks of which 0 didn't need to be rerun and all succeeded.



I wonder if that's related to this other situation I had: logs from IRC July 18th:
15:28:38 <yocton> In https://autobuilder.yoctoproject.org/valkyrie/?#/builders/29/builds/4233, the Source Mirror Selftest/sstatetests.SStateMirrors.* fail because of a missing object
15:28:38 <yocton> sstate:lzip-native:x86_64-linux:1.24:r0:x86_64:12:2216[...]25_deploy_source_date_epoch.tar.zst is missing (either on disk or via CDN/HTTP)
15:28:38 <yocton> but sstate:lzip-native:aarch64-linux:1.24:r0:aarch64:12:2216[...]25_deploy_source_date_epoch.tar.zst is present on both disk and CDN/HTTP (x86_64 <-> aarch64)
15:29:40 <yocton> does that ring a bell?
18:16:22 <yocton> ... and now the sstate object appeared... Maybe a random build I ran in the meantime noticed and recreated it? At least the mirror selftest should go green now.

I will check AB sstate/hashserv data around the task differences that make the test fail:
* virtual:native:$poky/meta/recipes-extended/xz/xz_5.4.7.bb:do_deploy_source_date_epoch
* $poky/meta/recipes-devtools/libtool/libtool-native_2.4.7.bb:do_deploy_source_date_epoch
* $poky/meta/recipes-core/gettext/gettext-minimal-native_0.22.5.bb:do_deploy_source_date_epoch
* virtual:native:$poky/meta/recipes-devtools/autoconf/autoconf_2.72e.bb:do_deploy_source_date_epoch
* $poky/meta/recipes-extended/texinfo-dummy-native/texinfo-dummy-native.bb:do_deploy_source_date_epoch
* $poky/meta/recipes-devtools/m4/m4-native_1.4.19.bb:do_deploy_source_date_epoch
* $poky/meta/recipes-devtools/quilt/quilt-native_0.67.bb:do_deploy_source_date_epoch
* virtual:native:$poky/meta/recipes-devtools/automake/automake_1.16.5.bb:do_deploy_source_date_epoch
Comment 47 Yoann Congal 2026-07-24 00:27:45 UTC
My hypothesis:
* I'll take flex-native as an example, but the others should be the same
* Tasks have 2 hashes that matter here:
  * taskhash: fully hashing task input/output/content
  * unihash: shared hash accross build and through equivalence (tasks with
    different content but the same output)
* in AB config, flex-native:deploy_source_date_epoch is particular case where
  unihash != taskhash[0].
  That mean that the task got its unihash not by its content but because the output
  matched a previously recorded unihash (hash equivalence)
* When running sstatetests.SStatePrintdiff tests the AB sstate server/hashserv
  is use to pull this task setscene into the build.
* In the test, task signature tree is checked but the printdiff code use is
  based on taskhash, not unihash.
* The unihash/taskhash mismatch of flex-native:deploy_source_date_epoch is then
  interpreted as a difference in the task signature tree. Since it is earlier
  in the tree than the searched difference it hides it (printdiff only print
  the deepest changing signature)
* ... and the test fails.

My idea to fix this: ensuring that unihash=taskhash for this kind of test:
BB_SIGNATURE_HANDLER = "OEBasicHash"
Note: this trick is already used in SStateHashSameSigs, added by:
oeqa/selftest/sstatetests: Ensure we don't use hashequiv for sstatesigs tests
https://git.openembedded.org/openembedded-core/commit/?id=6af559467f869afabad16cd84ae1af4bc7af0950

I started a build with a WIP patch:
* oe-selftest-armhost › 4414 https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/4414
* oe-selftest-debian › 4361 https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/4361
* oe-selftest-fedora › 4180 https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/4180

[0]:
$ bitbake-diffsigs -t flex-native deploy_source_date_epoch -d
Computed task hash is 713435c822ecf9d73aed6ed84a347ff34b1bdf5107135debfb2a02cb7c6dbb2a

❯ bitbake-hashclient -n --address wss://hashserv.yoctoproject.org/ws   get oe.sstatesig.OEOuthashBasic 713435c822ecf9d73aed6ed84a347ff34b1bdf5107135debfb2a02cb7c6dbb2a
{
    "PN": null,
    "PR": null,
    "PV": null,
    "created": "2026-05-25T22:42:07.856329",
    "id": 251312670,
    "method": "oe.sstatesig.OEOuthashBasic",
    "outhash": "9fdd0906df872848ea2d4bd2bbb9d36a2305533f7544f7026fe55940acf3eed8",
    "outhash_siginfo": null,
    "owner": "pokybuild",
    "task": null,
    "taskhash": "713435c822ecf9d73aed6ed84a347ff34b1bdf5107135debfb2a02cb7c6dbb2a",
    "unihash": "d5b4e1bf47986e2cf03c1e1eb14104967a6792253c837598424bc549353fcc62"
}
Comment 48 Yoann Congal 2026-07-24 09:33:57 UTC
(In reply to Yoann Congal from comment #47)
> * In the test, task signature tree is checked but the printdiff code use is
>   based on taskhash, not unihash.

This is wrong :( printdiff actually uses unihash and the whole hypothesis, while tempting, does not work.

============================

New hypothesis:
* The YP sstate server serves (at least) 2 kinds of objects:
  * Sstate object with actual tasks content (.tar.zst)
  * siginfo data about the sstate objects (.tar.zst.siginfo)
* Normal builds use the Sstate objects but printdiff also needs siginfo [0]
* In the case of flex-native:do_deploy_source_date_epoch
  (unihash=d5b4e1bf47...), the sstate object is correctly served by the server,
  but not the siginfo file:
  200 http://sstate.yoctoproject.org/all/universal/d5/b4/sstate:flex-native:x86_64-linux:2.6.4:r0:x86_64:12:d5b4e1bf47986e2cf03c1e1eb14104967a6792253c837598424bc549353fcc62_deploy_source_date_epoch.tar.zst
  404 http://sstate.yoctoproject.org/all/universal/d5/b4/sstate:flex-native:x86_64-linux:2.6.4:r0:x86_64:12:d5b4e1bf47986e2cf03c1e1eb14104967a6792253c837598424bc549353fcc62_deploy_source_date_epoch.tar.zst.siginfo
* During the pre-build part of the SStatePrintdiff, the sstate object is
  pulled and build succeeds. But when the printdiff code try to check the task,
  it looks for the missing siginfo URL and, unable to validate the task
  signature, reports it as a difference -- even though the sstate object itself
  was correctly restored.

Open question:
* Is a sstate object without its accompanying siginfo file on the sstate server OK?
* If not, why is it missing?
  * Never created in the first place?
  * Removed at some point? Do we ever remove files from sstate server?

Note: switching to OEBasicHash worked because it changed the hash so sstate
pointed to objects with siginfo (either preexisting or rebuilt). But, it does
not fix the underlying issue.

[0]: https://git.openembedded.org/bitbake/tree/lib/bb/runqueue.py?id=c251833d2062ef92fc7e77ab6456af928ad1d31a#n1765:
     def print_diffscenetasks(self):
         [...]
         valid_new = self.validate_hashes(tocheck, self.cooker.data, 0, True, summary=False)
                                                                        ^^^^ = siginfo
Comment 49 Yoann Congal 2026-07-24 10:57:32 UTC
After discussion with Paul, we may try to move the sstate objects from the sstate server. That will hoepfully force the test pre-build to recreate them with siginfo data.

When ran locally, here are the 404 siginfo URLs for which there are matching sstate objects:
http://sstate.yoctoproject.org/all/5a/8a/sstate:libmpc-native::1.3.1:r0::12:5a8a4f88273f67b4731b57508fedb0c970f336c7c91fe793a4dbf49222ae04cf_populate_lic.tar.zst.siginfo
http://sstate.yoctoproject.org/all/6e/72/sstate:cmake-native::3.28.3:r0::12:6e726283e249916a535c446651391c5a1603b8ca59bec241e40ba5b0885ba913_populate_lic.tar.zst.siginfo
http://sstate.yoctoproject.org/all/universal/04/32/sstate:zlib-native:x86_64-linux:1.3.1:r0:x86_64:12:04321f742448739b035aa4c442d0500e259b78565af69f2e51d907fb59290a98_deploy_source_date_epoch.tar.zst.siginfo
http://sstate.yoctoproject.org/all/universal/09/62/sstate:libmpc-native:x86_64-linux:1.3.1:r0:x86_64:12:09628205d50c87e5917c63d3814bef25851174e338d9b24907e6eca9137c5e22_deploy_source_date_epoch.tar.zst.siginfo
http://sstate.yoctoproject.org/all/universal/0c/f0/sstate:zstd-native:x86_64-linux:1.5.5:r0:x86_64:12:0cf060875d1aa86788ecf2e54844630f83313a868d7c67a99276f4cce0951826_create_spdx.tar.zst.siginfo
http://sstate.yoctoproject.org/all/universal/0f/27/sstate:autoconf-native:x86_64-linux:2.72e:r0:x86_64:12:0f27277124e07cce7fd2ead225de318113d14cd2cd0a70e459de68ff4a625eaa_deploy_source_date_epoch.tar.zst.siginfo
http://sstate.yoctoproject.org/all/universal/14/b3/sstate:gnu-config-native:x86_64-linux:20240101+git:r0:x86_64:12:14b36aa9638ea023d8ecaf2df0777d1274711c033b0785ff7b5b497061b59d73_deploy_source_date_epoch.tar.zst.siginfo
http://sstate.yoctoproject.org/all/universal/15/ac/sstate:quilt-native:x86_64-linux:0.67:r0:x86_64:12:15acc17df638fbd123dc957a43bc3ef2846d6fd9e2c5cfde25a26a113c507b61_deploy_source_date_epoch.tar.zst.siginfo
http://sstate.yoctoproject.org/all/universal/16/26/sstate:ncurses-native:x86_64-linux:6.4:r0:x86_64:12:16263037d4faa2dd9af5b99b55e7e94dd2024dd918ad890418a14d9717562f92_deploy_source_date_epoch.tar.zst.siginfo
http://sstate.yoctoproject.org/all/universal/18/f6/sstate:gettext-minimal-native:x86_64-linux:0.22.5:r0:x86_64:12:18f6cd23826e3bcdc89ecf604a122b4e86f1863dfdaad684d7ef24f374dbc75f_deploy_source_date_epoch.tar.zst.siginfo
http://sstate.yoctoproject.org/all/universal/1e/9b/sstate:gmp-native:x86_64-linux:6.3.0:r0:x86_64:12:1e9bd21a6959fa57e776efee4591dc210415689018a595f73152483afb794b13_deploy_source_date_epoch.tar.zst.siginfo
http://sstate.yoctoproject.org/all/universal/25/69/sstate:mpfr-native:x86_64-linux:4.2.1:r0:x86_64:12:256971d162c7f33816c6115c283aae027ad79c1741404d6be8a9d2d7345d29fc_recipe_qa.tar.zst.siginfo
http://sstate.yoctoproject.org/all/universal/2a/ce/sstate:texinfo-dummy-native:x86_64-linux:1.0:r0:x86_64:12:2aceacbe381f81d1851c0fbf48d05351260828bcca7c33e5e2972dd11248a2f5_deploy_source_date_epoch.tar.zst.siginfo
http://sstate.yoctoproject.org/all/universal/30/c7/sstate:sqlite3-native:x86_64-linux:3.45.3:r0:x86_64:12:30c72cc8b2dfb6bd3baac437214891abe50edb0faed2deaead6520bcd5fa228e_deploy_source_date_epoch.tar.zst.siginfo
http://sstate.yoctoproject.org/all/universal/31/38/sstate:mpfr-native:x86_64-linux:4.2.1:r0:x86_64:12:3138a4546799db4d3f5034a2158f31bd8fc00664c706b21dff37598b46323b17_deploy_source_date_epoch.tar.zst.siginfo
http://sstate.yoctoproject.org/all/universal/5d/ee/sstate:cmake-native:x86_64-linux:3.28.3:r0:x86_64:12:5dee2c0fd71eae3eac2028c1763692ecc5d390bd9dd781d0c916b9338b619332_deploy_source_date_epoch.tar.zst.siginfo
http://sstate.yoctoproject.org/all/universal/65/fb/sstate:m4-native:x86_64-linux:1.4.19:r0:x86_64:12:65fbefe3d0085032c49b1c0fb7eb8353044b932c112184d3ada73fee45e4a573_deploy_source_date_epoch.tar.zst.siginfo
http://sstate.yoctoproject.org/all/universal/6b/18/sstate:attr-native:x86_64-linux:2.5.1:r0:x86_64:12:6b1818cf376a9b440072ee8da78bd9903c781d2f09be297203442e4729574017_deploy_source_date_epoch.tar.zst.siginfo
http://sstate.yoctoproject.org/all/universal/74/79/sstate:gnu-config-native:x86_64-linux:20240101+git:r0:x86_64:12:7479109c9e5420a3b09aab5b622c4003a1d4c01dd059da890f052d32bbd46bb6_populate_sysroot.tar.zst.siginfo
http://sstate.yoctoproject.org/all/universal/87/9d/sstate:xz-native:x86_64-linux:5.4.7:r0:x86_64:12:879d3dd0984f74e23d8f8221f6f3da3f58a4586af2806fc9fa8cf0693953c267_deploy_source_date_epoch.tar.zst.siginfo
http://sstate.yoctoproject.org/all/universal/8b/58/sstate:make-native:x86_64-linux:4.4.1:r0:x86_64:12:8b58ad79c4c02429be3c6048d35dbe2ff2b907792bd8a3496d9c292ccf137be1_deploy_source_date_epoch.tar.zst.siginfo
http://sstate.yoctoproject.org/all/universal/90/32/sstate:gdbm-native:x86_64-linux:1.23:r0:x86_64:12:9032763d0cca1018dea83bacc772e2ee9a034d8361edba7d4bd14a051491964b_deploy_source_date_epoch.tar.zst.siginfo
http://sstate.yoctoproject.org/all/universal/97/02/sstate:zstd-native:x86_64-linux:1.5.5:r0:x86_64:12:97024d8336341895eb2fa597d37fe06ecb88a3aa8b277cc0833b89431c6a21f5_deploy_source_date_epoch.tar.zst.siginfo
http://sstate.yoctoproject.org/all/universal/9c/0f/sstate:autoconf-archive-native:x86_64-linux:2023.02.20:r0:x86_64:12:9c0f3efb3af60c975314b057a784385ec3dcdcaf8bf921e483221f8cb17372eb_deploy_source_date_epoch.tar.zst.siginfo
http://sstate.yoctoproject.org/all/universal/aa/7c/sstate:pkgconfig-native:x86_64-linux:0.29.2+git:r0:x86_64:12:aa7c648fc20bd39bf54c3f1ca478f0ea1c592639d2d9c36a9730e51c1511e77d_deploy_source_date_epoch.tar.zst.siginfo
http://sstate.yoctoproject.org/all/universal/ae/08/sstate:automake-native:x86_64-linux:1.16.5:r0:x86_64:12:ae08a46ff7bca7d1563afa16290084612502c5ea3ae73a06d2be104589873995_deploy_source_date_epoch.tar.zst.siginfo
http://sstate.yoctoproject.org/all/universal/af/97/sstate:bison-native:x86_64-linux:3.8.2:r0:x86_64:12:af979a7469c7fb06238383bbfa2980f685554aacf4d15d5f8d69a453b0c20c07_deploy_source_date_epoch.tar.zst.siginfo
http://sstate.yoctoproject.org/all/universal/bb/d3/sstate:patch-native:x86_64-linux:2.7.6:r0:x86_64:12:bbd343afd5ff08c454dfe6faee199204aefa311d91d1c5f6dc449d3b6d5d1969_deploy_source_date_epoch.tar.zst.siginfo
http://sstate.yoctoproject.org/all/universal/c5/c0/sstate:bzip2-native:x86_64-linux:1.0.8:r0:x86_64:12:c5c050091fba8c170feaa4237f5ba485782e62e7204d7a4272162307465399e5_populate_sysroot.tar.zst.siginfo
http://sstate.yoctoproject.org/all/universal/ca/15/sstate:acl-native:x86_64-linux:2.3.2:r0:x86_64:12:ca15c1444217d43d99f2d3cb6a1d8842aad162e5fd6d1cc0148be69d2e4709e2_deploy_source_date_epoch.tar.zst.siginfo
http://sstate.yoctoproject.org/all/universal/d5/b4/sstate:flex-native:x86_64-linux:2.6.4:r0:x86_64:12:d5b4e1bf47986e2cf03c1e1eb14104967a6792253c837598424bc549353fcc62_deploy_source_date_epoch.tar.zst.siginfo
http://sstate.yoctoproject.org/all/universal/db/c5/sstate:automake-native:x86_64-linux:1.16.5:r0:x86_64:12:dbc54da45050daadce47a9c8b9757748bca6f7f01ba64bb9eb347cc88991f7f1_populate_sysroot.tar.zst.siginfo
http://sstate.yoctoproject.org/all/universal/dd/dc/sstate:re2c-native:x86_64-linux:3.1:r0:x86_64:12:dddc0e5fad9ecafeb9656825bdcfb90de1009c6c1b140eceaa4a51ed581b7697_deploy_source_date_epoch.tar.zst.siginfo
http://sstate.yoctoproject.org/all/universal/e2/63/sstate:readline-native:x86_64-linux:8.2:r0:x86_64:12:e263a149f8b7bdd6c487292249dae6ec8ddd7f755753bbbfccbc774302600257_populate_sysroot.tar.zst.siginfo
http://sstate.yoctoproject.org/all/universal/e4/60/sstate:ninja-native:x86_64-linux:1.11.1:r0:x86_64:12:e4609ec64be3d0d5e00ab751bd1d0ad8d457435e8f6f31fd2f009de86c550d0a_deploy_source_date_epoch.tar.zst.siginfo
http://sstate.yoctoproject.org/all/universal/e6/28/sstate:rsync-native:x86_64-linux:3.2.7:r0:x86_64:12:e628006c33f330651a8ce49ba7f71da528ff0387f76a1f6800cf020bade2173d_deploy_source_date_epoch.tar.zst.siginfo
http://sstate.yoctoproject.org/all/universal/e8/38/sstate:popt-native:x86_64-linux:1.19:r0:x86_64:12:e838ba5d7ba778808b914245d847ae95bdcf92524c40ff724057ff6cb5eeb5ec_deploy_source_date_epoch.tar.zst.siginfo
http://sstate.yoctoproject.org/all/universal/f5/2a/sstate:libtool-native:x86_64-linux:2.4.7:r0:x86_64:12:f52a4494bbea9cbb699e24130666ea696ce20eea9aebedc286ea0e93afdae653_deploy_source_date_epoch.tar.zst.siginfo
Comment 50 Paul Barker 2026-07-24 14:56:19 UTC
I moved the sstate objects corresponding to the missing siginfo files listed by Yoann to /srv/autobuilder/valkyrie.yocto.io/pub/non-release/sstate-missing-siginfo. Hopefully builds start working now!
Comment 51 Yoann Congal 2026-07-26 06:52:58 UTC
(In reply to Paul Barker from comment #50)
> I moved the sstate objects corresponding to the missing siginfo files listed
> by Yoann to
> /srv/autobuilder/valkyrie.yocto.io/pub/non-release/sstate-missing-siginfo.
> Hopefully builds start working now!

With some more cycles of :
* Building
* Listing the sstate object without their accompanying .siginfo file
* Moving the sstate file to /srv/autobuilder/valkyrie.yocto.io/pub/non-release/sstate-missing-siginfo/
builds are bask to green:
* oe-selftest-armhost › 4434: https://autobuilder.yoctoproject.org/valkyrie/?#/builders/23/builds/4434
* oe-selftest-debian › 4372: https://autobuilder.yoctoproject.org/valkyrie/?#/builders/35/builds/4372
* oe-selftest-fedora › 4191: https://autobuilder.yoctoproject.org/valkyrie/?#/builders/48/builds/4191

So scarthgap is fixed :)

Still open questions:
* Is a sstate object without its accompanying siginfo file on the sstate server legitimate?
* If not, why is it missing?
  * Removed at some point? (a overzealous cleanup process? a NFS corruption?)
* Should we actively search and cleanup the other sstate objects without siginfo files?
Comment 52 Richard Purdie 2026-07-27 09:34:52 UTC
The autobuilder is deleting stale files in sstate (files over 29 days old), however that cleanup hasn't been run since July 3rd. We don't have the list of files removed unfortunately.

In general, an sstate object shouldn't exist without the corresponding siginfo file. The code goes to great lengths to always touch both the sstate file and the corresponding siginfo when it uses them.

I compared the code in sstate.bbclass between master and scarthgap and I couldn't see any difference in how they were handling files. I also couldn't spot a code patch where it would use an sstate file and not touch the siginfo file, if it had permission.

So the lack of siginfo files could explain the issue but how we get there, I'm not sure. I would note e.g.:

-rw-rw-r--   1 pokybuild pokybuild     3201 Apr 21 16:25 sstate:lib32-cups:x86-pokymllib32-linux:2.4.16:r0:x86:14:0000bb7b1e84f87ec4439153e077f8439d2cc9a3317bcb82b2ab1f0ec0fcc4b5_create_recipe_spdx.tar.zst
(10:08:01) RP: -rw-rw-r--   1 pokybuild pokybuild    10293 Apr 17 01:56 sstate:lib32-cups:x86-pokymllib32-linux:2.4.16:r0:x86:14:0000bb7b1e84f87ec4439153e077f8439d2cc9a3317bcb82b2ab1f0ec0fcc4b5_create_recipe_spdx.tar.zst.siginfo

so files with differing stamps are rare but do exist.
Comment 53 Richard Purdie 2026-07-27 10:19:56 UTC
Looking at atime:

/srv/autobuilder/valkyrie.yocto.io/pub/sstate/00/00$ ls -la *cursor* -u
-rw-rw-r-- 1 pokybuild pokybuild  2550 Jul 10 06:57 sstate:xcursor-transparent-theme:all-poky-linux:0.1.1+git:r0:allarch:14:0000d41f45c9a9cefc712ab5fdcff363e9e0fa9b8a5d99ee867f3cef5b0de372_create_spdx.tar.zst
-rw-rw-r-- 1 pokybuild pokybuild 15939 Jul 19 11:36 sstate:xcursor-transparent-theme:all-poky-linux:0.1.1+git:r0:allarch:14:0000d41f45c9a9cefc712ab5fdcff363e9e0fa9b8a5d99ee867f3cef5b0de372_create_spdx.tar.zst.siginfo
-rw-rw-r-- 1 pokybuild pokybuild  1709 Jul 16 12:16 sstate:xcursor-transparent-theme:x86-64-v3-poky-linux:0.1.1+git:r0:x86-64-v3:14:0000a48455f6501c8460af3043078b91726d1a1967094a1536dc4dc731e78ee7_create_recipe_spdx.tar.zst
-rw-rw-r-- 1 pokybuild pokybuild  9171 Jul 10 17:19 sstate:xcursor-transparent-theme:x86-64-v3-poky-linux:0.1.1+git:r0:x86-64-v3:14:0000a48455f6501c8460af3043078b91726d1a1967094a1536dc4dc731e78ee7_create_recipe_spdx.tar.zst.siginfo

i.e. 6 days and 9 days apart and both ways around :/
Comment 54 Richard Purdie 2026-07-27 10:20:50 UTC
Some files show the issue on mtime but not atime:

pokybuild@debian12-vk-1:/srv/autobuilder/valkyrie.yocto.io/pub/sstate/00/00$ ls -la *cups*
-rw-rw-r-- 1 pokybuild pokybuild  3201 Apr 21 16:25 sstate:lib32-cups:x86-pokymllib32-linux:2.4.16:r0:x86:14:0000bb7b1e84f87ec4439153e077f8439d2cc9a3317bcb82b2ab1f0ec0fcc4b5_create_recipe_spdx.tar.zst
-rw-rw-r-- 1 pokybuild pokybuild 10293 Apr 17 01:56 sstate:lib32-cups:x86-pokymllib32-linux:2.4.16:r0:x86:14:0000bb7b1e84f87ec4439153e077f8439d2cc9a3317bcb82b2ab1f0ec0fcc4b5_create_recipe_spdx.tar.zst.siginfo
pokybuild@debian12-vk-1:/srv/autobuilder/valkyrie.yocto.io/pub/sstate/00/00$ ls -la *cups* -u
-rw-rw-r-- 1 pokybuild pokybuild  3201 Jul 10 16:19 sstate:lib32-cups:x86-pokymllib32-linux:2.4.16:r0:x86:14:0000bb7b1e84f87ec4439153e077f8439d2cc9a3317bcb82b2ab1f0ec0fcc4b5_create_recipe_spdx.tar.zst
-rw-rw-r-- 1 pokybuild pokybuild 10293 Jul 10 15:37 sstate:lib32-cups:x86-pokymllib32-linux:2.4.16:r0:x86:14:0000bb7b1e84f87ec4439153e077f8439d2cc9a3317bcb82b2ab1f0ec0fcc4b5_create_recipe_spdx.tar.zst.siginfo
pokybuild@debian12-vk-1:/srv/autobuilder/valkyrie.yocto.io/pub/sstate/00/00$ ls -la *cups* -c
-rw-rw-r-- 1 pokybuild pokybuild  3201 Jul  4 01:35 sstate:lib32-cups:x86-pokymllib32-linux:2.4.16:r0:x86:14:0000bb7b1e84f87ec4439153e077f8439d2cc9a3317bcb82b2ab1f0ec0fcc4b5_create_recipe_spdx.tar.zst
-rw-rw-r-- 1 pokybuild pokybuild 10293 Jul  4 01:35 sstate:lib32-cups:x86-pokymllib32-linux:2.4.16:r0:x86:14:0000bb7b1e84f87ec4439153e077f8439d2cc9a3317bcb82b2ab1f0ec0fcc4b5_create_recipe_spdx.tar.zst.siginfo
Comment 55 Richard Purdie 2026-07-27 10:33:19 UTC
I have a theory - there are some sstate tasks which might never execute if there are higher level tasks in the task graph which "mask" them and mean they are not needed. In those cases, these lower level pieces of the dependency chart might not actually get accessed at all.

Even running the build wouldn't trigger them to get rebuilt either since the upper level artefacts would mean the lower ones were not needed.

Where setscene tasks are skipped, we do therefore really need to still update the atime of the files. I'm not sure how we do that...
Comment 56 Richard Purdie 2026-07-27 10:42:29 UTC
I think this can be fixed in https://git.openembedded.org/openembedded-core/tree/meta/classes-global/sstate.bbclass#n980 where we need to touch both the file and any siginfo and sig files
Comment 59 Yoann Congal 2026-11-12 16:00:02 UTC
Patches merged:

wrynose
* cceec3584f3 - sstate: Update unpack touch code to be consistent (7 weeks ago) <Richard Purdie>
* 53a663be633 - sstate: Update mtime/atime for sig/siginfo files in sstate_checkhashes() (7 weeks ago) <Richard Purdie>

scarthgap
* 147004ef609 - sstate: Update unpack touch code to be consistent (6 weeks ago) <Richard Purdie>
* 2bd595a9d95 - sstate: Update mtime/atime for sig/siginfo files in sstate_checkhashes() (6 weeks ago) <Richard Purdie>

Closing