Created attachment 5252 [details] the corrupted sstate object The build: oe-selftest-debian ubuntu2404-vk-2 wrynose completed at 2026-10-05 17:55:12+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/5010 ... created the sstate object (hinted by timestamps only): sstate_popt-native_x86_64-linux_1.19_r0_x86_64_14_54faa65665868426a9f37eec2276da4d0d818788bb5ec81714b2fee7836191b0_create_recipe_spdx.tar.zst (attached to this bug, for backup and further inspection) The object can be downloaded here (for now): http://sstate.yoctoproject.org/all/universal/54/fa/sstate:popt-native:x86_64-linux:1.19:r0:x86_64:14:54faa65665868426a9f37eec2276da4d0d818788bb5ec81714b2fee7836191b0_create_recipe_spdx.tar.zst This sstate object is corrupted because it contains a "recipe-deploy/x86_64/static/static-popt-native.spdx.json" containing 20704 NUL bytes where a valid JSON is expected: ❯ zstd -d < sstate_popt-native_[...]_create_recipe_spdx.tar.zst | tar xf - recipe-deploy/x86_64/static/static-popt-native.spdx.json -O |hexdump 0000000 0000 0000 0000 0000 0000 0000 0000 0000 * 00050e0 Note: This "0" block is in fact a "hole" as in "sparse tar" feature. Interestingly, 20704 is the size of the correct expected JSON (rebuilt locally): ❯ head correct/static-popt-native.spdx.json { "@context": "https://spdx.org/rdf/3.0.1/spdx-context.jsonld", "@graph": [ { "type": "CreationInfo", "@id": "_:CreationInfo0", "created": "2026-10-08T18:44:49Z", "createdBy": [ "http://spdx.org/spdxdocs/3f48ce2be4187531be6c5eeb177ec40304d25644/bitbake-addba517-4804-5ae3-87c2-0c3a1a5812ba/bitbake/agent/John_Doe" ], ❯ stat correct/static-popt-native.spdx.json -c %s 20704 The impact of this corruption can be seen in JSON parsing failure in SPDX related tasks: https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/5020 https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/5026 https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/5027 https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/4834 https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/4840 Logs: ERROR: rpm-native-1_4.20.1-r0 do_create_recipe_spdx: Error executing a python function in exec_func_python() autogenerated: [...] Exception: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Created attachment 5253 [details] The correct JSON file
I now believe this is a race-condition in ext4. On a ext4 filesystem, under load (e.g. lots of sync()), there is a window of time where a recently written+closed file reports "stat.st_blocks = 0". See this reproducer: https://gist.github.com/ycongal-smile/ee5c363a10f12fc014262930369fbb6f ❯ python3 st_blocks_race.py 347/1000 files holding 20704 bytes reported st_blocks == 0 Then, "tar --sparse" (that we use in sstate) optimize fully sparse files that have "st_size != 0" but no blocks (stat.st_blocks == 0)[1]. If tar does this check in the race window, then it will store into the archive a file that is just a sparse hole of the size of the file. Which is exactly what we observe. Neither tar nor the kernel seem to have a fix/workaround. Note: The same bug on btrfs: https://lists.gnu.org/archive/html/bug-tar/2016-07/msg00000.html * There are POSIX-level debate whether a filesystem can report st_blocks=0 on written files. [1]: https://cgit.git.savannah.gnu.org/cgit/tar.git/tree/src/sparse.c#n269
From a failing build: oe-selftest-debian debian12-vk-6 wrynose completed at 2026-10-08 16:56:12+00:00 https://autobuilder.yoctoproject.org/valkyrie/?#/builders/35/builds/5027 If I prevent the corrupted sstate object to be reused with https://git.openembedded.org/openembedded-core-contrib/commit/?h=stable/wrynose-nut&id=ecd7d7370301218f89ce460e393cb48978f15673 meta/recipes-support/popt/popt_1.19.bb: +# XXX to workaround a corrupted sstate object (BZ#16446) +PR = "r1" +HASHEQUIV_HASH_VERSION .= ".1" Build succeed: oe-selftest-debian debian13-vk-2 wrynose completed at 2026-10-09 10:13:12+00:00 https://autobuilder.yoctoproject.org/valkyrie/?#/builders/35/builds/5030
I removed the corrupted sstate object from the shared sstate AB directory