Bug 16446

Summary:

Autobuilder produced a corrupted sstate object (popt-native:create_recipe_spdx)

Product:

[Infrastructure] AutoBuilder

Reporter:

Yoann Congal <yoann.congal>

Component:

autobuilder

Assignee:

Richard Purdie <richard.purdie>

Status:

NEW ---

QA Contact:

Severity:

normal

Priority:

Undecided

CC:

infras.ab.watcher, Infras.watcher, paul

Version:

6.0.9

Target Milestone:

0.0.0

Hardware:

x86

OS:

Multiple

Whiteboard:

OS type for building Yocto:

---

Type of Regression:

---

Verified:

Documentation change:

No (bug/feature does not impact docs)

Attachments:

Description	Flags
the corrupted sstate object	none
The correct JSON file	none

Description Yoann Congal 2026-10-08 19:25:41 UTC

Created attachment 5252 [details]
the corrupted sstate object

The build: oe-selftest-debian ubuntu2404-vk-2 wrynose completed at 2026-10-05 17:55:12+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/5010

... created the sstate object (hinted by timestamps only):
sstate_popt-native_x86_64-linux_1.19_r0_x86_64_14_54faa65665868426a9f37eec2276da4d0d818788bb5ec81714b2fee7836191b0_create_recipe_spdx.tar.zst
(attached to this bug, for backup and further inspection)
The object can be downloaded here (for now):
http://sstate.yoctoproject.org/all/universal/54/fa/sstate:popt-native:x86_64-linux:1.19:r0:x86_64:14:54faa65665868426a9f37eec2276da4d0d818788bb5ec81714b2fee7836191b0_create_recipe_spdx.tar.zst

This sstate object is corrupted because it contains a
"recipe-deploy/x86_64/static/static-popt-native.spdx.json" containing 20704 NUL
bytes where a valid JSON is expected:
  ❯ zstd -d < sstate_popt-native_[...]_create_recipe_spdx.tar.zst | tar xf - recipe-deploy/x86_64/static/static-popt-native.spdx.json -O |hexdump 
  0000000 0000 0000 0000 0000 0000 0000 0000 0000
  *
  00050e0

Note: This "0" block is in fact a "hole" as in "sparse tar" feature.

Interestingly, 20704 is the size of the correct expected JSON (rebuilt locally):
  ❯ head correct/static-popt-native.spdx.json
  {
    "@context": "https://spdx.org/rdf/3.0.1/spdx-context.jsonld",
    "@graph": [
      {
        "type": "CreationInfo",
        "@id": "_:CreationInfo0",
        "created": "2026-10-08T18:44:49Z",
        "createdBy": [
          "http://spdx.org/spdxdocs/3f48ce2be4187531be6c5eeb177ec40304d25644/bitbake-addba517-4804-5ae3-87c2-0c3a1a5812ba/bitbake/agent/John_Doe"
        ],
  
  ❯ stat correct/static-popt-native.spdx.json -c %s
  20704

The impact of this corruption can be seen in JSON parsing failure in SPDX related tasks:
  https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/5020
  https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/5026
  https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/5027
  https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/4834
  https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/4840
Logs:
  ERROR: rpm-native-1_4.20.1-r0 do_create_recipe_spdx: Error executing a python function in exec_func_python() autogenerated:
  [...]
  Exception: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Comment 1 Yoann Congal 2026-10-08 19:26:14 UTC

Created attachment 5253 [details]
The correct JSON file

Comment 2 Yoann Congal 2026-10-08 22:37:09 UTC

I now believe this is a race-condition in ext4.

On a ext4 filesystem, under load (e.g. lots of sync()), there is a window of
time where a recently written+closed file reports "stat.st_blocks = 0".
See this reproducer: https://gist.github.com/ycongal-smile/ee5c363a10f12fc014262930369fbb6f
  ❯ python3 st_blocks_race.py 
  347/1000 files holding 20704 bytes reported st_blocks == 0

Then, "tar --sparse" (that we use in sstate) optimize fully sparse files that
have "st_size != 0" but no blocks (stat.st_blocks == 0)[1]. If tar does this check
in the race window, then it will store into the archive a file that is just a
sparse hole of the size of the file.
Which is exactly what we observe.

Neither tar nor the kernel seem to have a fix/workaround.

Note: The same bug on btrfs: https://lists.gnu.org/archive/html/bug-tar/2016-07/msg00000.html
 * There are POSIX-level debate whether a filesystem can report st_blocks=0
   on written files.

[1]: https://cgit.git.savannah.gnu.org/cgit/tar.git/tree/src/sparse.c#n269

Comment 3 Yoann Congal 2026-10-09 11:43:38 UTC

From a failing build:
oe-selftest-debian debian12-vk-6 wrynose completed at 2026-10-08 16:56:12+00:00
https://autobuilder.yoctoproject.org/valkyrie/?#/builders/35/builds/5027

If I prevent the corrupted sstate object to be reused with
https://git.openembedded.org/openembedded-core-contrib/commit/?h=stable/wrynose-nut&id=ecd7d7370301218f89ce460e393cb48978f15673
meta/recipes-support/popt/popt_1.19.bb:
+# XXX to workaround a corrupted sstate object (BZ#16446)
+PR = "r1"
+HASHEQUIV_HASH_VERSION .= ".1"  

Build succeed:
oe-selftest-debian debian13-vk-2 wrynose completed at 2026-10-09 10:13:12+00:00
https://autobuilder.yoctoproject.org/valkyrie/?#/builders/35/builds/5030

Comment 4 Yoann Congal 2026-10-09 13:21:17 UTC

I removed the corrupted sstate object from the shared sstate AB directory