I created a new workflow with github/gitea actions and observed, that whenever I bitbake unintaive-tarball a different sha256sum is created. Is this a known problem? I will switch to something closer to master and give it another try.
It isn't a known issue, no. I'd be interested to understand a bit more about where the differences are inside the tarballs. diffoscope is good at helping identify those issues. Please do let us know if that happens with master too.
Please run the tests that Richard requested.- YP bug review
Maybe my test case is flawed and I need SSTATE involved? Here is what I do: 1) uninative-tarball --runall=fetch 2) create a tarball from the sources and add them to some web server 3) site.conf excerpt: UNINATIVE_TARBALL="" UNINATIVE_CHECKSUM[x86_64]="954182f691bb2dbae157ee991654ad2fd4cb51c7f3d3ab429e9f84654c8dc990" INHERIT += "own-mirrors" SOURCE_MIRROR_URL = "file:///workspace/rber/yocto-build-docker-action/workdir/source_mirror/" SRCREV:pn-ascii-invaders = "2644be38afc06f7f7bba55034966dd66d9ad524f" BB_NO_NETWORK = "1 4) local SSTATE_DIR, DL_DIR empty, 5) bitbake uninative-tarball If I do that 2 times I get different results. === The first problem I detected was, that if I INHERIT += "image-buildinfo" it creates in the tarball a buildinfo file and DATETIME = 20230624080445 differs with different builds. So I removed the image-buildinfo class from my configuration. === Still the checksum differs. I tried with diffoscope, but it seems to run out of memory all the time (magic.MagicException: b'Output buffer space exceeded 1038+20') Here[1] is some diffoscope output when I only compare the etc: [1] https://pastebin.com/9TKsfach It's not master (yet) I will try it as soon as I find some more time, but here is my build output attached[2]. [2] https://drive.google.com/drive/folders/1Y1vKYCi-4TEyRqlcnyHhPeVcn-wUpzSb?usp=sharing
I am staring to suspect that something is wrong with the xz compression ;) 1) md5sum on unpacked files: if [ -d unzip ]; then rm -rf unzip fi mkdir unzip pushd unzip tar xf ../x86_64-nativesdk-libc.tar.xz find -type f \( -not -name "md5sum.txt" \) -exec md5sum '{}' \; > md5sum.txt mv md5sum.txt ../../3.7-md5sum.txt popd If I diff the md5sums they are identical ;) 2) filesize is different: + diff 3.7-ls 3.7.ori-ls 1c1 < -rw-r--r-- 1 rber rber 10517496 Jun 26 19:30 x86_64-nativesdk-libc.tar.xz --- > -rw-r--r-- 1 rber rber 10472856 Jun 26 18:57 x86_64-nativesdk-libc.tar.xz 3) sha256sum is different + diff 3.7-sha256sum 3.7.ori-sha256sum 1c1 < 0d2fbb42ac43942f79f01228e9178fa020b9349c607d041020aafe72de828698 x86_64-nativesdk-libc.tar.xz --- > 237fb8bc21f5ab092c590bfbb0300a57d161af78317e05d4c81bb5dbaf416520 x86_64-nativesdk-libc.tar.xz 4) xz analysis is different: + diff 3.7-xz-lvv 3.7.ori-xz-lvv 4c4 < Compressed size: 10.0 MiB (10,517,496 B) --- > Compressed size: 10.0 MiB (10,472,856 B) 6c6 < Ratio: 0.020 --- > Ratio: 0.019 11c11 < 1 3 0 0 10,517,496 537,323,520 0.020 CRC64 0 --- > 1 3 0 0 10,472,856 537,323,520 0.019 CRC64 0 14,16c14,16 < 1 1 12 0 2,838,320 201,326,592 0.014 CRC64 52bfb8d78735c230 20 cu 2,838,290 65 MiB --lzma2=dict=64MiB < 1 2 2,838,332 201,326,592 3,000,080 201,326,592 0.015 CRC64 bec544b6964ca059 20 cu 3,000,049 65 MiB --lzma2=dict=64MiB < 1 3 5,838,412 402,653,184 4,679,040 134,670,336 0.035 CRC64 15177e5b5aee9516 20 cu 4,679,012 65 MiB --lzma2=dict=64MiB --- > 1 1 12 0 2,815,944 201,326,592 0.014 CRC64 1294ec05da356697 20 cu 2,815,913 65 MiB --lzma2=dict=64MiB > 1 2 2,815,956 201,326,592 2,978,612 201,326,592 0.015 CRC64 a0dcc97d77de4299 20 cu 2,978,583 65 MiB --lzma2=dict=64MiB > 1 3 5,794,568 402,653,184 4,678,244 134,670,336 0.035 CRC64 7a5c0920eedcf24a 20 cu 4,678,215 65 MiB --lzma2=dict=64MiB full report: + cat 3.7-xz-lvv x86_64-nativesdk-libc.tar.xz (1/1) Streams: 1 Blocks: 3 Compressed size: 10.0 MiB (10,517,496 B) Uncompressed size: 512.4 MiB (537,323,520 B) Ratio: 0.020 Check: CRC64 Stream padding: 0 B Streams: Stream Blocks CompOffset UncompOffset CompSize UncompSize Ratio Check Padding 1 3 0 0 10,517,496 537,323,520 0.020 CRC64 0 Blocks: Stream Block CompOffset UncompOffset TotalSize UncompSize Ratio Check CheckVal Header Flags CompSize MemUsage Filters 1 1 12 0 2,838,320 201,326,592 0.014 CRC64 52bfb8d78735c230 20 cu 2,838,290 65 MiB --lzma2=dict=64MiB 1 2 2,838,332 201,326,592 3,000,080 201,326,592 0.015 CRC64 bec544b6964ca059 20 cu 3,000,049 65 MiB --lzma2=dict=64MiB 1 3 5,838,412 402,653,184 4,679,040 134,670,336 0.035 CRC64 15177e5b5aee9516 20 cu 4,679,012 65 MiB --lzma2=dict=64MiB Memory needed: 65 MiB Sizes in headers: Yes Minimum XZ Utils version: 5.0.0 + cat 3.7.ori-xz-lvv x86_64-nativesdk-libc.tar.xz (1/1) Streams: 1 Blocks: 3 Compressed size: 10.0 MiB (10,472,856 B) Uncompressed size: 512.4 MiB (537,323,520 B) Ratio: 0.019 Check: CRC64 Stream padding: 0 B Streams: Stream Blocks CompOffset UncompOffset CompSize UncompSize Ratio Check Padding 1 3 0 0 10,472,856 537,323,520 0.019 CRC64 0 Blocks: Stream Block CompOffset UncompOffset TotalSize UncompSize Ratio Check CheckVal Header Flags CompSize MemUsage Filters 1 1 12 0 2,815,944 201,326,592 0.014 CRC64 1294ec05da356697 20 cu 2,815,913 65 MiB --lzma2=dict=64MiB 1 2 2,815,956 201,326,592 2,978,612 201,326,592 0.015 CRC64 a0dcc97d77de4299 20 cu 2,978,583 65 MiB --lzma2=dict=64MiB 1 3 5,794,568 402,653,184 4,678,244 134,670,336 0.035 CRC64 7a5c0920eedcf24a 20 cu 4,678,215 65 MiB --lzma2=dict=64MiB Memory needed: 65 MiB Sizes in headers: Yes Minimum XZ Utils version: 5.0.0
In order to get a reproducible tar archive we need to take care about *) owner *) group *) timestamps Owner and group are taking care of[1] like that: SDKTAROPTS = "--owner=root --group=root" But timestamps are not! Something like this would fix it: SDKTAROPTS_ADDON ?= "--mtime='1970-01-01 00:00:00'" d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; tar ${SDKTAROPTS} ${SDKTAROPTS_ADDON} -cf - . | xz ${SDK_XZ_OPTIONS} > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE}') [1] https://git.yoctoproject.org/poky/tree/meta/classes-recipe/populate_sdk_base.bbclass
For mtime we often do things like "--clamp-mtime --mtime=@${SOURCE_DATE_EPOCH}", would that work here?
@RP: I will give it a try.
it seems to do the trick: cat 3.7/sha256sum.txt 772ed808389b3e3051c69e58e2b63c022ce5c7d5d241b1fb26ca465c0089a94e cat 3.7.ori/sha256sum.txt 772ed808389b3e3051c69e58e2b63c022ce5c7d5d241b1fb26ca465c0089a94e I will create a patch and send it upstream.
patch: https://lists.openembedded.org/g/openembedded-core/message/183628
Underway on the email list. Thanks Robert! -- YP bug review
https://patchwork.yoctoproject.org/project/oe-core/patch/20230629140258.5635-1-Robert.Berger@ReliableEmbeddedSystems.com/
v2 is on it's way: https://lists.openembedded.org/g/openembedded-core/message/183694
https://patchwork.yoctoproject.org/project/oe-core/patch/20230630131430.5051-1-Robert.Berger@ReliableEmbeddedSystems.com/
Oneline change patch on list. To be merged.
My understanding is, that this patch is OK and should go upstream. It it trapped since 2023-06-30 in patchwork. Is there anything else I should/can do about it to get it upstream? ... and .. if we believe what the doc says[1] and the uninative tarball is rebuilt whenever the eSDK is built and included in the eSDK, than the eSDK is non reproducible as well without this patch. [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=13920
It looks like it made it upstream[1] [1] https://git.yoctoproject.org/poky/tree/meta/classes-recipe/populate_sdk_base.bbclass#n288 So we can close it, I guess.
https://git.yoctoproject.org/poky/commit/?id=23549b87c597f1c7b8492dbdacaf88e4e4b6ee76