15148 2023-06-20 15:03:58 +0000 Is uninative-tarball reproducible? 2024-03-22 17:58:28 +0000 1 1 7 Build System, Metadata & Runtime OE-Core core 4.1.4 All arm RESOLVED FIXED Medium+ normal 5.1 1 pokylinux richard.purdie JPEWhacker meta.mr.watcher meta.watcher randy.macleod richard.purdie --- --- Don't know oldest_to_newest 95754 0 pokylinux 2023-06-20 15:03:58 +0000 I created a new workflow with github/gitea actions and observed, that whenever I bitbake unintaive-tarball a different sha256sum is created. Is this a known problem? I will switch to something closer to master and give it another try. 95772 1 richard.purdie 2023-06-21 09:27:17 +0000 It isn't a known issue, no. I'd be interested to understand a bit more about where the differences are inside the tarballs. diffoscope is good at helping identify those issues. Please do let us know if that happens with master too. 95797 2 randy.macleod 2023-06-22 14:36:49 +0000 Please run the tests that Richard requested.- YP bug review 95825 3 pokylinux 2023-06-27 06:40:24 +0000 Maybe my test case is flawed and I need SSTATE involved? Here is what I do: 1) uninative-tarball --runall=fetch 2) create a tarball from the sources and add them to some web server 3) site.conf excerpt: UNINATIVE_TARBALL="" UNINATIVE_CHECKSUM[x86_64]="954182f691bb2dbae157ee991654ad2fd4cb51c7f3d3ab429e9f84654c8dc990" INHERIT += "own-mirrors" SOURCE_MIRROR_URL = "file:///workspace/rber/yocto-build-docker-action/workdir/source_mirror/" SRCREV:pn-ascii-invaders = "2644be38afc06f7f7bba55034966dd66d9ad524f" BB_NO_NETWORK = "1 4) local SSTATE_DIR, DL_DIR empty, 5) bitbake uninative-tarball If I do that 2 times I get different results. === The first problem I detected was, that if I INHERIT += "image-buildinfo" it creates in the tarball a buildinfo file and DATETIME = 20230624080445 differs with different builds. So I removed the image-buildinfo class from my configuration. === Still the checksum differs. I tried with diffoscope, but it seems to run out of memory all the time (magic.MagicException: b'Output buffer space exceeded 1038+20') Here[1] is some diffoscope output when I only compare the etc: [1] https://pastebin.com/9TKsfach It's not master (yet) I will try it as soon as I find some more time, but here is my build output attached[2]. [2] https://drive.google.com/drive/folders/1Y1vKYCi-4TEyRqlcnyHhPeVcn-wUpzSb?usp=sharing 95827 4 pokylinux 2023-06-27 15:45:17 +0000 I am staring to suspect that something is wrong with the xz compression ;) 1) md5sum on unpacked files: if [ -d unzip ]; then rm -rf unzip fi mkdir unzip pushd unzip tar xf ../x86_64-nativesdk-libc.tar.xz find -type f \( -not -name "md5sum.txt" \) -exec md5sum '{}' \; > md5sum.txt mv md5sum.txt ../../3.7-md5sum.txt popd If I diff the md5sums they are identical ;) 2) filesize is different: + diff 3.7-ls 3.7.ori-ls 1c1 < -rw-r--r-- 1 rber rber 10517496 Jun 26 19:30 x86_64-nativesdk-libc.tar.xz --- > -rw-r--r-- 1 rber rber 10472856 Jun 26 18:57 x86_64-nativesdk-libc.tar.xz 3) sha256sum is different + diff 3.7-sha256sum 3.7.ori-sha256sum 1c1 < 0d2fbb42ac43942f79f01228e9178fa020b9349c607d041020aafe72de828698 x86_64-nativesdk-libc.tar.xz --- > 237fb8bc21f5ab092c590bfbb0300a57d161af78317e05d4c81bb5dbaf416520 x86_64-nativesdk-libc.tar.xz 4) xz analysis is different: + diff 3.7-xz-lvv 3.7.ori-xz-lvv 4c4 < Compressed size: 10.0 MiB (10,517,496 B) --- > Compressed size: 10.0 MiB (10,472,856 B) 6c6 < Ratio: 0.020 --- > Ratio: 0.019 11c11 < 1 3 0 0 10,517,496 537,323,520 0.020 CRC64 0 --- > 1 3 0 0 10,472,856 537,323,520 0.019 CRC64 0 14,16c14,16 < 1 1 12 0 2,838,320 201,326,592 0.014 CRC64 52bfb8d78735c230 20 cu 2,838,290 65 MiB --lzma2=dict=64MiB < 1 2 2,838,332 201,326,592 3,000,080 201,326,592 0.015 CRC64 bec544b6964ca059 20 cu 3,000,049 65 MiB --lzma2=dict=64MiB < 1 3 5,838,412 402,653,184 4,679,040 134,670,336 0.035 CRC64 15177e5b5aee9516 20 cu 4,679,012 65 MiB --lzma2=dict=64MiB --- > 1 1 12 0 2,815,944 201,326,592 0.014 CRC64 1294ec05da356697 20 cu 2,815,913 65 MiB --lzma2=dict=64MiB > 1 2 2,815,956 201,326,592 2,978,612 201,326,592 0.015 CRC64 a0dcc97d77de4299 20 cu 2,978,583 65 MiB --lzma2=dict=64MiB > 1 3 5,794,568 402,653,184 4,678,244 134,670,336 0.035 CRC64 7a5c0920eedcf24a 20 cu 4,678,215 65 MiB --lzma2=dict=64MiB full report: + cat 3.7-xz-lvv x86_64-nativesdk-libc.tar.xz (1/1) Streams: 1 Blocks: 3 Compressed size: 10.0 MiB (10,517,496 B) Uncompressed size: 512.4 MiB (537,323,520 B) Ratio: 0.020 Check: CRC64 Stream padding: 0 B Streams: Stream Blocks CompOffset UncompOffset CompSize UncompSize Ratio Check Padding 1 3 0 0 10,517,496 537,323,520 0.020 CRC64 0 Blocks: Stream Block CompOffset UncompOffset TotalSize UncompSize Ratio Check CheckVal Header Flags CompSize MemUsage Filters 1 1 12 0 2,838,320 201,326,592 0.014 CRC64 52bfb8d78735c230 20 cu 2,838,290 65 MiB --lzma2=dict=64MiB 1 2 2,838,332 201,326,592 3,000,080 201,326,592 0.015 CRC64 bec544b6964ca059 20 cu 3,000,049 65 MiB --lzma2=dict=64MiB 1 3 5,838,412 402,653,184 4,679,040 134,670,336 0.035 CRC64 15177e5b5aee9516 20 cu 4,679,012 65 MiB --lzma2=dict=64MiB Memory needed: 65 MiB Sizes in headers: Yes Minimum XZ Utils version: 5.0.0 + cat 3.7.ori-xz-lvv x86_64-nativesdk-libc.tar.xz (1/1) Streams: 1 Blocks: 3 Compressed size: 10.0 MiB (10,472,856 B) Uncompressed size: 512.4 MiB (537,323,520 B) Ratio: 0.019 Check: CRC64 Stream padding: 0 B Streams: Stream Blocks CompOffset UncompOffset CompSize UncompSize Ratio Check Padding 1 3 0 0 10,472,856 537,323,520 0.019 CRC64 0 Blocks: Stream Block CompOffset UncompOffset TotalSize UncompSize Ratio Check CheckVal Header Flags CompSize MemUsage Filters 1 1 12 0 2,815,944 201,326,592 0.014 CRC64 1294ec05da356697 20 cu 2,815,913 65 MiB --lzma2=dict=64MiB 1 2 2,815,956 201,326,592 2,978,612 201,326,592 0.015 CRC64 a0dcc97d77de4299 20 cu 2,978,583 65 MiB --lzma2=dict=64MiB 1 3 5,794,568 402,653,184 4,678,244 134,670,336 0.035 CRC64 7a5c0920eedcf24a 20 cu 4,678,215 65 MiB --lzma2=dict=64MiB Memory needed: 65 MiB Sizes in headers: Yes Minimum XZ Utils version: 5.0.0 95835 5 pokylinux 2023-06-29 10:21:00 +0000 In order to get a reproducible tar archive we need to take care about *) owner *) group *) timestamps Owner and group are taking care of[1] like that: SDKTAROPTS = "--owner=root --group=root" But timestamps are not! Something like this would fix it: SDKTAROPTS_ADDON ?= "--mtime='1970-01-01 00:00:00'" d.setVar('SDK_ARCHIVE_CMD', 'cd ${SDK_OUTPUT}/${SDKPATH}; tar ${SDKTAROPTS} ${SDKTAROPTS_ADDON} -cf - . | xz ${SDK_XZ_OPTIONS} > ${SDKDEPLOYDIR}/${TOOLCHAIN_OUTPUTNAME}.${SDK_ARCHIVE_TYPE}') [1] https://git.yoctoproject.org/poky/tree/meta/classes-recipe/populate_sdk_base.bbclass 95836 6 richard.purdie 2023-06-29 10:50:59 +0000 For mtime we often do things like "--clamp-mtime --mtime=@${SOURCE_DATE_EPOCH}", would that work here? 95837 7 pokylinux 2023-06-29 11:31:08 +0000 @RP: I will give it a try. 95838 8 pokylinux 2023-06-29 12:52:18 +0000 it seems to do the trick: cat 3.7/sha256sum.txt 772ed808389b3e3051c69e58e2b63c022ce5c7d5d241b1fb26ca465c0089a94e cat 3.7.ori/sha256sum.txt 772ed808389b3e3051c69e58e2b63c022ce5c7d5d241b1fb26ca465c0089a94e I will create a patch and send it upstream. 95840 9 pokylinux 2023-06-29 14:06:11 +0000 patch: https://lists.openembedded.org/g/openembedded-core/message/183628 95844 10 randy.macleod 2023-06-29 14:44:45 +0000 Underway on the email list. Thanks Robert! -- YP bug review 95870 11 pokylinux 2023-06-30 07:00:13 +0000 https://patchwork.yoctoproject.org/project/oe-core/patch/20230629140258.5635-1-Robert.Berger@ReliableEmbeddedSystems.com/ 95873 12 pokylinux 2023-06-30 13:18:23 +0000 v2 is on it's way: https://lists.openembedded.org/g/openembedded-core/message/183694 95874 13 pokylinux 2023-06-30 15:42:25 +0000 https://patchwork.yoctoproject.org/project/oe-core/patch/20230630131430.5051-1-Robert.Berger@ReliableEmbeddedSystems.com/ 96626 14 randy.macleod 2023-10-05 15:01:06 +0000 Oneline change patch on list. To be merged. 97705 15 pokylinux 2023-12-22 12:16:09 +0000 My understanding is, that this patch is OK and should go upstream. It it trapped since 2023-06-30 in patchwork. Is there anything else I should/can do about it to get it upstream? ... and .. if we believe what the doc says[1] and the uninative tarball is rebuilt whenever the eSDK is built and included in the eSDK, than the eSDK is non reproducible as well without this patch. [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=13920 98069 16 pokylinux 2024-01-31 12:00:41 +0000 It looks like it made it upstream[1] [1] https://git.yoctoproject.org/poky/tree/meta/classes-recipe/populate_sdk_base.bbclass#n288 So we can close it, I guess. 98593 17 richard.purdie 2024-03-22 17:58:28 +0000 https://git.yoctoproject.org/poky/commit/?id=23549b87c597f1c7b8492dbdacaf88e4e4b6ee76