If you try to build the same image on 2 different builders and compare files-in-image.txt buildhistory reports, you'll find that there are many small size changes in binaries. At least some of these random changes are caused by random-seed used by toolchain to generate symbol names (i.e. different builds having different lenghts of randomly generated symbol names cause few byte difference in binary size) http://blog.mindfab.net/2013/12/on-way-to-deterministic-binariy-gcc.html The problem is how to pass -frandom-seed=$(shell md5sum $< | sed 's/\(.*\) .*/\1/') in ideally all components without modifying them all. Putting that just to TUNE_CCARGS wont work because it cannot expand Makefile's $<. I was even thinking about patching our gcc-cross to just do this automagically. Any other suggestions? Does anyone know about some work in (gcc) upstream to make this easier (-frandom-seed-from-md5 would be nice).
this is only for C++ code btw. and uniqueness has to be preserved across a single build. Using md5 is way too much heavyduty, easiest is to use the relative source filename, however, I dont think there is anywork in gcc being done in this regard, secondly you could have two files with same name sitting in different directories and cd into the directory before building it. So even if gcc tried to use source file name it will fail, now one could say then use full path but that defeats the purpose too since if you change your builddir the seeds will change too. so may be for us the approach left is to identify which packages are exhibiting the behavior and fix them one by one.
I've tried appending "-frandom-seed=1" to TUNE_CCARGS just to see if this is the only reason for different filesizes (I know that passing constant isn't correct solution). But even with this, the filesizes are different, one of the reasons is that cmake's link command will generate different order of *.o files for final binary linking which also seems to influence the final file size. Small improvement is that I was able to generate the same binary (same md5) on 2 different builders when I've also used the same final link command.
*** Bug 4731 has been marked as a duplicate of this bug. ***
Debian has interesting project to generate reproducible builds: https://wiki.debian.org/ReproducibleBuilds
There is a bit of ambiguity to this bug since things like paths in dwarf sections will also cause binary differences. Also will we use the build-compare tool to test for "real" differences in binaries? Or are we only satisfied with exact matches in file size in the buildhistory reports?
(In reply to comment #5) > There is a bit of ambiguity to this bug since things like paths in dwarf > sections will also cause binary differences. > > Also will we use the build-compare tool to test for "real" differences in > binaries? Or are we only satisfied with exact matches in file size in the > buildhistory reports? You're right, we need to put some scope on this problem. For the purposes of this bug and a first round of development, I'd be happy to see that we have basic "identical" binaries as defined by build-compare and ignoring DWARF sections. I don't think its reasonable to have removed all differences but we should perhaps catalogue the list of them for OE-Core.
(In reply to comment #6) > You're right, we need to put some scope on this problem. For the purposes of > this bug and a first round of development, I'd be happy to see that we have > basic "identical" binaries as defined by build-compare and ignoring DWARF > sections. I don't think its reasonable to have removed all differences but > we should perhaps catalogue the list of them for OE-Core. To be clear, I realise there are some things we can perhaps do with DWARF sections too but that should be a separate bug.
Clearing 2.0 milestone
debbindiff - now renamed to diffoscope shows the differences quite nicely {noformat} martin@ljama:~$ diffoscope libcrypto.so.1.0.0.BUILD.PLAT-17601-* --- libcrypto.so.1.0.0.BUILD.PLAT-17601-1 +++ libcrypto.so.1.0.0.BUILD.PLAT-17601-3 ├── readelf --wide --notes {} │ @@ -1,5 +1,5 @@ │ │ Displaying notes found at file offset 0x000000f4 with length 0x00000024: │ Owner Data size Description │ GNU 0x00000014 NT_GNU_BUILD_ID (unique build ID bitstring) │ - Build ID: f817e0c916f5820025a29ebfe7fc2fbd78a19466 │ + Build ID: 2a721745d4c16c6cdbef1cbbf232762463fa13bb ├── readelf --wide --decompress --hex-dump=.rodata {} │ @@ -67,15 +67,15 @@ │ 0x000bf360 5f424e5f 41534d5f 4746326d 202d4453 _BN_ASM_GF2m -DS │ 0x000bf370 4841315f 41534d20 2d445348 41323536 HA1_ASM -DSHA256 │ 0x000bf380 5f41534d 202d4453 48413531 325f4153 _ASM -DSHA512_AS │ 0x000bf390 4d202d44 4145535f 41534d20 2d444748 M -DAES_ASM -DGH │ 0x000bf3a0 4153485f 41534d00 4f70656e 53534c20 ASH_ASM.OpenSSL │ 0x000bf3b0 312e302e 31702039 204a756c 20323031 1.0.1p 9 Jul 201 │ 0x000bf3c0 35006275 696c7420 6f6e3a20 54756520 5.built on: Tue │ - 0x000bf3d0 4d617220 20382032 303a3030 3a333720 Mar 8 20:00:37 │ + 0x000bf3d0 4d617220 20382032 323a3334 3a303420 Mar 8 22:34:04 │ 0x000bf3e0 32303136 00706c61 74666f72 6d3a206c 2016.platform: l │ 0x000bf3f0 696e7578 2d61726d 7634004f 50454e53 inux-armv4.OPENS │ 0x000bf400 534c4449 523a2022 2f757372 2f6c6962 SLDIR: "/usr/lib │ 0x000bf410 2f73736c 22006578 5f646174 612e6300 /ssl".ex_data.c. │ 0x000bf420 66697073 206d6f64 65206e6f 74207375 fips mode not su │ 0x000bf430 70706f72 74656400 6e6f2064 796e6c6f pported.no dynlo │ 0x000bf440 636b2063 72656174 65206361 6c6c6261 ck create callba ├── readelf --wide --decompress --hex-dump=.gnu_debuglink {} │ @@ -1,5 +1,5 @@ │ │ Hex dump of section '.gnu_debuglink': │ 0x00000000 6c696263 72797074 6f2e736f 2e312e30 libcrypto.so.1.0 │ - 0x00000010 2e300000 1cfb126f .0.....o │ + 0x00000010 2e300000 acc4be31 .0.....1 │ ╵ {noformat}
it will be good to have a global level tool/script which I could run on rootfs and it will run diffoscope on all binary artifacts and produce an output like this binary/library has an issue and it belongs to this package which came from this recipe. May be a build QA check which is optional to begin with. Then we can open multiple tickets to fix the packages one by one.
*** Bug 10705 has been marked as a duplicate of this bug. ***
There are several things which end up going into build output which may be useful to virtualize so that it is consistent between machines and different users: - Build directory (can be seen embedded in shared libraries from /lib using grep) - Username (cat /proc/version) - Hostname (cat /proc/version) - Timezone (cat /proc/version) Virtualizing the build directory may be useful to allow moving the build directory to another location without invalidating the sstate-cache.
adding Sonia and José to the bug as this would need testing additions.
Created attachment 3657 [details] class needed to create reproducible binaries
As a part of initial investigation, I wanted to binary compare packages created on a single machine but built in two different folders. I did : local conf: PACKAGE_CLASSES ?= "package_rpm package_deb package_ipk" $ bitbake core-image-minimal And binary compared individual packages (3868 in each case) core-image-minimal (default) ============================ IPK RPM DEB ---------------- Same: 0 0 25 Diff: 3868 3868 3843 Total: 3868 3868 3868 Following some basic recommendations for reproducible binaries, I created a "reproducible.bbclass" (attached) and re-built the core-image-minimal: PACKAGE_CLASSES ?= "package_rpm package_deb package_ipk" INHERIT += "reproducible" And got these results: core-image-minimal (with INHERIT +="reproducible") ================================================== IPK RPM DEB ---------------- Same: 0 530 3617 Diff: 3868 3338 251 Total: 3868 3868 3868 Obviously, this is still work in progress.
Comparison of packages built on two different systems (Ubuntu 16 vs container with Ubuntu 14) commit 9fe7a69535f9443175da8289633a761f58c372ff core-image-minimal (with INHERIT +="reproducible") ================================================== IPK RPM DEB ---------------- Same: 0 0 3595 Diff: 3859 3859 264 Total: 3859 3859 3859
(In reply to comment #16) > Comparison of packages built on two different systems (Ubuntu 16 vs > container with Ubuntu 14) > > commit 9fe7a69535f9443175da8289633a761f58c372ff > > > core-image-minimal (with INHERIT +="reproducible") > ================================================== > IPK RPM DEB > ---------------- > Same: 0 0 3595 > Diff: 3859 3859 264 > Total: 3859 3859 3859 SOURCE_DATE_EPOCH should be set to something may be to 0 to begin with but we need a better way to extract this value. We dont have changeling to record date so there must be another value may be use the last commit time/date for the recipe if using git otherwise, may be PR server can help
Based on https://reproducible-builds.org/specs/source-date-epoch/ The recommendation says: The value SHOULD be set to the last modification time of the source, incorporating any packaging-specific modifications. For example, in Debian, the timestamp of the latest entry in debian/changelog. In my testing I set the value based on the repo git last entry, applied globally. Ideally, we would modify individual recipes themselves based on their respective source date.
Is following sequence of patchwork IDs the latest and complete set of changes which are currently available? 139554 139558 139556 139555 139557 139559 139599 139600 139999
I keep the latest patches here: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=jurob/reproducible_binaries Also, I typically use these settings: BUILD_REPRODUCIBLE_BINARIES="1" REPRODUCIBLE_TIMESTAMP_ROOTFS="1483228802" LDCONFIGDEPEND="" IMAGE_DEPENDS_cpio_append=" cpio-replacement-native" EXTRANATIVEPATH+="cpio-native" IMAGE_CMD_CPIO="cpio --ignore-devno --reproducible " IMAGE_CMD_TAR="tar -v --sort=name " PACKAGE_CLASSES="package_rpm package_deb package_ipk" IMAGE_FSTYPES+="jffs2 cpio hdddirect qcow2 iso hddimg" # optionally DISTRO_FEATURES_append=" ptest" EXTRA_IMAGE_FEATURES+="ptest-pkgs" INHERIT+="rm_work" Once the patches are more stable and there is some consensus/feedback I will send them to the mailing list again. With the settings/patches above the latest stats are as follows: core-image-sato ======================== IPK RPM DEB ---------------- Same: 20 777 6883 Diff: 6955 6198 92 Total: 6975 6975 6975
I've tried to use all the changes from your poky-contrib branch: 7c5e8d3c6d kernel.bbclass: improve reproducibility 565607bd2c gdbm: improve reproducibility 023b035b65 bash: improve reproducibility 3423049c8d python3: improve reproducibility ea5e8ec943 python2.7: improve reproducibility 6894ffbc71 image_types.bbclass: improve cpio image reproducibility 6b8e01d209 cpio: provide cpio-replacement-native 88a42c2bec image.bbclass: support binary reproducibility 265dc4e80c busybox.inc: improve reproducibility a76225f28b rootfs-postcommands.bbclass: support binary reproducibility f3392be6a7 image-prelink.bbclass: support binary reproducibility aa714d6efd reproducible_build.bbclass: initial support for binary reproducibility and even before "enabling" them by setting those new variables, I've tried to run: bitbake -k core-image-base 2>&1 | tee log.qemux86 delete sstate-cache and again bitbake -k core-image-base 2>&1 | tee log.qemux86 surprisingly there wasn't as many changes in buildhistory as I was used to before, will try to redo this test on 2 different hosts. One issue still shown in buildhistory is locale generation difference: -- a/images/qemux86/glibc/core-image-base/files-in-image.txt +++ b/images/qemux86/glibc/core-image-base/files-in-image.txt @@ -2142,7 +2142,7 @@ drwxr-xr-x root root 20480 ./var/lib/opkg/info -rw-r--r-- root root 1083 ./var/lib/opkg/info/avahi-locale-cs.control -rw-r--r-- root root 42 ./var/lib/opkg/info/avahi-locale-cs.list -rw-r--r-- root root 1098 ./var/lib/opkg/info/avahi-locale-en-gb.control --rw-r--r-- root root 107 ./var/lib/opkg/info/avahi-locale-en-gb.list +-rw-r--r-- root root 45 ./var/lib/opkg/info/avahi-locale-en-gb.list -rw-r--r-- root root 79 ./var/lib/opkg/info/base-files.conffiles -rw-r--r-- root root 640 ./var/lib/opkg/info/base-files.control -rw-r--r-- root root 578 ./var/lib/opkg/info/base-files.list @@ -2191,7 +2191,7 @@ drwxr-xr-x root root 20480 ./var/lib/opkg/info -rw-r--r-- root root 11757 ./var/lib/opkg/info/kbd-keymaps.list -rw-r--r-- root root 515 ./var/lib/opkg/info/kbd.list -rw-r--r-- root root 565 ./var/lib/opkg/info/kbd-locale-cs.control --rw-r--r-- root root 40 ./var/lib/opkg/info/kbd-locale-cs.list +-rw-r--r-- root root 115 ./var/lib/opkg/info/kbd-locale-cs.list -rwxr-xr-x root root 400 ./var/lib/opkg/info/kbd.postinst -rwxr-xr-x root root 316 ./var/lib/opkg/info/kbd.prerm -rw-r--r-- root root 543 ./var/lib/opkg/info/kernel-4.10.17-yocto-standard.control @@ -2286,9 +2286,9 @@ drwxr-xr-x root root 20480 ./var/lib/opkg/info -rwxr-xr-x root root 396 ./var/lib/opkg/info/libglib-2.0-0.postinst -rwxr-xr-x root root 316 ./var/lib/opkg/info/libglib-2.0-0.postrm -rw-r--r-- root root 1223 ./var/lib/opkg/info/libglib-2.0-locale-cs.control --rw-r--r-- root root 118 ./var/lib/opkg/info/libglib-2.0-locale-cs.list +-rw-r--r-- root root 43 ./var/lib/opkg/info/libglib-2.0-locale-cs.list -rw-r--r-- root root 1241 ./var/lib/opkg/info/libglib-2.0-locale-en-gb.control --rw-r--r-- root root 46 ./var/lib/opkg/info/libglib-2.0-locale-en-gb.list +-rw-r--r-- root root 108 ./var/lib/opkg/info/libglib-2.0-locale-en-gb.list -rw-r--r-- root root 585 ./var/lib/opkg/info/libgpg-error0.control -rw-r--r-- root root 59 ./var/lib/opkg/info/libgpg-error0.list -rwxr-xr-x root root 90 ./var/lib/opkg/info/libgpg-error0.postinst Have you seen this as well in your tests?
the "short" version of one of locale packages contains this: OE qemux86@ ~/build/oe-core $ cat tmp-glibc/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/var/lib/opkg/info/libglib-2.0-locale-cs.list /usr/share/locale/cs/LC_MESSAGES/glib20.mo I'll re-run the build to see if I can get the contents of "longer" version seen in previous build.
(In reply to comment #21) > I've tried to use all the changes from your poky-contrib branch: > > 7c5e8d3c6d kernel.bbclass: improve reproducibility > 565607bd2c gdbm: improve reproducibility > 023b035b65 bash: improve reproducibility > 3423049c8d python3: improve reproducibility > ea5e8ec943 python2.7: improve reproducibility > 6894ffbc71 image_types.bbclass: improve cpio image reproducibility > 6b8e01d209 cpio: provide cpio-replacement-native > 88a42c2bec image.bbclass: support binary reproducibility > 265dc4e80c busybox.inc: improve reproducibility > a76225f28b rootfs-postcommands.bbclass: support binary reproducibility > f3392be6a7 image-prelink.bbclass: support binary reproducibility > aa714d6efd reproducible_build.bbclass: initial support for binary > reproducibility > > and even before "enabling" them by setting those new variables, I've tried > to run: > > bitbake -k core-image-base 2>&1 | tee log.qemux86 > > delete sstate-cache and again > > bitbake -k core-image-base 2>&1 | tee log.qemux86 > > surprisingly there wasn't as many changes in buildhistory as I was used to > before, will try to redo this test on 2 different hosts. > > One issue still shown in buildhistory is locale generation difference: > -- a/images/qemux86/glibc/core-image-base/files-in-image.txt > +++ b/images/qemux86/glibc/core-image-base/files-in-image.txt > @@ -2142,7 +2142,7 @@ drwxr-xr-x root root 20480 > ./var/lib/opkg/info > -rw-r--r-- root root 1083 > ./var/lib/opkg/info/avahi-locale-cs.control > -rw-r--r-- root root 42 > ./var/lib/opkg/info/avahi-locale-cs.list > -rw-r--r-- root root 1098 > ./var/lib/opkg/info/avahi-locale-en-gb.control > --rw-r--r-- root root 107 > ./var/lib/opkg/info/avahi-locale-en-gb.list > +-rw-r--r-- root root 45 > ./var/lib/opkg/info/avahi-locale-en-gb.list > -rw-r--r-- root root 79 > ./var/lib/opkg/info/base-files.conffiles > -rw-r--r-- root root 640 > ./var/lib/opkg/info/base-files.control > -rw-r--r-- root root 578 > ./var/lib/opkg/info/base-files.list > @@ -2191,7 +2191,7 @@ drwxr-xr-x root root 20480 > ./var/lib/opkg/info > -rw-r--r-- root root 11757 > ./var/lib/opkg/info/kbd-keymaps.list > -rw-r--r-- root root 515 ./var/lib/opkg/info/kbd.list > -rw-r--r-- root root 565 > ./var/lib/opkg/info/kbd-locale-cs.control > --rw-r--r-- root root 40 > ./var/lib/opkg/info/kbd-locale-cs.list > +-rw-r--r-- root root 115 > ./var/lib/opkg/info/kbd-locale-cs.list > -rwxr-xr-x root root 400 ./var/lib/opkg/info/kbd.postinst > -rwxr-xr-x root root 316 ./var/lib/opkg/info/kbd.prerm > -rw-r--r-- root root 543 > ./var/lib/opkg/info/kernel-4.10.17-yocto-standard.control > @@ -2286,9 +2286,9 @@ drwxr-xr-x root root 20480 > ./var/lib/opkg/info > -rwxr-xr-x root root 396 > ./var/lib/opkg/info/libglib-2.0-0.postinst > -rwxr-xr-x root root 316 > ./var/lib/opkg/info/libglib-2.0-0.postrm > -rw-r--r-- root root 1223 > ./var/lib/opkg/info/libglib-2.0-locale-cs.control > --rw-r--r-- root root 118 > ./var/lib/opkg/info/libglib-2.0-locale-cs.list > +-rw-r--r-- root root 43 > ./var/lib/opkg/info/libglib-2.0-locale-cs.list > -rw-r--r-- root root 1241 > ./var/lib/opkg/info/libglib-2.0-locale-en-gb.control > --rw-r--r-- root root 46 > ./var/lib/opkg/info/libglib-2.0-locale-en-gb.list > +-rw-r--r-- root root 108 > ./var/lib/opkg/info/libglib-2.0-locale-en-gb.list > -rw-r--r-- root root 585 > ./var/lib/opkg/info/libgpg-error0.control > -rw-r--r-- root root 59 > ./var/lib/opkg/info/libgpg-error0.list > -rwxr-xr-x root root 90 > ./var/lib/opkg/info/libgpg-error0.postinst > > Have you seen this as well in your tests? No, I have never noticed this one before, but thanks for pointing it out. There seem to be some elusive bugs in locale generation, often resulting with incorrect ownership (there is a bug filed regarding this). Ross also mentioned he noticed on ocassions whole blocks of locales missing... So this may give us more clues on what's wrong.
In next build I got "smaller" libglib-2.0-locale-en-gb.list (before it was about locale-cs.list) --rw-r--r-- root root 108 ./var/lib/opkg/info/libglib-2.0-locale-en-gb.list +-rw-r--r-- root root 46 ./var/lib/opkg/info/libglib-2.0-locale-en-gb.list cat ../tmp-glibc/A/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/libglib-2.0-locale-en-gb.list /usr/share/locale/en_GB/LC_MESSAGES/glib20.mo In the next build it was the bigger one again: --rw-r--r-- root root 46 ./var/lib/opkg/info/libglib-2.0-locale-en-gb.list +-rw-r--r-- root root 108 ./var/lib/opkg/info/libglib-2.0-locale-en-gb.list And the contents is: cat ../tmp-glibc/E/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./vaoe-linux/core-image-base/1.0-r0/rootfs/r/lib/opkg/info/libglib-2.0-locale-en-gb.list /usr/share/locale/en_GB/LC_MESSAGES/ /usr/share/locale/en_GB/LC_MESSAGES/glib20.mo /usr/share/locale/en_GB/ It seems to be caused by the order in which the locale packages are installed and which packages will end up "owning" the directory with given locale. Build A is the one with smaller libglib-2.0-locale-en-gb, build B the bigger: OE qemux86@ ~/build/oe-core/buildhistory $ grep "/usr/share/locale/en_GB/" ../tmp-glibc/A/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/*.list ../tmp-glibc/A/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/avahi-locale-en-gb.list:/usr/share/locale/en_GB/LC_MESSAGES/avahi.mo ../tmp-glibc/A/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/glibc-locale-en-gb.list:/usr/share/locale/en_GB/LC_MESSAGES/ ../tmp-glibc/A/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/glibc-locale-en-gb.list:/usr/share/locale/en_GB/ ../tmp-glibc/A/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/libglib-2.0-locale-en-gb.list:/usr/share/locale/en_GB/LC_MESSAGES/glib20.mo OE qemux86@ ~/build/oe-core/buildhistory $ grep "/usr/share/locale/en_GB/" ../tmp-glibc/B/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/*.list ../tmp-glibc/B/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/avahi-locale-en-gb.list:/usr/share/locale/en_GB/LC_MESSAGES/avahi.mo ../tmp-glibc/B/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/libglib-2.0-locale-en-gb.list:/usr/share/locale/en_GB/LC_MESSAGES/ ../tmp-glibc/B/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/libglib-2.0-locale-en-gb.list:/usr/share/locale/en_GB/LC_MESSAGES/glib20.mo ../tmp-glibc/B/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/libglib-2.0-locale-en-gb.list:/usr/share/locale/en_GB/
Another causes for random changes detected in 6 builds on the same host with the same metadata: 1) sbindir owned by base-files or libc6: diff -uNr ../tmp-glibc/F/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/base-files.list ../tmp-glibc/G/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/base-files.list --- ../tmp-glibc/F/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/base-files.list 2017-04-25 00:16:53.000000000 +0200 +++ ../tmp-glibc/G/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/base-files.list 2017-04-25 00:16:53.000000000 +0200 @@ -26,7 +26,6 @@ /etc/skel/ /etc/hostname /var/spool/ -/sbin/ /var/lib/ /var/lib/misc/ /var/volatile/tmp/ diff -uNr ../tmp-glibc/F/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/libc6.list ../tmp-glibc/G/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/libc6.list --- ../tmp-glibc/F/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/libc6.list 2017-04-25 00:16:53.000000000 +0200 +++ ../tmp-glibc/G/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/libc6.list 2017-04-25 00:16:53.000000000 +0200 @@ -19,6 +19,7 @@ /lib/librt.so.1 /lib/libm-2.25.so /lib/libpthread.so.0 +/sbin/ /lib/libdl.so.2 /lib/libnss_dns-2.25.so /lib/libresolv-2.25.so 2) EXTENDPRAUTO bumped from unknown reasons in linux-yocto: diff -uNr ../tmp-glibc/E/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/kernel-4.10.17-yocto-standard.control ../tmp-glibc/F/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/kernel-4.10.17-yocto-standard.control --- ../tmp-glibc/E/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/kernel-4.10.17-yocto-standard.control 2017-04-25 00:16:53.000000000 +0200 +++ ../tmp-glibc/F/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/kernel-4.10.17-yocto-standard.control 2017-04-25 00:16:53.000000000 +0200 @@ -1,5 +1,5 @@ Package: kernel-4.10.17-yocto-standard -Version: 4.10.17+git0+4b89f331d4_6648a34e00-r0.7 +Version: 4.10.17+git0+4b89f331d4_6648a34e00-r0.10 Description: Linux kernel Linux kernel. Section: kernel That results also in different status file sizes ls -la ../tmp-glibc/*/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/status -rw-r--r-- 1 bitbake bitbake 31605 Jun 16 16:33 ../tmp-glibc/A/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/status -rw-r--r-- 1 bitbake bitbake 31605 Jun 17 14:20 ../tmp-glibc/B/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/status -rw-r--r-- 1 bitbake bitbake 31626 Apr 25 00:16 ../tmp-glibc/E/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/status -rw-r--r-- 1 bitbake bitbake 31632 Apr 25 00:16 ../tmp-glibc/F/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/status -rw-r--r-- 1 bitbake bitbake 31632 Apr 25 00:16 ../tmp-glibc/G/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/status -rw-r--r-- 1 bitbake bitbake 31632 Apr 25 00:16 ../tmp-glibc/H/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/status And different EXTENDPRAUTO for packages which depend on it diff -uNr ../tmp-glibc/B/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/udev.control ../tmp-glibc/E/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/udev.control --- ../tmp-glibc/B/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/udev.control 2017-06-17 14:20:40.547985099 +0200 +++ ../tmp-glibc/E/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/udev.control 2017-04-25 00:16:53.000000000 +0200 @@ -1,5 +1,5 @@ Package: udev -Version: 1:232-r0.9 +Version: 1:232-r0.10 Description: A System and service manager systemd is a system and service manager for Linux, compatible with SysV and LSB init scripts. systemd provides aggressive parallelization But good thing is that I still don't see any binary differences from linker mangling I've used to see before.
Thanks, these are very useful observations. I have also noticed some weirdness recently: https://bugzilla.yoctoproject.org/show_bug.cgi?id=11551 And there is this one, probably related: https://bugzilla.yoctoproject.org/show_bug.cgi?id=11299 Correct me if I'm wrong, but my understanding is that none of the issues you mentioned is actually related to the patches for binary reproducibility, as you did not set BUILD_REPRODUCIBLE_BINARIES="1"
I did 6 builds from the same metadata (oe-core from last week + the changes from your poky-contrib branch). First 2 (in directories A and B) were built withoutBUILD_REPRODUCIBLE_BINARIES="1". Remaining 4 (E, F, G, H) were built after adding following to conf/local.conf: BUILD_REPRODUCIBLE_BINARIES = "1" REPRODUCIBLE_TIMESTAMP_ROOTFS = "1493072213" REPRODUCIBLE_TIMESTAMP_IMAGE_PRELINK = "1493072213" The issues I was seeing aren't imho caused by the reproducible changes. Main source of changes are PR service and order of package installation by opkg.
(In reply to comment #23) > (In reply to comment #21) > > I've tried to use all the changes from your poky-contrib branch: > > > > 7c5e8d3c6d kernel.bbclass: improve reproducibility > > 565607bd2c gdbm: improve reproducibility > > 023b035b65 bash: improve reproducibility > > 3423049c8d python3: improve reproducibility > > ea5e8ec943 python2.7: improve reproducibility > > 6894ffbc71 image_types.bbclass: improve cpio image reproducibility > > 6b8e01d209 cpio: provide cpio-replacement-native > > 88a42c2bec image.bbclass: support binary reproducibility > > 265dc4e80c busybox.inc: improve reproducibility > > a76225f28b rootfs-postcommands.bbclass: support binary reproducibility > > f3392be6a7 image-prelink.bbclass: support binary reproducibility > > aa714d6efd reproducible_build.bbclass: initial support for binary > > reproducibility > > > > and even before "enabling" them by setting those new variables, I've tried > > to run: > > > > bitbake -k core-image-base 2>&1 | tee log.qemux86 > > > > delete sstate-cache and again > > > > bitbake -k core-image-base 2>&1 | tee log.qemux86 > > > > surprisingly there wasn't as many changes in buildhistory as I was used to > > before, will try to redo this test on 2 different hosts. > > > > One issue still shown in buildhistory is locale generation difference: > > -- a/images/qemux86/glibc/core-image-base/files-in-image.txt > > +++ b/images/qemux86/glibc/core-image-base/files-in-image.txt > > @@ -2142,7 +2142,7 @@ drwxr-xr-x root root 20480 > > ./var/lib/opkg/info > > -rw-r--r-- root root 1083 > > ./var/lib/opkg/info/avahi-locale-cs.control > > -rw-r--r-- root root 42 > > ./var/lib/opkg/info/avahi-locale-cs.list > > -rw-r--r-- root root 1098 > > ./var/lib/opkg/info/avahi-locale-en-gb.control > > --rw-r--r-- root root 107 > > ./var/lib/opkg/info/avahi-locale-en-gb.list > > +-rw-r--r-- root root 45 > > ./var/lib/opkg/info/avahi-locale-en-gb.list > > -rw-r--r-- root root 79 > > ./var/lib/opkg/info/base-files.conffiles > > -rw-r--r-- root root 640 > > ./var/lib/opkg/info/base-files.control > > -rw-r--r-- root root 578 > > ./var/lib/opkg/info/base-files.list > > @@ -2191,7 +2191,7 @@ drwxr-xr-x root root 20480 > > ./var/lib/opkg/info > > -rw-r--r-- root root 11757 > > ./var/lib/opkg/info/kbd-keymaps.list > > -rw-r--r-- root root 515 ./var/lib/opkg/info/kbd.list > > -rw-r--r-- root root 565 > > ./var/lib/opkg/info/kbd-locale-cs.control > > --rw-r--r-- root root 40 > > ./var/lib/opkg/info/kbd-locale-cs.list > > +-rw-r--r-- root root 115 > > ./var/lib/opkg/info/kbd-locale-cs.list > > -rwxr-xr-x root root 400 ./var/lib/opkg/info/kbd.postinst > > -rwxr-xr-x root root 316 ./var/lib/opkg/info/kbd.prerm > > -rw-r--r-- root root 543 > > ./var/lib/opkg/info/kernel-4.10.17-yocto-standard.control > > @@ -2286,9 +2286,9 @@ drwxr-xr-x root root 20480 > > ./var/lib/opkg/info > > -rwxr-xr-x root root 396 > > ./var/lib/opkg/info/libglib-2.0-0.postinst > > -rwxr-xr-x root root 316 > > ./var/lib/opkg/info/libglib-2.0-0.postrm > > -rw-r--r-- root root 1223 > > ./var/lib/opkg/info/libglib-2.0-locale-cs.control > > --rw-r--r-- root root 118 > > ./var/lib/opkg/info/libglib-2.0-locale-cs.list > > +-rw-r--r-- root root 43 > > ./var/lib/opkg/info/libglib-2.0-locale-cs.list > > -rw-r--r-- root root 1241 > > ./var/lib/opkg/info/libglib-2.0-locale-en-gb.control > > --rw-r--r-- root root 46 > > ./var/lib/opkg/info/libglib-2.0-locale-en-gb.list > > +-rw-r--r-- root root 108 > > ./var/lib/opkg/info/libglib-2.0-locale-en-gb.list > > -rw-r--r-- root root 585 > > ./var/lib/opkg/info/libgpg-error0.control > > -rw-r--r-- root root 59 > > ./var/lib/opkg/info/libgpg-error0.list > > -rwxr-xr-x root root 90 > > ./var/lib/opkg/info/libgpg-error0.postinst > > > > Have you seen this as well in your tests? > > No, I have never noticed this one before, but thanks for pointing it out. > There seem to be some elusive bugs in locale generation, often resulting > with incorrect ownership (there is a bug filed regarding this). Ross also > mentioned he noticed on ocassions whole blocks of locales missing... So this > may give us more clues on what's wrong. Added Ross to CC list, hoping he'll be able to identify the root cause of the problem ...
Yes, glibc-locale is a nuisance to say the least.
Incremental progress. Moving to 2.5 M1.
The scope of this bug is not very well defined. Some of the initial issues have been resolved a while ago: 1. dwarf sections: reproducibility addressed fixed via new DEBUG_FLAGS -ffile-prefix-map 2. -frandom-seed=xxx : generally implemented AFAIK There have also been multiple patches addressing reproducibility of various packages already accepted. In addition there are several supplemental Bugzilla entries addressing binary reproducibility: https://bugzilla.yoctoproject.org/show_bug.cgi?id=11176 https://bugzilla.yoctoproject.org/show_bug.cgi?id=11177 https://bugzilla.yoctoproject.org/show_bug.cgi?id=11178 https://bugzilla.yoctoproject.org/show_bug.cgi?id=11179 I opened new entries https://bugzilla.yoctoproject.org/show_bug.cgi?id=12421 https://bugzilla.yoctoproject.org/show_bug.cgi?id=12422 (With more to follow.) So I will close this one particular Bugzilla entry as RESOLVED/OBSOLETE as it is being replaced/superseded by several more granulated entries. If there is a particular issue I may have missed which is present here but not addressed in the entries above, please open a new Bugzilla entry.