Bug 5866

Summary: Reproducible builds: identical binaries
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Martin Jansa <Martin.Jansa>
Component: devtools / tool chainAssignee: Juro Bystricky <juro.bystricky>
Status: RESOLVED OBSOLETE QA Contact: Apoorv <apoorv.sangal>
Severity: enhancement    
Priority: Medium+ CC: benjamin.esquivel, bluelightning, joshuagloe, juro.bystricky, kergoth, meta.mr.watcher, meta.watcher, net147, raj.khem, randy.macleod, richard.purdie, ross.burton, sgw, tom.zanussi
Version: 1.5.1   
Target Milestone: 2.5 M1   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
class needed to create reproducible binaries none

Description Martin Jansa 2014-02-22 13:07:04 UTC 
If you try to build the same image on 2 different builders and compare files-in-image.txt buildhistory reports, you'll find that there are many small size changes in binaries.

At least some of these random changes are caused by random-seed used by toolchain to generate symbol names (i.e. different builds having different lenghts of randomly generated symbol names cause few byte difference in binary size)

http://blog.mindfab.net/2013/12/on-way-to-deterministic-binariy-gcc.html

The problem is how to pass
-frandom-seed=$(shell md5sum $< | sed 's/\(.*\) .*/\1/')

in ideally all components without modifying them all. Putting that just to TUNE_CCARGS wont work because it cannot expand Makefile's $<. I was even thinking about patching our gcc-cross to just do this automagically.

Any other suggestions? Does anyone know about some work in (gcc) upstream to make this easier (-frandom-seed-from-md5 would be nice).
Comment 1 Khem Raj 2014-02-28 02:42:13 UTC 
this is only for C++ code btw. and uniqueness has to be preserved across a single build. Using md5 is way too much heavyduty, easiest is to use the relative source filename, however, I dont think there is anywork in gcc being done in this regard, secondly you could have two files with same name sitting in different directories and cd into the directory before building it. So even if gcc tried to use source file name it will fail, now one could say then use full path but that defeats the purpose too since if you change your builddir the seeds will change too. 

so may be for us the approach left is to identify which packages are exhibiting the behavior and fix them one by one.
Comment 2 Martin Jansa 2014-03-03 20:10:49 UTC 
I've tried appending "-frandom-seed=1" to TUNE_CCARGS just to see if this is the only reason for different filesizes (I know that passing constant isn't correct solution).

But even with this, the filesizes are different, one of the reasons is that cmake's link command will generate different order of *.o files for final binary linking which also seems to influence the final file size.

Small improvement is that I was able to generate the same binary (same md5) on 2 different builders when I've also used the same final link command.
Comment 3 Tom Zanussi 2015-01-30 15:25:12 UTC 
*** Bug 4731 has been marked as a duplicate of this bug. ***
Comment 4 Martin Jansa 2015-02-17 17:44:35 UTC 
Debian has interesting project to generate reproducible builds:
https://wiki.debian.org/ReproducibleBuilds
Comment 5 Randy Witt 2015-05-20 18:08:15 UTC 
There is a bit of ambiguity to this bug since things like paths in dwarf sections will also cause binary differences.

Also will we use the build-compare tool to test for "real" differences in binaries? Or are we only satisfied with exact matches in file size in the buildhistory reports?
Comment 6 Richard Purdie 2015-06-16 10:08:45 UTC 
(In reply to comment #5)
> There is a bit of ambiguity to this bug since things like paths in dwarf
> sections will also cause binary differences.
> 
> Also will we use the build-compare tool to test for "real" differences in
> binaries? Or are we only satisfied with exact matches in file size in the
> buildhistory reports?

You're right, we need to put some scope on this problem. For the purposes of this bug and a first round of development, I'd be happy to see that we have basic "identical" binaries as defined by build-compare and ignoring DWARF sections. I don't think its reasonable to have removed all differences but we should perhaps catalogue the list of them for OE-Core.
Comment 7 Richard Purdie 2015-06-16 10:10:13 UTC 
(In reply to comment #6)

> You're right, we need to put some scope on this problem. For the purposes of
> this bug and a first round of development, I'd be happy to see that we have
> basic "identical" binaries as defined by build-compare and ignoring DWARF
> sections. I don't think its reasonable to have removed all differences but
> we should perhaps catalogue the list of them for OE-Core.

To be clear, I realise there are some things we can perhaps do with DWARF sections too but that should be a separate bug.
Comment 8 Michael Halstead 2015-08-12 16:01:57 UTC 
Clearing 2.0 milestone
Comment 9 Martin Jansa 2016-08-08 11:20:16 UTC 
debbindiff - now renamed to diffoscope shows the differences quite nicely

{noformat}
martin@ljama:~$ diffoscope libcrypto.so.1.0.0.BUILD.PLAT-17601-*
--- libcrypto.so.1.0.0.BUILD.PLAT-17601-1
+++ libcrypto.so.1.0.0.BUILD.PLAT-17601-3
├── readelf --wide --notes {}
│ @@ -1,5 +1,5 @@
│  
│  Displaying notes found at file offset 0x000000f4 with length 0x00000024:
│    Owner                 Data size    Description
│    GNU                  0x00000014    NT_GNU_BUILD_ID (unique build ID bitstring)
│ -    Build ID: f817e0c916f5820025a29ebfe7fc2fbd78a19466
│ +    Build ID: 2a721745d4c16c6cdbef1cbbf232762463fa13bb
├── readelf --wide --decompress --hex-dump=.rodata {}
│ @@ -67,15 +67,15 @@
│    0x000bf360 5f424e5f 41534d5f 4746326d 202d4453 _BN_ASM_GF2m -DS
│    0x000bf370 4841315f 41534d20 2d445348 41323536 HA1_ASM -DSHA256
│    0x000bf380 5f41534d 202d4453 48413531 325f4153 _ASM -DSHA512_AS
│    0x000bf390 4d202d44 4145535f 41534d20 2d444748 M -DAES_ASM -DGH
│    0x000bf3a0 4153485f 41534d00 4f70656e 53534c20 ASH_ASM.OpenSSL 
│    0x000bf3b0 312e302e 31702039 204a756c 20323031 1.0.1p 9 Jul 201
│    0x000bf3c0 35006275 696c7420 6f6e3a20 54756520 5.built on: Tue 
│ -  0x000bf3d0 4d617220 20382032 303a3030 3a333720 Mar  8 20:00:37 
│ +  0x000bf3d0 4d617220 20382032 323a3334 3a303420 Mar  8 22:34:04 
│    0x000bf3e0 32303136 00706c61 74666f72 6d3a206c 2016.platform: l
│    0x000bf3f0 696e7578 2d61726d 7634004f 50454e53 inux-armv4.OPENS
│    0x000bf400 534c4449 523a2022 2f757372 2f6c6962 SLDIR: "/usr/lib
│    0x000bf410 2f73736c 22006578 5f646174 612e6300 /ssl".ex_data.c.
│    0x000bf420 66697073 206d6f64 65206e6f 74207375 fips mode not su
│    0x000bf430 70706f72 74656400 6e6f2064 796e6c6f pported.no dynlo
│    0x000bf440 636b2063 72656174 65206361 6c6c6261 ck create callba
├── readelf --wide --decompress --hex-dump=.gnu_debuglink {}
│ @@ -1,5 +1,5 @@
│  
│  Hex dump of section '.gnu_debuglink':
│    0x00000000 6c696263 72797074 6f2e736f 2e312e30 libcrypto.so.1.0
│ -  0x00000010 2e300000 1cfb126f                   .0.....o
│ +  0x00000010 2e300000 acc4be31                   .0.....1
│  
╵
{noformat}
Comment 10 Khem Raj 2016-08-08 15:24:52 UTC 
it will be good to have a global level tool/script which I could run on rootfs and it will run diffoscope on all binary artifacts and produce an output like this binary/library has an issue and it belongs to this package which came from this recipe.

May be a build QA check which is optional to begin with.

Then we can open multiple tickets to fix the packages one by one.
Comment 11 Juro Bystricky 2016-11-22 19:08:30 UTC 
*** Bug 10705 has been marked as a duplicate of this bug. ***
Comment 12 Jonathan Liu 2016-11-23 00:24:41 UTC 
There are several things which end up going into build output which may be useful to virtualize so that it is consistent between machines and different users:
- Build directory (can be seen embedded in shared libraries from /lib using grep)
- Username (cat /proc/version)
- Hostname (cat /proc/version)
- Timezone (cat /proc/version)

Virtualizing the build directory may be useful to allow moving the build directory to another location without invalidating the sstate-cache.
Comment 13 Benjamin Esquivel 2016-12-01 17:47:18 UTC 
adding Sonia and José to the bug as this would need testing additions.
Comment 14 Juro Bystricky 2017-03-19 18:43:17 UTC 
Created attachment 3657 [details]
class needed to create reproducible binaries
Comment 15 Juro Bystricky 2017-03-19 18:44:12 UTC 
As a part of initial investigation, I wanted to binary compare packages created on a single machine but built in two different folders. I did :

local conf:
PACKAGE_CLASSES ?= "package_rpm package_deb package_ipk"

$ bitbake core-image-minimal 

And binary compared individual packages (3868 in each case)

core-image-minimal (default)
============================
       IPK    RPM   DEB
       ----------------
Same:     0     0    25
Diff:  3868  3868  3843
Total: 3868  3868  3868


Following some basic recommendations for reproducible binaries, I created a "reproducible.bbclass" (attached) and re-built the core-image-minimal:

PACKAGE_CLASSES ?= "package_rpm package_deb package_ipk"
INHERIT += "reproducible"

And got these results:
core-image-minimal (with INHERIT +="reproducible")
==================================================

       IPK    RPM   DEB
       ----------------
Same:     0   530  3617
Diff:  3868  3338   251
Total: 3868  3868  3868


Obviously, this is still work in progress.
Comment 16 Juro Bystricky 2017-03-23 17:25:07 UTC 
Comparison of packages built on two different systems (Ubuntu 16 vs container with Ubuntu 14)

commit 9fe7a69535f9443175da8289633a761f58c372ff


core-image-minimal (with INHERIT +="reproducible")
==================================================
      IPK    RPM   DEB
       ----------------
Same:     0     0  3595
Diff:  3859  3859   264
Total: 3859  3859  3859
Comment 17 Khem Raj 2017-03-27 03:14:19 UTC 
(In reply to comment #16)
> Comparison of packages built on two different systems (Ubuntu 16 vs
> container with Ubuntu 14)
> 
> commit 9fe7a69535f9443175da8289633a761f58c372ff
> 
> 
> core-image-minimal (with INHERIT +="reproducible")
> ==================================================
>       IPK    RPM   DEB
>        ----------------
> Same:     0     0  3595
> Diff:  3859  3859   264
> Total: 3859  3859  3859

SOURCE_DATE_EPOCH should be set to something may be to 0 to begin with
but we need a better way to extract this value. We dont have changeling to record date so there must be another value

may be use the last commit time/date for the recipe if using git
otherwise, may be PR server can help
Comment 18 Juro Bystricky 2017-03-27 15:47:58 UTC 
Based on 
https://reproducible-builds.org/specs/source-date-epoch/
The recommendation says:
The value SHOULD be set to the last modification time of the source, incorporating any packaging-specific modifications. For example, in Debian, the timestamp of the latest entry in debian/changelog. 
In my testing I set the value based on the repo git last entry, applied globally.
Ideally, we would modify individual recipes themselves based on their respective source date.
Comment 19 Martin Jansa 2017-06-14 16:49:47 UTC 
Is following sequence of patchwork IDs the latest and complete set of changes which are currently available?

139554 139558 139556 139555 139557 139559 139599 139600 139999
Comment 20 Juro Bystricky 2017-06-14 17:56:40 UTC 
I keep the latest patches here:
http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=jurob/reproducible_binaries

Also, I typically use these settings:

BUILD_REPRODUCIBLE_BINARIES="1"
REPRODUCIBLE_TIMESTAMP_ROOTFS="1483228802"
LDCONFIGDEPEND=""
IMAGE_DEPENDS_cpio_append=" cpio-replacement-native"
EXTRANATIVEPATH+="cpio-native"
IMAGE_CMD_CPIO="cpio --ignore-devno --reproducible "
IMAGE_CMD_TAR="tar -v --sort=name "

PACKAGE_CLASSES="package_rpm package_deb package_ipk"
IMAGE_FSTYPES+="jffs2 cpio hdddirect qcow2 iso hddimg"

# optionally
DISTRO_FEATURES_append=" ptest"
EXTRA_IMAGE_FEATURES+="ptest-pkgs"
INHERIT+="rm_work"

Once the patches are more stable and there is some consensus/feedback I will send them to the mailing list again.
With the settings/patches above the latest stats are as follows:

core-image-sato
========================
      IPK    RPM   DEB
       ----------------
Same:    20   777  6883
Diff:  6955  6198    92
Total: 6975  6975  6975
Comment 21 Martin Jansa 2017-06-16 14:26:04 UTC 
I've tried to use all the changes from your poky-contrib branch:

7c5e8d3c6d kernel.bbclass: improve reproducibility
565607bd2c gdbm: improve reproducibility
023b035b65 bash: improve reproducibility
3423049c8d python3: improve reproducibility
ea5e8ec943 python2.7: improve reproducibility
6894ffbc71 image_types.bbclass: improve cpio image reproducibility
6b8e01d209 cpio: provide cpio-replacement-native
88a42c2bec image.bbclass: support binary reproducibility
265dc4e80c busybox.inc: improve reproducibility
a76225f28b rootfs-postcommands.bbclass: support binary reproducibility
f3392be6a7 image-prelink.bbclass: support binary reproducibility
aa714d6efd reproducible_build.bbclass: initial support for binary reproducibility

and even before "enabling" them by setting those new variables, I've tried to run:

bitbake -k core-image-base 2>&1 | tee log.qemux86

delete sstate-cache and again

bitbake -k core-image-base 2>&1 | tee log.qemux86

surprisingly there wasn't as many changes in buildhistory as I was used to before, will try to redo this test on 2 different hosts.

One issue still shown in buildhistory is locale generation difference:
 -- a/images/qemux86/glibc/core-image-base/files-in-image.txt
 +++ b/images/qemux86/glibc/core-image-base/files-in-image.txt
 @@ -2142,7 +2142,7 @@ drwxr-xr-x root       root            20480 ./var/lib/opkg/info
 -rw-r--r-- root       root             1083 ./var/lib/opkg/info/avahi-locale-cs.control
 -rw-r--r-- root       root               42 ./var/lib/opkg/info/avahi-locale-cs.list
 -rw-r--r-- root       root             1098 ./var/lib/opkg/info/avahi-locale-en-gb.control
--rw-r--r-- root       root              107 ./var/lib/opkg/info/avahi-locale-en-gb.list
+-rw-r--r-- root       root               45 ./var/lib/opkg/info/avahi-locale-en-gb.list
 -rw-r--r-- root       root               79 ./var/lib/opkg/info/base-files.conffiles
 -rw-r--r-- root       root              640 ./var/lib/opkg/info/base-files.control
 -rw-r--r-- root       root              578 ./var/lib/opkg/info/base-files.list
@@ -2191,7 +2191,7 @@ drwxr-xr-x root       root            20480 ./var/lib/opkg/info
 -rw-r--r-- root       root            11757 ./var/lib/opkg/info/kbd-keymaps.list
 -rw-r--r-- root       root              515 ./var/lib/opkg/info/kbd.list
 -rw-r--r-- root       root              565 ./var/lib/opkg/info/kbd-locale-cs.control
--rw-r--r-- root       root               40 ./var/lib/opkg/info/kbd-locale-cs.list
+-rw-r--r-- root       root              115 ./var/lib/opkg/info/kbd-locale-cs.list
 -rwxr-xr-x root       root              400 ./var/lib/opkg/info/kbd.postinst
 -rwxr-xr-x root       root              316 ./var/lib/opkg/info/kbd.prerm
 -rw-r--r-- root       root              543 ./var/lib/opkg/info/kernel-4.10.17-yocto-standard.control
@@ -2286,9 +2286,9 @@ drwxr-xr-x root       root            20480 ./var/lib/opkg/info
 -rwxr-xr-x root       root              396 ./var/lib/opkg/info/libglib-2.0-0.postinst
 -rwxr-xr-x root       root              316 ./var/lib/opkg/info/libglib-2.0-0.postrm
 -rw-r--r-- root       root             1223 ./var/lib/opkg/info/libglib-2.0-locale-cs.control
--rw-r--r-- root       root              118 ./var/lib/opkg/info/libglib-2.0-locale-cs.list
+-rw-r--r-- root       root               43 ./var/lib/opkg/info/libglib-2.0-locale-cs.list
 -rw-r--r-- root       root             1241 ./var/lib/opkg/info/libglib-2.0-locale-en-gb.control
--rw-r--r-- root       root               46 ./var/lib/opkg/info/libglib-2.0-locale-en-gb.list
+-rw-r--r-- root       root              108 ./var/lib/opkg/info/libglib-2.0-locale-en-gb.list
 -rw-r--r-- root       root              585 ./var/lib/opkg/info/libgpg-error0.control
 -rw-r--r-- root       root               59 ./var/lib/opkg/info/libgpg-error0.list
 -rwxr-xr-x root       root               90 ./var/lib/opkg/info/libgpg-error0.postinst

Have you seen this as well in your tests?
Comment 22 Martin Jansa 2017-06-16 14:37:57 UTC 
the "short" version of one of locale packages contains this:

OE qemux86@ ~/build/oe-core $ cat tmp-glibc/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/var/lib/opkg/info/libglib-2.0-locale-cs.list
/usr/share/locale/cs/LC_MESSAGES/glib20.mo

I'll re-run the build to see if I can get the contents of "longer" version seen in previous build.
Comment 23 Juro Bystricky 2017-06-16 15:43:01 UTC 
(In reply to comment #21)
> I've tried to use all the changes from your poky-contrib branch:
> 
> 7c5e8d3c6d kernel.bbclass: improve reproducibility
> 565607bd2c gdbm: improve reproducibility
> 023b035b65 bash: improve reproducibility
> 3423049c8d python3: improve reproducibility
> ea5e8ec943 python2.7: improve reproducibility
> 6894ffbc71 image_types.bbclass: improve cpio image reproducibility
> 6b8e01d209 cpio: provide cpio-replacement-native
> 88a42c2bec image.bbclass: support binary reproducibility
> 265dc4e80c busybox.inc: improve reproducibility
> a76225f28b rootfs-postcommands.bbclass: support binary reproducibility
> f3392be6a7 image-prelink.bbclass: support binary reproducibility
> aa714d6efd reproducible_build.bbclass: initial support for binary
> reproducibility
> 
> and even before "enabling" them by setting those new variables, I've tried
> to run:
> 
> bitbake -k core-image-base 2>&1 | tee log.qemux86
> 
> delete sstate-cache and again
> 
> bitbake -k core-image-base 2>&1 | tee log.qemux86
> 
> surprisingly there wasn't as many changes in buildhistory as I was used to
> before, will try to redo this test on 2 different hosts.
> 
> One issue still shown in buildhistory is locale generation difference:
>  -- a/images/qemux86/glibc/core-image-base/files-in-image.txt
>  +++ b/images/qemux86/glibc/core-image-base/files-in-image.txt
>  @@ -2142,7 +2142,7 @@ drwxr-xr-x root       root            20480
> ./var/lib/opkg/info
>  -rw-r--r-- root       root             1083
> ./var/lib/opkg/info/avahi-locale-cs.control
>  -rw-r--r-- root       root               42
> ./var/lib/opkg/info/avahi-locale-cs.list
>  -rw-r--r-- root       root             1098
> ./var/lib/opkg/info/avahi-locale-en-gb.control
> --rw-r--r-- root       root              107
> ./var/lib/opkg/info/avahi-locale-en-gb.list
> +-rw-r--r-- root       root               45
> ./var/lib/opkg/info/avahi-locale-en-gb.list
>  -rw-r--r-- root       root               79
> ./var/lib/opkg/info/base-files.conffiles
>  -rw-r--r-- root       root              640
> ./var/lib/opkg/info/base-files.control
>  -rw-r--r-- root       root              578
> ./var/lib/opkg/info/base-files.list
> @@ -2191,7 +2191,7 @@ drwxr-xr-x root       root            20480
> ./var/lib/opkg/info
>  -rw-r--r-- root       root            11757
> ./var/lib/opkg/info/kbd-keymaps.list
>  -rw-r--r-- root       root              515 ./var/lib/opkg/info/kbd.list
>  -rw-r--r-- root       root              565
> ./var/lib/opkg/info/kbd-locale-cs.control
> --rw-r--r-- root       root               40
> ./var/lib/opkg/info/kbd-locale-cs.list
> +-rw-r--r-- root       root              115
> ./var/lib/opkg/info/kbd-locale-cs.list
>  -rwxr-xr-x root       root              400 ./var/lib/opkg/info/kbd.postinst
>  -rwxr-xr-x root       root              316 ./var/lib/opkg/info/kbd.prerm
>  -rw-r--r-- root       root              543
> ./var/lib/opkg/info/kernel-4.10.17-yocto-standard.control
> @@ -2286,9 +2286,9 @@ drwxr-xr-x root       root            20480
> ./var/lib/opkg/info
>  -rwxr-xr-x root       root              396
> ./var/lib/opkg/info/libglib-2.0-0.postinst
>  -rwxr-xr-x root       root              316
> ./var/lib/opkg/info/libglib-2.0-0.postrm
>  -rw-r--r-- root       root             1223
> ./var/lib/opkg/info/libglib-2.0-locale-cs.control
> --rw-r--r-- root       root              118
> ./var/lib/opkg/info/libglib-2.0-locale-cs.list
> +-rw-r--r-- root       root               43
> ./var/lib/opkg/info/libglib-2.0-locale-cs.list
>  -rw-r--r-- root       root             1241
> ./var/lib/opkg/info/libglib-2.0-locale-en-gb.control
> --rw-r--r-- root       root               46
> ./var/lib/opkg/info/libglib-2.0-locale-en-gb.list
> +-rw-r--r-- root       root              108
> ./var/lib/opkg/info/libglib-2.0-locale-en-gb.list
>  -rw-r--r-- root       root              585
> ./var/lib/opkg/info/libgpg-error0.control
>  -rw-r--r-- root       root               59
> ./var/lib/opkg/info/libgpg-error0.list
>  -rwxr-xr-x root       root               90
> ./var/lib/opkg/info/libgpg-error0.postinst
> 
> Have you seen this as well in your tests?

No, I have never noticed this one before, but thanks for pointing it out. There seem to be some elusive bugs in locale generation, often resulting with incorrect ownership (there is a bug filed regarding this). Ross also mentioned he noticed on ocassions whole blocks of locales missing... So this may give us more clues on what's wrong.
Comment 24 Martin Jansa 2017-06-19 17:17:23 UTC 
In next build I got "smaller" libglib-2.0-locale-en-gb.list (before it was about locale-cs.list)

--rw-r--r-- root       root              108 ./var/lib/opkg/info/libglib-2.0-locale-en-gb.list
+-rw-r--r-- root       root               46 ./var/lib/opkg/info/libglib-2.0-locale-en-gb.list

cat ../tmp-glibc/A/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/libglib-2.0-locale-en-gb.list 
/usr/share/locale/en_GB/LC_MESSAGES/glib20.mo

In the next build it was the bigger one again:

--rw-r--r-- root       root               46 ./var/lib/opkg/info/libglib-2.0-locale-en-gb.list
+-rw-r--r-- root       root              108 ./var/lib/opkg/info/libglib-2.0-locale-en-gb.list

And the contents is:
cat ../tmp-glibc/E/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./vaoe-linux/core-image-base/1.0-r0/rootfs/r/lib/opkg/info/libglib-2.0-locale-en-gb.list 
/usr/share/locale/en_GB/LC_MESSAGES/
/usr/share/locale/en_GB/LC_MESSAGES/glib20.mo
/usr/share/locale/en_GB/

It seems to be caused by the order in which the locale packages are installed and which packages will end up "owning" the directory with given locale.

Build A is the one with smaller libglib-2.0-locale-en-gb, build B the bigger:

OE qemux86@ ~/build/oe-core/buildhistory $ grep "/usr/share/locale/en_GB/" ../tmp-glibc/A/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/*.list
../tmp-glibc/A/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/avahi-locale-en-gb.list:/usr/share/locale/en_GB/LC_MESSAGES/avahi.mo
../tmp-glibc/A/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/glibc-locale-en-gb.list:/usr/share/locale/en_GB/LC_MESSAGES/
../tmp-glibc/A/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/glibc-locale-en-gb.list:/usr/share/locale/en_GB/
../tmp-glibc/A/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/libglib-2.0-locale-en-gb.list:/usr/share/locale/en_GB/LC_MESSAGES/glib20.mo

OE qemux86@ ~/build/oe-core/buildhistory $ grep "/usr/share/locale/en_GB/" ../tmp-glibc/B/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/*.list
../tmp-glibc/B/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/avahi-locale-en-gb.list:/usr/share/locale/en_GB/LC_MESSAGES/avahi.mo
../tmp-glibc/B/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/libglib-2.0-locale-en-gb.list:/usr/share/locale/en_GB/LC_MESSAGES/
../tmp-glibc/B/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/libglib-2.0-locale-en-gb.list:/usr/share/locale/en_GB/LC_MESSAGES/glib20.mo
../tmp-glibc/B/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/libglib-2.0-locale-en-gb.list:/usr/share/locale/en_GB/
Comment 25 Martin Jansa 2017-06-19 17:26:59 UTC 
Another causes for random changes detected in 6 builds on the same host with the same metadata:

1) sbindir owned by base-files or libc6:

diff -uNr ../tmp-glibc/F/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/base-files.list ../tmp-glibc/G/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/base-files.list
--- ../tmp-glibc/F/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/base-files.list      2017-04-25 00:16:53.000000000 +0200
+++ ../tmp-glibc/G/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/base-files.list      2017-04-25 00:16:53.000000000 +0200
@@ -26,7 +26,6 @@
 /etc/skel/
 /etc/hostname
 /var/spool/
-/sbin/
 /var/lib/
 /var/lib/misc/
 /var/volatile/tmp/

diff -uNr ../tmp-glibc/F/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/libc6.list ../tmp-glibc/G/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/libc6.list
--- ../tmp-glibc/F/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/libc6.list   2017-04-25 00:16:53.000000000 +0200
+++ ../tmp-glibc/G/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/libc6.list   2017-04-25 00:16:53.000000000 +0200
@@ -19,6 +19,7 @@
 /lib/librt.so.1
 /lib/libm-2.25.so
 /lib/libpthread.so.0
+/sbin/
 /lib/libdl.so.2
 /lib/libnss_dns-2.25.so
 /lib/libresolv-2.25.so

2) EXTENDPRAUTO bumped from unknown reasons in linux-yocto:

diff -uNr ../tmp-glibc/E/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/kernel-4.10.17-yocto-standard.control ../tmp-glibc/F/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/kernel-4.10.17-yocto-standard.control
--- ../tmp-glibc/E/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/kernel-4.10.17-yocto-standard.control       2017-04-25 00:16:53.000000000 +0200
+++ ../tmp-glibc/F/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/kernel-4.10.17-yocto-standard.control       2017-04-25 00:16:53.000000000 +0200
@@ -1,5 +1,5 @@
 Package: kernel-4.10.17-yocto-standard
-Version: 4.10.17+git0+4b89f331d4_6648a34e00-r0.7
+Version: 4.10.17+git0+4b89f331d4_6648a34e00-r0.10
 Description: Linux kernel
  Linux kernel.
 Section: kernel

That results also in different status file sizes
ls -la ../tmp-glibc/*/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/status
-rw-r--r-- 1 bitbake bitbake 31605 Jun 16 16:33 ../tmp-glibc/A/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/status
-rw-r--r-- 1 bitbake bitbake 31605 Jun 17 14:20 ../tmp-glibc/B/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/status
-rw-r--r-- 1 bitbake bitbake 31626 Apr 25 00:16 ../tmp-glibc/E/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/status
-rw-r--r-- 1 bitbake bitbake 31632 Apr 25 00:16 ../tmp-glibc/F/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/status
-rw-r--r-- 1 bitbake bitbake 31632 Apr 25 00:16 ../tmp-glibc/G/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/status
-rw-r--r-- 1 bitbake bitbake 31632 Apr 25 00:16 ../tmp-glibc/H/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/status

And different EXTENDPRAUTO for packages which depend on it
diff -uNr ../tmp-glibc/B/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/udev.control ../tmp-glibc/E/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/udev.control
--- ../tmp-glibc/B/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/udev.control 2017-06-17 14:20:40.547985099 +0200
+++ ../tmp-glibc/E/work/qemux86-oe-linux/core-image-base/1.0-r0/rootfs/./var/lib/opkg/info/udev.control 2017-04-25 00:16:53.000000000 +0200
@@ -1,5 +1,5 @@
 Package: udev
-Version: 1:232-r0.9
+Version: 1:232-r0.10
 Description: A System and service manager
  systemd is a system and service manager for Linux, compatible with SysV
  and LSB init scripts. systemd provides aggressive parallelization

But good thing is that I still don't see any binary differences from linker mangling I've used to see before.
Comment 26 Juro Bystricky 2017-06-20 15:26:41 UTC 
Thanks, these are very useful observations.
I have also noticed some weirdness recently:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=11551

And there is this one, probably related:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=11299

Correct me if I'm wrong, but my understanding is that none of the issues you mentioned is actually related to the patches for binary reproducibility, as you did not set 
BUILD_REPRODUCIBLE_BINARIES="1"
Comment 27 Martin Jansa 2017-06-20 15:37:17 UTC 
I did 6 builds from the same metadata (oe-core from last week + the changes from your poky-contrib branch).

First 2 (in directories A and B) were built withoutBUILD_REPRODUCIBLE_BINARIES="1".

Remaining 4 (E, F, G, H) were built after adding following to conf/local.conf:
BUILD_REPRODUCIBLE_BINARIES = "1"
REPRODUCIBLE_TIMESTAMP_ROOTFS = "1493072213"
REPRODUCIBLE_TIMESTAMP_IMAGE_PRELINK = "1493072213"

The issues I was seeing aren't imho caused by the reproducible changes. Main source of changes are PR service and order of package installation by opkg.
Comment 28 Juro Bystricky 2017-06-22 20:41:55 UTC 
(In reply to comment #23)
> (In reply to comment #21)
> > I've tried to use all the changes from your poky-contrib branch:
> > 
> > 7c5e8d3c6d kernel.bbclass: improve reproducibility
> > 565607bd2c gdbm: improve reproducibility
> > 023b035b65 bash: improve reproducibility
> > 3423049c8d python3: improve reproducibility
> > ea5e8ec943 python2.7: improve reproducibility
> > 6894ffbc71 image_types.bbclass: improve cpio image reproducibility
> > 6b8e01d209 cpio: provide cpio-replacement-native
> > 88a42c2bec image.bbclass: support binary reproducibility
> > 265dc4e80c busybox.inc: improve reproducibility
> > a76225f28b rootfs-postcommands.bbclass: support binary reproducibility
> > f3392be6a7 image-prelink.bbclass: support binary reproducibility
> > aa714d6efd reproducible_build.bbclass: initial support for binary
> > reproducibility
> > 
> > and even before "enabling" them by setting those new variables, I've tried
> > to run:
> > 
> > bitbake -k core-image-base 2>&1 | tee log.qemux86
> > 
> > delete sstate-cache and again
> > 
> > bitbake -k core-image-base 2>&1 | tee log.qemux86
> > 
> > surprisingly there wasn't as many changes in buildhistory as I was used to
> > before, will try to redo this test on 2 different hosts.
> > 
> > One issue still shown in buildhistory is locale generation difference:
> >  -- a/images/qemux86/glibc/core-image-base/files-in-image.txt
> >  +++ b/images/qemux86/glibc/core-image-base/files-in-image.txt
> >  @@ -2142,7 +2142,7 @@ drwxr-xr-x root       root            20480
> > ./var/lib/opkg/info
> >  -rw-r--r-- root       root             1083
> > ./var/lib/opkg/info/avahi-locale-cs.control
> >  -rw-r--r-- root       root               42
> > ./var/lib/opkg/info/avahi-locale-cs.list
> >  -rw-r--r-- root       root             1098
> > ./var/lib/opkg/info/avahi-locale-en-gb.control
> > --rw-r--r-- root       root              107
> > ./var/lib/opkg/info/avahi-locale-en-gb.list
> > +-rw-r--r-- root       root               45
> > ./var/lib/opkg/info/avahi-locale-en-gb.list
> >  -rw-r--r-- root       root               79
> > ./var/lib/opkg/info/base-files.conffiles
> >  -rw-r--r-- root       root              640
> > ./var/lib/opkg/info/base-files.control
> >  -rw-r--r-- root       root              578
> > ./var/lib/opkg/info/base-files.list
> > @@ -2191,7 +2191,7 @@ drwxr-xr-x root       root            20480
> > ./var/lib/opkg/info
> >  -rw-r--r-- root       root            11757
> > ./var/lib/opkg/info/kbd-keymaps.list
> >  -rw-r--r-- root       root              515 ./var/lib/opkg/info/kbd.list
> >  -rw-r--r-- root       root              565
> > ./var/lib/opkg/info/kbd-locale-cs.control
> > --rw-r--r-- root       root               40
> > ./var/lib/opkg/info/kbd-locale-cs.list
> > +-rw-r--r-- root       root              115
> > ./var/lib/opkg/info/kbd-locale-cs.list
> >  -rwxr-xr-x root       root              400 ./var/lib/opkg/info/kbd.postinst
> >  -rwxr-xr-x root       root              316 ./var/lib/opkg/info/kbd.prerm
> >  -rw-r--r-- root       root              543
> > ./var/lib/opkg/info/kernel-4.10.17-yocto-standard.control
> > @@ -2286,9 +2286,9 @@ drwxr-xr-x root       root            20480
> > ./var/lib/opkg/info
> >  -rwxr-xr-x root       root              396
> > ./var/lib/opkg/info/libglib-2.0-0.postinst
> >  -rwxr-xr-x root       root              316
> > ./var/lib/opkg/info/libglib-2.0-0.postrm
> >  -rw-r--r-- root       root             1223
> > ./var/lib/opkg/info/libglib-2.0-locale-cs.control
> > --rw-r--r-- root       root              118
> > ./var/lib/opkg/info/libglib-2.0-locale-cs.list
> > +-rw-r--r-- root       root               43
> > ./var/lib/opkg/info/libglib-2.0-locale-cs.list
> >  -rw-r--r-- root       root             1241
> > ./var/lib/opkg/info/libglib-2.0-locale-en-gb.control
> > --rw-r--r-- root       root               46
> > ./var/lib/opkg/info/libglib-2.0-locale-en-gb.list
> > +-rw-r--r-- root       root              108
> > ./var/lib/opkg/info/libglib-2.0-locale-en-gb.list
> >  -rw-r--r-- root       root              585
> > ./var/lib/opkg/info/libgpg-error0.control
> >  -rw-r--r-- root       root               59
> > ./var/lib/opkg/info/libgpg-error0.list
> >  -rwxr-xr-x root       root               90
> > ./var/lib/opkg/info/libgpg-error0.postinst
> > 
> > Have you seen this as well in your tests?
> 
> No, I have never noticed this one before, but thanks for pointing it out.
> There seem to be some elusive bugs in locale generation, often resulting
> with incorrect ownership (there is a bug filed regarding this). Ross also
> mentioned he noticed on ocassions whole blocks of locales missing... So this
> may give us more clues on what's wrong.

Added Ross to CC list, hoping he'll be able to identify the root cause of the problem ...
Comment 29 Ross Burton 2017-06-23 16:21:35 UTC 
Yes, glibc-locale is a nuisance to say the least.
Comment 30 Juro Bystricky 2017-08-30 23:12:16 UTC 
Incremental progress.  Moving to 2.5 M1.
Comment 31 Juro Bystricky 2017-12-13 22:34:42 UTC 
The scope of this bug is not very well defined. 
Some of the initial issues have been resolved a while ago:

1. dwarf sections: reproducibility addressed fixed via new DEBUG_FLAGS -ffile-prefix-map

2. -frandom-seed=xxx : generally implemented AFAIK

There have also been multiple patches addressing reproducibility of various packages already accepted. 

In addition there are several supplemental Bugzilla entries addressing binary reproducibility:

https://bugzilla.yoctoproject.org/show_bug.cgi?id=11176
https://bugzilla.yoctoproject.org/show_bug.cgi?id=11177
https://bugzilla.yoctoproject.org/show_bug.cgi?id=11178
https://bugzilla.yoctoproject.org/show_bug.cgi?id=11179

I opened new entries

https://bugzilla.yoctoproject.org/show_bug.cgi?id=12421
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12422

(With more to follow.)

So I will close this one particular Bugzilla entry as RESOLVED/OBSOLETE as it is being replaced/superseded by several more granulated entries.
If there is a particular issue I may have missed which is present here but not addressed in the entries above, please open a new Bugzilla entry.