Created attachment 3671 [details] diffoscope output (HTML format) When binary comparing two python packages in "deploy/deb" folders, ie: /data/master-master/poky/build-repro1-master/tmp/deploy/deb/core2-64/python-curses_2.7.13-r1_amd64.deb /data/master-master/poky/build-repro2-master/tmp/deploy/deb/core2-64/python-curses_2.7.13-r1_amd64.deb they show binary differences. Using diffoscope (resulting files attached), the difference can be shown to be caused by a time stamp in .pyc header. To achieve a true binary identical files, this stamps should be identical. I speculate the stamps correspond to the date/time when the .pyc was created/compiled. I believe the stamp is needed to detect the code needs to be recompiled if the python source code changes. One way to fix this is ensure the stamps correspond to the time/date of the source code, not the time of the compilation. stamp1 (0x58d40f23): Thu Mar 23 11:08:35 PDT 2017 .deb file: Mar 23 11:09 stamp2 (0x:58d41807) Thu Mar 23 11:46:31 PDT 2017 .deb file: Mar 23 11:47 With some minimal requirements for binary reproducibility, core-image-minimal debian packages are distributed roughly like this: Different: 242 Same: 3617 Total: 3859 Once the above time stamp issue is addressed, I estimate additional ~130 packages would also binary identical.
Created attachment 3672 [details] diffoscope output (text format)
Another possible solution would be not to include .pyc files in the packages at all.
There's an upstream bug and an initial set of patches: https://bugs.python.org/issue29708 https://github.com/python/cpython/pull/296
Just FYI, the patch is for Python3, I doubt there will be a backport to 2.7. AFAIK we will need it, as for example in core-image-minimal we have only python2.7 modules. But it should be relatively easy to backport.
(In reply to comment #4) > Just FYI, the patch is for Python3, I doubt there will be a backport to 2.7. > AFAIK we will need it, as for example in core-image-minimal we have only > python2.7 modules. > But it should be relatively easy to backport. Don't be so sure, see the reproducible.patch in: https://build.opensuse.org/package/rdiff/devel:languages:python:Factory/python?linkrev=base&rev=203
There might not be an official backport of the patch, but the patches author backported it to 2.7 for Suse's use (it does look like it was trivial). As an aside we're looking at dropping Python2 from default images in the YP 2.4 timeframe, but that doesn't mean we shouldn't make python2 reproducible if we can.
Seems both patches are going to be upstreamed. Should we wait for new python releases or patch the python recipes meanwhile?
I will test the patches.
I have two patches (python2.7 and puthon3), so far they seem to work fine: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=jurob/reproducible_binaries&id=b860255cb7ec426cd202fbd03d5ad813095f630e http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=jurob/reproducible_binaries&id=fdb2116f77529ff36bd1e00cfd2ff302c1bf3644
will be send to the mailing list as a part of "reproducible binaries" patch set.
Two patches (python2.7 and python3 merged with master : http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=07348bb76f6617477e164db9245a1a98f1a18569 http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=6fd0e23dbd47bee2cd6debc4c8234375aa0e0e27