Bug 12421 - Host-compiled Python bytecode not reproducible
Summary: Host-compiled Python bytecode not reproducible
Status: RESOLVED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: core (show other bugs)
Version: 2.5
Hardware: x86 Multiple
: Medium+ enhancement
Target Milestone: 2.7 M2
Assignee: Mingli Yu
QA Contact:
URL:
Whiteboard:
Depends on: 11178
Blocks:
  Show dependency tree
 
Reported: 2017-12-13 22:13 UTC by Juro Bystricky
Modified: 2019-01-30 03:12 UTC (History)
4 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Yes (doc changes required)

Attachments
new problem (build host uninative reference) (58.17 KB, text/html)
2018-04-22 10:48 UTC, Juro Bystricky 		no flags Details
notorious python-xcbgen (131.44 KB, text/html)
2018-04-22 10:49 UTC, Juro Bystricky 		no flags Details
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Juro Bystricky 2017-12-13 22:13:56 UTC 
This is an offshoot of
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5866

providing a better specified deliverable.

The goal of this enhancement is to build reproducible packages and images for:

$ MACHINE=qemux86 bitbake core-image-minimal

It is assumed the packages are Debian. 

The resulting Debian packages of this build are placed in the folder <builddir>/tmp/deploy/deb
Repeating the build in a different folder will place the packages in:
<builddir-different>/tmp/deploy/deb

The build is deemed reproducible if all packages in both build folders are binary identical.

This enhancement strongly depends on 
SOURCE_DATE_EPOCH support.
https://bugzilla.yoctoproject.org/show_bug.cgi?id=11178
Comment 2 Ross Burton 2018-04-19 04:49:59 UTC 
That xcb-proto patch has been integrated.  Is that really the last piece for reproducible builds in this limited scope?
Comment 3 Juro Bystricky 2018-04-19 07:10:36 UTC 
I will re-check. I am pretty sure this was the last one binary different for package for core-image-minimal, qemux86, qemux86-64. But the master is a moving target, so things may have changed.

BTW, RPM and IPK packages are binary identical as well.

On the downside, MACHINE like qemuarm, qemumips need som more work.
But that is subject to another bug/enhancement which I will file.
Comment 4 Juro Bystricky 2018-04-22 10:46:54 UTC 
So I checked core-image-minimal (qemux86-64) with poky-contrib/ross/sumo.
(4ce51e515b575d89972a835cdead8df06b212558 go_runtime_1.9: pass BUILD_ flags)

I got two packages that are binary different (out of 3950):
python-xcbgen_1.12

and a new kid on the block (new, but easy to fix):
bash-ptest_4.4.12

(I will attach the diffoscope outputs).
So it seems python-xcbgen_1.12 patch has not been merged.
Comment 5 Juro Bystricky 2018-04-22 10:48:18 UTC 
Created attachment 4280 [details]
new problem (build host uninative reference)
Comment 6 Juro Bystricky 2018-04-22 10:49:25 UTC 
Created attachment 4281 [details]
notorious python-xcbgen
Comment 7 Ross Burton 2018-11-01 14:48:30 UTC 
The only concrete mention in this bug is python-xcbgen, which should be covered now that Python has the SOURCE_DATE_EPOCH support integrated.
Comment 8 Ross Burton 2018-11-01 15:58:09 UTC 
Recap:

python-xcbgen isn't reproducible because it contains .pyc files which are compiled by the *host* python.
Juro submitted a patch which made it use py3native if reprodicible builds were requested.  I was instinctively against that because I don't want to pull py3native further into the bottom of the build tree.

Then python-xcbgen got more complicated with pyc files being the wrong version and being regenerated and now its got a sstate hack to work around the fact that we're using host python to byte compile target code.

I'm now thinking we need to two part plan:
1) just inherit python3native in this recipe, remove the cruft. Live with the build impact (rpm depends on it anyway, so its built early anyway).
2) Figure out a neater way to handle pyc files generally.  Target packages and native that use py3native will need py3native to be present so they get built with the right version of Python.  Native packages that don't use py3native are an interesting case...