Bug 13323

Summary:	Add reproducibility automated testing
Product:	[QA/Testing] Functional (self) Testing	Reporter:	Richard Purdie <richard.purdie>
Component:	general	Assignee:	Joshua Watt <JPEWhacker>
Status:	RESOLVED FIXED	QA Contact:
Severity:	enhancement
Priority:	High	CC:	alessioigorbogani, JPEWhacker, randy.macleod, ross.burton
Version:	unspecified
Target Milestone:	2.8 M3
Hardware:	x86
OS:	Multiple
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	No (bug/feature does not impact docs)
Bug Depends on:
Bug Blocks:	13324

Description Richard Purdie 2019-05-02 13:24:38 UTC

We need to have tests that confirm our builds are reproducible. Initially we want a test which:

a) runs a build in the usual way/location (can be sstate accelerated)

b) changes to a new TMPDIR and rebuilds without using sstate

c) compares the build in a) and b) and lists the differences both in packages themselves and in images.

We'd start with a test for core-image-minimal, ultimately moving to more complex images such as sato-sdk and perhaps world builds.

Comment 1 Ross Burton 2019-07-25 14:42:10 UTC

The test suite is merged in oe-core now but is skipped currently.  We can close this when the test is enabled.

Comment 2 Ross Burton 2019-07-25 15:19:26 UTC

With a couple of patches I'm about to send, the current failure list is:

The following deb packages are missing or different:
ca-certificates_20190110-r0_all.deb
libical-dbg_2.0.0-r0_amd64.deb
libical_2.0.0-r0_amd64.deb
perl-ptest_5.30.0-r0_amd64.deb
systemd-dbg_242+0+db2e367bfc-r0_amd64.deb
systemd_242+0+db2e367bfc-r0_amd64.deb
kernel-image-bzimage-5.0.19-yocto-standard_5.0.19+git0+c2e34d9ab2_00638cdd8f-r0_amd64.deb

Comment 3 Ross Burton 2019-07-25 17:54:11 UTC

So if the test uses reproducible_build_simple instead of the convoluted magic in the normal class, more recipes are reproducible.  With that change it's just:

libical-dbg_2.0.0-r0_amd64.deb
libical_2.0.0-r0_amd64.deb
systemd-dbg_242+0+db2e367bfc-r0_amd64.deb
systemd_242+0+db2e367bfc-r0_amd64.deb
kernel-image-bzimage-5.0.19-yocto-standard_5.0.19+git0+c2e34d9ab2_00638cdd8f-r0_amd64.deb

The libical one is build paths in the cmake files, iirc.

Comment 4 Ross Burton 2019-07-26 10:34:42 UTC

ical hopefully fixed with v3 of the upgrade.

systemd is because it embeds the build timestamp as a build epoch.

Comment 5 Ross Burton 2019-07-26 20:46:32 UTC

With a bit of a hack to fix systemd, and a little patch to the kernel, I have the test passing here.

Comment 6 Ross Burton 2019-08-02 10:19:45 UTC

For a laugh I changed the QA test to build core-image-sato instead of -minimal.

gstreamer1.0-libav-dbg_1.16.0-r0_amd64.deb
gstreamer1.0-libav_1.16.0-r0_amd64.deb
jpeg-tools_2.0.2-r0_amd64.deb
libatk-1.0-doc_2.32.0-r0_amd64.deb
libatspi-src_2.32.1-r0_amd64.deb
libical-dbg_3.0.5-r0_amd64.deb
libical_3.0.5-r0_amd64.deb
libjpeg-dbg_2.0.2-r0_amd64.deb
libjpeg62_2.0.2-r0_amd64.deb
librsvg-2-2_2.40.20-r0_amd64.deb
librsvg-2-dbg_2.40.20-r0_amd64.deb
libsoup-2.4-doc_2.66.2-r0_amd64.deb
libtag-c0_1.11.1-r0_amd64.deb
libtag-dbg_1.11.1-r0_amd64.deb
libturbojpeg0_2.0.2-r0_amd64.deb
pango-src_1.42.4-r0_amd64.deb
xserver-xorg-src_1.20.4-r0_amd64.deb

The ones which impact *just* the -src are where absolute build paths are embedded in generated sources, I've a fix for xserver-xorg already.

Comment 7 Joshua Watt 2019-08-27 19:37:20 UTC

This patch series is the initial attempt to enable the reproducible OEQA test for core-image-minimal:

http://lists.openembedded.org/pipermail/openembedded-core/2019-August/286154.html

Comment 8 Joshua Watt 2019-09-05 14:45:49 UTC

The autobuilder is now running the reproducible build test.