Bug 11242 - binary reproducibility: ipk packages never identical
Summary: binary reproducibility: ipk packages never identical
Status: RESOLVED FIXED
Alias: None
Product: opkg
Classification: Yocto Project Subprojects
Component: opkg (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Undecided normal
Target Milestone: ---
Assignee: Alejandro del Castillo
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2017-03-24 17:12 UTC by Juro Bystricky
Modified: 2018-02-01 21:08 UTC (History)
2 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Don't know


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Juro Bystricky 2017-03-24 17:12:49 UTC
When trying to achieve binary reproducible packages, .ipk packages are not comparing favourably with neither RPM nor DEB packages.

For core-image-minimal 
(see  https://bugzilla.yoctoproject.org/show_bug.cgi?id=5866 )
we get something like this:

core-image-minimal (with INHERIT +="reproducible")
==================================================

       IPK    RPM   DEB
       ----------------
Same:     0   530  3617
Diff:  3868  3338   251
Total: 3868  3868  3868
Comment 1 Alejandro del Castillo 2018-01-29 10:27:05 UTC
Gzip by default adds a timestamp to the archive. Modified opkg-build to supress timestamping the gzip archive. Also, added deterministic directory entries sorting, as well as support for SOURCE_DATE_EPOCH. With those fixes, opkg now looks more inline with deb and rpm:

Comparing 2781 files in git/oe-core/build//tmp-glibc/deploy/ipk/ to test_oe/openembedded-core/build//tmp-glibc/deploy/ipk/
Same:  2713 
Different:  68 
Total: 2781
Comment 2 Juro Bystricky 2018-01-29 15:57:57 UTC
(In reply to comment #1)
> Gzip by default adds a timestamp to the archive. Modified opkg-build to
> supress timestamping the gzip archive. Also, added deterministic directory
> entries sorting, as well as support for SOURCE_DATE_EPOCH. With those fixes,
> opkg now looks more inline with deb and rpm:
> 
> Comparing 2781 files in git/oe-core/build//tmp-glibc/deploy/ipk/ to
> test_oe/openembedded-core/build//tmp-glibc/deploy/ipk/
> Same:  2713 
> Different:  68 
> Total: 2781

Great news, is there a patch in ML or upstream?
Comment 3 Alejandro del Castillo 2018-01-29 16:24:39 UTC
yep, I posted to the ML today [1]. Usually I leave it there for a few days before pulling into the trunk. Once I do that, I'll close this bug (usually I don't get a whole lot of feedback from the ML, but still like to leave the patch there for a few days).


[1] https://groups.google.com/forum/#!topic/opkg-devel/sFGFB-W9xM8
Comment 4 Juro Bystricky 2018-01-29 16:50:54 UTC
I hade a look at the patch, I like the way you check if really gzip or pigz. I filed a bug complaining about gzip != pigz a while ago.
Comment 5 Alejandro del Castillo 2018-01-29 16:57:50 UTC
took me a few hours to realize that OE uses pgzip....and that -n on gzip in pgzip is -nT...really annoying, glad there is bug, the discrepancy makes no sense!
Comment 6 Alejandro del Castillo 2018-02-01 18:01:33 UTC
fixed on 682f8c5e35b8854a9bb858b8ee1714d27e0c00db
Comment 7 Juro Bystricky 2018-02-01 19:32:43 UTC
Thanks, however the fix still needs to percolate to opkg-utils recipe in yocto. AFAIK the recipe presently uses a tarball based on release ver 0.3.5. Are you planning a new release as well (i.e. 0.3.6) ? There seems to be quite a number of additional changes since 0.3.5 in master. In any case, the recipe needs to be modified, be it another version or simply git master. If there are no takers (designated maintainers) I don't mind doing it myself.
Comment 8 Alejandro del Castillo 2018-02-01 19:51:45 UTC
I am planning the 0.3.6 release of opkg & opkg-utils by mid February. With the release I usually post new recipes as well. Since we are so close to the release, is it OK to wait? if not, the fix can certainly be backported to 0.3.5.
Comment 9 Juro Bystricky 2018-02-01 21:08:14 UTC
(In reply to comment #8)
> I am planning the 0.3.6 release of opkg & opkg-utils by mid February. With
> the release I usually post new recipes as well. Since we are so close to the
> release, is it OK to wait? if not, the fix can certainly be backported to
> 0.3.5.

Mid February sounds good. Looking forward to it.