Bug 12425

Summary: Upgrade rpm to 4.14.0+ to support binary reproducibility
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Juro Bystricky <juro.bystricky>
Component: coreAssignee: Alexander Kanavin <alex.kanavin>
Status: RESOLVED FIXED QA Contact:
Severity: enhancement    
Priority: Medium+ CC: meta.mr.watcher, meta.watcher, richard.purdie
Version: 2.5   
Target Milestone: 2.5 M2   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Don't know
Attachments:
Description Flags
patch to upgrade rpm_git.bb to 4.14.0 none

Description Juro Bystricky 2017-12-14 23:28:09 UTC 
Created attachment 4155 [details]
patch to upgrade rpm_git.bb to 4.14.0

The RPM version 4.14.0 has improved support for binary reproducibility:

Release notes:

Add some support for reproducible builds via SOURCE_DATE_EPOCH standard (https://reproducible-builds.org/specs/source-date-epoch):

    Support setting build time from SOURCE_DATE_EPOCH
    Support setting file mtime from SOURCE_DATE_EPOCH
    Support setting buildhost via %_buildhost macro (RhBug:1309367)
    Option to set SOURCE_DATE_EPOCH from %changelog
    Do not store digests of %ghost files
    Do not store directory sizes in packages

I did some preliminary POC, and it works as advertised.
Upgrade seems fairly straightforward, I modified some existing patches but some patches need to be still evaluated/modified.

I attached a working recipe patch as well.
Comment 1 Ross Burton 2017-12-21 09:43:39 UTC 
Alex has sent an upgrade for this so reassigning to him.
Comment 2 Juro Bystricky 2017-12-26 20:13:20 UTC 
I patched the current master 8240adb06b05f693a53c6cdda9e6ffc51e036c82 with 
http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=ross/mut&id=0b3b0d5cc3f0598bdab4322d164bf755b606270e

Then I built core-image-minimal, I get a lot of errors building grub & grub-efi:
(log.do_package):

ERROR: objcopy failed with exit code 139 (cmd was 'i586-poky-linux-objcopy' --only-keep-debug '/tmp/ramdisk/build-repro-minimal-git-rpm-no-repro-2017-12-26/work/i586-poky-linux/grub-efi/2.02-r0/package/usr/lib/grub/i386-efi/zfs.module' '/tmp/ramdisk/build-repro-minimal-git-rpm-no-repro-2017-12-26/work/i586-poky-linux/grub-efi/2.02-r0/package/usr/lib/grub/i386-efi/.debug/zfs.module'):
i586-poky-linux-objcopy: /tmp/ramdisk/build-repro-minimal-git-rpm-no-repro-2017-12-26/work/i586-poky-linux/grub-efi/2.02-r0/package/usr/lib/grub/i386-efi/zfs.module: invalid string offset 65536 >= 1589 for section `rtab'
i586-poky-linux-objcopy: /tmp/ramdisk/build-repro-minimal-git-rpm-no-repro-2017-12-26/work/i586-poky-linux/grub-efi/2.02-r0/package/usr/lib/grub/i386-efi/zfs.module: invalid string offset 19181 >= 1589 for section `rtab'
...
...
i586-poky-linux-objcopy: /tmp/ramdisk/build-repro-minimal-git-rpm-no-repro-2017-12-26/work/i586-poky-linux/grub-efi/2.02-r0/package/usr/lib/grub/i386-efi/zfs.module: invalid relocation type 112
i586-poky-linux-objcopy: /tmp/ramdisk/build-repro-minimal-git-rpm-no-repro-2017-12-26/work/i586-poky-linux/grub-efi/2.02-r0/package/usr/lib/grub/i386-efi/zfs.module(bug_loc): relocation 54 has invalid symbol index 7548416
Segmentation fault (core dumped)
DEBUG: Python function split_and_strip_files finished
DEBUG: Python function do_package finished
ERROR: Function failed: split_and_strip_files


All other packages build fine.
Comment 3 Juro Bystricky 2017-12-26 20:55:30 UTC 
In addition, I think the default rpm build should be built with 

%clamp_mtime_to_source_date_epoch 1

instead of 

%clamp_mtime_to_source_date_epoch 0

This way we get about 4000 binary identical RPM packages (out of about 4023) built for core-image-minimal. (providing SOURCE_DATE_EPOCH is in environment)
as opposed to about 530.
Comment 4 Alexander Kanavin 2018-01-02 11:54:36 UTC 
(In reply to comment #2)
> I patched the current master 8240adb06b05f693a53c6cdda9e6ffc51e036c82 with 
> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=ross/
> mut&id=0b3b0d5cc3f0598bdab4322d164bf755b606270e

You need the 'package.bbclass: replace rpm/debugedit with dwarfsrcfiles' patch for this.
Comment 5 Alexander Kanavin 2018-01-02 11:55:58 UTC 
(In reply to comment #3)
> In addition, I think the default rpm build should be built with 
> 
> %clamp_mtime_to_source_date_epoch 1
> 
> instead of 
> 
> %clamp_mtime_to_source_date_epoch 0
> 
> This way we get about 4000 binary identical RPM packages (out of about 4023)
> built for core-image-minimal. (providing SOURCE_DATE_EPOCH is in environment)
> as opposed to about 530.

This should be a separate patch, as it's unrelated to 4.14 upgrade itself. Can you prepare it?
Comment 6 Juro Bystricky 2018-01-02 16:11:04 UTC 
(In reply to comment #5)
> (In reply to comment #3)
> > In addition, I think the default rpm build should be built with 
> > 
> > %clamp_mtime_to_source_date_epoch 1
> > 
> > instead of 
> > 
> > %clamp_mtime_to_source_date_epoch 0
> > 
> > This way we get about 4000 binary identical RPM packages (out of about 4023)
> > built for core-image-minimal. (providing SOURCE_DATE_EPOCH is in environment)
> > as opposed to about 530.
> 
> This should be a separate patch, as it's unrelated to 4.14 upgrade itself.
> Can you prepare it?

Yes, I'll send the patch to OE mailing list.
Comment 7 Juro Bystricky 2018-01-02 17:55:48 UTC 
(In reply to comment #5)
> (In reply to comment #3)
> > In addition, I think the default rpm build should be built with 
> > 
> > %clamp_mtime_to_source_date_epoch 1
> > 
> > instead of 
> > 
> > %clamp_mtime_to_source_date_epoch 0
> > 
> > This way we get about 4000 binary identical RPM packages (out of about 4023)
> > built for core-image-minimal. (providing SOURCE_DATE_EPOCH is in environment)
> > as opposed to about 530.
> 
> This should be a separate patch, as it's unrelated to 4.14 upgrade itself.
> Can you prepare it?

Yes, I'll send the patch to OE mailing list.(In reply to comment #4)
> (In reply to comment #2)
> > I patched the current master 8240adb06b05f693a53c6cdda9e6ffc51e036c82 with 
> > http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=ross/
> > mut&id=0b3b0d5cc3f0598bdab4322d164bf755b606270e
> 
> You need the 'package.bbclass: replace rpm/debugedit with dwarfsrcfiles'
> patch for this.

Thanks, this fixed the problem.
Comment 8 Juro Bystricky 2018-01-03 00:23:06 UTC 
(In reply to comment #5)
> (In reply to comment #3)
> > In addition, I think the default rpm build should be built with 
> > 
> > %clamp_mtime_to_source_date_epoch 1
> > 
> > instead of 
> > 
> > %clamp_mtime_to_source_date_epoch 0
> > 
> > This way we get about 4000 binary identical RPM packages (out of about 4023)
> > built for core-image-minimal. (providing SOURCE_DATE_EPOCH is in environment)
> > as opposed to about 530.
> 
> This should be a separate patch, as it's unrelated to 4.14 upgrade itself.
> Can you prepare it?

Yes, I'll send the patch to OE mailing list.(In reply to comment #4)
> (In reply to comment #2)
> > I patched the current master 8240adb06b05f693a53c6cdda9e6ffc51e036c82 with 
> > http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=ross/
> > mut&id=0b3b0d5cc3f0598bdab4322d164bf755b606270e
> 
> You need the 'package.bbclass: replace rpm/debugedit with dwarfsrcfiles'
> patch for this.

Thanks, this fixed the problem.(In reply to comment #6)
> (In reply to comment #5)
> > (In reply to comment #3)
> > > In addition, I think the default rpm build should be built with 
> > > 
> > > %clamp_mtime_to_source_date_epoch 1
> > > 
> > > instead of 
> > > 
> > > %clamp_mtime_to_source_date_epoch 0
> > > 
> > > This way we get about 4000 binary identical RPM packages (out of about 4023)
> > > built for core-image-minimal. (providing SOURCE_DATE_EPOCH is in environment)
> > > as opposed to about 530.
> > 
> > This should be a separate patch, as it's unrelated to 4.14 upgrade itself.
> > Can you prepare it?
> 
> Yes, I'll send the patch to OE mailing list.

Sent patch:
http://lists.openembedded.org/pipermail/openembedded-core/2018-January/145987.html