Bug 11247

Summary: binary reproducibility: ncurses-staticdev_6.0+20161126-r0_amd64.deb
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Juro Bystricky <juro.bystricky>
Component: coreAssignee: Ross Burton <ross.burton>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium+ CC: meta.mr.watcher, meta.watcher
Version: 2.3   
Target Milestone: 2.4 M1   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Don't know
Attachments:
Description Flags
diffoscope output (HTML format)
none
diffoscope output after "configure" patched
none
Patch that replaces the ncurses AR command line options. none

Description Juro Bystricky 2017-03-24 19:55:54 UTC
Created attachment 3674 [details]
diffoscope output (HTML format)

Two independent builds don't binary compare (diffoscope log attached).

ncurses-staticdev_6.0+20161126-r0_amd64.deb

It seems the various .a libraries were linked wih AR not configured with 
"--enable-deterministic-archives"

One possible fix could be to pass correct flags to AR such as Dcvr, or post-process the archives:
objcopy --enable-deterministic-archives libfoo.a

Other fixes may be possible.
Comment 1 Ross Burton 2017-03-30 14:50:10 UTC
Our ar is built with deterministic archives.  I wonder if its using the host ar?  Can you re-run with something like MACHINE=qemuarm?
Comment 2 Juro Bystricky 2017-03-30 15:30:28 UTC
I will re-run with MACHINE=qemuarm.
Another explanation would be the ncurses Makefile uses AR with -u flag
Comment 3 Ross Burton 2017-03-30 15:35:25 UTC
Grepping log.do_compile shows:

  x86_64-poky-linux-ar -curvU

       U   Do not operate in deterministic mode.  This is the inverse of the D modifier, above: added files and the archive index
           will get their actual UID, GID, timestamp, and file mode values.

           This is the default unless binutils was configured with --enable-deterministic-archives.

Yet another reason to put ncurses on my list of Software I Hate.
Comment 4 Ross Burton 2017-03-30 15:38:16 UTC
aclocal.m4 says this:

AC_CACHE_CHECK(for options to update archives, cf_cv_ar_flags,[
        cf_cv_ar_flags=unknown
        for cf_ar_flags in -curvU -curv curv -crv crv -cqv cqv -rv rv

Can you verify that removing the -curvU entry solves the problem?
Comment 5 Juro Bystricky 2017-03-30 15:49:29 UTC
Will verify, no problem.
Comment 6 Juro Bystricky 2017-03-30 21:36:40 UTC
Created attachment 3682 [details]
diffoscope output after "configure" patched
Comment 7 Juro Bystricky 2017-03-30 21:54:32 UTC
Created attachment 3683 [details]
Patch that replaces the ncurses AR command line options.
Comment 8 Juro Bystricky 2017-03-30 21:54:52 UTC
So patching aclocal.m4 did not help. I had to patch "configure" instead.
After building ncurses with the patched recipe (two separate builds), the difference between the .deb packages was reduced as expected (see attached ncurses-config.html).
The remaining difference should be addressed in a separate patch, similarily the way debian does it:

examine the modification time of all files in the package
build directory. If the file is newer than the date of the latest
debian/changelog entry, we assume the file is a result of a build, and its
modification time will be set to the date of the latest debian/changelog entry.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=759886

(I also attached to ncurses recipe patch)
Comment 9 Ross Burton 2017-04-03 13:33:58 UTC
The ar change is merged in OE-Core 46c757d0ca7ff294a7e55c130698fd256b69b62e.
Comment 10 Ross Burton 2017-06-01 15:23:04 UTC
Merged in oe-core 46c757d0ca7ff294a7e55c130698fd256b69b62e.