Created attachment 4812 [details] LTO reproducible build test Using DISTRO_FEATURES_append = " lto" or adding "-flto" (Link Time Optimization) to a Makefile makes the resulting binary non-reproducible, in that it depends on the full absolute build path to recipe-sysroot-native, so installing Yocto in a different directory results in a different binary. This is despite the fact that the CFLAGS contains a -fdebug-prefix-map for recipe-sysroot-native. The -frandom-seed flag does not help. Attached is a recipe that demonstrates the problem, without having to install two instances of Yocto. Instructions: bitbake lto-test (fails with error message) (open a new shell and execute the commands given in error message)
Hi Tony, Thanks reporting this issue. We believe that this is still a problem with the upstream toolchain. If you know otherwise and ideally can point to other distros that use LTO in a reproducible build that would be useful but note that Yocto is more strict since we do not use the same patch in our reproducible build tests. The 3.99 milestone means that we almost certainly won't get to this in 3.4 unless someone signs up for the bug/enhancement.
May I suggest adding a note to the "Reproducible Builds" documentation?
I added Michael who does lots of work on the YP docs and can either add a note or show you how to do it since we're using Sphinx now: http://docs.yoctoproject.org/test-manual/reproducible-builds.html?highlight=reproducible My concern is that while we could document the current limitation, we do want to know what the upstream status is. Are you able to spend some time looking into that Tony? Ideally, there should be an upstream gcc bug that we track.
https://reproducible-builds.org/docs/randomness/ "When Link-Time Optimizations are turned on, GCC users will write random identifiers to binary objects they create. Using -frandom-seed can be used for this particular case. As it will hash arbitrary data, passing the file name should work in most cases." In my simple lto-test program, the -frandom-seed flag does not seem to be necessary to produce a deterministic binary, and the test still fails when using -frandom-seed. However, looking at the gcc source, it looks like it might be necessary in more complex programs, but I am not sure. If you look at the differences in the LTO binaries with different sysroot-native paths: strings lto-test.lto1.sysroot0.try0 > strings0 strings lto-test.lto1.sysroot1.try0 > strings1 diff strings0 strings1 < lto_test.c.79b8ff9a > lto_test.c.bba5e59b Which brings us to: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=66305#c6 "-ffat-lto-objects create unreproducible objects" This looks related, specifically the comment "LOCATION_FILE is unaffected by -ffile-prefix-map, unlike fold_builtin_FILE in gcc/builtins.c for example." However, this comment is about the path to the source file being compiled rather than the location of the sysroot-native containing gcc itself. I could always submit a new gcc bug upstream. However, it would take some effort to come up with a test case without Yocto providing the sysroot-native, and I am very busy with other things. Alternately, I could add a comment to the gcc bug above linking it to this Yocto bug.
(In reply to comment #3) > I added Michael who does lots of work on the YP docs and can either add a > note or show you how to do it since we're using Sphinx now: > > http://docs.yoctoproject.org/test-manual/reproducible-builds. > html?highlight=reproducible > > My concern is that while we could document the current limitation, we do want > to know what the upstream status is. Are you able to spend some time looking > into that Tony? Ideally, there should be an upstream gcc bug that we track. Hi, I'll be happy to propose a note corresponding to this issue, referring to this bug until it is fixed. Thanks for adding me to the CC list. Michael.
I have submitted a new upstream bug report: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101473
Submitted a documentation patch on this issue: https://lists.yoctoproject.org/g/docs/message/1514 Reviews are welcome!
Created attachment 4813 [details] gcc: Backport patch to make LTO builds more reproducible This patch is for Yocto 3.3.1; however, I expect that this feature will be be targeted for Yocto 3.4 or later. So this patch is just for reference.
Created attachment 4814 [details] bitbake.conf: add DEBUG_PREFIX_MAP to TARGET_LDFLAGS
Created attachment 4815 [details] net-snmp: fix QA Issue after LDFLAGS change
The original issue has now been fixed in upstream gcc, but Yocto needs additional changes to make LTO builds reproducible. See the upstream bug report for details: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=101473 And the gcc fix: https://gcc.gnu.org/g:7cc2df084b7977653a9b59cbc34a9ad500ae619c I have attached a backport of the gcc fix above for gcc 10.2 in Yocto 3.3.1. However, this is just for reference in case anyone wants to try it out; I would expect that this feature will be be targeted for Yocto 3.4 or later. As discussed in the gcc bug report above, Yocto needs to be changed to add -f*-prefix-map to LDFLAGS instead of just CFLAGS. I have attached a patch to do that. Alternately I suppose the -f*-prefix-map could be moved to CC instead, but I haven't tested that. /usr/sbin/lspci from pciutils is a good program to test that shows how adding -f*-prefix-map to LDFLAGS makes a program binary reproducible with LTO enabled. Adding -f*-prefix-map to LDFLAGS broke package QA on net-snmp. I have attached a patch that fixes that. That was the only broken package in my particular build, but there might be other packages that have been broken that I am not building. With all these patches, there are still many shared libraries in /lib and /usr/lib that are still not reproducible with LTO enabled because x86_64-poky-linux-libtool filters out -f*-prefix-map flags when linking shared libraries. /usr/lib/libbz2.so.1.0.6 from the libbz2 package from the bzip2 recipe is a good example. I do not have a patch for that problem. I will leave that to someone else. Finally, /usr/lib/grub/i386-pc/*.mod files from the grub package are not reproducible with LTO enabled. I haven't investigated this problem. Maybe grub needs LTO disabled, or -f*-prefix-map, or -frandom-seed.
Tony, I've been trying to make the time to review your comment and read the links provide but that isn't happening today. Sigh. Would it make sense to send the patch you have to the list for review and to have it tested on the autobuilder cluster?
(In reply to comment #12) > Would it make sense to send the patch you have to the list for review and to > have it tested on the autobuilder cluster? I am a developer but I am not a Yocto developer, so I am not familiar with your procedures. What version of Yocto should we target for this feature? I am working with 3.3 Hardknott, but that seems the wrong version to target. The LDFLAGS change broke net-snmp (but I have a patch for that), and it will possibly break other things as well. So are we looking at 3.4 Honister or 3.5 Kirkstone? If you want to target Kirkstone, is now the right time to send patches to the mailing list? To get reproducible LTO builds, we need: *) The gcc fix *) The LDFLAGS patch, along with fixes for everything it breaks *) A fix for libtool, which I don't have *) Fixes for individual packages like grub, if needed *) Add LTO to reproducible builds testing *) Once everything works, update the documentation again Of course it is possible to target a subset of these changes for a particular Yocto release, which would make LTO builds more reproducible without getting them to 100%. So if you think now is the appropriate time to send the patches to the mailing list, let me know which Yocto version to target, and I will look into remaking the patches against that version (although I haven't worked with a development version of Yocto before, so I am not sure what is involved).
Hi Tony, Thanks for your prompt, clear response. I've added Richard (project lead) and Khem (key maintainer and toolchain geek!) to the bug so they can get up to speed and perhaps comment on the timing of these improvements. The right branch to target is 'master' which will be the 3.4 release in October of 2021. We have a (long!) document about our workflow but honestly, I've never read it! https://wiki.yoctoproject.org/wiki/Yocto_Project_Design_and_Development_Process All you really need to know is that we release in October and April of each year and that there are 4 milestone periods of 1.5 months each (M1-> M4). The first 3 periods are for new development and the last is for bug fixes. M3 is around September 1 so it's possible that, with you pushing, we could get -flto builds to be reproducible. Since this work involves several components of the system, it might be best to wait until Monday and send your existing patch along with a cover letter to: openembedded-core@lists.openembedded.org including the list that you presented in your previous comment: "To get reproducible LTO builds, we need: ..." Submission _guidelines_ are here: https://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded but basically, just work on master, do some testing and send a decently commented commit log. Oh we do try to track patches so that we can eventually drop them so please read: https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines#Patch_Header_Recommendations:_Upstream-Status By sending this patch + description, people other than Richard, Khem and I will have a chance to comment on the work and timing of the fix. Thanks again, ../Randy.
Tony, I've giving this issue to Sundeep but he won't get to it for a while. Free free to comment and take the issue back if you like.
Hi, The reproducibility issue is not observed in the current master and Kirkstone branch. LTO disabled 9f965b136260fe34b7482ccabcf0924fb7dbc7ec790952d9241f71bf9a5f7e09 *lto-test.lto0.sysroot0.try0 9f965b136260fe34b7482ccabcf0924fb7dbc7ec790952d9241f71bf9a5f7e09 *lto-test.lto0.sysroot0.try1 9f965b136260fe34b7482ccabcf0924fb7dbc7ec790952d9241f71bf9a5f7e09 *lto-test.lto0.sysroot1.try0 9f965b136260fe34b7482ccabcf0924fb7dbc7ec790952d9241f71bf9a5f7e09 *lto-test.lto0.sysroot1.try1 LTO enabled f3a2139f52e2bbdc44b73d1eb607f94db3a089ef6a9c8f1ec0b0f1992a556682 *lto-test.lto1.sysroot0.try0 f3a2139f52e2bbdc44b73d1eb607f94db3a089ef6a9c8f1ec0b0f1992a556682 *lto-test.lto1.sysroot0.try1 f3a2139f52e2bbdc44b73d1eb607f94db3a089ef6a9c8f1ec0b0f1992a556682 *lto-test.lto1.sysroot1.try0 f3a2139f52e2bbdc44b73d1eb607f94db3a089ef6a9c8f1ec0b0f1992a556682 *lto-test.lto1.sysroot1.try1 PASS - LTO build is reproducible
Can you please check and let me know if its fine to resolve it
I haven't tested this in a while, but digging into the git history it looks like the following things fixed the general problems: upgrade to gcc 12 poky commit 488aa36a61ad ("bitbake.conf: add DEBUG_PREFIX_MAP to TARGET_LDFLAGS") poky commit 2f743b473938 ("libtool: Fix lto option passing for reproducible builds") There might still be issues with individual packages like grub, but I don't have time to test right now. So I would be OK with resolving this bug, and opening bugs for individual packages later if they turn out to have problems.
Thanks for the confirmation. Closing the issue as resolved.
Resolved as Fixed
Also removing the corresponding comment from the docs: https://lists.yoctoproject.org/g/docs/message/4502