Bug 13763

Summary: reproducible_build.bbclass incorrectly set SOURCE_DATE_EPOCH=0 in do_deploy_source_date_epoch, do_configure, do_compile
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Rick Liu <rick.liu>
Component: coreAssignee: Joshua Watt <JPEWhacker>
Status: RESOLVED FIXED QA Contact:
Severity: major    
Priority: High CC: JPEWhacker, meta.mr.watcher, meta.watcher, randy.macleod, richard.purdie
Version: 3.0   
Target Milestone: 3.1 M3   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Don't know
Attachments:
Description Flags
complete temp logs (2020-0129)
none
rpm-build-date.list
none
reproducible_build.bbclass debug verbose patch
none
brcm-u-boot logs
none
alsa-utils log files
none
alsa-state log files none

Description Rick Liu 2020-01-30 00:04:03 UTC
Created attachment 4618 [details]
complete temp logs (2020-0129)

Environment:
Build Configuration:
BB_VERSION           = "1.44.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "ubuntu-18.04"
TARGET_SYS           = "aarch64-poky-linux"
MACHINE              = "bcm958742t"
DISTRO               = "poky"
DISTRO_VERSION       = "3.0+snapshot-20200129"
TUNE_FEATURES        = "aarch64 crypto cortexa72"



From "log.do_unpack",
I see do_create_source_date_epoch_stamp() is executed
and generated correct "SOURCE_DATE_EPOCH: 1579843392".
(but this correct SOURCE_DATE_EPOCH is never exported)

However, after do_patch(),
when do_deploy_source_date_epoch() is executed, 
from "run.do_deploy_source_date_epoch":
export SOURCE_DATE_EPOCH="0"

It does not set SOURCE_DATE_EPOCH to the correct EPOCH "1579843392"
from do_create_source_date_epoch_stamp().

It's the same for do_configure() and do_compile():
export SOURCE_DATE_EPOCH="0" is first set
before do_configure() and do_compile() start.
Comment 1 Rick Liu 2020-01-30 01:17:52 UTC
After some further debugging,
it looks like after do_create_source_date_epoch_stamp(),
the line 169 ~ line179 are not executed.

http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/classes/reproducible_build.bbclass
python () {
    if d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1':
        d.appendVarFlag("do_unpack", "postfuncs", " do_create_source_date_epoch_stamp")
        epochfile = d.getVar('SDE_FILE')
        source_date_epoch = "0"
        if os.path.isfile(epochfile):
            with open(epochfile, 'r') as f:
                source_date_epoch = f.read()
            bb.debug(1, "SOURCE_DATE_EPOCH: %s" % source_date_epoch)
        d.setVar('SOURCE_DATE_EPOCH', source_date_epoch)
}
Comment 2 Richard Purdie 2020-01-30 23:33:48 UTC
Does this issue reproduce every time? Were seeing issues which would be caused by something like this but we've yet to be able to trigger it reliably...
Comment 3 Rick Liu 2020-01-31 00:30:54 UTC
Created attachment 4619 [details]
rpm-build-date.list
Comment 4 Rick Liu 2020-01-31 00:33:44 UTC
Yes, I could consistently reproduce the issue for my recipe.
For Yocto's own recipes,
I see most of packages are good,
but some of them have the same issue for RPM Build Date.

eg.
alsa-state (has issue) and alsa-utils (look good)

alsa-state-0.2.0-r5.aarch64.rpm
Release     : r5                            Build Date: Wednesday, December 31, 1969 PM04:00:00 PST

alsa-states-0.2.0-r5.aarch64.rpm
Release     : r5                            Build Date: Wednesday, December 31, 1969 PM04:00:00 PST

alsa-utils-1.2.1-r0.aarch64.rpm
Release     : r0                            Build Date: Thursday, November 14, 2019 AM09:02:29 PST

alsa-utils-aconnect-1.2.1-r0.aarch64.rpm
Release     : r0                            Build Date: Thursday, November 14, 2019 AM09:02:29 PST

alsa-utils-alsactl-1.2.1-r0.aarch64.rpm
Release     : r0                            Build Date: Thursday, November 14, 2019 AM09:02:29 PST
Comment 5 Rick Liu 2020-01-31 02:24:06 UTC
Created attachment 4620 [details]
reproducible_build.bbclass debug verbose patch
Comment 6 Rick Liu 2020-01-31 02:27:11 UTC
Created attachment 4621 [details]
brcm-u-boot logs


brcm-u-boot.env:
bitbake mc:ns3:brcm-u-boot -f -e output

brcm-u-boot.log:
bitbake mc:ns3:brcm-u-boot -f output (fresh build with empty yocto-cache)

temp:
bitbake logs for brcm-u-boot
Comment 7 Rick Liu 2020-01-31 02:33:02 UTC
From brcm-u-boot.env:
WORKDIR="/home/rl891036/scrach/SOC-11124-timestamp/ns3-bcm958742t/poky/build/tmp/work/bcm958742t-poky-linux/brcm-u-boot/gitAUTOINC+749bcf8ae5-r0"


From brcm-u-boot.log:
epochfile=/home/rl891036/scrach/SOC-11124-timestamp/ns3-bcm958742t/poky/build/tmp/work/bcm958742t-poky-linux/brcm-u-boot/1.0-r0/source-date-epoch/__source_date_epoch.txt
epochfile not exist: /home/rl891036/scrach/SOC-11124-timestamp/ns3-bcm958742t/poky/build/tmp/work/bcm958742t-poky-linux/brcm-u-boot/1.0-r0/source-date-epoch/__source_date_epoch.txt
d.setVar: SOURCE_DATE_EPOCH=0 (from anonymous)


From temp/log.do_unpack:
DEBUG: Executing python function do_create_source_date_epoch_stamp
sourcedir: /home/rl891036/scrach/SOC-11124-timestamp/ns3-bcm958742t/poky/build/tmp/work/bcm958742t-poky-linux/brcm-u-boot/gitAUTOINC+749bcf8ae5-r0/git
DEBUG: git repository: /home/rl891036/scrach/SOC-11124-timestamp/ns3-bcm958742t/poky/build/tmp/work/bcm958742t-poky-linux/brcm-u-boot/gitAUTOINC+749bcf8ae5-r0/git/.git
SOURCE_DATE_EPOCH: 1579843392
WARNING: cerate SDE_DIR!
WARNING: write source_date_epoch=1579843392
WARNING: d.setVar: SOURCE_DATE_EPOCH=1579843392 (from do_create_source_date_epoch_stamp)


But
from log.do_configure, log.do_compile:
SOURCE_DATE_EPOCH=0
SOURCE_DATE_EPOCH=0

from log.do_package_write_rpm:
DEBUG: SOURCE_DATE_EPOCH=0
Comment 8 Rick Liu 2020-01-31 02:53:43 UTC
Created attachment 4622 [details]
alsa-utils log files

alsa-utils has correct SOURCE_DATE_EPOCH=1573750949

alsa-utils.env:
(bitbake mc:ns3:alsa-utils -f -e output)
WORKDIR="/home/rl891036/scrach/SOC-11124-timestamp/ns3-bcm958742t/poky/build/tmp/work/aarch64-poky-linux/alsa-utils/1.2.1-r0"

alsa-utils.log:
(bitbake mc:ns3:alsa-utils -f output)
sourcedir: /home/rl891036/scrach/SOC-11124-timestamp/ns3-bcm958742t/poky/build/tmp/work/aarch64-poky-linux/alsa-utils/1.2.1-r0/alsa-utils-1.2.1
SOURCE_DATE_EPOCH: 1573750949
WARNING: mc:ns3:alsa-utils-1.2.1-r0 do_unpack: cerate SDE_DIR!
WARNING: mc:ns3:alsa-utils-1.2.1-r0 do_unpack: write source_date_epoch=1573750949
WARNING: mc:ns3:alsa-utils-1.2.1-r0 do_unpack: d.setVar: SOURCE_DATE_EPOCH=1573750949 (from do_create_source_date_epoch_stamp)
epochfile=/home/rl891036/scrach/SOC-11124-timestamp/ns3-bcm958742t/poky/build/tmp/work/aarch64-poky-linux/alsa-utils/1.2.1-r0/source-date-epoch/__source_date_epoch.txt
WARNING: read from epochfile: SOURCE_DATE_EPOCH=1573750949
WARNING: d.setVar: SOURCE_DATE_EPOCH=1573750949 (from anonymous)


temp:
(bitbake logs for alsa-utils)
run.do_configure, run.do_compile:
export SOURCE_DATE_EPOCH="1573750949"
Comment 9 Rick Liu 2020-01-31 03:07:41 UTC
Created attachment 4625 [details]
alsa-state log files

alsa-state has incorrect SOURCE_DATE_EPOCH=0

alsa-state.env:
(bitbake mc:ns3:alsa-state -f -e output)
WORKDIR="/home/rl891036/scrach/SOC-11124-timestamp/ns3-bcm958742t/poky/build/tmp/work/aarch64-poky-linux/alsa-state/0.2.0-r5"

alsa-state.log:
(bitbake mc:ns3:alsa-state -f output)
epochfile=/home/rl891036/scrach/SOC-11124-timestamp/ns3-bcm958742t/poky/build/tmp/work/aarch64-poky-linux/alsa-state/0.2.0-r5/source-date-epoch/__source_date_epoch.txt
epochfile not exist: /home/rl891036/scrach/SOC-11124-timestamp/ns3-bcm958742t/poky/build/tmp/work/aarch64-poky-linux/alsa-state/0.2.0-r5/source-date-epoch/__source_date_epoch.txt
d.setVar: SOURCE_DATE_EPOCH=0 (from anonymous)
WARNING: mc:ns3:alsa-state-0.2.0-r5 do_deploy_source_date_epoch_setscene: read from SDE_FILE: SOURCE_DATE_EPOCH=0
WARNING: mc:ns3:alsa-state-0.2.0-r5 do_deploy_source_date_epoch_setscene: d.setVar: SOURCE_DATE_EPOCH=0 (from do_deploy_source_date_epoch_setscene)
NOTE: Setscene tasks completed
epochfile=/home/rl891036/scrach/SOC-11124-timestamp/ns3-bcm958742t/poky/build/tmp/work/aarch64-poky-linux/alsa-state/0.2.0-r5/source-date-epoch/__source_date_epoch.txt
WARNING: read from epochfile: SOURCE_DATE_EPOCH=0
epochfile=/home/rl891036/scrach/SOC-11124-timestamp/ns3-bcm958742t/poky/build/tmp/work/aarch64-poky-linux/alsa-state/0.2.0-r5/source-date-epoch/__source_date_epoch.txt
WARNING: read from epochfile: SOURCE_DATE_EPOCH=0
WARNING: d.setVar: SOURCE_DATE_EPOCH=0 (from anonymous)
epochfile=/home/rl891036/scrach/SOC-11124-timestamp/ns3-bcm958742t/poky/build/tmp/work/aarch64-poky-linux/alsa-state/0.2.0-r5/source-date-epoch/__source_date_epoch.txt
WARNING: read from epochfile: SOURCE_DATE_EPOCH=0
WARNING: d.setVar: SOURCE_DATE_EPOCH=0 (from anonymous)
epochfile=/home/rl891036/scrach/SOC-11124-timestamp/ns3-bcm958742t/poky/build/tmp/work/aarch64-poky-linux/alsa-state/0.2.0-r5/source-date-epoch/__source_date_epoch.txt
WARNING: read from epochfile: SOURCE_DATE_EPOCH=0
WARNING: d.setVar: SOURCE_DATE_EPOCH=0 (from anonymous)
WARNING: mc:ns3:alsa-state-0.2.0-r5 do_unpack: Reusing SOURCE_DATE_EPOCH from: /home/rl891036/scrach/SOC-11124-timestamp/ns3-bcm958742t/poky/build/tmp/work/aarch64-poky-linux/alsa-state/0.2.0-r5/source-date-epoch/__source_date_epoch.txt
epochfile=/home/rl891036/scrach/SOC-11124-timestamp/ns3-bcm958742t/poky/build/tmp/work/aarch64-poky-linux/alsa-state/0.2.0-r5/source-date-epoch/__source_date_epoch.txt
WARNING: read from epochfile: SOURCE_DATE_EPOCH=0
WARNING: d.setVar: SOURCE_DATE_EPOCH=0 (from anonymous)
epochfile=/home/rl891036/scrach/SOC-11124-timestamp/ns3-bcm958742t/poky/build/tmp/work/aarch64-poky-linux/alsa-state/0.2.0-r5/source-date-epoch/__source_date_epoch.txt
WARNING: read from epochfile: SOURCE_DATE_EPOCH=0
WARNING: d.setVar: SOURCE_DATE_EPOCH=0 (from anonymous)


temp:
(bitbake logs for alsa-state)
run.do_configure, run.do_compile:
export SOURCE_DATE_EPOCH="0"
Comment 10 Joshua Watt 2020-01-31 15:16:25 UTC
Thank you so much for providing these logs, it has been extremely helpful! I think I might have an idea as to what the problem is. In this specific recipe, there is a bad interaction between the anonymous SOURCE_DATE_EPOCH code and the AUTOINC code. The problem is that the PV variable is set to the default value of "1.0-r1" when the anon python code looks for the source_date_epoch.txt file (this is quite obvious in the logs with your recording patch). The problem is that this recipe uses AUTOREV, which changes PV in an anon python function *after* the SDE check. This means that the do_deploy_source_date_epoch task writes the timestamp out the correct path, as can be seen in the logs.

I'm not sure how to fix it exactly yet, but I'm confident we can figure out something now that we know the problem.
Comment 11 Rick Liu 2020-01-31 18:04:16 UTC
Right now, in my specific recipe, 
I use a work-around similar to kernel class (LINE294) [http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/classes/kernel.bbclass#n294]
to re-export "${SOURCE_DATE_EPOCH}",
but I re-export based on "${SDE_FILE}" content:

do_configure_prepend() {
    export SOURCE_DATE_EPOCH="$(cat ${SDE_FILE})"
}

do_compile_prepend() {
    export SOURCE_DATE_EPOCH="$(cat ${SDE_FILE})"
}

do_package_write_rpm_prepend() {
    epochfile = d.getVar('SDE_FILE')
    if os.path.isfile(epochfile):
            with open(epochfile, 'r') as my_epoch:
                source_date_epoch = my_epoch.read()
                bb.debug(1, f"SOURCE_DATE_EPOCH={source_date_epoch} (read from epochfile)")
                my_epoch.close()

    d.setVar('SOURCE_DATE_EPOCH', source_date_epoch)
}
Comment 12 Rick Liu 2020-01-31 18:15:38 UTC
I'm kinda new to Yocto, so I have 2 questions.

1.
I know in Bitbake-style python function,
we could export/update variable back to database "d" through d.setVar()
so the variable can be available in subsequent tasks.

Are we able to do the same in shell functions?
When I export a variable in shell function,
its scope seems within that function only.


2.
Since I have many recipes using AUCOINC,
I tried to put my workaround hack into reproducible_build.bbclass.
But I run into issue that some recipes are not using shell function do_configure / do_compile (they are using python),
so I got syntax error.

Is there a way to prepend do_configure / do_compile differently based on their function type?

eg.
do_configure_prepend { # for shell }
do_configure_prepend { # for python }
Comment 13 Joshua Watt 2020-01-31 19:32:24 UTC
(In reply to comment #12)
> I'm kinda new to Yocto, so I have 2 questions.
> 
> 1.
> I know in Bitbake-style python function,
> we could export/update variable back to database "d" through d.setVar()
> so the variable can be available in subsequent tasks.

Not quite. If you change a variable in a given task, it doesn't propagate to other subsequent tasks (at least AFAIK).

> 
> Are we able to do the same in shell functions?
> When I export a variable in shell function,
> its scope seems within that function only.
> 
> 
> 2.
> Since I have many recipes using AUCOINC,
> I tried to put my workaround hack into reproducible_build.bbclass.
> But I run into issue that some recipes are not using shell function
> do_configure / do_compile (they are using python),
> so I got syntax error.
> 
> Is there a way to prepend do_configure / do_compile differently based on
> their function type?
> 
> eg.
> do_configure_prepend { # for shell }
> do_configure_prepend { # for python }

No, but depending on what you need to do, you can add a prefunc that will execute before the given task. Because this executes independently, it can be either shell or python independent of the main task:

 python do_my_pre_func() {
      d.setVar('FOO', 'BAR')
 }

 do_configure[prefuncs] += "do_my_pre_func"

Note that because these execute in the same context as the do_configure task, changes to bitbake variables in the prefunc *will* be visible in do_configure
Comment 14 Joshua Watt 2020-01-31 19:33:44 UTC
Patch submitted to ML: http://lists.openembedded.org/pipermail/openembedded-core/2020-January/292273.html

If you could give this a try and report back it would be a huge help!
Comment 15 Rick Liu 2020-01-31 20:20:11 UTC
Thank you a lot for the prompt fix.
I'll try out your patch today.
Comment 16 Rick Liu 2020-01-31 20:51:29 UTC
>> 1.
>> I know in Bitbake-style python function,
>> we could export/update variable back to database "d" through d.setVar()
>> so the variable can be available in subsequent tasks.
>
>Not quite. If you change a variable in a given task, it doesn't propagate to >other subsequent tasks (at least AFAIK).

Yes, I just verified this.
Even using d.setVar in one task to update/create variable,
the change won't propagate to next tasks (even through d.getVar).
Comment 17 Rick Liu 2020-01-31 21:33:39 UTC
> http://lists.openembedded.org/pipermail/openembedded-core/2020-January/292273.html
I confirm the patch is working for my recipe.
Comment 18 Joshua Watt 2020-01-31 21:59:22 UTC
Excellent. Thanks again, this was extremely helpful in tracking down the bug, and this bug has been giving us a lot of failures on the autobuilder so we're really happy to be able to resolve it!
Comment 19 Joshua Watt 2020-02-03 20:18:48 UTC
Landed in OE-core as b3313a10a
Comment 20 Joshua Watt 2020-02-04 16:15:29 UTC
*** Bug 13754 has been marked as a duplicate of this bug. ***