| Summary: | externalsrc do_package sets back timestamps ${EXTERNALSRC}, breaking make based builds | ||
|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] OE-Core | Reporter: | Marcus Comstedt <marcus.comstedt> |
| Component: | oe-core other | Assignee: | Marcus Comstedt <marcus.comstedt> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | normal | ||
| Priority: | Medium | CC: | randy.macleod |
| Version: | 5.0.4 | ||
| Target Milestone: | 5.2 | ||
| Hardware: | Other | ||
| OS: | arm64 | ||
| Whiteboard: | NIR 11/29 | ||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |
I find it a bit curious that things are done with files in a directory
"sstate-whatever" in the first place, since externalsrc.bbclass has
# sstate is never going to work for external source trees, disable it
d.setVar('SSTATE_SKIP_CREATION', '1')
Is the manipulation of dates of files in sstate-build-package correct and
required for externalsrc recipes? (If it is, I guess the correct fix might
be to just remove "-l" from the cpio command line, since that was only
added as an optimization anyway...)
We've actually just made some changes to sstate in master that remove that utime() call. Can you try replicating your problem with at least oe-core 475759fdab7200488b2a568b2ba1aa31a456d113, or try cherry-picking that commit to scarthgap (if it applies)? Thanks for the pointer. I was able to cherry-pick 475759fdab7200488b2a568b2ba1aa31a456d113 without any problems, and yes it does indeed seem to fix the issue. Any chance of the fix getting backported to the scarthgap branch, considering this issue appears to be a regression from dunfell? (The utime() call seems to also be present in kirkstone.) Marcus, Ross may nix this suggestion but I think the best approach is to send a styhead?, and scarthgap back-port patch to the oe-core list for review. I got an automatically generated email pointing out that this bug is still in state "NEEDINFO". Do you still need some info from me? In that case I'd be happy to provide it... Hi Marcus, Sorry, I wasn't clear about what I was asking in my previous comment. Since you were able to cherry-pick 475759fdab7200488b2a568b2ba1aa31a456d113 and have things work for you, can you send that backport to the list please? https://docs.yoctoproject.org/dev/contributor-guide/submit-changes.html I don't see anyone having done that already: https://lore.kernel.org/openembedded-core/?q=do_package%2Fsstate%2Fsstatesig If so, then just move the bug out of NEEDINFO and the nag emails will stop. Randy Ok. I was waiting for Ross to chime in, but I guess they are not on CC for this bug. I've sent a mail to the list now. Not sure what I was supposed to change the Status to, but I guess "IN PROGRESS REVIEW" is appropriate since it has been sent to the list for review... The fix has been cherry-picked into 4.0.24 as 0c93bb692b3, into 5.0.6 as 9df0bf5775e, and into 5.1.2 as 0e6b2c761f6. |
Yocto release: scarthgap openembedded-core commit: 136a255674 The issue occurs when building a recipe using externalsrc. What happens is that the mtime of a file in ${EXTERNALSRC} (which is outside of ${TOPDIR}) is changed to a date before it was last modified. This is problematic, because now if a second build tree exists using the same recipe (and thus the same ${EXTERNALSRC}) but with a different ${TMPDIR} and ${SSTATE_DIR}, the newly changed source file will not be recompiled because the mtime is older than the .o files in the second tree. The result is that the wrong (old) code is packaged. In the following run, GPIO.c contains a change made on Nov 8: ---8<--- kobold|master:~/COTM/Mercury_SW/RESA_ACU/yocto% ls -l ../../ACUFE/GPIO.c -rw-r--r-- 3 marcus marcus 6139 Nov 8 10:53 ../../ACUFE/GPIO.c kobold|master:~/COTM/Mercury_SW/RESA_ACU/yocto% bitbake acufe NOTE: Started PRServer with DBfile: /home/marcus/COTM/Mercury_SW/yocto/cache/prserv.sqlite3, Address: 127.0.0.1:42677, PID: 22488 WARNING: XSCT has been deprecated. It will still be available for several releases. In the future, it's recommended to start new projects with SDT workflow. Loading cache: 100% |############################################| Time: 0:00:01 Loaded 5337 entries from dependency cache. Parsing recipes: 100% |##########################################| Time: 0:00:01 Parsing of 3263 .bb files complete (3246 cached, 17 parsed). 5352 targets, 890 skipped, 0 masked, 0 errors. NOTE: Resolving any missing task queue dependencies Build Configuration: BB_VERSION = "2.8.0" BUILD_SYS = "x86_64-linux" NATIVELSBSTRING = "gentoo-2.17" TARGET_SYS = "aarch64-oe-linux-musl" MACHINE = "rapu-acu-mk2" DISTRO = "openwrt" DISTRO_VERSION = "v22.03.1" TUNE_FEATURES = "aarch64 crc cortexa72-cortexa53 crypto" TARGET_FPU = "" XILINX_RELEASE_VERSION = "v2022.2" XILINX_XSCT_VERSION = "2022.2" meta-openwrt = "HEAD:24d0bcafaf097f001164fa9a0f43aa9ca254b7db" meta-oe meta-initramfs meta-networking meta-webserver meta-filesystems meta-perl meta-python = "HEAD:2338409efc51cf2022ff5610a9fb689251706e2b" meta-java = "HEAD:ac65b6e5bbd2c9b55a64a13ab4c55d212a8b585f" meta-intel-fpga = "HEAD:03de87a47b70d7cec1d6cb657714b0ef55e47cae" meta-de10-nano = "master:cb00843fc6638f7e9ef0fbb9230094b34fb4128b" meta-raspberrypi = "HEAD:1918a27419dcd5e79954c0dc0edddcde91057a7e" meta-xilinx-core meta-xilinx-bsp meta-xilinx-standalone = "HEAD:03d3b2ce359a1c2959f04588cf2c4a1b0dcb4de8" meta-xilinx-tools = "HEAD:338f31d1a7ee07e6408925b2101ff8d364792367" meta-requtech = "master:cb00843fc6638f7e9ef0fbb9230094b34fb4128b" meta = "HEAD:136a25567499191b23a4d000a06bf83a473224ca" Sstate summary: Wanted 60 Local 53 Mirrors 0 Missed 7 Current 1115 (88% match, 99% complete) Removing 1 stale sstate objects for arch rapu_acu_mk2: 100% |####| Time: 0:00:00 Removing 6 stale sstate objects for arch cortexa72-cortexa53-crypto: 16% || ETARemoving 6 stale sstate objects for arch cortexa72-cortexa53-crypto: 33% || ETARemoving 6 stale sstate objects for arch cortexa72-cortexa53-crypto: 50% || ETARemoving 6 stale sstate objects for arch cortexa72-cortexa53-crypto: 66% || ETARemoving 6 stale sstate objects for arch cortexa72-cortexa53-crypto: 83% || ETARemoving 6 stale sstate objects for arch cortexa72-cortexa53-crypto: 100% || ETARemoving 6 stale sstate objects for arch cortexa72-cortexa53-crypto: 100% || Time: 0:00:00 NOTE: Executing Tasks NOTE: acufe: compiling from external source tree /home/marcus/COTM/Mercury_SW/yocto/../ACUFE NOTE: Tasks Summary: Attempted 2624 tasks of which 2612 didn't need to be rerun and all succeeded. Summary: There was 1 WARNING message. kobold|master:~/COTM/Mercury_SW/RESA_ACU/yocto% ls -l ../../ACUFE/GPIO.c -rw-r--r-- 3 marcus marcus 6139 Nov 7 15:22 ../../ACUFE/GPIO.c kobold|master:~/COTM/Mercury_SW/RESA_ACU/yocto% ---8<--- Note how the date has been set back to Nov 7. This happens because of the following: 1) When the sources are copied from ${EXTERNALSRC}/ to ${WORKDIR}/package/usr/src/debug/${BPN}/${PV}/, bitbake makes hardlinks and not copies (note that the link count in the ls -l above is 3). I can see this happen in strace: [pid 20907] link("/home/marcus/COTM/Mercury_SW/ACUFE/GPIO.c", "/home/marcus/COTM/Mercury_SW/yocto/build/tmp-resa-openwrt-musl/work/cortexa72-cortexa53-crypto-oe-linux-musl/acufe/1.0/package/usr/src/debug/acufe/1.0/GPIO.c") = 0 Pid 20907 seems to be [pid 20907] execve("/home/marcus/COTM/Mercury_SW/yocto/build/tmp-resa-openwrt-musl/hosttools/cpio", ["cpio", "-pd0mlL", "--no-preserve-owner", "/home/marcus/COTM/Mercury_SW/yoc"...], 0x556d04b4cf70 /* 93 vars */ <unfinished ...> which suggests that copydebugsources() in openembedded-core/meta/lib/oe/package.py is the culprit. 2) ${WORKDIR}/package is temporarily renamed to ${WORKDIR}/sstate-build-package/package (it gets renamed back later). From the strace: [pid 20849] rename("/home/marcus/COTM/Mercury_SW/yocto/build/tmp-resa-openwrt-musl/work/cortexa72-cortexa53-crypto-oe-linux-musl/acufe/1.0/package", "/home/marcus/COTM/Mercury_SW/yocto/build/tmp-resa-openwrt-musl/work/cortexa72-cortexa53-crypto-oe-linux-musl/acufe/1.0/sstate-build-package/package") = 0 3) utime is called on the link in ${WORKDIR}/sstate-build-package//package/usr/src/debug/${BPN}/${PV}/. Since this is now the same inode as the one in ${EXTERNALSRC}/, the mtime of the file in ${EXTERNALSRC}/ is changed. From the strace: [pid 20849] utimensat(AT_FDCWD, "/home/marcus/COTM/Mercury_SW/yocto/build/tmp-resa-openwrt-musl/work/cortexa72-cortexa53-crypto-oe-linux-musl/acufe/1.0/sstate-build-package//package/usr/src/debug/acufe/1.0/GPIO.c", [{tv_sec=1730989324, tv_nsec=0} /* 2024-11-07T15:22:04+0100 */, {tv_sec=1730989324, tv_nsec=0} /* 2024-11-07T15:22:04+0100 */], AT_SYMLINK_NOFOLLOW) = 0