Bug 13280 - Race condition in wic to package IMAGE_BOOT_FILES before EXTRA_IMAGEDEPENDS are built
Summary: Race condition in wic to package IMAGE_BOOT_FILES before EXTRA_IMAGEDEPENDS a...
Status: RESOLVED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: oe-core other (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 2.8 M1
Assignee: Richard Purdie
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2019-04-12 21:10 UTC by Denys Dmytriyenko
Modified: 2019-06-04 10:48 UTC (History)
2 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Don't know


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Denys Dmytriyenko 2019-04-12 21:10:53 UTC
It seems wic has a race condition for not respecting EXTRA_IMAGEDEPENDS to finish building before trying to package resulting files listed in IMAGE_BOOT_FILES.

It manifests when rootfs is ready, but U-boot needs to be rebuilt, with the following example config, like in beaglebone-yocto:
IMAGE_BOOT_FILES = "u-boot.img"
EXTRA_IMAGEDEPENDS = "u-boot"

Error log looks like:
| ERROR: _exec_cmd: install -m 0644 -D /home/pokybuild/yocto-worker/beaglebone/build/build/tmp/deploy/images/beaglebone-yocto/u-boot.img /home/pokybuild/yocto-worker/beaglebone/build/build/tmp/work/beaglebone_yocto-poky-linux-gnueabi/core-image-sato-sdk-ptest/1.0-r0/deploy-core-image-sato-sdk-ptest-image-complete/core-image-sato-sdk-ptest-beaglebone-yocto-20190410232147/tmp.wic.4xf3vi9f/boot.1/u-boot.img returned '1' instead of 0
| output: install: cannot stat '/home/pokybuild/yocto-worker/beaglebone/build/build/tmp/deploy/images/beaglebone-yocto/u-boot.img': No such file or directory
| ERROR: Function failed: do_image_wic (log file is located at /home/pokybuild/yocto-worker/beaglebone/build/build/tmp/work/beaglebone_yocto-poky-linux-gnueabi/core-image-sato-sdk-ptest/1.0-r0/temp/log.do_image_wic.41960)

Encountered here:
https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/490
https://autobuilder.yoctoproject.org/typhoon/#/builders/46/builds/489

Discussion on the mailing list:
http://lists.openembedded.org/pipermail/openembedded-core/2019-April/281149.html
Comment 1 Richard Purdie 2019-04-12 21:26:56 UTC
Just to add, I have a suspicion that in a failed build the autobuilder shows the error due to the use of "bitbake -k" (continue mode) which most builds therefore wouldn't see. You therefore work best with an underlying uboot failure then the -k option to expose the race.
Comment 2 Richard Purdie 2019-05-31 16:58:06 UTC
This isn't what EXTRA_IMAGEDEPENDS is designed to do. These dependencies are added to do_image_complete which clearly isn't what is needed here. I think the fix might be:

diff --git a/meta-yocto-bsp/conf/machine/beaglebone-yocto.conf b/meta-yocto-bsp/conf/machine/beaglebone-yocto.conf
index bc18ee8c47e..35814cd8f1a 100644
--- a/meta-yocto-bsp/conf/machine/beaglebone-yocto.conf
+++ b/meta-yocto-bsp/conf/machine/beaglebone-yocto.conf
@@ -18,7 +18,7 @@ IMAGE_FSTYPES += "tar.bz2 jffs2 wic wic.bmap"
 EXTRA_IMAGECMD_jffs2 = "-lnp "
 WKS_FILE ?= "beaglebone-yocto.wks"
 IMAGE_INSTALL_append = " kernel-devicetree kernel-image-zimage"
-do_image_wic[depends] += "mtools-native:do_populate_sysroot dosfstools-native:do_populate_sysroot"
+do_image_wic[depends] += "mtools-native:do_populate_sysroot dosfstools-native:do_populate_sysroot u-boot:do_deploy"
 
 SERIAL_CONSOLES ?= "115200;ttyS0 115200;ttyO0"
 SERIAL_CONSOLES_CHECK = "${SERIAL_CONSOLES}"
Comment 4 Denys Dmytriyenko 2019-06-02 19:28:07 UTC
Richard,

I already had this workaround for a month:
http://git.yoctoproject.org/cgit/cgit.cgi/meta-ti/commit/?id=484c2d0d126722bcbb1f3aa3eac01472361c145a

But I don't believe it is a proper fix... It used to work just fine with EXTRA_IMAGEDEPENDS for long time until we moved to thud and later - it would never fail in do_image_wic like that, maybe we were just lucky before. But now we've seen this breakage intermittently even w/o u-boot build failing, so some dependencies got rearranged and do_image_wic gets ahead of u-boot:do_deploy sometimes...

BTW, why does EXTRA_IMAGEDEPEDNS add to do_image_complete? It kind of makes the name of the variable slightly confusing, as those are rather POST image depends. I know those deps are not meant to go into the image, so maybe EXTRA_POSTIMAGEDEPS is a better name, though historical reasons, I guess...
Comment 5 Richard Purdie 2019-06-04 10:48:11 UTC
(In reply to comment #4)
> Richard,
> 
> I already had this workaround for a month:
> http://git.yoctoproject.org/cgit/cgit.cgi/meta-ti/commit/
> ?id=484c2d0d126722bcbb1f3aa3eac01472361c145a
> 
> But I don't believe it is a proper fix... It used to work just fine with
> EXTRA_IMAGEDEPENDS for long time until we moved to thud and later - it would
> never fail in do_image_wic like that, maybe we were just lucky before. But
> now we've seen this breakage intermittently even w/o u-boot build failing,
> so some dependencies got rearranged and do_image_wic gets ahead of
> u-boot:do_deploy sometimes...

Having looked at this quite carefully I believe its just been luck and this is the correct fix. I think you could test that by executing "bitbake image:do_image_wic" explicitly even pre thud and see the problem if you'd cleaned u-boot prior to that.

Put another way, if you specify a wks which needs the u-boot deploy artefacts, you also have to add a dependency for that to do_image_wic.

> BTW, why does EXTRA_IMAGEDEPEDNS add to do_image_complete? It kind of makes
> the name of the variable slightly confusing, as those are rather POST image
> depends. I know those deps are not meant to go into the image, so maybe
> EXTRA_POSTIMAGEDEPS is a better name, though historical reasons, I guess...

Its a very old variable, it even predates task dependencies and the multithreaded runqueue in bitbake.
We made it do what its contents mean rather than its name. Ideally we should probably clean up some of these things but that is a people/time problem.