Bug 12070

Summary: Race condition for do_image_complete due to do_populate_lic task not completed for EXTRA_IMAGEDEPENDS and EFI_PROVIDER
Product: [Build System, Metadata & Runtime] OE-Core Reporter: olivierthomas.j <olivierthomas.j>
Component: deploymentAssignee: Armin Kuster <akuster>
Status: RESOLVED WONTFIX QA Contact:
Severity: major    
Priority: Medium CC: akuster, dl9pf, gauragup, randy.macleod
Version: 2.3.2   
Target Milestone: 2.3.5   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description olivierthomas.j 2017-09-08 22:26:16 UTC
I observed a consistent race condition on slow running machines, when "EXTRA_IMAGEDEPENDS" and "EFI_PROVIDER" is used.

In meta/classes/image.bbclass only the "do_populate_sysroot" task is added as a dependency to "do_image_complete" for all "EXTRA_IMAGEDEPENDS":
deps = ""
    for dep in (d.getVar('EXTRA_IMAGEDEPENDS') or "").split():
        deps += " %s:do_populate_sysroot" % dep
    d.appendVarFlag('do_image_complete', 'depends', deps)

In meta/classes/grub-efi.bbclass (this could very well be any EFI_PROVIDER) the "do_deploy" task is added as a dependency to "do_bootimg":
do_bootimg[depends] += "${MLPREFIX}grub-efi:do_deploy"

and in meta/classes/license.bbclass we include all dependencies from "EXTRA_IMAGEDEPENDS" and do_bootimg[depends] for manifest deployment:
def get_deployed_dependencies(d):
    """
    Get all the deployed dependencies of an image
    """

    deploy = {}
    # Get all the dependencies for the current task (rootfs).
    # Also get EXTRA_IMAGEDEPENDS because the bootloader is
    # usually in this var and not listed in rootfs.
    # At last, get the dependencies from boot classes because
    # it might contain the bootloader.
    taskdata = d.getVar("BB_TASKDEPDATA", False)
    depends = list(set([dep[0] for dep
                    in list(taskdata.values())
                    if not dep[0].endswith("-native")]))
    extra_depends = d.getVar("EXTRA_IMAGEDEPENDS")
    boot_depends = get_boot_dependencies(d)
    depends.extend(extra_depends.split())
    depends.extend(boot_depends)
    depends = list(set(depends))

To summarize:
1. IMAGE_POSTPROCESS_COMMAND_prepend = "write_deploy_manifest; is called after do_image_complete
   Call order: write_deploy_manifest -> license_deployed_manifest -> get_deployed_dependencies
2. get_deployed_dependencies includes:
   "EXTRA_IMAGEDEPENDS" and grub-efi as dependencies for write_deploy_manifest
3. do_image_complete does not depend on EXTRA_IMAGEDEPENDS do_populate_lic task, so it fails to find the files since the task has not been completed (EXTRA_IMAGEDEPENDS = "qemu-native qemu-helper-native ovmf"):

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:write_deploy_manifest(d)
     0003:
NOTE: recipe eudev-3.2.1-r0: task do_package_qa: Started
File: '/data/build-dir/SANDBOX-VDYOCTO10-JOB1/poky/meta/classes/license.bbclass', lineno: 33, function: write_deploy_manifest
     0029:        'w+').write(output)
     0030:}
     0031:
     0032:python write_deploy_manifest() {
 *** 0033:    license_deployed_manifest(d)
     0034:}
     0035:
     0036:python license_create_manifest() {
     0037:    import oe.packagedata
File: '/data/build-dir/SANDBOX-VDYOCTO10-JOB1/poky/meta/classes/license.bbclass', lineno: 192, function: license_deployed_manifest
     0188:        man_dic[dep]["IMAGE_MANIFEST"] = True
     0189:        man_dic[dep]["PN"] = dep
     0190:        man_dic[dep]["FILES"] = \
     0191:            " ".join(get_deployed_files(dep_dic[dep]))
 *** 0192:        with open(os.path.join(lic_dir, dep, "recipeinfo"), "r") as f:
     0193:            for line in f.readlines():
     0194:                key,val = line.split(": ", 1)
     0195:                man_dic[dep][key] = val[:-1]
     0196:
Exception: FileNotFoundError: [Errno 2] No such file or directory: '/data/build-dir/SANDBOX-VDYOCTO10-JOB1/poky/build/odm/tmp/deploy/licenses/ovmf/recipeinfo

4. do_image_complete does not depend on grub-efi do_populate_lic task, so it fails to find the files since the task has not been completed:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:write_deploy_manifest(d)
     0003:
File: '/data/build-dir/SANDBOX-VDYOCTO10-JOB1/poky/meta/classes/license.bbclass', lineno: 33, function: write_deploy_manifest
     0029:        'w+').write(output)
     0030:}
     0031:
     0032:python write_deploy_manifest() {
 *** 0033:    license_deployed_manifest(d)
     0034:}
     0035:
     0036:python license_create_manifest() {
     0037:    import oe.packagedata
File: '/data/build-dir/SANDBOX-VDYOCTO10-JOB1/poky/meta/classes/license.bbclass', lineno: 192, function: license_deployed_manifest
     0188:        man_dic[dep]["IMAGE_MANIFEST"] = True
     0189:        man_dic[dep]["PN"] = dep
     0190:        man_dic[dep]["FILES"] = \
     0191:            " ".join(get_deployed_files(dep_dic[dep]))
 *** 0192:        with open(os.path.join(lic_dir, dep, "recipeinfo"), "r") as f:
     0193:            for line in f.readlines():
     0194:                key,val = line.split(": ", 1)
     0195:                man_dic[dep][key] = val[:-1]
     0196:
Exception: FileNotFoundError: [Errno 2] No such file or directory: '/data/build-dir/SANDBOX-VDYOCTO10-JOB1/poky/build/odm/tmp/deploy/licenses/grub-efi/recipeinfo'

This may be split up into two bugs, one for grub-efi and EXTRA_IMAGEDEPENDS.
Comment 1 Armin Kuster 2017-09-18 23:26:34 UTC
Oliver,

Have you tried this scenario against Master?
How often does this occur?
Is there a test case or scenario that will help me reproduce this.
Host info and local.conf may be helpful.

By slow machine, can you define that?

- armin
Comment 2 Stephen K Jolley 2017-09-21 14:35:46 UTC
Please review and answer the question.
Comment 3 Jan-Simon Moeller 2017-10-15 06:48:58 UTC
Here it happens on pyro.

For me slow manchine means 8 core 16G ram but quite slow i/o . Happens quite reliably e.g.

ovmf:
https://build.automotivelinux.org/view/jjb-release-jobs/job/release-jjb-eel-snapshot/MACHINE=intel-corei7-64,label=agl-test-slave/7/consoleFull

But also with other packages, e.g.: 
systemd-boot - 
https://build.automotivelinux.org/view/jjb-release-jobs/job/release-jjb-eel-snapshot/MACHINE=intel-corei7-64,label=agl-test-slave/6/console

rmc-db - 
https://build.automotivelinux.org/view/jjb-release-jobs/job/release-jjb-eel-snapshot/MACHINE=intel-corei7-64,label=agl-test-slave/5/console


Interestingly it only fails on builds with meta-intel for me. Six other builds from the same set complete w/o issues.
Comment 4 Jan-Simon Moeller 2017-10-15 21:08:59 UTC
Correction: it turns out that with PARALLEL_MAKE and BB_NUMBER_THREADS both set to 1  the problem is reproducible on all machines.
Comment 5 Armin Kuster 2017-10-15 21:24:08 UTC
(In reply to comment #4)
> Correction: it turns out that with PARALLEL_MAKE and BB_NUMBER_THREADS both
> set to 1  the problem is reproducible on all machines.

Perfect. Thanks Jan
Comment 6 Jan-Simon Moeller 2017-10-16 19:10:59 UTC
http://lists.openembedded.org/pipermail/openembedded-core/2017-September/142760.html 

might be a solution.

@Armin: could you consider that for pyro ?
Comment 7 Jan-Simon Moeller 2017-10-17 08:58:09 UTC
As temp solution in my local.conf I used

do_image_complete[deptask] += "do_populate_lic" 

and the builds went through finally.
Comment 8 Gaurav Gupta 2018-01-02 21:31:15 UTC
Fix/Change in Comment#6 solves the EXTRA_IMAGEDEPENDS issue but the grub-efi issue is still there. Temp solution given by Jan-Simon is able to make it go past the grub-efi issue.
Comment 9 Armin Kuster 2018-08-05 19:24:25 UTC
At this time we will stick with the work around.