| Summary: | Race condition for do_image_complete due to do_populate_lic task not completed for EXTRA_IMAGEDEPENDS and EFI_PROVIDER | ||
|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] OE-Core | Reporter: | olivierthomas.j <olivierthomas.j> |
| Component: | deployment | Assignee: | Armin Kuster <akuster> |
| Status: | RESOLVED WONTFIX | QA Contact: | |
| Severity: | major | ||
| Priority: | Medium | CC: | akuster, dl9pf, gauragup, randy.macleod |
| Version: | 2.3.2 | ||
| Target Milestone: | 2.3.5 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |
Oliver, Have you tried this scenario against Master? How often does this occur? Is there a test case or scenario that will help me reproduce this. Host info and local.conf may be helpful. By slow machine, can you define that? - armin Please review and answer the question. Here it happens on pyro. For me slow manchine means 8 core 16G ram but quite slow i/o . Happens quite reliably e.g. ovmf: https://build.automotivelinux.org/view/jjb-release-jobs/job/release-jjb-eel-snapshot/MACHINE=intel-corei7-64,label=agl-test-slave/7/consoleFull But also with other packages, e.g.: systemd-boot - https://build.automotivelinux.org/view/jjb-release-jobs/job/release-jjb-eel-snapshot/MACHINE=intel-corei7-64,label=agl-test-slave/6/console rmc-db - https://build.automotivelinux.org/view/jjb-release-jobs/job/release-jjb-eel-snapshot/MACHINE=intel-corei7-64,label=agl-test-slave/5/console Interestingly it only fails on builds with meta-intel for me. Six other builds from the same set complete w/o issues. Correction: it turns out that with PARALLEL_MAKE and BB_NUMBER_THREADS both set to 1 the problem is reproducible on all machines. (In reply to comment #4) > Correction: it turns out that with PARALLEL_MAKE and BB_NUMBER_THREADS both > set to 1 the problem is reproducible on all machines. Perfect. Thanks Jan http://lists.openembedded.org/pipermail/openembedded-core/2017-September/142760.html might be a solution. @Armin: could you consider that for pyro ? As temp solution in my local.conf I used do_image_complete[deptask] += "do_populate_lic" and the builds went through finally. Fix/Change in Comment#6 solves the EXTRA_IMAGEDEPENDS issue but the grub-efi issue is still there. Temp solution given by Jan-Simon is able to make it go past the grub-efi issue. At this time we will stick with the work around. |
I observed a consistent race condition on slow running machines, when "EXTRA_IMAGEDEPENDS" and "EFI_PROVIDER" is used. In meta/classes/image.bbclass only the "do_populate_sysroot" task is added as a dependency to "do_image_complete" for all "EXTRA_IMAGEDEPENDS": deps = "" for dep in (d.getVar('EXTRA_IMAGEDEPENDS') or "").split(): deps += " %s:do_populate_sysroot" % dep d.appendVarFlag('do_image_complete', 'depends', deps) In meta/classes/grub-efi.bbclass (this could very well be any EFI_PROVIDER) the "do_deploy" task is added as a dependency to "do_bootimg": do_bootimg[depends] += "${MLPREFIX}grub-efi:do_deploy" and in meta/classes/license.bbclass we include all dependencies from "EXTRA_IMAGEDEPENDS" and do_bootimg[depends] for manifest deployment: def get_deployed_dependencies(d): """ Get all the deployed dependencies of an image """ deploy = {} # Get all the dependencies for the current task (rootfs). # Also get EXTRA_IMAGEDEPENDS because the bootloader is # usually in this var and not listed in rootfs. # At last, get the dependencies from boot classes because # it might contain the bootloader. taskdata = d.getVar("BB_TASKDEPDATA", False) depends = list(set([dep[0] for dep in list(taskdata.values()) if not dep[0].endswith("-native")])) extra_depends = d.getVar("EXTRA_IMAGEDEPENDS") boot_depends = get_boot_dependencies(d) depends.extend(extra_depends.split()) depends.extend(boot_depends) depends = list(set(depends)) To summarize: 1. IMAGE_POSTPROCESS_COMMAND_prepend = "write_deploy_manifest; is called after do_image_complete Call order: write_deploy_manifest -> license_deployed_manifest -> get_deployed_dependencies 2. get_deployed_dependencies includes: "EXTRA_IMAGEDEPENDS" and grub-efi as dependencies for write_deploy_manifest 3. do_image_complete does not depend on EXTRA_IMAGEDEPENDS do_populate_lic task, so it fails to find the files since the task has not been completed (EXTRA_IMAGEDEPENDS = "qemu-native qemu-helper-native ovmf"): The stack trace of python calls that resulted in this exception/failure was: File: 'exec_python_func() autogenerated', lineno: 2, function: <module> 0001: *** 0002:write_deploy_manifest(d) 0003: NOTE: recipe eudev-3.2.1-r0: task do_package_qa: Started File: '/data/build-dir/SANDBOX-VDYOCTO10-JOB1/poky/meta/classes/license.bbclass', lineno: 33, function: write_deploy_manifest 0029: 'w+').write(output) 0030:} 0031: 0032:python write_deploy_manifest() { *** 0033: license_deployed_manifest(d) 0034:} 0035: 0036:python license_create_manifest() { 0037: import oe.packagedata File: '/data/build-dir/SANDBOX-VDYOCTO10-JOB1/poky/meta/classes/license.bbclass', lineno: 192, function: license_deployed_manifest 0188: man_dic[dep]["IMAGE_MANIFEST"] = True 0189: man_dic[dep]["PN"] = dep 0190: man_dic[dep]["FILES"] = \ 0191: " ".join(get_deployed_files(dep_dic[dep])) *** 0192: with open(os.path.join(lic_dir, dep, "recipeinfo"), "r") as f: 0193: for line in f.readlines(): 0194: key,val = line.split(": ", 1) 0195: man_dic[dep][key] = val[:-1] 0196: Exception: FileNotFoundError: [Errno 2] No such file or directory: '/data/build-dir/SANDBOX-VDYOCTO10-JOB1/poky/build/odm/tmp/deploy/licenses/ovmf/recipeinfo 4. do_image_complete does not depend on grub-efi do_populate_lic task, so it fails to find the files since the task has not been completed: The stack trace of python calls that resulted in this exception/failure was: File: 'exec_python_func() autogenerated', lineno: 2, function: <module> 0001: *** 0002:write_deploy_manifest(d) 0003: File: '/data/build-dir/SANDBOX-VDYOCTO10-JOB1/poky/meta/classes/license.bbclass', lineno: 33, function: write_deploy_manifest 0029: 'w+').write(output) 0030:} 0031: 0032:python write_deploy_manifest() { *** 0033: license_deployed_manifest(d) 0034:} 0035: 0036:python license_create_manifest() { 0037: import oe.packagedata File: '/data/build-dir/SANDBOX-VDYOCTO10-JOB1/poky/meta/classes/license.bbclass', lineno: 192, function: license_deployed_manifest 0188: man_dic[dep]["IMAGE_MANIFEST"] = True 0189: man_dic[dep]["PN"] = dep 0190: man_dic[dep]["FILES"] = \ 0191: " ".join(get_deployed_files(dep_dic[dep])) *** 0192: with open(os.path.join(lic_dir, dep, "recipeinfo"), "r") as f: 0193: for line in f.readlines(): 0194: key,val = line.split(": ", 1) 0195: man_dic[dep][key] = val[:-1] 0196: Exception: FileNotFoundError: [Errno 2] No such file or directory: '/data/build-dir/SANDBOX-VDYOCTO10-JOB1/poky/build/odm/tmp/deploy/licenses/grub-efi/recipeinfo' This may be split up into two bugs, one for grub-efi and EXTRA_IMAGEDEPENDS.