12070 2017-09-08 22:26:16 +0000 Race condition for do_image_complete due to do_populate_lic task not completed for EXTRA_IMAGEDEPENDS and EFI_PROVIDER 2018-08-05 19:24:25 +0000 1 1 7 Build System, Metadata & Runtime OE-Core deployment 2.3.2 x86 Multiple RESOLVED WONTFIX Medium major 2.3.5 1 olivierthomas.j akuster akuster dl9pf gauragup randy.macleod --- --- No (bug/feature does not impact docs) oldest_to_newest 76707 0 olivierthomas.j 2017-09-08 22:26:16 +0000 I observed a consistent race condition on slow running machines, when "EXTRA_IMAGEDEPENDS" and "EFI_PROVIDER" is used. In meta/classes/image.bbclass only the "do_populate_sysroot" task is added as a dependency to "do_image_complete" for all "EXTRA_IMAGEDEPENDS": deps = "" for dep in (d.getVar('EXTRA_IMAGEDEPENDS') or "").split(): deps += " %s:do_populate_sysroot" % dep d.appendVarFlag('do_image_complete', 'depends', deps) In meta/classes/grub-efi.bbclass (this could very well be any EFI_PROVIDER) the "do_deploy" task is added as a dependency to "do_bootimg": do_bootimg[depends] += "${MLPREFIX}grub-efi:do_deploy" and in meta/classes/license.bbclass we include all dependencies from "EXTRA_IMAGEDEPENDS" and do_bootimg[depends] for manifest deployment: def get_deployed_dependencies(d): """ Get all the deployed dependencies of an image """ deploy = {} # Get all the dependencies for the current task (rootfs). # Also get EXTRA_IMAGEDEPENDS because the bootloader is # usually in this var and not listed in rootfs. # At last, get the dependencies from boot classes because # it might contain the bootloader. taskdata = d.getVar("BB_TASKDEPDATA", False) depends = list(set([dep[0] for dep in list(taskdata.values()) if not dep[0].endswith("-native")])) extra_depends = d.getVar("EXTRA_IMAGEDEPENDS") boot_depends = get_boot_dependencies(d) depends.extend(extra_depends.split()) depends.extend(boot_depends) depends = list(set(depends)) To summarize: 1. IMAGE_POSTPROCESS_COMMAND_prepend = "write_deploy_manifest; is called after do_image_complete Call order: write_deploy_manifest -> license_deployed_manifest -> get_deployed_dependencies 2. get_deployed_dependencies includes: "EXTRA_IMAGEDEPENDS" and grub-efi as dependencies for write_deploy_manifest 3. do_image_complete does not depend on EXTRA_IMAGEDEPENDS do_populate_lic task, so it fails to find the files since the task has not been completed (EXTRA_IMAGEDEPENDS = "qemu-native qemu-helper-native ovmf"): The stack trace of python calls that resulted in this exception/failure was: File: 'exec_python_func() autogenerated', lineno: 2, function: <module> 0001: *** 0002:write_deploy_manifest(d) 0003: NOTE: recipe eudev-3.2.1-r0: task do_package_qa: Started File: '/data/build-dir/SANDBOX-VDYOCTO10-JOB1/poky/meta/classes/license.bbclass', lineno: 33, function: write_deploy_manifest 0029: 'w+').write(output) 0030:} 0031: 0032:python write_deploy_manifest() { *** 0033: license_deployed_manifest(d) 0034:} 0035: 0036:python license_create_manifest() { 0037: import oe.packagedata File: '/data/build-dir/SANDBOX-VDYOCTO10-JOB1/poky/meta/classes/license.bbclass', lineno: 192, function: license_deployed_manifest 0188: man_dic[dep]["IMAGE_MANIFEST"] = True 0189: man_dic[dep]["PN"] = dep 0190: man_dic[dep]["FILES"] = \ 0191: " ".join(get_deployed_files(dep_dic[dep])) *** 0192: with open(os.path.join(lic_dir, dep, "recipeinfo"), "r") as f: 0193: for line in f.readlines(): 0194: key,val = line.split(": ", 1) 0195: man_dic[dep][key] = val[:-1] 0196: Exception: FileNotFoundError: [Errno 2] No such file or directory: '/data/build-dir/SANDBOX-VDYOCTO10-JOB1/poky/build/odm/tmp/deploy/licenses/ovmf/recipeinfo 4. do_image_complete does not depend on grub-efi do_populate_lic task, so it fails to find the files since the task has not been completed: The stack trace of python calls that resulted in this exception/failure was: File: 'exec_python_func() autogenerated', lineno: 2, function: <module> 0001: *** 0002:write_deploy_manifest(d) 0003: File: '/data/build-dir/SANDBOX-VDYOCTO10-JOB1/poky/meta/classes/license.bbclass', lineno: 33, function: write_deploy_manifest 0029: 'w+').write(output) 0030:} 0031: 0032:python write_deploy_manifest() { *** 0033: license_deployed_manifest(d) 0034:} 0035: 0036:python license_create_manifest() { 0037: import oe.packagedata File: '/data/build-dir/SANDBOX-VDYOCTO10-JOB1/poky/meta/classes/license.bbclass', lineno: 192, function: license_deployed_manifest 0188: man_dic[dep]["IMAGE_MANIFEST"] = True 0189: man_dic[dep]["PN"] = dep 0190: man_dic[dep]["FILES"] = \ 0191: " ".join(get_deployed_files(dep_dic[dep])) *** 0192: with open(os.path.join(lic_dir, dep, "recipeinfo"), "r") as f: 0193: for line in f.readlines(): 0194: key,val = line.split(": ", 1) 0195: man_dic[dep][key] = val[:-1] 0196: Exception: FileNotFoundError: [Errno 2] No such file or directory: '/data/build-dir/SANDBOX-VDYOCTO10-JOB1/poky/build/odm/tmp/deploy/licenses/grub-efi/recipeinfo' This may be split up into two bugs, one for grub-efi and EXTRA_IMAGEDEPENDS. 76953 1 akuster 2017-09-18 23:26:34 +0000 Oliver, Have you tried this scenario against Master? How often does this occur? Is there a test case or scenario that will help me reproduce this. Host info and local.conf may be helpful. By slow machine, can you define that? - armin 77017 2 sjolley.yp.pm 2017-09-21 14:35:46 +0000 Please review and answer the question. 77598 3 dl9pf 2017-10-15 06:48:58 +0000 Here it happens on pyro. For me slow manchine means 8 core 16G ram but quite slow i/o . Happens quite reliably e.g. ovmf: https://build.automotivelinux.org/view/jjb-release-jobs/job/release-jjb-eel-snapshot/MACHINE=intel-corei7-64,label=agl-test-slave/7/consoleFull But also with other packages, e.g.: systemd-boot - https://build.automotivelinux.org/view/jjb-release-jobs/job/release-jjb-eel-snapshot/MACHINE=intel-corei7-64,label=agl-test-slave/6/console rmc-db - https://build.automotivelinux.org/view/jjb-release-jobs/job/release-jjb-eel-snapshot/MACHINE=intel-corei7-64,label=agl-test-slave/5/console Interestingly it only fails on builds with meta-intel for me. Six other builds from the same set complete w/o issues. 77600 4 dl9pf 2017-10-15 21:08:59 +0000 Correction: it turns out that with PARALLEL_MAKE and BB_NUMBER_THREADS both set to 1 the problem is reproducible on all machines. 77601 5 akuster 2017-10-15 21:24:08 +0000 (In reply to comment #4) > Correction: it turns out that with PARALLEL_MAKE and BB_NUMBER_THREADS both > set to 1 the problem is reproducible on all machines. Perfect. Thanks Jan 77613 6 dl9pf 2017-10-16 19:10:59 +0000 http://lists.openembedded.org/pipermail/openembedded-core/2017-September/142760.html might be a solution. @Armin: could you consider that for pyro ? 77631 7 dl9pf 2017-10-17 08:58:09 +0000 As temp solution in my local.conf I used do_image_complete[deptask] += "do_populate_lic" and the builds went through finally. 78871 8 gauragup 2018-01-02 21:31:15 +0000 Fix/Change in Comment#6 solves the EXTRA_IMAGEDEPENDS issue but the grub-efi issue is still there. Temp solution given by Jan-Simon is able to make it go past the grub-efi issue. 81247 9 akuster 2018-08-05 19:24:25 +0000 At this time we will stick with the work around.