14721 2022-02-12 19:49:02 +0000 Intermittent runqueue issue re-running populate_sysroot_setscene after populate_sysroot 2023-03-07 17:12:19 +0000 1 1 7 Build System, Metadata & Runtime BitBake bitbake unspecified x86 Multiple RESOLVED FIXED Medium+ normal 4.2 M3 1 denis richard.purdie luca.ceresoli poky.bs.watcher poky.watcher randy.macleod --- --- No (bug/feature does not impact docs) oldest_to_newest 92547 0 denis 2022-02-12 19:49:02 +0000 Got qemu-native:do_prepare_recipe_sysroot failure due to zlib.h not being ready from zlib-native:do_populate_sysroot ERROR: qemu-native-6.2.0-r0 do_prepare_recipe_sysroot: Error executing a python function in exec_func_python() autogenerated: The stack trace of python calls that resulted in this exception/failure was: File: 'exec_func_python() autogenerated', lineno: 2, function: <module> 0001: *** 0002:extend_recipe_sysroot(d) 0003: File: '/OE/new-arago-master/meta/classes/staging.bbclass', lineno: 583, function: extend_recipe_sysroot 0579: if "/bin/" in l or "/sbin/" in l: 0580: # defer /*bin/* files until last in case they need libs 0581: binfiles[l] = (targetdir, dest) 0582: else: *** 0583: staging_copyfile(l, targetdir, dest, postinsts, seendirs) 0584: 0585: # Handle deferred binfiles 0586: for l in binfiles: 0587: (targetdir, dest) = binfiles[l] File: '/OE/new-arago-master/meta/classes/staging.bbclass', lineno: 157, function: staging_copyfile 0153: os.symlink(linkto, dest) 0154: #bb.warn(c) 0155: else: 0156: try: *** 0157: os.link(c, dest) 0158: except OSError as err: 0159: if err.errno == errno.EXDEV: 0160: bb.utils.copyfile(c, dest) 0161: else: Exception: FileNotFoundError: [Errno 2] No such file or directory: '/OE/new-arago-master/build/tmp-default-glibc/sysroots-components/x86_64/zlib-native/usr/include/zlib.h' -> '/OE/new-arago-master/build/tmp-default-glibc/work/x86_64-linux/qemu-native/6.2.0-r0/recipe-sysroot-native/usr/include/zlib.h' Cooker logs for all qemu-native and zlib-native tasks show that zlib-native successfully executed do_populate_sysroot task and then decided to re-run do_populate_sysroot_setscene task in the middle of qemu-native:do_prepare_recipe_sysroot NOTE: recipe zlib-native-1.2.11-r0: task do_fetch: Started NOTE: recipe zlib-native-1.2.11-r0: task do_fetch: Succeeded NOTE: recipe zlib-native-1.2.11-r0: task do_prepare_recipe_sysroot: Started NOTE: recipe zlib-native-1.2.11-r0: task do_prepare_recipe_sysroot: Succeeded NOTE: Running task 151 of 9535 (/OE/new-arago-master/meta/recipes-devtools/qemu/qemu-native_6.2.0.bb:do_fetch) NOTE: recipe qemu-native-6.2.0-r0: task do_fetch: Started NOTE: recipe qemu-native-6.2.0-r0: task do_fetch: Succeeded NOTE: Running task 2026 of 9535 (/OE/new-arago-master/meta/recipes-devtools/qemu/qemu-native_6.2.0.bb:do_unpack) NOTE: recipe zlib-native-1.2.11-r0: task do_unpack: Started NOTE: recipe qemu-native-6.2.0-r0: task do_unpack: Started NOTE: recipe zlib-native-1.2.11-r0: task do_unpack: Succeeded NOTE: recipe zlib-native-1.2.11-r0: task do_patch: Started NOTE: recipe zlib-native-1.2.11-r0: task do_patch: Succeeded NOTE: recipe zlib-native-1.2.11-r0: task do_deploy_source_date_epoch: Started NOTE: recipe zlib-native-1.2.11-r0: task do_create_srcipk: Started NOTE: recipe zlib-native-1.2.11-r0: task do_create_srcipk: Succeeded NOTE: recipe zlib-native-1.2.11-r0: task do_deploy_source_date_epoch: Succeeded NOTE: recipe qemu-native-6.2.0-r0: task do_unpack: Succeeded NOTE: Running task 2170 of 9535 (/OE/new-arago-master/meta/recipes-devtools/qemu/qemu-native_6.2.0.bb:do_patch) NOTE: recipe zlib-native-1.2.11-r0: task do_configure: Started NOTE: recipe qemu-native-6.2.0-r0: task do_patch: Started NOTE: recipe zlib-native-1.2.11-r0: task do_configure: Succeeded NOTE: recipe qemu-native-6.2.0-r0: task do_patch: Succeeded NOTE: Running task 2279 of 9535 (/OE/new-arago-master/meta/recipes-devtools/qemu/qemu-native_6.2.0.bb:do_create_srcipk) NOTE: Running task 2280 of 9535 (/OE/new-arago-master/meta/recipes-devtools/qemu/qemu-native_6.2.0.bb:do_deploy_source_date_epoch) NOTE: recipe zlib-native-1.2.11-r0: task do_compile: Started NOTE: recipe qemu-native-6.2.0-r0: task do_create_srcipk: Started NOTE: recipe qemu-native-6.2.0-r0: task do_deploy_source_date_epoch: Started NOTE: recipe qemu-native-6.2.0-r0: task do_create_srcipk: Succeeded NOTE: Running task 2352 of 9535 (/OE/new-arago-master/meta/recipes-devtools/qemu/qemu-native_6.2.0.bb:do_populate_lic) NOTE: recipe qemu-native-6.2.0-r0: task do_deploy_source_date_epoch: Succeeded NOTE: Task /OE/new-arago-master/meta/recipes-devtools/qemu/qemu-native_6.2.0.bb:do_deploy_source_date_epoch unihash changed to 93fe89d6405b5f00058aa4eb64beef13e6269f5e7f10cfd703e96cc0affe53d1 NOTE: recipe qemu-native-6.2.0-r0: task do_populate_lic: Started NOTE: recipe zlib-native-1.2.11-r0: task do_populate_lic: Started NOTE: recipe zlib-native-1.2.11-r0: task do_compile: Succeeded NOTE: recipe qemu-native-6.2.0-r0: task do_populate_lic: Succeeded NOTE: recipe zlib-native-1.2.11-r0: task do_populate_lic: Succeeded NOTE: Task /OE/new-arago-master/meta/recipes-devtools/qemu/qemu-native_6.2.0.bb:do_populate_lic unihash changed to a6dced3dae65779c96d0299625aabe4c857cbd0fda6dc11c5d2d5497a302f821 NOTE: recipe zlib-native-1.2.11-r0: task do_install: Started NOTE: recipe zlib-native-1.2.11-r0: task do_install: Succeeded NOTE: recipe zlib-native-1.2.11-r0: task do_populate_sysroot: Started NOTE: recipe zlib-native-1.2.11-r0: task do_populate_sysroot: Succeeded NOTE: recipe zlib-native-1.2.11-r0: task do_deploy_source_date_epoch: Started NOTE: recipe zlib-native-1.2.11-r0: task do_deploy_source_date_epoch: Succeeded NOTE: recipe zlib-native-1.2.11-r0: task do_populate_lic: Started NOTE: recipe zlib-native-1.2.11-r0: task do_populate_lic: Succeeded NOTE: Running task 5223 of 9535 (/OE/new-arago-master/meta/recipes-devtools/qemu/qemu-native_6.2.0.bb:do_prepare_recipe_sysroot) NOTE: recipe zlib-native-1.2.11-r0: task do_populate_sysroot_setscene: Started NOTE: recipe qemu-native-6.2.0-r0: task do_prepare_recipe_sysroot: Started ERROR: qemu-native-6.2.0-r0 do_prepare_recipe_sysroot: Error executing a python function in exec_func_python() autogenerated: Exception: FileNotFoundError: [Errno 2] No such file or directory: '/OE/new-arago-master/build/tmp-default-glibc/sysroots-components/x86_64/zlib-native/usr/include/zlib.h' -> '/OE/new-arago-master/build/tmp-default-glibc/work/x86_64-linux/qemu-native/6.2.0-r0/recipe-sysroot-native/usr/include/zlib.h' NOTE: recipe zlib-native-1.2.11-r0: task do_populate_sysroot_setscene: Succeeded ERROR: Logfile of failure stored in: /OE/new-arago-master/build/tmp-default-glibc/work/x86_64-linux/qemu-native/6.2.0-r0/temp/log.do_prepare_recipe_sysroot.1169796 NOTE: recipe qemu-native-6.2.0-r0: task do_prepare_recipe_sysroot: Failed ERROR: Task (/OE/new-arago-master/meta/recipes-devtools/qemu/qemu-native_6.2.0.bb:do_prepare_recipe_sysroot) failed with exit code '1' NOTE: recipe zlib-native-1.2.11-r0: task do_deploy_source_date_epoch: Started NOTE: recipe zlib-native-1.2.11-r0: task do_deploy_source_date_epoch: Succeeded NOTE: recipe zlib-native-1.2.11-r0: task do_populate_lic: Started NOTE: recipe zlib-native-1.2.11-r0: task do_populate_lic: Succeeded 92548 1 denis 2022-02-13 22:34:07 +0000 Another instance of the same issue: NOTE: recipe util-linux-native-2.37.3-r0: task do_fetch: Started NOTE: recipe util-linux-native-2.37.3-r0: task do_fetch: Succeeded NOTE: recipe vala-native-0.54.6-r0: task do_fetch: Started NOTE: recipe vala-native-0.54.6-r0: task do_fetch: Succeeded NOTE: recipe util-linux-native-2.37.3-r0: task do_unpack: Started NOTE: recipe vala-native-0.54.6-r0: task do_unpack: Started NOTE: recipe vala-native-0.54.6-r0: task do_unpack: Succeeded NOTE: recipe vala-native-0.54.6-r0: task do_patch: Started NOTE: recipe vala-native-0.54.6-r0: task do_patch: Succeeded NOTE: recipe util-linux-native-2.37.3-r0: task do_unpack: Succeeded NOTE: recipe util-linux-native-2.37.3-r0: task do_patch: Started NOTE: recipe vala-native-0.54.6-r0: task do_create_srcipk: Started NOTE: recipe vala-native-0.54.6-r0: task do_deploy_source_date_epoch: Started NOTE: recipe vala-native-0.54.6-r0: task do_create_srcipk: Succeeded NOTE: recipe vala-native-0.54.6-r0: task do_deploy_source_date_epoch: Succeeded NOTE: recipe util-linux-native-2.37.3-r0: task do_patch: Succeeded NOTE: recipe util-linux-native-2.37.3-r0: task do_deploy_source_date_epoch: Started NOTE: recipe util-linux-native-2.37.3-r0: task do_create_srcipk: Started NOTE: recipe util-linux-native-2.37.3-r0: task do_create_srcipk: Succeeded NOTE: recipe vala-native-0.54.6-r0: task do_populate_lic: Started NOTE: recipe util-linux-native-2.37.3-r0: task do_populate_lic: Started NOTE: recipe util-linux-native-2.37.3-r0: task do_deploy_source_date_epoch: Succeeded NOTE: recipe vala-native-0.54.6-r0: task do_populate_lic: Succeeded NOTE: recipe util-linux-native-2.37.3-r0: task do_populate_lic: Succeeded NOTE: recipe util-linux-native-2.37.3-r0: task do_prepare_recipe_sysroot: Started NOTE: recipe util-linux-native-2.37.3-r0: task do_prepare_recipe_sysroot: Succeeded NOTE: recipe util-linux-native-2.37.3-r0: task do_configure: Started NOTE: recipe util-linux-native-2.37.3-r0: task do_configure: Succeeded NOTE: recipe util-linux-native-2.37.3-r0: task do_compile: Started NOTE: recipe util-linux-native-2.37.3-r0: task do_compile: Succeeded NOTE: recipe util-linux-native-2.37.3-r0: task do_install: Started NOTE: recipe util-linux-native-2.37.3-r0: task do_install: Succeeded NOTE: recipe util-linux-native-2.37.3-r0: task do_populate_sysroot: Started NOTE: recipe util-linux-native-2.37.3-r0: task do_populate_sysroot: Succeeded NOTE: recipe util-linux-native-2.37.3-r0: task do_deploy_source_date_epoch: Started NOTE: recipe util-linux-native-2.37.3-r0: task do_deploy_source_date_epoch: Succeeded NOTE: recipe util-linux-native-2.37.3-r0: task do_populate_lic: Started NOTE: recipe util-linux-native-2.37.3-r0: task do_populate_lic: Succeeded NOTE: recipe util-linux-native-2.37.3-r0: task do_populate_sysroot_setscene: Started NOTE: recipe vala-native-0.54.6-r0: task do_prepare_recipe_sysroot: Started ERROR: vala-native-0.54.6-r0 do_prepare_recipe_sysroot: Manifest /OE/new-arago-master/build/tmp-default-glibc/sstate-control/manifest-x86_64_ubuntu-20.04-util-linux-native.populate_sysroot not found in x86_64 x86_64_ubuntu-20.04 (variant '')? ERROR: Logfile of failure stored in: /OE/new-arago-master/build/tmp-default-glibc/work/x86_64-linux/vala-native/0.54.6-r0/temp/log.do_prepare_recipe_sysroot.1195006 NOTE: recipe vala-native-0.54.6-r0: task do_prepare_recipe_sysroot: Failed NOTE: recipe util-linux-native-2.37.3-r0: task do_populate_sysroot_setscene: Succeeded NOTE: recipe util-linux-native-2.37.3-r0: task do_deploy_source_date_epoch: Started NOTE: recipe util-linux-native-2.37.3-r0: task do_populate_lic: Started NOTE: recipe util-linux-native-2.37.3-r0: task do_deploy_source_date_epoch: Succeeded NOTE: recipe util-linux-native-2.37.3-r0: task do_populate_lic: Succeeded 92599 2 denis 2022-02-18 23:49:08 +0000 FYI, just got another instance of the second error with vala-native and util-linux-native, but with more recent master and a different platform. BTW, I have hash equiv enabled in local.conf: BB_HASHSERVE_UPSTREAM = "typhoon.yocto.io:8687" BB_HASHSERVE = "auto" BB_SIGNATURE_HANDLER = "OEEquivHash" I'll probably need to disable it to not hit the issue, unless more logs are needed. 92602 3 richard.purdie 2022-02-20 16:04:46 +0000 The real problem here is that I need to find a way to reproduce the issue. We have seen errors like this in a past as it is something wrong with the state engine in runqueue.py. I remember how hard this was to debug and fix last time, I think I spent about two weeks trying to resolve a previous issue like this. I don't have that length of time available :( 92884 4 richard.purdie 2022-03-30 17:53:08 +0000 The issue here is that if XXX-native:do_populate_sysroot runs, XXX-native:do_populate_sysroot_setscene should never run after that. The code is supposed to enforce that but doesn't seem to be the case here. 93348 5 luca.ceresoli 2022-05-27 18:57:27 +0000 Richard pointed out on IRC that the following failure has different effects but the same reason ("they both end up with overlapping files as a result of setscene graph corruption so I'm happy for them to be the same issue"): stdio: ERROR: Setscene task .../netbase_6.3.bb:do_package in both covered and notcovered. (many of these, all for netbase) https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/3602 93727 6 luca.ceresoli 2022-08-05 14:32:02 +0000 Two new errors, probably caused by the same bug: ERROR: ca-certificates-20211016-r0 do_package_write_rpm: Error executing a python function in exec_func_python() autogenerated: The stack trace of python calls that resulted in this exception/failure was: File: 'exec_func_python() autogenerated', lineno: 2, function: <module> 0001: *** 0002:write_specfile(d) 0003: File: '/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/classes/package_rpm.bbclass', lineno: 342, function: write_specfile 0338: localdata.setVar('PKG', pkgname) 0339: 0340: localdata.setVar('OVERRIDES', d.getVar("OVERRIDES", False) + ":" + pkg) 0341: *** 0342: conffiles = get_conffiles(pkg, d) 0343: dirfiles = localdata.getVar('DIRFILES') 0344: if dirfiles is not None: 0345: dirfiles = dirfiles.split() 0346: File: '/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/classes/package.bbclass', lineno: 302, function: get_conffiles 0298:def get_conffiles(pkg, d): 0299: pkgdest = d.getVar('PKGDEST') 0300: root = os.path.join(pkgdest, pkg) 0301: cwd = os.getcwd() *** 0302: os.chdir(root) 0303: 0304: conffiles = d.getVar('CONFFILES:%s' % pkg); 0305: if conffiles == None: 0306: conffiles = d.getVar('CONFFILES') Exception: FileNotFoundError: [Errno 2] No such file or directory: '/home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1389915/tmp/work/all-poky-linux/ca-certificates/20211016-r0/packages-split/ca-certificates-src' Logs: https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/3895/steps/14/logs/stdio oe-selftest-fedora on fedora35-ty-2 https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3940/steps/14/logs/stdio oe-selftest-ubuntu on ubuntu2204-ty-1 Discussed these with Richard on IRC: "it ran packaging without having do_package present". Richard has sent a series that hopefully fixes the bug: https://lists.openembedded.org/g/bitbake-devel/message/13873 94650 7 denis 2023-01-10 16:15:46 +0000 Was this fixed? I still see this marked as NEW. Richard, is this issue similar to what you mentioned on the tech call regarding uninative and tasks running out of order? 94676 8 richard.purdie 2023-01-11 22:54:56 +0000 (In reply to comment #7) > Was this fixed? I still see this marked as NEW. I don't know, I've not been able to reproduce the issue. It is possible something like: https://git.yoctoproject.org/poky/commit/bitbake/lib/bb/runqueue.py?id=5fdd28e37face59410da781d5d70178565e69e5e fixed it. > Richard, is this issue similar to what you mentioned on the tech call > regarding uninative and tasks running out of order? No, nothing related to that. Are you still seeing it? 95073 9 richard.purdie 2023-03-07 13:00:52 +0000 I'm going to assume the above fix was the issue. If it still occurs, please reopen or open a new bug. 95082 10 denis 2023-03-07 17:12:19 +0000 Sounds good, thanks