| Summary: | Intermittent runqueue issue re-running populate_sysroot_setscene after populate_sysroot | ||
|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] BitBake | Reporter: | Denys Dmytriyenko <denis> |
| Component: | bitbake | Assignee: | Richard Purdie <richard.purdie> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | normal | ||
| Priority: | Medium+ | CC: | luca.ceresoli, poky.bs.watcher, poky.watcher, randy.macleod |
| Version: | unspecified | ||
| Target Milestone: | 4.2 M3 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |
Another instance of the same issue: NOTE: recipe util-linux-native-2.37.3-r0: task do_fetch: Started NOTE: recipe util-linux-native-2.37.3-r0: task do_fetch: Succeeded NOTE: recipe vala-native-0.54.6-r0: task do_fetch: Started NOTE: recipe vala-native-0.54.6-r0: task do_fetch: Succeeded NOTE: recipe util-linux-native-2.37.3-r0: task do_unpack: Started NOTE: recipe vala-native-0.54.6-r0: task do_unpack: Started NOTE: recipe vala-native-0.54.6-r0: task do_unpack: Succeeded NOTE: recipe vala-native-0.54.6-r0: task do_patch: Started NOTE: recipe vala-native-0.54.6-r0: task do_patch: Succeeded NOTE: recipe util-linux-native-2.37.3-r0: task do_unpack: Succeeded NOTE: recipe util-linux-native-2.37.3-r0: task do_patch: Started NOTE: recipe vala-native-0.54.6-r0: task do_create_srcipk: Started NOTE: recipe vala-native-0.54.6-r0: task do_deploy_source_date_epoch: Started NOTE: recipe vala-native-0.54.6-r0: task do_create_srcipk: Succeeded NOTE: recipe vala-native-0.54.6-r0: task do_deploy_source_date_epoch: Succeeded NOTE: recipe util-linux-native-2.37.3-r0: task do_patch: Succeeded NOTE: recipe util-linux-native-2.37.3-r0: task do_deploy_source_date_epoch: Started NOTE: recipe util-linux-native-2.37.3-r0: task do_create_srcipk: Started NOTE: recipe util-linux-native-2.37.3-r0: task do_create_srcipk: Succeeded NOTE: recipe vala-native-0.54.6-r0: task do_populate_lic: Started NOTE: recipe util-linux-native-2.37.3-r0: task do_populate_lic: Started NOTE: recipe util-linux-native-2.37.3-r0: task do_deploy_source_date_epoch: Succeeded NOTE: recipe vala-native-0.54.6-r0: task do_populate_lic: Succeeded NOTE: recipe util-linux-native-2.37.3-r0: task do_populate_lic: Succeeded NOTE: recipe util-linux-native-2.37.3-r0: task do_prepare_recipe_sysroot: Started NOTE: recipe util-linux-native-2.37.3-r0: task do_prepare_recipe_sysroot: Succeeded NOTE: recipe util-linux-native-2.37.3-r0: task do_configure: Started NOTE: recipe util-linux-native-2.37.3-r0: task do_configure: Succeeded NOTE: recipe util-linux-native-2.37.3-r0: task do_compile: Started NOTE: recipe util-linux-native-2.37.3-r0: task do_compile: Succeeded NOTE: recipe util-linux-native-2.37.3-r0: task do_install: Started NOTE: recipe util-linux-native-2.37.3-r0: task do_install: Succeeded NOTE: recipe util-linux-native-2.37.3-r0: task do_populate_sysroot: Started NOTE: recipe util-linux-native-2.37.3-r0: task do_populate_sysroot: Succeeded NOTE: recipe util-linux-native-2.37.3-r0: task do_deploy_source_date_epoch: Started NOTE: recipe util-linux-native-2.37.3-r0: task do_deploy_source_date_epoch: Succeeded NOTE: recipe util-linux-native-2.37.3-r0: task do_populate_lic: Started NOTE: recipe util-linux-native-2.37.3-r0: task do_populate_lic: Succeeded NOTE: recipe util-linux-native-2.37.3-r0: task do_populate_sysroot_setscene: Started NOTE: recipe vala-native-0.54.6-r0: task do_prepare_recipe_sysroot: Started ERROR: vala-native-0.54.6-r0 do_prepare_recipe_sysroot: Manifest /OE/new-arago-master/build/tmp-default-glibc/sstate-control/manifest-x86_64_ubuntu-20.04-util-linux-native.populate_sysroot not found in x86_64 x86_64_ubuntu-20.04 (variant '')? ERROR: Logfile of failure stored in: /OE/new-arago-master/build/tmp-default-glibc/work/x86_64-linux/vala-native/0.54.6-r0/temp/log.do_prepare_recipe_sysroot.1195006 NOTE: recipe vala-native-0.54.6-r0: task do_prepare_recipe_sysroot: Failed NOTE: recipe util-linux-native-2.37.3-r0: task do_populate_sysroot_setscene: Succeeded NOTE: recipe util-linux-native-2.37.3-r0: task do_deploy_source_date_epoch: Started NOTE: recipe util-linux-native-2.37.3-r0: task do_populate_lic: Started NOTE: recipe util-linux-native-2.37.3-r0: task do_deploy_source_date_epoch: Succeeded NOTE: recipe util-linux-native-2.37.3-r0: task do_populate_lic: Succeeded FYI, just got another instance of the second error with vala-native and util-linux-native, but with more recent master and a different platform. BTW, I have hash equiv enabled in local.conf: BB_HASHSERVE_UPSTREAM = "typhoon.yocto.io:8687" BB_HASHSERVE = "auto" BB_SIGNATURE_HANDLER = "OEEquivHash" I'll probably need to disable it to not hit the issue, unless more logs are needed. The real problem here is that I need to find a way to reproduce the issue. We have seen errors like this in a past as it is something wrong with the state engine in runqueue.py. I remember how hard this was to debug and fix last time, I think I spent about two weeks trying to resolve a previous issue like this. I don't have that length of time available :( The issue here is that if XXX-native:do_populate_sysroot runs, XXX-native:do_populate_sysroot_setscene should never run after that. The code is supposed to enforce that but doesn't seem to be the case here. Richard pointed out on IRC that the following failure has different effects but the same reason ("they both end up with overlapping files as a result of setscene graph corruption so I'm happy for them to be the same issue"):
stdio: ERROR: Setscene task .../netbase_6.3.bb:do_package in both covered and notcovered. (many of these, all for netbase)
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/3602
Two new errors, probably caused by the same bug:
ERROR: ca-certificates-20211016-r0 do_package_write_rpm: Error executing a python function in exec_func_python() autogenerated:
The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
0001:
*** 0002:write_specfile(d)
0003:
File: '/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/classes/package_rpm.bbclass', lineno: 342, function: write_specfile
0338: localdata.setVar('PKG', pkgname)
0339:
0340: localdata.setVar('OVERRIDES', d.getVar("OVERRIDES", False) + ":" + pkg)
0341:
*** 0342: conffiles = get_conffiles(pkg, d)
0343: dirfiles = localdata.getVar('DIRFILES')
0344: if dirfiles is not None:
0345: dirfiles = dirfiles.split()
0346:
File: '/home/pokybuild/yocto-worker/oe-selftest-fedora/build/meta/classes/package.bbclass', lineno: 302, function: get_conffiles
0298:def get_conffiles(pkg, d):
0299: pkgdest = d.getVar('PKGDEST')
0300: root = os.path.join(pkgdest, pkg)
0301: cwd = os.getcwd()
*** 0302: os.chdir(root)
0303:
0304: conffiles = d.getVar('CONFFILES:%s' % pkg);
0305: if conffiles == None:
0306: conffiles = d.getVar('CONFFILES')
Exception: FileNotFoundError: [Errno 2] No such file or directory: '/home/pokybuild/yocto-worker/oe-selftest-fedora/build/build-st-1389915/tmp/work/all-poky-linux/ca-certificates/20211016-r0/packages-split/ca-certificates-src'
Logs:
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/3895/steps/14/logs/stdio
oe-selftest-fedora on fedora35-ty-2
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/3940/steps/14/logs/stdio
oe-selftest-ubuntu on ubuntu2204-ty-1
Discussed these with Richard on IRC: "it ran packaging without having do_package present".
Richard has sent a series that hopefully fixes the bug:
https://lists.openembedded.org/g/bitbake-devel/message/13873
Was this fixed? I still see this marked as NEW. Richard, is this issue similar to what you mentioned on the tech call regarding uninative and tasks running out of order? (In reply to comment #7) > Was this fixed? I still see this marked as NEW. I don't know, I've not been able to reproduce the issue. It is possible something like: https://git.yoctoproject.org/poky/commit/bitbake/lib/bb/runqueue.py?id=5fdd28e37face59410da781d5d70178565e69e5e fixed it. > Richard, is this issue similar to what you mentioned on the tech call > regarding uninative and tasks running out of order? No, nothing related to that. Are you still seeing it? I'm going to assume the above fix was the issue. If it still occurs, please reopen or open a new bug. Sounds good, thanks |
Got qemu-native:do_prepare_recipe_sysroot failure due to zlib.h not being ready from zlib-native:do_populate_sysroot ERROR: qemu-native-6.2.0-r0 do_prepare_recipe_sysroot: Error executing a python function in exec_func_python() autogenerated: The stack trace of python calls that resulted in this exception/failure was: File: 'exec_func_python() autogenerated', lineno: 2, function: <module> 0001: *** 0002:extend_recipe_sysroot(d) 0003: File: '/OE/new-arago-master/meta/classes/staging.bbclass', lineno: 583, function: extend_recipe_sysroot 0579: if "/bin/" in l or "/sbin/" in l: 0580: # defer /*bin/* files until last in case they need libs 0581: binfiles[l] = (targetdir, dest) 0582: else: *** 0583: staging_copyfile(l, targetdir, dest, postinsts, seendirs) 0584: 0585: # Handle deferred binfiles 0586: for l in binfiles: 0587: (targetdir, dest) = binfiles[l] File: '/OE/new-arago-master/meta/classes/staging.bbclass', lineno: 157, function: staging_copyfile 0153: os.symlink(linkto, dest) 0154: #bb.warn(c) 0155: else: 0156: try: *** 0157: os.link(c, dest) 0158: except OSError as err: 0159: if err.errno == errno.EXDEV: 0160: bb.utils.copyfile(c, dest) 0161: else: Exception: FileNotFoundError: [Errno 2] No such file or directory: '/OE/new-arago-master/build/tmp-default-glibc/sysroots-components/x86_64/zlib-native/usr/include/zlib.h' -> '/OE/new-arago-master/build/tmp-default-glibc/work/x86_64-linux/qemu-native/6.2.0-r0/recipe-sysroot-native/usr/include/zlib.h' Cooker logs for all qemu-native and zlib-native tasks show that zlib-native successfully executed do_populate_sysroot task and then decided to re-run do_populate_sysroot_setscene task in the middle of qemu-native:do_prepare_recipe_sysroot NOTE: recipe zlib-native-1.2.11-r0: task do_fetch: Started NOTE: recipe zlib-native-1.2.11-r0: task do_fetch: Succeeded NOTE: recipe zlib-native-1.2.11-r0: task do_prepare_recipe_sysroot: Started NOTE: recipe zlib-native-1.2.11-r0: task do_prepare_recipe_sysroot: Succeeded NOTE: Running task 151 of 9535 (/OE/new-arago-master/meta/recipes-devtools/qemu/qemu-native_6.2.0.bb:do_fetch) NOTE: recipe qemu-native-6.2.0-r0: task do_fetch: Started NOTE: recipe qemu-native-6.2.0-r0: task do_fetch: Succeeded NOTE: Running task 2026 of 9535 (/OE/new-arago-master/meta/recipes-devtools/qemu/qemu-native_6.2.0.bb:do_unpack) NOTE: recipe zlib-native-1.2.11-r0: task do_unpack: Started NOTE: recipe qemu-native-6.2.0-r0: task do_unpack: Started NOTE: recipe zlib-native-1.2.11-r0: task do_unpack: Succeeded NOTE: recipe zlib-native-1.2.11-r0: task do_patch: Started NOTE: recipe zlib-native-1.2.11-r0: task do_patch: Succeeded NOTE: recipe zlib-native-1.2.11-r0: task do_deploy_source_date_epoch: Started NOTE: recipe zlib-native-1.2.11-r0: task do_create_srcipk: Started NOTE: recipe zlib-native-1.2.11-r0: task do_create_srcipk: Succeeded NOTE: recipe zlib-native-1.2.11-r0: task do_deploy_source_date_epoch: Succeeded NOTE: recipe qemu-native-6.2.0-r0: task do_unpack: Succeeded NOTE: Running task 2170 of 9535 (/OE/new-arago-master/meta/recipes-devtools/qemu/qemu-native_6.2.0.bb:do_patch) NOTE: recipe zlib-native-1.2.11-r0: task do_configure: Started NOTE: recipe qemu-native-6.2.0-r0: task do_patch: Started NOTE: recipe zlib-native-1.2.11-r0: task do_configure: Succeeded NOTE: recipe qemu-native-6.2.0-r0: task do_patch: Succeeded NOTE: Running task 2279 of 9535 (/OE/new-arago-master/meta/recipes-devtools/qemu/qemu-native_6.2.0.bb:do_create_srcipk) NOTE: Running task 2280 of 9535 (/OE/new-arago-master/meta/recipes-devtools/qemu/qemu-native_6.2.0.bb:do_deploy_source_date_epoch) NOTE: recipe zlib-native-1.2.11-r0: task do_compile: Started NOTE: recipe qemu-native-6.2.0-r0: task do_create_srcipk: Started NOTE: recipe qemu-native-6.2.0-r0: task do_deploy_source_date_epoch: Started NOTE: recipe qemu-native-6.2.0-r0: task do_create_srcipk: Succeeded NOTE: Running task 2352 of 9535 (/OE/new-arago-master/meta/recipes-devtools/qemu/qemu-native_6.2.0.bb:do_populate_lic) NOTE: recipe qemu-native-6.2.0-r0: task do_deploy_source_date_epoch: Succeeded NOTE: Task /OE/new-arago-master/meta/recipes-devtools/qemu/qemu-native_6.2.0.bb:do_deploy_source_date_epoch unihash changed to 93fe89d6405b5f00058aa4eb64beef13e6269f5e7f10cfd703e96cc0affe53d1 NOTE: recipe qemu-native-6.2.0-r0: task do_populate_lic: Started NOTE: recipe zlib-native-1.2.11-r0: task do_populate_lic: Started NOTE: recipe zlib-native-1.2.11-r0: task do_compile: Succeeded NOTE: recipe qemu-native-6.2.0-r0: task do_populate_lic: Succeeded NOTE: recipe zlib-native-1.2.11-r0: task do_populate_lic: Succeeded NOTE: Task /OE/new-arago-master/meta/recipes-devtools/qemu/qemu-native_6.2.0.bb:do_populate_lic unihash changed to a6dced3dae65779c96d0299625aabe4c857cbd0fda6dc11c5d2d5497a302f821 NOTE: recipe zlib-native-1.2.11-r0: task do_install: Started NOTE: recipe zlib-native-1.2.11-r0: task do_install: Succeeded NOTE: recipe zlib-native-1.2.11-r0: task do_populate_sysroot: Started NOTE: recipe zlib-native-1.2.11-r0: task do_populate_sysroot: Succeeded NOTE: recipe zlib-native-1.2.11-r0: task do_deploy_source_date_epoch: Started NOTE: recipe zlib-native-1.2.11-r0: task do_deploy_source_date_epoch: Succeeded NOTE: recipe zlib-native-1.2.11-r0: task do_populate_lic: Started NOTE: recipe zlib-native-1.2.11-r0: task do_populate_lic: Succeeded NOTE: Running task 5223 of 9535 (/OE/new-arago-master/meta/recipes-devtools/qemu/qemu-native_6.2.0.bb:do_prepare_recipe_sysroot) NOTE: recipe zlib-native-1.2.11-r0: task do_populate_sysroot_setscene: Started NOTE: recipe qemu-native-6.2.0-r0: task do_prepare_recipe_sysroot: Started ERROR: qemu-native-6.2.0-r0 do_prepare_recipe_sysroot: Error executing a python function in exec_func_python() autogenerated: Exception: FileNotFoundError: [Errno 2] No such file or directory: '/OE/new-arago-master/build/tmp-default-glibc/sysroots-components/x86_64/zlib-native/usr/include/zlib.h' -> '/OE/new-arago-master/build/tmp-default-glibc/work/x86_64-linux/qemu-native/6.2.0-r0/recipe-sysroot-native/usr/include/zlib.h' NOTE: recipe zlib-native-1.2.11-r0: task do_populate_sysroot_setscene: Succeeded ERROR: Logfile of failure stored in: /OE/new-arago-master/build/tmp-default-glibc/work/x86_64-linux/qemu-native/6.2.0-r0/temp/log.do_prepare_recipe_sysroot.1169796 NOTE: recipe qemu-native-6.2.0-r0: task do_prepare_recipe_sysroot: Failed ERROR: Task (/OE/new-arago-master/meta/recipes-devtools/qemu/qemu-native_6.2.0.bb:do_prepare_recipe_sysroot) failed with exit code '1' NOTE: recipe zlib-native-1.2.11-r0: task do_deploy_source_date_epoch: Started NOTE: recipe zlib-native-1.2.11-r0: task do_deploy_source_date_epoch: Succeeded NOTE: recipe zlib-native-1.2.11-r0: task do_populate_lic: Started NOTE: recipe zlib-native-1.2.11-r0: task do_populate_lic: Succeeded