| Summary: | multiconfig race in sstate/SDE with combined tmp/work-shared | ||
|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] OE-Core | Reporter: | Denys Dmytriyenko <denis> |
| Component: | core | Assignee: | Richard Purdie <richard.purdie> |
| Status: | RESOLVED OBSOLETE | QA Contact: | |
| Severity: | normal | ||
| Priority: | Medium+ | CC: | meta.mr.watcher, meta.watcher, quaresma.jose, randy.macleod |
| Version: | 5.0 | ||
| Target Milestone: | 5.1 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |
Posted an RFC patch to re-enable TCLIBCAPPEND in Poky: https://patchwork.yoctoproject.org/project/yocto/patch/20231108191537.3956782-1-denis@denix.org/ I've got a more local workaround/solution that I posted to meta-ti: https://patchwork.yoctoproject.org/project/ti/patch/20231112190743.1496752-1-denis@denix.org/ I still believe re-enabling TCLIBCAPPEND in Poky is a good generic change going forward... I've dropped TCLIBCAPPEND from core and meta-yocto. If there are specific issues with different libcs causing races/corruption when they shouldn't, please open specific bugs. There are some cases where separate TMPDIRs are needed for multiconfigs, depending on what the differences are between them. The multiconfigs themselves are the places where this will be known or not though so TCLIBCAPPEND doesn't make much sense even there. |
There seems to be a race condition in sstate/SDE code during multiconfig build that uses work-shared in a single TMPDIR. 1. meta-ti-bsp has a "k3r5" multiconfig that builds few baremetal components. So, the toolchain is built twice - glibc and baremetal. And both create gcc-source in work-shared 2. unlike OE-Core "nodistro" and derivatives, which use TCLIBCAPPEND and result in separate TMPDIRs for those multiconfig builds ("tmp-glibc" and "tmp-baremetal"), Poky disables TCLIBCAPPEND and ends up with a single TMPDIR="tmp", which exposes this race condition, as both multiconfigs are trying to access the same tmp/work-shared 3. this happens often, but not always, and when it fails, re-running bitbake the second time works 4. the workaround/fix is to enable TCLIBCAPPEND in Poky by removing line #31: https://git.yoctoproject.org/meta-yocto/tree/meta-poky/conf/distro/poky.conf#n31 Logs from master: ERROR: mc:k3r5:gcc-source-13.2.0-13.2.0-r0 do_deploy_source_date_epoch: Error executing a python function in exec_func_python() autogenerated: The stack trace of python calls that resulted in this exception/failure was: File: 'exec_func_python() autogenerated', lineno: 2, function: <module> 0001: *** 0002:sstate_hardcode_path(d) 0003: File: '/OE/poky-ti-master/meta/classes-global/sstate.bbclass', lineno: 635, function: sstate_hardcode_path 0631: bb.note("Removing hardcoded paths from sstate package: '%s'" % (sstate_hardcode_cmd)) 0632: subprocess.check_output(sstate_hardcode_cmd, shell=True, cwd=sstate_builddir) 0633: 0634: # If the fixmefn is empty, remove it.. *** 0635: if os.stat(fixmefn).st_size == 0: 0636: os.remove(fixmefn) 0637: else: 0638: bb.note("Replacing absolute paths in fixmepath file: '%s'" % (sstate_filelist_relative_cmd)) 0639: subprocess.check_output(sstate_filelist_relative_cmd, shell=True) Exception: FileNotFoundError: [Errno 2] No such file or directory: '/OE/poky-ti-master/build/tmp/work-shared/gcc-13.2.0-r0/sstate-build-deploy_source_date_epoch/fixmepath' ERROR: Logfile of failure stored in: /OE/poky-ti-master/build/tmp/work-shared/gcc-13.2.0-r0/temp/log.do_deploy_source_date_epoch.18437 ERROR: Task (mc:k3r5:/OE/poky-ti-master/meta/recipes-devtools/gcc/gcc-source_13.2.bb:do_deploy_source_date_epoch) failed with exit code '1' Logs from Kirkstone: ERROR: mc:k3r5:gcc-source-11.4.0-11.4.0-r0 do_deploy_source_date_epoch: ExecutionError('/OE/poky-ti-kirkstone/build/tmp/work-shared/gcc-11.4.0-r0/temp/run.sstate_create_package.6615', 2, None, None) ERROR: Logfile of failure stored in: /OE/poky-ti-kirkstone/build/tmp/work-shared/gcc-11.4.0-r0/temp/log.do_deploy_source_date_epoch.6615 Log data follows: | DEBUG: Executing python function sstate_task_prefunc | DEBUG: Python function sstate_task_prefunc finished | DEBUG: Executing shell function do_deploy_source_date_epoch | Deploying SDE from /OE/poky-ti-kirkstone/build/tmp/work-shared/gcc-11.4.0-r0/source-date-epoch/__source_date_epoch.txt -> /OE/poky-ti-kirkstone/build/tmp/work-shared/gcc-11.4.0-r0/deploy-source-date-epoch. | DEBUG: Shell function do_deploy_source_date_epoch finished | DEBUG: Executing python function sstate_task_postfunc | NOTE: Using umask 0o002 (not 22) for sstate packaging | DEBUG: Executing python function target_add_sysroot_deps | DEBUG: Python function target_add_sysroot_deps finished | DEBUG: Executing python function sstate_hardcode_path | NOTE: Removing hardcoded paths from sstate package: 'find /OE/poky-ti-kirkstone/build/tmp/work-shared/gcc-11.4.0-r0/sstate-build-deploy_source_date_epoch/ \( -name "*.la" -o -name "*-config" -o -name "*_config" -o -name "postinst-*" \) -type f | xargs grep -l -e '/OE/poky-ti-kirkstone/build/tmp/work-shared/gcc-11.4.0-r0/recipe-sysroot' -e '/OE/poky-ti-kirkstone/build/tmp/work-shared/gcc-11.4.0-r0/recipe-sysroot-native' -e '/OE/poky-ti-kirkstone/build/tmp/hosttools' | tee /OE/poky-ti-kirkstone/build/tmp/work-shared/gcc-11.4.0-r0/sstate-build-deploy_source_date_epoch/fixmepath | xargs --no-run-if-empty sed -i -e 's:/OE/poky-ti-kirkstone/build/tmp/work-shared/gcc-11.4.0-r0/recipe-sysroot-native:FIXMESTAGINGDIRHOST:g' -e 's:/OE/poky-ti-kirkstone/build/tmp/work-shared/gcc-11.4.0-r0/recipe-sysroot:FIXMESTAGINGDIRTARGET:g' -e 's:/OE/poky-ti-kirkstone/build/tmp/hosttools:FIXME_HOSTTOOLS_DIR:g'' | DEBUG: Python function sstate_hardcode_path finished | DEBUG: Executing python function sstate_report_unihash | DEBUG: Reported task e6d371fa6ff4876196c8b71ae2018b9240a4c136c8938660b99d50b8d48d76a7 as unihash e6d371fa6ff4876196c8b71ae2018b9240a4c136c8938660b99d50b8d48d76a7 to unix:///OE/poky-ti-kirkstone/build/hashserve.sock | DEBUG: Python function sstate_report_unihash finished | DEBUG: Executing shell function sstate_create_package | /OE/poky-ti-kirkstone/build/tmp/work-shared/gcc-11.4.0-r0/temp/run.sstate_create_package.6615: 194: cd: can't cd to /OE/poky-ti-kirkstone/build/tmp/work-shared/gcc-11.4.0-r0/sstate-build-deploy_source_date_epoch/ | WARNING: exit code 2 from a shell command. | DEBUG: Python function sstate_task_postfunc finished ERROR: Task (mc:k3r5:/OE/poky-ti-kirkstone/meta/recipes-devtools/gcc/gcc-source_11.4.bb:do_deploy_source_date_epoch) failed with exit code '1' And Kirkstone with OEBasicHash: ERROR: mc:k3r5:gcc-source-11.4.0-11.4.0-r0 do_deploy_source_date_epoch: ExecutionError('/OE/poky-ti-kirkstone/build/tmp/work-shared/gcc-11.4.0-r0/temp/run.do_deploy_source_date_epoch.41323', 2, None, None) ERROR: Logfile of failure stored in: /OE/poky-ti-kirkstone/build/tmp/work-shared/gcc-11.4.0-r0/temp/log.do_deploy_source_date_epoch.41323 Log data follows: | DEBUG: Executing python function sstate_task_prefunc | DEBUG: Python function sstate_task_prefunc finished | DEBUG: Executing shell function do_deploy_source_date_epoch | /OE/poky-ti-kirkstone/build/tmp/work-shared/gcc-11.4.0-r0/temp/run.do_deploy_source_date_epoch.41323: 159: cd: can't cd to /OE/poky-ti-kirkstone/build/tmp/work-shared/gcc-11.4.0-r0/deploy-source-date-epoch | WARNING: exit code 2 from a shell command. ERROR: Task (mc:k3r5:/OE/poky-ti-kirkstone/meta/recipes-devtools/gcc/gcc-source_11.4.bb:do_deploy_source_date_epoch) failed with exit code '1'