10449 2016-10-18 08:35:37 +0000 tar-native - task do_populate_sysroot_setscene fails - likely race 2017-01-26 16:09:04 +0000 1 1 7 Build System, Metadata & Runtime OE-Core core 2.1.2 x86 Multiple RESOLVED FIXED Medium+ major 2.3 M2 1 dl9pf ross.burton meta.mr.watcher meta.watcher --- --- No (bug/feature does not impact docs) oldest_to_newest 67321 0 dl9pf 2016-10-18 08:35:37 +0000 There seems to be a race between do_populate_sysroot_setscene for tar-native and its do_populate_lic_setscene (which uses tar). Outcome is: Exception: CalledProcessError: Command 'cd /w/workspace/ci-meta-agl-devel/repoclone/build/tmp/work/x86_64-linux/tar-native/1.28-r0/license-destdir; find . -type d -print | tar -cf - -C /w/workspace/ci-meta-agl-devel/repoclone/build/tmp/work/x86_64-linux/tar-native/1.28-r0/license-destdir -p --files-from - --no-recursion | tar -xf - -C /w/workspace/ci-meta-agl-devel/repoclone/build/tmp/deploy/licenses' returned non-zero exit status 126 with output /bin/sh: /w/workspace/ci-meta-agl-devel/repoclone/build/tmp/sysroots/x86_64-linux/usr/bin/tar-native/tar: Text file busy Look at this sequence: NOTE: recipe tar-native-1.28-r0: task do_populate_sysroot_setscene: Started NOTE: recipe tar-native-1.28-r0: task do_populate_lic_setscene: Started NOTE: recipe pbzip2-native-1.1.13-r0: task do_populate_lic_setscene: Succeeded NOTE: Running setscene task 69 of 1892 (/w/workspace/ci-meta-agl-devel/repoclone/meta-agl/meta-agl/recipes-core/packagegroups/packagegroup-core-boot-agl.bb, do_populate_lic_setscene) NOTE: recipe qemuwrapper-cross-1.0-r0: task do_populate_sysroot_setscene: Succeeded NOTE: Running setscene task 70 of 1892 (/w/workspace/ci-meta-agl-devel/repoclone/meta-agl/meta-agl/recipes-core/packagegroups/packagegroup-core-boot-agl.bb, do_package_qa_setscene) ERROR: tar-native-1.28-r0 do_populate_lic_setscene: Error executing a python function in exec_python_func() autogenerated: The stack trace of python calls that resulted in this exception/failure was: File: 'exec_python_func() autogenerated', lineno: 2, function: <module> 0001: *** 0002:do_populate_lic_setscene(d) 0003: File: '/w/workspace/ci-meta-agl-devel/repoclone/poky/meta/classes/license.bbclass', lineno: 664, function: do_populate_lic_setscene 0660: 0661:do_populate_lic_setscene[dirs] = "${LICSSTATEDIR}/${PN}" 0662:do_populate_lic_setscene[cleandirs] = "${LICSSTATEDIR}" 0663:python do_populate_lic_setscene () { *** 0664: sstate_setscene(d) 0665:} 0666:addtask do_populate_lic_setscene File: '/w/workspace/ci-meta-agl-devel/repoclone/poky/meta/classes/sstate.bbclass', lineno: 648, function: sstate_setscene 0644: break 0645: 0646:def sstate_setscene(d): 0647: shared_state = sstate_state_fromvars(d) *** 0648: accelerate = sstate_installpkg(shared_state, d) 0649: if not accelerate: 0650: raise bb.build.FuncFailed("No suitable staging package found") 0651: 0652:python sstate_task_prefunc () { File: '/w/workspace/ci-meta-agl-devel/repoclone/poky/meta/classes/sstate.bbclass', lineno: 320, function: sstate_installpkg 0316: 0317: for state in ss['dirs']: 0318: prepdir(state[1]) 0319: os.rename(sstateinst + state[0], state[1]) *** 0320: sstate_install(ss, d) 0321: 0322: for plain in ss['plaindirs']: 0323: workdir = d.getVar('WORKDIR', True) 0324: src = sstateinst + "/" + plain.replace(workdir, '') File: '/w/workspace/ci-meta-agl-devel/repoclone/poky/meta/classes/sstate.bbclass', lineno: 268, function: sstate_install 0264: 0265: # Run the actual file install 0266: for state in ss['dirs']: 0267: if os.path.exists(state[1]): *** 0268: oe.path.copyhardlinktree(state[1], state[2]) 0269: 0270: for postinst in (d.getVar('SSTATEPOSTINSTFUNCS', True) or '').split(): 0271: # All hooks should run in the SSTATE_INSTDIR 0272: bb.build.exec_func(postinst, d, (sstateinst,)) File: '/w/workspace/ci-meta-agl-devel/repoclone/poky/meta/lib/oe/path.py', lineno: 81, function: copyhardlinktree 0077: if (os.stat(src).st_dev == os.stat(dst).st_dev): 0078: # Need to copy directories only with tar first since cp will error if two 0079: # writers try and create a directory at the same time 0080: cmd = 'cd %s; find . -type d -print | tar -cf - -C %s -p --files-from - --no-recursion | tar -xf - -C %s' % (src, src, dst) *** 0081: check_output(cmd, shell=True, stderr=subprocess.STDOUT) 0082: cmd = 'cd %s; find . -print0 | cpio --null -pdlu %s' % (src, dst) 0083: check_output(cmd, shell=True, stderr=subprocess.STDOUT) 0084: else: 0085: copytree(src, dst) File: '/w/workspace/ci-meta-agl-devel/repoclone/poky/meta/lib/oe/path.py', lineno: 146, function: check_output 0142: if retcode: 0143: cmd = kwargs.get("args") 0144: if cmd is None: 0145: cmd = popenargs[0] *** 0146: raise CalledProcessError(retcode, cmd, output=output) 0147: return output 0148: 0149:def find(dir, **walkoptions): 0150: """ Given a directory, recurses into that directory, Exception: CalledProcessError: Command 'cd /w/workspace/ci-meta-agl-devel/repoclone/build/tmp/work/x86_64-linux/tar-native/1.28-r0/license-destdir; find . -type d -print | tar -cf - -C /w/workspace/ci-meta-agl-devel/repoclone/build/tmp/work/x86_64-linux/tar-native/1.28-r0/license-destdir -p --files-from - --no-recursion | tar -xf - -C /w/workspace/ci-meta-agl-devel/repoclone/build/tmp/deploy/licenses' returned non-zero exit status 126 with output /bin/sh: /w/workspace/ci-meta-agl-devel/repoclone/build/tmp/sysroots/x86_64-linux/usr/bin/tar-native/tar: Text file busy tar: -: Cannot write: Broken pipe tar: Error is not recoverable: exiting now WARNING: Logfile for failed setscene task is /w/workspace/ci-meta-agl-devel/repoclone/build/tmp/work/x86_64-linux/tar-native/1.28-r0/temp/log.do_populate_lic_setscene.4534 WARNING: Setscene task 49 (virtual:native:/w/workspace/ci-meta-agl-devel/repoclone/poky/meta/recipes-extended/tar/tar_1.28.bb, do_populate_lic_setscene) failed with exit code '1' - real task will be run instead A solution might be to serialize the the tar task. Tar-native (sysroot-setscene) basically needs to run on its own. Env is Ubuntu 16.04, 64bit. 8 core, 16GB with relatively slow I/O - which might trigger this. I'm not sure if I got the syntax right ... would this solve it: do_populate_lic_setscene[depends] = "do_populate_sysroot_setscene" or is it then: do_populate_lic_setscene_class-native[depends] = "tar-native:do_populate_sysroot_setscene" 67322 1 dl9pf 2016-10-18 09:06:00 +0000 Note: The issue might be *more generic* as *any* setscene tasks might fail with the bin/tar-native/tar: Text file busy not only tar-native:do_populate_lic_setscene is affected. 67323 2 dl9pf 2016-10-18 09:12:00 +0000 Found another one ... so it's a more generic race. Tar-native should be run as single tasks with no parallel tasks at all. ERROR: pseudo-native-1.7.5-r0 do_populate_lic_setscene: Error executing a python function in exec_python_func() autogenerated: 0150: """ Given a directory, recurses into that directory, Exception: CalledProcessError: Command 'cd /w/workspace/ci-meta-agl/repoclone/build/tmp/work/x86_64-linux/pseudo-native/1.7.5-r0/license-destdir; find . -type d -print | tar -cf - -C /w/workspace/ci-meta-agl/repoclone/build/tmp/work/x86_64-linux/pseudo-native/1.7.5-r0/license-destdir -p --files-from - --no-recursion | tar -xf - -C /w/workspace/ci-meta-agl/repoclone/build/tmp/deploy/licenses' returned non-zero exit status 126 with output /bin/sh: /w/workspace/ci-meta-agl/repoclone/build/tmp/sysroots/x86_64-linux/usr/bin/tar-native/tar: Text file busy /bin/sh: /w/workspace/ci-meta-agl/repoclone/build/tmp/sysroots/x86_64-linux/usr/bin/tar-native/tar: Text file busy 70183 3 ross.burton 2017-01-26 16:09:04 +0000 Recipe-specific sysroots (oe-core 809746f56df4b91af014bf6a3f28997d6698ac78) makes this impossible, marking as resolved.