14301 2021-03-11 18:24:32 +0000 Incremental build in Docker fails with Invalid cross-device link error 2021-10-18 18:04:51 +0000 1 1 7 Build System, Metadata & Runtime Meta-yocto meta-yocto unspecified x86 Multiple RESOLVED FIXED Medium normal 3.4 1 devendra.tewari devendra.tewari devendra.tewari poky.bs.watcher poky.watcher randy.macleod --- --- Don't know oldest_to_newest 89598 0 devendra.tewari 2021-03-11 18:24:32 +0000 Incremental build in Docker on macOS BigSur fails with errors of the kind reproduced below. The error does not occur when shutil.move is used instead of os.rename, please see https://stackoverflow.com/questions/42392600/oserror-errno-18-invalid-cross-device-link. ERROR: core-image-base-1.0-r0 do_deploy_source_date_epoch: Error executing a python function in exec_python_func() autogenerated: The stack trace of python calls that resulted in this exception/failure was: File: 'exec_python_func() autogenerated', lineno: 2, function: <module> 0001: *** 0002:sstate_task_postfunc(d) 0003: File: '/home/pi/yocto-raspberry-pi/poky/meta/classes/sstate.bbclass', lineno: 770, function: sstate_task_postfunc 0766: 0767: omask = os.umask(0o002) 0768: if omask != 0o002: 0769: bb.note("Using umask 0o002 (not %0o) for sstate packaging" % omask) *** 0770: sstate_package(shared_state, d) 0771: os.umask(omask) 0772: 0773: sstateinst = d.getVar("SSTATE_INSTDIR") 0774: d.setVar('SSTATE_FIXMEDIR', shared_state['fixmedir']) File: '/home/pi/yocto-raspberry-pi/poky/meta/classes/sstate.bbclass', lineno: 677, function: sstate_package 0673: if sharedworkdir in plain: 0674: pdir = plain.replace(sharedworkdir, sstatebuild) 0675: bb.utils.mkdirhier(plain) 0676: bb.utils.mkdirhier(pdir) *** 0677: os.rename(plain, pdir) 0678: 0679: d.setVar('SSTATE_BUILDDIR', sstatebuild) 0680: d.setVar('SSTATE_INSTDIR', sstatebuild) 0681: Exception: OSError: [Errno 18] Invalid cross-device link: '/home/pi/yocto-raspberry-pi/build/tmp/work/raspberrypi0_wifi-poky-linux-gnueabi/core-image-base/1.0-r0/deploy-source-date-epoch' -> '/home/pi/yocto-raspberry-pi/build/tmp/work/raspberrypi0_wifi-poky-linux-gnueabi/core-image-base/1.0-r0/sstate-build-deploy_source_date_epoch//deploy-source-date-epoch' ERROR: Logfile of failure stored in: /home/pi/yocto-raspberry-pi/build/tmp/work/raspberrypi0_wifi-poky-linux-gnueabi/core-image-base/1.0-r0/temp/log.do_deploy_source_date_epoch.77 ERROR: Task (/home/pi/yocto-raspberry-pi/poky/meta/recipes-core/images/core-image-base.bb:do_deploy_source_date_epoch) failed with exit code '1' NOTE: Tasks Summary: Attempted 4634 tasks of which 4633 didn't need to be rerun and 1 failed. 89599 1 4776 devendra.tewari 2021-03-11 18:29:32 +0000 Created attachment 4776 Use shutil.move instead of os.rename 89643 2 randy.macleod 2021-03-18 14:40:18 +0000 The rename is faster but we could trap the error and fall-back to the move if the host/filesystem has an error. Can you send a patch? 89657 3 devendra.tewari 2021-03-18 21:37:07 +0000 According to documentation, shutil.move falls back to os.rename when source and destination are on the same filesystem, but trapping error with os.rename and trying shutil.move looks safer. I'll attach a patch. 89658 4 4777 devendra.tewari 2021-03-18 21:38:47 +0000 Created attachment 4777 Use shutil.move when os.rename fails 89717 5 randy.macleod 2021-03-23 20:08:31 +0000 Thanks Devendra, Are you able to send he patch to the oe-core email list, like this: git send-email -M -1 --to="openembedded-core@lists.openembedded.org" See: https://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded#Sending_using_git-send-email 89718 6 devendra.tewari 2021-03-23 20:37:35 +0000 Thanks, Randy. I've submitted the patch via git send-email. 89812 7 randy.macleod 2021-03-29 14:50:07 +0000 Hello Devendra, I didn't see the email. Did I miss it? Did it get merged? ../Randy 89813 8 devendra.tewari 2021-03-29 15:19:12 +0000 Sorry, looks like I didn't have an smtp server setup and the mail didn't go out. Sent it out again, and confirmed it got sent at https://lists.openembedded.org/g/openembedded-core/topic/patch_use_shutil_move_when/81698791. 91781 9 devendra.tewari 2021-10-18 18:04:51 +0000 Patches have been integrated. 4776 2021-03-11 18:29:32 +0000 2021-03-18 21:38:47 +0000 Use shutil.move instead of os.rename use-shutil-move-instead-of-os-rename.patch application/octet-stream 1945 devendra.tewari ZGlmZiAtLWdpdCBhL21ldGEvY2xhc3Nlcy9zc3RhdGUuYmJjbGFzcyBiL21ldGEvY2xhc3Nlcy9z c3RhdGUuYmJjbGFzcwppbmRleCBmNTc5MTY4MTYyLi4yZTg3Njk3ZTNlIDEwMDY0NAotLS0gYS9t ZXRhL2NsYXNzZXMvc3N0YXRlLmJiY2xhc3MKKysrIGIvbWV0YS9jbGFzc2VzL3NzdGF0ZS5iYmNs YXNzCkBAIC0zODQsNiArMzg0LDcgQEAgZGVmIHNzdGF0ZV9pbnN0YWxscGtnKHNzLCBkKToKIGRl ZiBzc3RhdGVfaW5zdGFsbHBrZ2RpcihzcywgZCk6CiAgICAgaW1wb3J0IG9lLnBhdGgKICAgICBp bXBvcnQgc3VicHJvY2VzcworICAgIGltcG9ydCBzaHV0aWwKIAogICAgIHNzdGF0ZWluc3QgPSBk LmdldFZhcigiU1NUQVRFX0lOU1RESVIiKQogICAgIGQuc2V0VmFyKCdTU1RBVEVfRklYTUVESVIn LCBzc1snZml4bWVkaXInXSkKQEAgLTQwMSw3ICs0MDIsNyBAQCBkZWYgc3N0YXRlX2luc3RhbGxw a2dkaXIoc3MsIGQpOgogCiAgICAgZm9yIHN0YXRlIGluIHNzWydkaXJzJ106CiAgICAgICAgIHBy ZXBkaXIoc3RhdGVbMV0pCi0gICAgICAgIG9zLnJlbmFtZShzc3RhdGVpbnN0ICsgc3RhdGVbMF0s IHN0YXRlWzFdKQorICAgICAgICBzaHV0aWwubW92ZShzc3RhdGVpbnN0ICsgc3RhdGVbMF0sIHN0 YXRlWzFdKQogICAgIHNzdGF0ZV9pbnN0YWxsKHNzLCBkKQogCiAgICAgZm9yIHBsYWluIGluIHNz WydwbGFpbmRpcnMnXToKQEAgLTQxMyw3ICs0MTQsNyBAQCBkZWYgc3N0YXRlX2luc3RhbGxwa2dk aXIoc3MsIGQpOgogICAgICAgICBkZXN0ID0gcGxhaW4KICAgICAgICAgYmIudXRpbHMubWtkaXJo aWVyKHNyYykKICAgICAgICAgcHJlcGRpcihkZXN0KQotICAgICAgICBvcy5yZW5hbWUoc3JjLCBk ZXN0KQorICAgICAgICBzaHV0aWwubW92ZShzcmMsIGRlc3QpCiAKICAgICByZXR1cm4gVHJ1ZQog CkBAIC02MzgsNiArNjM5LDcgQEAgcHl0aG9uIHNzdGF0ZV9oYXJkY29kZV9wYXRoICgpIHsKIAog ZGVmIHNzdGF0ZV9wYWNrYWdlKHNzLCBkKToKICAgICBpbXBvcnQgb2UucGF0aAorICAgIGltcG9y dCBzaHV0aWwKIAogICAgIHRtcGRpciA9IGQuZ2V0VmFyKCdUTVBESVInKQogCkBAIC02NjQsNyAr NjY2LDcgQEAgZGVmIHNzdGF0ZV9wYWNrYWdlKHNzLCBkKToKICAgICAgICAgICAgICAgICAgICAg Y29udGludWUKICAgICAgICAgICAgICAgICBiYi5lcnJvcigic3N0YXRlIGZvdW5kIGFuIGFic29s dXRlIHBhdGggc3ltbGluayAlcyBwb2ludGluZyBhdCAlcy4gUGxlYXNlIHJlcGxhY2UgdGhpcyB3 aXRoIGEgcmVsYXRpdmUgbGluay4iICUgKHNyY3BhdGgsIGxpbmspKQogICAgICAgICBiYi5kZWJ1 ZygyLCAiUHJlcGFyaW5nIHRyZWUgJXMgZm9yIHBhY2thZ2luZyBhdCAlcyIgJSAoc3RhdGVbMV0s IHNzdGF0ZWJ1aWxkICsgc3RhdGVbMF0pKQotICAgICAgICBvcy5yZW5hbWUoc3RhdGVbMV0sIHNz dGF0ZWJ1aWxkICsgc3RhdGVbMF0pCisgICAgICAgIHNodXRpbC5tb3ZlKHN0YXRlWzFdLCBzc3Rh dGVidWlsZCArIHN0YXRlWzBdKQogCiAgICAgd29ya2RpciA9IGQuZ2V0VmFyKCdXT1JLRElSJykK ICAgICBzaGFyZWR3b3JrZGlyID0gb3MucGF0aC5qb2luKGQuZ2V0VmFyKCdUTVBESVInKSwgIndv cmstc2hhcmVkIikKQEAgLTY3NCw3ICs2NzYsNyBAQCBkZWYgc3N0YXRlX3BhY2thZ2Uoc3MsIGQp OgogICAgICAgICAgICAgcGRpciA9IHBsYWluLnJlcGxhY2Uoc2hhcmVkd29ya2Rpciwgc3N0YXRl YnVpbGQpCiAgICAgICAgIGJiLnV0aWxzLm1rZGlyaGllcihwbGFpbikKICAgICAgICAgYmIudXRp bHMubWtkaXJoaWVyKHBkaXIpCi0gICAgICAgIG9zLnJlbmFtZShwbGFpbiwgcGRpcikKKyAgICAg ICAgc2h1dGlsLm1vdmUocGxhaW4sIHBkaXIpCiAKICAgICBkLnNldFZhcignU1NUQVRFX0JVSUxE RElSJywgc3N0YXRlYnVpbGQpCiAgICAgZC5zZXRWYXIoJ1NTVEFURV9JTlNURElSJywgc3N0YXRl YnVpbGQpCg== 4777 2021-03-18 21:38:47 +0000 2021-03-18 21:38:47 +0000 Use shutil.move when os.rename fails use-shutil-when-os-rename-fails.patch.txt text/plain 2380 devendra.tewari ZGlmZiAtLWdpdCBhL21ldGEvY2xhc3Nlcy9zc3RhdGUuYmJjbGFzcyBiL21ldGEvY2xhc3Nlcy9z c3RhdGUuYmJjbGFzcwppbmRleCBmNTc5MTY4MTYyLi5mOTRhYTk2ZDcwIDEwMDY0NAotLS0gYS9t ZXRhL2NsYXNzZXMvc3N0YXRlLmJiY2xhc3MKKysrIGIvbWV0YS9jbGFzc2VzL3NzdGF0ZS5iYmNs YXNzCkBAIC0zODQsNiArMzg0LDcgQEAgZGVmIHNzdGF0ZV9pbnN0YWxscGtnKHNzLCBkKToKIGRl ZiBzc3RhdGVfaW5zdGFsbHBrZ2RpcihzcywgZCk6CiAgICAgaW1wb3J0IG9lLnBhdGgKICAgICBp bXBvcnQgc3VicHJvY2VzcworICAgIGltcG9ydCBzaHV0aWwKIAogICAgIHNzdGF0ZWluc3QgPSBk LmdldFZhcigiU1NUQVRFX0lOU1RESVIiKQogICAgIGQuc2V0VmFyKCdTU1RBVEVfRklYTUVESVIn LCBzc1snZml4bWVkaXInXSkKQEAgLTQwMSw3ICs0MDIsMTEgQEAgZGVmIHNzdGF0ZV9pbnN0YWxs cGtnZGlyKHNzLCBkKToKIAogICAgIGZvciBzdGF0ZSBpbiBzc1snZGlycyddOgogICAgICAgICBw cmVwZGlyKHN0YXRlWzFdKQotICAgICAgICBvcy5yZW5hbWUoc3N0YXRlaW5zdCArIHN0YXRlWzBd LCBzdGF0ZVsxXSkKKyAgICAgICAgdHJ5OgorICAgICAgICAgICAgb3MucmVuYW1lKHNzdGF0ZWlu c3QgKyBzdGF0ZVswXSwgc3RhdGVbMV0pCisgICAgICAgICAgICBicmVhaworICAgICAgICBleGNl cHQgT1NFcnJvcjoKKyAgICAgICAgICAgIHNodXRpbC5tb3ZlKHNzdGF0ZWluc3QgKyBzdGF0ZVsw XSwgc3RhdGVbMV0pCiAgICAgc3N0YXRlX2luc3RhbGwoc3MsIGQpCiAKICAgICBmb3IgcGxhaW4g aW4gc3NbJ3BsYWluZGlycyddOgpAQCAtNDEzLDcgKzQxOCwxMSBAQCBkZWYgc3N0YXRlX2luc3Rh bGxwa2dkaXIoc3MsIGQpOgogICAgICAgICBkZXN0ID0gcGxhaW4KICAgICAgICAgYmIudXRpbHMu bWtkaXJoaWVyKHNyYykKICAgICAgICAgcHJlcGRpcihkZXN0KQotICAgICAgICBvcy5yZW5hbWUo c3JjLCBkZXN0KQorICAgICAgICB0cnk6CisgICAgICAgICAgICBvcy5yZW5hbWUoc3JjLCBkZXN0 KQorICAgICAgICAgICAgYnJlYWsKKyAgICAgICAgZXhjZXB0IE9TRXJyb3I6CisgICAgICAgICAg ICBzaHV0aWwubW92ZShzcmMsIGRlc3QpCiAKICAgICByZXR1cm4gVHJ1ZQogCkBAIC02MzgsNiAr NjQ3LDcgQEAgcHl0aG9uIHNzdGF0ZV9oYXJkY29kZV9wYXRoICgpIHsKIAogZGVmIHNzdGF0ZV9w YWNrYWdlKHNzLCBkKToKICAgICBpbXBvcnQgb2UucGF0aAorICAgIGltcG9ydCBzaHV0aWwKIAog ICAgIHRtcGRpciA9IGQuZ2V0VmFyKCdUTVBESVInKQogCkBAIC02NjQsNyArNjc0LDExIEBAIGRl ZiBzc3RhdGVfcGFja2FnZShzcywgZCk6CiAgICAgICAgICAgICAgICAgICAgIGNvbnRpbnVlCiAg ICAgICAgICAgICAgICAgYmIuZXJyb3IoInNzdGF0ZSBmb3VuZCBhbiBhYnNvbHV0ZSBwYXRoIHN5 bWxpbmsgJXMgcG9pbnRpbmcgYXQgJXMuIFBsZWFzZSByZXBsYWNlIHRoaXMgd2l0aCBhIHJlbGF0 aXZlIGxpbmsuIiAlIChzcmNwYXRoLCBsaW5rKSkKICAgICAgICAgYmIuZGVidWcoMiwgIlByZXBh cmluZyB0cmVlICVzIGZvciBwYWNrYWdpbmcgYXQgJXMiICUgKHN0YXRlWzFdLCBzc3RhdGVidWls ZCArIHN0YXRlWzBdKSkKLSAgICAgICAgb3MucmVuYW1lKHN0YXRlWzFdLCBzc3RhdGVidWlsZCAr IHN0YXRlWzBdKQorICAgICAgICB0cnk6CisgICAgICAgICAgICBvcy5yZW5hbWUoc3RhdGVbMV0s IHNzdGF0ZWJ1aWxkICsgc3RhdGVbMF0pCisgICAgICAgICAgICBicmVhaworICAgICAgICBleGNl cHQgT1NFcnJvcjoKKyAgICAgICAgICAgIHNodXRpbC5tb3ZlKHN0YXRlWzFdLCBzc3RhdGVidWls ZCArIHN0YXRlWzBdKQogCiAgICAgd29ya2RpciA9IGQuZ2V0VmFyKCdXT1JLRElSJykKICAgICBz aGFyZWR3b3JrZGlyID0gb3MucGF0aC5qb2luKGQuZ2V0VmFyKCdUTVBESVInKSwgIndvcmstc2hh cmVkIikKQEAgLTY3NCw3ICs2ODgsMTEgQEAgZGVmIHNzdGF0ZV9wYWNrYWdlKHNzLCBkKToKICAg ICAgICAgICAgIHBkaXIgPSBwbGFpbi5yZXBsYWNlKHNoYXJlZHdvcmtkaXIsIHNzdGF0ZWJ1aWxk KQogICAgICAgICBiYi51dGlscy5ta2RpcmhpZXIocGxhaW4pCiAgICAgICAgIGJiLnV0aWxzLm1r ZGlyaGllcihwZGlyKQotICAgICAgICBvcy5yZW5hbWUocGxhaW4sIHBkaXIpCisgICAgICAgIHRy eToKKyAgICAgICAgICAgIG9zLnJlbmFtZShwbGFpbiwgcGRpcikKKyAgICAgICAgICAgIGJyZWFr CisgICAgICAgIGV4Y2VwdCBPU0Vycm9yOgorICAgICAgICAgICAgc2h1dGlsLm1vdmUocGxhaW4s IHBkaXIpCiAKICAgICBkLnNldFZhcignU1NUQVRFX0JVSUxERElSJywgc3N0YXRlYnVpbGQpCiAg ICAgZC5zZXRWYXIoJ1NTVEFURV9JTlNURElSJywgc3N0YXRlYnVpbGQpCg==