Bug 14301

Summary: Incremental build in Docker fails with Invalid cross-device link error
Product: [Build System, Metadata & Runtime] Meta-yocto Reporter: Devendra Tewari <devendra.tewari>
Component: meta-yoctoAssignee: Devendra Tewari <devendra.tewari>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium CC: devendra.tewari, poky.bs.watcher, poky.watcher, randy.macleod
Version: unspecified   
Target Milestone: 3.4   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Don't know
Attachments:
Description Flags
Use shutil.move instead of os.rename
none
Use shutil.move when os.rename fails none

Description Devendra Tewari 2021-03-11 18:24:32 UTC 
Incremental build in Docker on macOS BigSur fails with errors of the kind reproduced below. The error does not occur when shutil.move is used instead of os.rename, please see https://stackoverflow.com/questions/42392600/oserror-errno-18-invalid-cross-device-link.

ERROR: core-image-base-1.0-r0 do_deploy_source_date_epoch: Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:sstate_task_postfunc(d)
     0003:
File: '/home/pi/yocto-raspberry-pi/poky/meta/classes/sstate.bbclass', lineno: 770, function: sstate_task_postfunc
     0766:
     0767:    omask = os.umask(0o002)
     0768:    if omask != 0o002:
     0769:       bb.note("Using umask 0o002 (not %0o) for sstate packaging" % omask)
 *** 0770:    sstate_package(shared_state, d)
     0771:    os.umask(omask)
     0772:
     0773:    sstateinst = d.getVar("SSTATE_INSTDIR")
     0774:    d.setVar('SSTATE_FIXMEDIR', shared_state['fixmedir'])
File: '/home/pi/yocto-raspberry-pi/poky/meta/classes/sstate.bbclass', lineno: 677, function: sstate_package
     0673:        if sharedworkdir in plain:
     0674:            pdir = plain.replace(sharedworkdir, sstatebuild)
     0675:        bb.utils.mkdirhier(plain)
     0676:        bb.utils.mkdirhier(pdir)
 *** 0677:        os.rename(plain, pdir)
     0678:
     0679:    d.setVar('SSTATE_BUILDDIR', sstatebuild)
     0680:    d.setVar('SSTATE_INSTDIR', sstatebuild)
     0681:
Exception: OSError: [Errno 18] Invalid cross-device link: '/home/pi/yocto-raspberry-pi/build/tmp/work/raspberrypi0_wifi-poky-linux-gnueabi/core-image-base/1.0-r0/deploy-source-date-epoch' -> '/home/pi/yocto-raspberry-pi/build/tmp/work/raspberrypi0_wifi-poky-linux-gnueabi/core-image-base/1.0-r0/sstate-build-deploy_source_date_epoch//deploy-source-date-epoch'

ERROR: Logfile of failure stored in: /home/pi/yocto-raspberry-pi/build/tmp/work/raspberrypi0_wifi-poky-linux-gnueabi/core-image-base/1.0-r0/temp/log.do_deploy_source_date_epoch.77
ERROR: Task (/home/pi/yocto-raspberry-pi/poky/meta/recipes-core/images/core-image-base.bb:do_deploy_source_date_epoch) failed with exit code '1'
NOTE: Tasks Summary: Attempted 4634 tasks of which 4633 didn't need to be rerun and 1 failed.
Comment 1 Devendra Tewari 2021-03-11 18:29:32 UTC 
Created attachment 4776 [details]
Use shutil.move instead of os.rename
Comment 2 Randy MacLeod 2021-03-18 14:40:18 UTC 
The rename is faster but we could trap the error and fall-back to the move if the host/filesystem has an error. Can you send a patch?
Comment 3 Devendra Tewari 2021-03-18 21:37:07 UTC 
According to documentation, shutil.move falls back to os.rename when source and destination are on the same filesystem, but trapping error with os.rename and trying shutil.move looks safer. I'll attach a patch.
Comment 4 Devendra Tewari 2021-03-18 21:38:47 UTC 
Created attachment 4777 [details]
Use shutil.move when os.rename fails
Comment 5 Randy MacLeod 2021-03-23 20:08:31 UTC 
Thanks Devendra,

Are you able to send he patch to the oe-core email list, like this:
   git send-email -M -1 --to="openembedded-core@lists.openembedded.org" 
See:
https://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded#Sending_using_git-send-email
Comment 6 Devendra Tewari 2021-03-23 20:37:35 UTC 
Thanks, Randy. I've submitted the patch via git send-email.
Comment 7 Randy MacLeod 2021-03-29 14:50:07 UTC 
Hello Devendra,
I didn't see the email. Did I miss it? Did it get merged? 
../Randy
Comment 8 Devendra Tewari 2021-03-29 15:19:12 UTC 
Sorry, looks like I didn't have an smtp server setup and the mail didn't go out. Sent it out again, and confirmed it got sent at https://lists.openembedded.org/g/openembedded-core/topic/patch_use_shutil_move_when/81698791.
Comment 9 Devendra Tewari 2021-10-18 18:04:51 UTC 
Patches have been integrated.