Bug 13923

oe-core is incompatible with Docker overlay(2) storage driver some files are in another overlay layer. Something that happens if parts of the build directory is pre-populated in a pre-built Docker image.

Scenario:
- Run bitbake --setscene-only <IMAGE>
- Add the resulting build directory to a Docker image
  (The purpose is to make a prebuilt image for faster build times)
- Use the image to build a newer image (including new external sstate changes and code changes)

With pre-populated docker mage content there is a risk of ERRNO 18 "Invalid cross-device link" even if being on the same mount point.
Docker are very clear about this limitation (https://docs.docker.com/storage/storagedriver/overlayfs-driver/) and states it is up to the application (bitbake / oe) to solve the issue.
- overlay(2) storage driver is the promoted and preferred solution from Docker over AUFS and other storage drivers.

Example from an image pre-populated via a bitbake --setscene <image>
file: '<builddir>/build/meta/classes/sstate.bbclass', lineno: 637, function: sstate_package
     0633:                if not link.startswith(tmpdir):
     0634:                    continue
     0635:                bb.error("sstate found an absolute path symlink %s pointing at %s. Please replace this with a relative link." % (srcpath, link))
     0636:        bb.debug(2, "Preparing tree %s for packaging at %s" % (state[1], sstatebuild + state[0]))
 *** 0637:        os.rename(state[1], sstatebuild + state[0])
     0638:
     0639:    workdir = d.getVar('WORKDIR')
     0640:    sharedworkdir = os.path.join(d.getVar('TMPDIR'), "work-shared")
     0641:    for plain in ss['plaindirs']:
Exception: OSError: [Errno 18] Invalid cross-device link: '<builddir>/build/tmp/work/<machine>-poky-linux-gnueabi/busybox/1.31.0-r0/pkgdata-pdata-input' -> '/home/svcj-build/build/build/tmp/work/<machine>-poky-linux-gnueabi/busybox/1.31.0-r0/sstate-build-packagedata/pkgdata-pdata-input'

According to Docker, the proper solution is to have a move fallback and not just relay on os.rename (python) and any other rename function.

Unfortunately, Bitbake/Poky uses os.rename and os.link heavily and this, appears to be incompatible with this way of working.

This may be a deliberate design choice and then this is no bug but I want to make you aware that it is not possible to have any performance gains by using a pre-populated /pre-built Docker image as a base for building with bitbake.

Unfortunately, there are some steps to create the docker image so I can't give a nice "steps-to-reproduce". Thus, this is more of an awareness issue.


Solution:
- It might be possible to replace all os.rename with bb.utils.movefile to avoid the error given above. There might be more issues though with os.link operations etc.

However, I am concerned that the way recipe-specific-sysroot works with hardlinks might prevent this use-case all together.