13923 2020-05-29 01:14:35 +0000 OE-Core not compatible with Docker overlayfs and pre-built images 2020-06-05 01:19:00 +0000 1 1 7 Build System, Metadata & Runtime OE-Core core unspecified x86 Multiple RESOLVED WONTFIX Medium normal 3.2 M2 1 tobias.hagelborn JPEWhacker meta.mr.watcher meta.watcher randy.macleod --- --- Yes (doc changes required) oldest_to_newest 87363 0 tobias.hagelborn 2020-05-29 01:14:35 +0000 oe-core is incompatible with Docker overlay(2) storage driver some files are in another overlay layer. Something that happens if parts of the build directory is pre-populated in a pre-built Docker image. Scenario: - Run bitbake --setscene-only <IMAGE> - Add the resulting build directory to a Docker image (The purpose is to make a prebuilt image for faster build times) - Use the image to build a newer image (including new external sstate changes and code changes) With pre-populated docker mage content there is a risk of ERRNO 18 "Invalid cross-device link" even if being on the same mount point. Docker are very clear about this limitation (https://docs.docker.com/storage/storagedriver/overlayfs-driver/) and states it is up to the application (bitbake / oe) to solve the issue. - overlay(2) storage driver is the promoted and preferred solution from Docker over AUFS and other storage drivers. Example from an image pre-populated via a bitbake --setscene <image> file: '<builddir>/build/meta/classes/sstate.bbclass', lineno: 637, function: sstate_package 0633: if not link.startswith(tmpdir): 0634: continue 0635: bb.error("sstate found an absolute path symlink %s pointing at %s. Please replace this with a relative link." % (srcpath, link)) 0636: bb.debug(2, "Preparing tree %s for packaging at %s" % (state[1], sstatebuild + state[0])) *** 0637: os.rename(state[1], sstatebuild + state[0]) 0638: 0639: workdir = d.getVar('WORKDIR') 0640: sharedworkdir = os.path.join(d.getVar('TMPDIR'), "work-shared") 0641: for plain in ss['plaindirs']: Exception: OSError: [Errno 18] Invalid cross-device link: '<builddir>/build/tmp/work/<machine>-poky-linux-gnueabi/busybox/1.31.0-r0/pkgdata-pdata-input' -> '/home/svcj-build/build/build/tmp/work/<machine>-poky-linux-gnueabi/busybox/1.31.0-r0/sstate-build-packagedata/pkgdata-pdata-input' According to Docker, the proper solution is to have a move fallback and not just relay on os.rename (python) and any other rename function. Unfortunately, Bitbake/Poky uses os.rename and os.link heavily and this, appears to be incompatible with this way of working. This may be a deliberate design choice and then this is no bug but I want to make you aware that it is not possible to have any performance gains by using a pre-populated /pre-built Docker image as a base for building with bitbake. Unfortunately, there are some steps to create the docker image so I can't give a nice "steps-to-reproduce". Thus, this is more of an awareness issue. Solution: - It might be possible to replace all os.rename with bb.utils.movefile to avoid the error given above. There might be more issues though with os.link operations etc. However, I am concerned that the way recipe-specific-sysroot works with hardlinks might prevent this use-case all together. 87423 1 JPEWhacker 2020-06-04 11:23:08 +0000 bitbake and OE make the assumption that TMPDIR is one a single filesystem, and changing that is not something we really want to fix. A different way to solve this might be instead to include the initial sstate cache in the docker image so that the first time the user runs a command, it restores (quickly) from sstate. 87424 2 JPEWhacker 2020-06-04 11:24:52 +0000 To clarify: we don't want to fix it because it would be quite disruptive and likely to easily break again without ongoing effort to test and maintain. 87431 3 tobias.hagelborn 2020-06-05 01:19:00 +0000 Thanks for having a look at this Joshua and I fully understand that you have to make this kind of assumptions on the underlying file system. It is unfortunate that Docker overlayfs has this kind of limitation. Unfortunately in-image sstate cache is not an option for us, I wanted to avoid the setscene procedure all together since it is actually very CPU intensive and takes 5 minutes for us with 16 active cores. (We do 10K+ builds like this every day)