Created attachment 4601 [details] Patch to apply on poky/meta layer Conflicting files in sysroot when switching provider by switching MACHINE Real-world-scenario: Building multiple products in one build-tree. The image contains virtual providers that are provided by different recipes on different machines. The recipes depending on the virtual provider is Consequences: Can't re-use builddir when switching products for our set of recipes which all contain this dependency setup. The example (patch provided): In the example, the main recipe foo depends on 2 machine-arch specific recipes bar-arch1 and bar-arch2 for different products. bar is provided to foo via virtual/bar. The two machines share architecture. For this a test64 machine is created that share architecture with qemux86-64. How to repeat: - checkout poky (zeus) - export MACHINE=qemux86-64 - bitbake -c install foo - export MACHINE=test64 - You should have gotten file conflict exceptions by now extend_recipe_sysroot: Exception: FileExistsError: [Errno 17] ==== ERROR: foo-1.0-r0 do_prepare_recipe_sysroot: Error executing a python function in exec_python_func() autogenerated: The stack trace of python calls that resulted in this exception/failure was: File: 'exec_python_func() autogenerated', lineno: 2, function: <module> 0001: *** 0002:extend_recipe_sysroot(d) 0003: File: '/virtualtest/meta/classes/staging.bbclass', lineno: 551, function: extend_recipe_sysroot 0547: dest = newmanifest[l] 0548: if l.endswith("/"): 0549: staging_copydir(l, targetdir, dest, seendirs) 0550: continue *** 0551: staging_copyfile(l, targetdir, dest, postinsts, seendirs) 0552: 0553: bb.note("Installed into sysroot: %s" % str(msg_adding)) 0554: bb.note("Skipping as already exists in sysroot: %s" % str(msg_exists)) 0555: File: '/virtualtest/meta/classes/staging.bbclass', lineno: 152, function: staging_copyfile 0148: os.symlink(linkto, dest) 0149: #bb.warn(c) 0150: else: 0151: try: *** 0152: os.link(c, dest) 0153: except OSError as err: 0154: if err.errno == errno.EXDEV: 0155: bb.utils.copyfile(c, dest) 0156: else: Exception: FileExistsError: [Errno 17] File exists: '/virtualtest/build/tmp/sysroots-components/test64/bar-arch2/usr/include/bar/bar.h' -> '/virtualtest/build/tmp/work/core2-64-poky-linux/foo/1.0-r0/recipe-sysroot/usr/include/bar/bar.h' ERROR: Logfile of failure stored in: /virtualtest/build/tmp/work/core2-64-poky-linux/foo/1.0-r0/temp/log.do_prepare_recipe_sysroot.6865 ERROR: Task (/virtualtest/meta/recipes-support/foo/foo_1.0.bb:do_prepare_recipe_sysroot) failed with exit code '1' ==== Issue relation: This issue seems to have very strong relationship to Bug 12180 but this one triggers the recipe sysroot update from a change of machines and virtual provider instead. Workaround: So far, we have tried, without success solutions along the lines of: do_prepare_recipe_sysroot[cleandirs] += "${@d.getVar('RECIPE_SYSROOT') if 'virtual/' in d.getVar('DEPENDS')}" We have also forced the recipe corresponding to 'foo' in the example to be machine specific. This works around the issue but is wrong by design, it does not have to be machine specific. Also, this reuires a prior knowledge of what recipes to update. Solution: I have looked into the staging.bbclass and extend_recipe_sysroot but it is a huge function and I can't determine the proper level at which to invalidate the sysroot.
Something very similar was fixed in master recently. Can you see if https://git.openembedded.org/openembedded-core/commit/?id=e3e5ace6e68d5fe68e4add301a44c1a1b8607411 fixes the issue?
I applied the suggested patch and re-run. I still get the same issue. I have also tried the suggested patch earlier in our current system and it did not solve the problems there. I should also update the procedure: (apply patch(es)) - export MACHINE=qemux86-64 - bitbake -c prepare_recipe_sysroot foo - export MACHINE=test64 - bitbake -c prepare_recipe_sysroot foo - export MACHINE=qemux86-64 - bitbake -c prepare_recipe_sysroot foo -- file conflict exceptions --- Exception: FileExistsError: [Errno 17] File exists: '/poky-master/build/tmp/sysroots-components/test64/bar-arch2/usr/include/bar/bar.h' -> '/poky-master/build/tmp/work/core2-64-poky-linux/foo/1.0-r0/recipe-sysroot/usr/include/bar/bar.h'
Adding and removing a dependency seems to do quite stable updates to the sysroot but this type of change in dependency does not seem to trigger delete/add in the same way. I just tried to verify that the signature of prepare_recipe_sysroot does change (Setting new dependency hash on bar1/bar2) with every invocation of bitbake with a new machine so that part seems ok from what I can see. ===== $> MACHINE=test64 bitbake -c prepare_recipe_sysroot foo ... MACHINE = "test64" ... NOTE: Tasks Summary: Attempted 252 tasks of which 252 didn't need to be rerun and all succeeded. > MACHINE=test64 bitbake-diffsigs -t foo prepare_recipe_sysroot NOTE: Starting bitbake server... runtaskdeps changed: [-bar-arch1/bar-arch1_1.0.bb:do_populate_sysroot, +bar-arch2/bar-arch2_1.0.bb:do_populate_sysroot, 'foo/foo_1.0.bb:do_fetch gcc/gcc-cross_9.2.bb:do_populate_sysroot gcc/gcc-runtime_9.2.bb:do_populate_sysroot glibc/glibc_2.30.bb:do_populate_sysroot'] bar-arch1/bar-arch1_1.0.bb:do_populate_sysroot with hash 3b0a0f9dcd26192034289c6fa0c361d025d23f5079ad3fa709c8abfd45e35145 changed to bar-arch2/bar-arch2_1.0.bb:do_populate_sysroot with hash ece77e7445ddc2fd6e77b0686754570e3c87b33b28e7e1d55eaa82128653cecd Dependency on task bar-arch2/bar-arch2_1.0.bb:do_populate_sysroot was added with hash ece77e7445ddc2fd6e77b0686754570e3c87b33b28e7e1d55eaa82128653cecd Dependency on task bar-arch1/bar-arch1_1.0.bb:do_populate_sysroot was removed with hash 3b0a0f9dcd26192034289c6fa0c361d025d23f5079ad3fa709c8abfd45e35145 storage/tobiasha/dists/poky-master/build> MACHINE=qemux86-64 bitbake-diffsigs -t foo prepare_recipe_sysroot NOTE: Starting bitbake server... runtaskdeps changed: [-bar-arch1/bar-arch1_1.0.bb:do_populate_sysroot, +bar-arch2/bar-arch2_1.0.bb:do_populate_sysroot, 'foo/foo_1.0.bb:do_fetch gcc/gcc-cross_9.2.bb:do_populate_sysroot gcc/gcc-runtime_9.2.bb:do_populate_sysroot glibc/glibc_2.30.bb:do_populate_sysroot'] bar-arch1/bar-arch1_1.0.bb:do_populate_sysroot with hash 3b0a0f9dcd26192034289c6fa0c361d025d23f5079ad3fa709c8abfd45e35145 changed to bar-arch2/bar-arch2_1.0.bb:do_populate_sysroot with hash ece77e7445ddc2fd6e77b0686754570e3c87b33b28e7e1d55eaa82128653cecd Dependency on task bar-arch2/bar-arch2_1.0.bb:do_populate_sysroot was added with hash ece77e7445ddc2fd6e77b0686754570e3c87b33b28e7e1d55eaa82128653cecd Dependency on task bar-arch1/bar-arch1_1.0.bb:do_populate_sysroot was removed with hash 3b0a0f9dcd26192034289c6fa0c361d025d23f5079ad3fa709c8abfd45e35145 $> MACHINE=qemux86-64 bitbake -c prepare_recipe_sysroot foo ... MACHINE = "qemux86-64" ... NOTE: Tasks Summary: Attempted 252 tasks of which 251 didn't need to be rerun and all succeeded. > MACHINE=test64 bitbake-diffsigs -t foo prepare_recipe_sysroot NOTE: Starting bitbake server... runtaskdeps changed: [-bar-arch2/bar-arch2_1.0.bb:do_populate_sysroot, +bar-arch1/bar-arch1_1.0.bb:do_populate_sysroot, 'foo/foo_1.0.bb:do_fetch gcc/gcc-cross_9.2.bb:do_populate_sysroot gcc/gcc-runtime_9.2.bb:do_populate_sysroot glibc/glibc_2.30.bb:do_populate_sysroot'] bar-arch2/bar-arch2_1.0.bb:do_populate_sysroot with hash ece77e7445ddc2fd6e77b0686754570e3c87b33b28e7e1d55eaa82128653cecd changed to bar-arch1/bar-arch1_1.0.bb:do_populate_sysroot with hash 3b0a0f9dcd26192034289c6fa0c361d025d23f5079ad3fa709c8abfd45e35145 Dependency on task bar-arch1/bar-arch1_1.0.bb:do_populate_sysroot was added with hash 3b0a0f9dcd26192034289c6fa0c361d025d23f5079ad3fa709c8abfd45e35145 Dependency on task bar-arch2/bar-arch2_1.0.bb:do_populate_sysroot was removed with hash ece77e7445ddc2fd6e77b0686754570e3c87b33b28e7e1d55eaa82128653cecd
Okay, thanks for the update. Reassigning to RP who understands this piece more than I do.
Sorry for the delay in replying. I think the issue is that foo is MACHINE specific (as its dependency virtual/bar is machine specific) but foo isn't marked as such. I'd guess if you add PACKAGE_ARCH = "${MACHINE_ARCH}" to the foo recipe, it will work. That said, the system should recover better from this situation so it is a valid bug.
There is code in staging.bbclass which is meant to handle this: # We want to remove anything which this task previously installed but is no longer a dependency However we hit: # We need to ensure not other task needs this dependency. We hold the sysroot # lock so we ca search the indexes to check and since the old bar-archX is referenced by index.do_install, it doesn't get removed. That explains why the code isn't helping. How to fix it...
Thanks for looking into this! Setting PACKAGE_ARCH = "${MACHINE_ARCH}" works for a specific recipe. I tried it in our environment but unfortunately the number of such cases exploded and it was not feasible as a full workaround. Thanks for the hint on where to look in staging.bbclass. I will probably not know how to correct this one but I can set some more debug print in those sections to better understand the issue.
I figured out a fix for this and also wrote up a test case for oe-selftest (oe-selftest -r sysroot), the patches are in master-next. It turned out not to be too bad to implement in the end.
This is great news! I will try to verify this against our build.
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=f135cb36596d9cc8996277f49b296b9f73b76b73 http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=3e9cdd878887d0e59754447f313c39d142b71971
Thanks for the fixes provided! Unfortunately we seem to have some added complexity in our setup because we still see errors that appear to be similar to the original. File: '/home/tobiasha/dists/cfp-master/meta/classes/staging.bbclass', lineno: 152, function: staging_copyfile 0148: os.symlink(linkto, dest) 0149: #bb.warn(c) 0150: else: 0151: try: *** 0152: os.link(c, dest) 0153: except OSError as err: 0154: if err.errno == errno.EXDEV: 0155: bb.utils.copyfile(c, dest) 0156: else: Exception: FileExistsError: [Errno 17] File exists: '<PATH>/build/tmp/sysroots-components/<ARCH>/<RECIPE>/sysroot-providers/virtual_video_driver' -> '<PATH>/ <RECIPE>/8.0.0-r0/recipe-sysroot/sysroot-providers/virtual_video_driver' There is also a new one I have not sen before: File: '<PATH>/meta/classes/staging.bbclass', lineno: 447, function: extend_recipe_sysroot 0443: bb.note("Task %s no longer depends on %s, removing from sysroot" % (mytaskname, l)) 0444: lnk = os.readlink(fl) 0445: sstate_clean_manifest(depdir + "/" + lnk, d, workdir) 0446: os.unlink(fl) *** 0447: os.unlink(fl + ".complete") 0448: 0449: msg_exists = [] 0450: msg_adding = [] 0451: Exception: FileNotFoundError: [Errno 2] No such file or directory: '<PATH>/build/tmp/work/cortexa9hf-neon-poky- linux-gnueabi/<foo>/1.1.0-r0/recipe-sysroot-native/installeddeps/<bar>.complete' It is too early to say if I can come up with a repeater test like a qa test for these issies.