13702 2019-12-12 11:42:56 +0000 Conflicting files in sysroot when switching provider by switching MACHINE 2020-04-29 05:43:34 +0000 1 1 7 Build System, Metadata & Runtime OE-Core core unspecified All Multiple RESOLVED FIXED https://bugzilla.yoctoproject.org/show_bug.cgi?id=12180 Medium+ major 3.2 M1 1 tobias.hagelborn richard.purdie akuster meta.mr.watcher meta.watcher --- --- No (bug/feature does not impact docs) oldest_to_newest 85799 0 4601 tobias.hagelborn 2019-12-12 11:42:56 +0000 Created attachment 4601 Patch to apply on poky/meta layer Conflicting files in sysroot when switching provider by switching MACHINE Real-world-scenario: Building multiple products in one build-tree. The image contains virtual providers that are provided by different recipes on different machines. The recipes depending on the virtual provider is Consequences: Can't re-use builddir when switching products for our set of recipes which all contain this dependency setup. The example (patch provided): In the example, the main recipe foo depends on 2 machine-arch specific recipes bar-arch1 and bar-arch2 for different products. bar is provided to foo via virtual/bar. The two machines share architecture. For this a test64 machine is created that share architecture with qemux86-64. How to repeat: - checkout poky (zeus) - export MACHINE=qemux86-64 - bitbake -c install foo - export MACHINE=test64 - You should have gotten file conflict exceptions by now extend_recipe_sysroot: Exception: FileExistsError: [Errno 17] ==== ERROR: foo-1.0-r0 do_prepare_recipe_sysroot: Error executing a python function in exec_python_func() autogenerated: The stack trace of python calls that resulted in this exception/failure was: File: 'exec_python_func() autogenerated', lineno: 2, function: <module> 0001: *** 0002:extend_recipe_sysroot(d) 0003: File: '/virtualtest/meta/classes/staging.bbclass', lineno: 551, function: extend_recipe_sysroot 0547: dest = newmanifest[l] 0548: if l.endswith("/"): 0549: staging_copydir(l, targetdir, dest, seendirs) 0550: continue *** 0551: staging_copyfile(l, targetdir, dest, postinsts, seendirs) 0552: 0553: bb.note("Installed into sysroot: %s" % str(msg_adding)) 0554: bb.note("Skipping as already exists in sysroot: %s" % str(msg_exists)) 0555: File: '/virtualtest/meta/classes/staging.bbclass', lineno: 152, function: staging_copyfile 0148: os.symlink(linkto, dest) 0149: #bb.warn(c) 0150: else: 0151: try: *** 0152: os.link(c, dest) 0153: except OSError as err: 0154: if err.errno == errno.EXDEV: 0155: bb.utils.copyfile(c, dest) 0156: else: Exception: FileExistsError: [Errno 17] File exists: '/virtualtest/build/tmp/sysroots-components/test64/bar-arch2/usr/include/bar/bar.h' -> '/virtualtest/build/tmp/work/core2-64-poky-linux/foo/1.0-r0/recipe-sysroot/usr/include/bar/bar.h' ERROR: Logfile of failure stored in: /virtualtest/build/tmp/work/core2-64-poky-linux/foo/1.0-r0/temp/log.do_prepare_recipe_sysroot.6865 ERROR: Task (/virtualtest/meta/recipes-support/foo/foo_1.0.bb:do_prepare_recipe_sysroot) failed with exit code '1' ==== Issue relation: This issue seems to have very strong relationship to Bug 12180 but this one triggers the recipe sysroot update from a change of machines and virtual provider instead. Workaround: So far, we have tried, without success solutions along the lines of: do_prepare_recipe_sysroot[cleandirs] += "${@d.getVar('RECIPE_SYSROOT') if 'virtual/' in d.getVar('DEPENDS')}" We have also forced the recipe corresponding to 'foo' in the example to be machine specific. This works around the issue but is wrong by design, it does not have to be machine specific. Also, this reuires a prior knowledge of what recipes to update. Solution: I have looked into the staging.bbclass and extend_recipe_sysroot but it is a huge function and I can't determine the proper level at which to invalidate the sysroot. 85813 1 ross.burton 2019-12-12 16:20:03 +0000 Something very similar was fixed in master recently. Can you see if https://git.openembedded.org/openembedded-core/commit/?id=e3e5ace6e68d5fe68e4add301a44c1a1b8607411 fixes the issue? 85825 2 tobias.hagelborn 2019-12-13 09:20:43 +0000 I applied the suggested patch and re-run. I still get the same issue. I have also tried the suggested patch earlier in our current system and it did not solve the problems there. I should also update the procedure: (apply patch(es)) - export MACHINE=qemux86-64 - bitbake -c prepare_recipe_sysroot foo - export MACHINE=test64 - bitbake -c prepare_recipe_sysroot foo - export MACHINE=qemux86-64 - bitbake -c prepare_recipe_sysroot foo -- file conflict exceptions --- Exception: FileExistsError: [Errno 17] File exists: '/poky-master/build/tmp/sysroots-components/test64/bar-arch2/usr/include/bar/bar.h' -> '/poky-master/build/tmp/work/core2-64-poky-linux/foo/1.0-r0/recipe-sysroot/usr/include/bar/bar.h' 85850 3 tobias.hagelborn 2019-12-18 09:57:14 +0000 Adding and removing a dependency seems to do quite stable updates to the sysroot but this type of change in dependency does not seem to trigger delete/add in the same way. I just tried to verify that the signature of prepare_recipe_sysroot does change (Setting new dependency hash on bar1/bar2) with every invocation of bitbake with a new machine so that part seems ok from what I can see. ===== $> MACHINE=test64 bitbake -c prepare_recipe_sysroot foo ... MACHINE = "test64" ... NOTE: Tasks Summary: Attempted 252 tasks of which 252 didn't need to be rerun and all succeeded. > MACHINE=test64 bitbake-diffsigs -t foo prepare_recipe_sysroot NOTE: Starting bitbake server... runtaskdeps changed: [-bar-arch1/bar-arch1_1.0.bb:do_populate_sysroot, +bar-arch2/bar-arch2_1.0.bb:do_populate_sysroot, 'foo/foo_1.0.bb:do_fetch gcc/gcc-cross_9.2.bb:do_populate_sysroot gcc/gcc-runtime_9.2.bb:do_populate_sysroot glibc/glibc_2.30.bb:do_populate_sysroot'] bar-arch1/bar-arch1_1.0.bb:do_populate_sysroot with hash 3b0a0f9dcd26192034289c6fa0c361d025d23f5079ad3fa709c8abfd45e35145 changed to bar-arch2/bar-arch2_1.0.bb:do_populate_sysroot with hash ece77e7445ddc2fd6e77b0686754570e3c87b33b28e7e1d55eaa82128653cecd Dependency on task bar-arch2/bar-arch2_1.0.bb:do_populate_sysroot was added with hash ece77e7445ddc2fd6e77b0686754570e3c87b33b28e7e1d55eaa82128653cecd Dependency on task bar-arch1/bar-arch1_1.0.bb:do_populate_sysroot was removed with hash 3b0a0f9dcd26192034289c6fa0c361d025d23f5079ad3fa709c8abfd45e35145 storage/tobiasha/dists/poky-master/build> MACHINE=qemux86-64 bitbake-diffsigs -t foo prepare_recipe_sysroot NOTE: Starting bitbake server... runtaskdeps changed: [-bar-arch1/bar-arch1_1.0.bb:do_populate_sysroot, +bar-arch2/bar-arch2_1.0.bb:do_populate_sysroot, 'foo/foo_1.0.bb:do_fetch gcc/gcc-cross_9.2.bb:do_populate_sysroot gcc/gcc-runtime_9.2.bb:do_populate_sysroot glibc/glibc_2.30.bb:do_populate_sysroot'] bar-arch1/bar-arch1_1.0.bb:do_populate_sysroot with hash 3b0a0f9dcd26192034289c6fa0c361d025d23f5079ad3fa709c8abfd45e35145 changed to bar-arch2/bar-arch2_1.0.bb:do_populate_sysroot with hash ece77e7445ddc2fd6e77b0686754570e3c87b33b28e7e1d55eaa82128653cecd Dependency on task bar-arch2/bar-arch2_1.0.bb:do_populate_sysroot was added with hash ece77e7445ddc2fd6e77b0686754570e3c87b33b28e7e1d55eaa82128653cecd Dependency on task bar-arch1/bar-arch1_1.0.bb:do_populate_sysroot was removed with hash 3b0a0f9dcd26192034289c6fa0c361d025d23f5079ad3fa709c8abfd45e35145 $> MACHINE=qemux86-64 bitbake -c prepare_recipe_sysroot foo ... MACHINE = "qemux86-64" ... NOTE: Tasks Summary: Attempted 252 tasks of which 251 didn't need to be rerun and all succeeded. > MACHINE=test64 bitbake-diffsigs -t foo prepare_recipe_sysroot NOTE: Starting bitbake server... runtaskdeps changed: [-bar-arch2/bar-arch2_1.0.bb:do_populate_sysroot, +bar-arch1/bar-arch1_1.0.bb:do_populate_sysroot, 'foo/foo_1.0.bb:do_fetch gcc/gcc-cross_9.2.bb:do_populate_sysroot gcc/gcc-runtime_9.2.bb:do_populate_sysroot glibc/glibc_2.30.bb:do_populate_sysroot'] bar-arch2/bar-arch2_1.0.bb:do_populate_sysroot with hash ece77e7445ddc2fd6e77b0686754570e3c87b33b28e7e1d55eaa82128653cecd changed to bar-arch1/bar-arch1_1.0.bb:do_populate_sysroot with hash 3b0a0f9dcd26192034289c6fa0c361d025d23f5079ad3fa709c8abfd45e35145 Dependency on task bar-arch1/bar-arch1_1.0.bb:do_populate_sysroot was added with hash 3b0a0f9dcd26192034289c6fa0c361d025d23f5079ad3fa709c8abfd45e35145 Dependency on task bar-arch2/bar-arch2_1.0.bb:do_populate_sysroot was removed with hash ece77e7445ddc2fd6e77b0686754570e3c87b33b28e7e1d55eaa82128653cecd 85866 4 ross.burton 2019-12-19 16:24:14 +0000 Okay, thanks for the update. Reassigning to RP who understands this piece more than I do. 86864 5 richard.purdie 2020-03-30 15:51:56 +0000 Sorry for the delay in replying. I think the issue is that foo is MACHINE specific (as its dependency virtual/bar is machine specific) but foo isn't marked as such. I'd guess if you add PACKAGE_ARCH = "${MACHINE_ARCH}" to the foo recipe, it will work. That said, the system should recover better from this situation so it is a valid bug. 86865 6 richard.purdie 2020-03-30 17:05:45 +0000 There is code in staging.bbclass which is meant to handle this: # We want to remove anything which this task previously installed but is no longer a dependency However we hit: # We need to ensure not other task needs this dependency. We hold the sysroot # lock so we ca search the indexes to check and since the old bar-archX is referenced by index.do_install, it doesn't get removed. That explains why the code isn't helping. How to fix it... 86867 7 tobias.hagelborn 2020-03-30 17:54:43 +0000 Thanks for looking into this! Setting PACKAGE_ARCH = "${MACHINE_ARCH}" works for a specific recipe. I tried it in our environment but unfortunately the number of such cases exploded and it was not feasible as a full workaround. Thanks for the hint on where to look in staging.bbclass. I will probably not know how to correct this one but I can set some more debug print in those sections to better understand the issue. 87056 8 richard.purdie 2020-04-21 08:55:17 +0000 I figured out a fix for this and also wrote up a test case for oe-selftest (oe-selftest -r sysroot), the patches are in master-next. It turned out not to be too bad to implement in the end. 87060 9 tobias.hagelborn 2020-04-22 00:01:37 +0000 This is great news! I will try to verify this against our build. 87086 10 richard.purdie 2020-04-26 06:02:48 +0000 http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=f135cb36596d9cc8996277f49b296b9f73b76b73 http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=3e9cdd878887d0e59754447f313c39d142b71971 87102 11 tobias.hagelborn 2020-04-29 05:43:34 +0000 Thanks for the fixes provided! Unfortunately we seem to have some added complexity in our setup because we still see errors that appear to be similar to the original. File: '/home/tobiasha/dists/cfp-master/meta/classes/staging.bbclass', lineno: 152, function: staging_copyfile 0148: os.symlink(linkto, dest) 0149: #bb.warn(c) 0150: else: 0151: try: *** 0152: os.link(c, dest) 0153: except OSError as err: 0154: if err.errno == errno.EXDEV: 0155: bb.utils.copyfile(c, dest) 0156: else: Exception: FileExistsError: [Errno 17] File exists: '<PATH>/build/tmp/sysroots-components/<ARCH>/<RECIPE>/sysroot-providers/virtual_video_driver' -> '<PATH>/ <RECIPE>/8.0.0-r0/recipe-sysroot/sysroot-providers/virtual_video_driver' There is also a new one I have not sen before: File: '<PATH>/meta/classes/staging.bbclass', lineno: 447, function: extend_recipe_sysroot 0443: bb.note("Task %s no longer depends on %s, removing from sysroot" % (mytaskname, l)) 0444: lnk = os.readlink(fl) 0445: sstate_clean_manifest(depdir + "/" + lnk, d, workdir) 0446: os.unlink(fl) *** 0447: os.unlink(fl + ".complete") 0448: 0449: msg_exists = [] 0450: msg_adding = [] 0451: Exception: FileNotFoundError: [Errno 2] No such file or directory: '<PATH>/build/tmp/work/cortexa9hf-neon-poky- linux-gnueabi/<foo>/1.1.0-r0/recipe-sysroot-native/installeddeps/<bar>.complete' It is too early to say if I can come up with a repeater test like a qa test for these issies. 4601 2019-12-12 11:42:56 +0000 2019-12-12 11:42:56 +0000 Patch to apply on poky/meta layer 0001-Virtual-providers-and-recipie-sysroot-with-multiple-.patch text/plain 3355 tobias.hagelborn RnJvbSAzNDAyMjBkOWZhZDFjZWU4Y2M4ODE4NDRlY2Q0MWNhN2I0ZjdhMjhiIE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBUb2JpYXMgSGFnZWxib3JuIDx0b2JpYXNoYUBheGlzLmNvbT4K RGF0ZTogV2VkLCAxMSBEZWMgMjAxOSAxMDoyMDoxOSArMDEwMApTdWJqZWN0OiBbUEFUQ0hdIFZp cnR1YWwgcHJvdmlkZXJzIGFuZCByZWNpcGllIHN5c3Jvb3Qgd2l0aCBtdWx0aXBsZSBwcm9kdWN0 cy4KCkZhaWxpbmcgcHJlcGFyaW5nIHJlY2lwZSBzeXNyb290IChmaWxlIGNvbmZsaWN0cykgd2hl biBzd2l0Y2hpbmcKdmlydHVhbCBwcm92aWRlcnMgYnkgc3dpdGNoaW5nIE1BQ0hJTkUgaW4gdGhl IHNhbWUgYnVpbGR0cmVlLgoKUHJldmVudHMgYnVpbGRpbmcgbXVsdGlwbGUgbWFjaGluZXMgaW4g b25lIGJ1aWxkLXRyZWUgaWYKdmlydHVhbCBwcm92aWRlcnMgZG8gc3dpdGNoIGluLWJldHdlZW4g cHJvZHVjdHMuCgpSZWZlcmVuY2U6IGh0dHBzOi8vYnVnemlsbGEueW9jdG9wcm9qZWN0Lm9yZy9z aG93X2J1Zy5jZ2k/aWQ9MTIxODAKQ2hhbmdlLUlkOiBJYTc5YTYxYjFiNWEyODcxNWI0NDQ1ZjI4 M2RhZjY4MDVlN2FjOWRmNQotLS0KIG1ldGEvY29uZi9tYWNoaW5lL3FlbXV4ODYtNjQuY29uZiAg ICAgICAgICAgICAgIHwgIDEgKwogbWV0YS9jb25mL21hY2hpbmUvdGVzdDY0LmNvbmYgICAgICAg ICAgICAgICAgICAgfCAgNyArKysrKysrCiBtZXRhL3JlY2lwZXMtc3VwcG9ydC9iYXItYXJjaDEv YmFyLWFyY2gxXzEuMC5iYiB8IDEwICsrKysrKysrKysKIG1ldGEvcmVjaXBlcy1zdXBwb3J0L2Jh ci1hcmNoMi9iYXItYXJjaDJfMS4wLmJiIHwgMTAgKysrKysrKysrKwogbWV0YS9yZWNpcGVzLXN1 cHBvcnQvZm9vL2Zvb18xLjAuYmIgICAgICAgICAgICAgfCAgMyArKysKIDUgZmlsZXMgY2hhbmdl ZCwgMzEgaW5zZXJ0aW9ucygrKQogY3JlYXRlIG1vZGUgMTAwNjQ0IG1ldGEvY29uZi9tYWNoaW5l L3Rlc3Q2NC5jb25mCiBjcmVhdGUgbW9kZSAxMDA2NDQgbWV0YS9yZWNpcGVzLXN1cHBvcnQvYmFy LWFyY2gxL2Jhci1hcmNoMV8xLjAuYmIKIGNyZWF0ZSBtb2RlIDEwMDY0NCBtZXRhL3JlY2lwZXMt c3VwcG9ydC9iYXItYXJjaDIvYmFyLWFyY2gyXzEuMC5iYgogY3JlYXRlIG1vZGUgMTAwNjQ0IG1l dGEvcmVjaXBlcy1zdXBwb3J0L2Zvby9mb29fMS4wLmJiCgpkaWZmIC0tZ2l0IGEvbWV0YS9jb25m L21hY2hpbmUvcWVtdXg4Ni02NC5jb25mIGIvbWV0YS9jb25mL21hY2hpbmUvcWVtdXg4Ni02NC5j b25mCmluZGV4IDY0OGNmMmZlOGYuLjFhM2UzYWM5MjEgMTAwNjQ0Ci0tLSBhL21ldGEvY29uZi9t YWNoaW5lL3FlbXV4ODYtNjQuY29uZgorKysgYi9tZXRhL2NvbmYvbWFjaGluZS9xZW11eDg2LTY0 LmNvbmYKQEAgLTYsNiArNiw3IEBAIFBSRUZFUlJFRF9QUk9WSURFUl92aXJ0dWFsL3hzZXJ2ZXIg Pz0gInhzZXJ2ZXIteG9yZyIKIFBSRUZFUlJFRF9QUk9WSURFUl92aXJ0dWFsL2xpYmdsID89ICJt ZXNhIgogUFJFRkVSUkVEX1BST1ZJREVSX3ZpcnR1YWwvbGliZ2xlczEgPz0gIm1lc2EiCiBQUkVG RVJSRURfUFJPVklERVJfdmlydHVhbC9saWJnbGVzMiA/PSAibWVzYSIKK1BSRUZFUlJFRF9QUk9W SURFUl92aXJ0dWFsL2JhciA/PSAiYmFyLWFyY2gxIgogCiByZXF1aXJlIGNvbmYvbWFjaGluZS9p bmNsdWRlL3FlbXUuaW5jCiBERUZBVUxUVFVORSA/PSAiY29yZTItNjQiCmRpZmYgLS1naXQgYS9t ZXRhL2NvbmYvbWFjaGluZS90ZXN0NjQuY29uZiBiL21ldGEvY29uZi9tYWNoaW5lL3Rlc3Q2NC5j b25mCm5ldyBmaWxlIG1vZGUgMTAwNjQ0CmluZGV4IDAwMDAwMDAwMDAuLjgxOTM3MzRhNzAKLS0t IC9kZXYvbnVsbAorKysgYi9tZXRhL2NvbmYvbWFjaGluZS90ZXN0NjQuY29uZgpAQCAtMCwwICsx LDcgQEAKKyNAVFlQRTogTWFjaGluZQorI0BOQU1FOiBUUllPVVQgeDg2LTY0IG1hY2hpbmUKKyNA REVTQ1JJUFRJT046IE1hY2hpbmUgY29uZmlndXJhdGlvbiBmb3IgcnVubmluZyBhbiB4ODYtNjQg c3lzdGVtIG9uIFFFTVUKKworcmVxdWlyZSBjb25mL21hY2hpbmUvcWVtdXg4Ni02NC5jb25mCisK K1BSRUZFUlJFRF9QUk9WSURFUl92aXJ0dWFsL2JhciA9ICJiYXItYXJjaDIiCmRpZmYgLS1naXQg YS9tZXRhL3JlY2lwZXMtc3VwcG9ydC9iYXItYXJjaDEvYmFyLWFyY2gxXzEuMC5iYiBiL21ldGEv cmVjaXBlcy1zdXBwb3J0L2Jhci1hcmNoMS9iYXItYXJjaDFfMS4wLmJiCm5ldyBmaWxlIG1vZGUg MTAwNjQ0CmluZGV4IDAwMDAwMDAwMDAuLjVjYWVjYzM5YjMKLS0tIC9kZXYvbnVsbAorKysgYi9t ZXRhL3JlY2lwZXMtc3VwcG9ydC9iYXItYXJjaDEvYmFyLWFyY2gxXzEuMC5iYgpAQCAtMCwwICsx LDEwIEBACitMSUNFTlNFID0gIkNMT1NFRCIKKworUFJPVklERVMgPSAidmlydHVhbC9iYXIiCitE RVBFTkRTPSIiCitQQUNLQUdFX0FSQ0ggPSAiJHtNQUNISU5FX0FSQ0h9IgorCitkb19pbnN0YWxs KCkgeworCWluc3RhbGwgLWQgJHtEfSR7aW5jbHVkZWRpcn0vYmFyCisJZWNobyAiIyBiYXIgMSIg PiAke0R9JHtpbmNsdWRlZGlyfS9iYXIvYmFyLmgKK30KXCBObyBuZXdsaW5lIGF0IGVuZCBvZiBm aWxlCmRpZmYgLS1naXQgYS9tZXRhL3JlY2lwZXMtc3VwcG9ydC9iYXItYXJjaDIvYmFyLWFyY2gy XzEuMC5iYiBiL21ldGEvcmVjaXBlcy1zdXBwb3J0L2Jhci1hcmNoMi9iYXItYXJjaDJfMS4wLmJi Cm5ldyBmaWxlIG1vZGUgMTAwNjQ0CmluZGV4IDAwMDAwMDAwMDAuLjQ1M2ZmZmU2ZGYKLS0tIC9k ZXYvbnVsbAorKysgYi9tZXRhL3JlY2lwZXMtc3VwcG9ydC9iYXItYXJjaDIvYmFyLWFyY2gyXzEu MC5iYgpAQCAtMCwwICsxLDEwIEBACitMSUNFTlNFID0gIkNMT1NFRCIKKworUFJPVklERVMgPSAi dmlydHVhbC9iYXIiCitERVBFTkRTPSIiCitQQUNLQUdFX0FSQ0ggPSAiJHtNQUNISU5FX0FSQ0h9 IgorCitkb19pbnN0YWxsKCkgeworCWluc3RhbGwgLWQgJHtEfSR7aW5jbHVkZWRpcn0vYmFyCisJ ZWNobyAiIyBiYXIgMiIgPiAke0R9JHtpbmNsdWRlZGlyfS9iYXIvYmFyLmgKK30KXCBObyBuZXds aW5lIGF0IGVuZCBvZiBmaWxlCmRpZmYgLS1naXQgYS9tZXRhL3JlY2lwZXMtc3VwcG9ydC9mb28v Zm9vXzEuMC5iYiBiL21ldGEvcmVjaXBlcy1zdXBwb3J0L2Zvby9mb29fMS4wLmJiCm5ldyBmaWxl IG1vZGUgMTAwNjQ0CmluZGV4IDAwMDAwMDAwMDAuLmQ5NzBhNGZmNGMKLS0tIC9kZXYvbnVsbAor KysgYi9tZXRhL3JlY2lwZXMtc3VwcG9ydC9mb28vZm9vXzEuMC5iYgpAQCAtMCwwICsxLDMgQEAK K1NVTU1BUlkgPSAiVmlydHVhbCBwcm92aWRlciBzeXNyb290IGV2YWx1YXRpb24iCitMSUNFTlNF ID0gIkNMT1NFRCIKK0RFUEVORFMgPSAidmlydHVhbC9iYXIiCi0tIAoyLjExLjAKCg==