14992 2022-12-21 02:40:19 +0000 layerindex: update.py fails to determine layer dependencies when collections contains "core <...>" 2023-10-07 17:56:10 +0000 1 1 6 Yocto Project Subprojects Layer Index Layer Index 4.2 x86 Multiple RESOLVED FIXED Medium+ normal 4.3 M4 1 tim.orling tim.orling liezhi.yang randy.macleod tgamblin apoorvsangal --- --- Don't know oldest_to_newest 94560 0 tim.orling 2022-12-21 02:40:19 +0000 Many/Most layers set BBFILES_COLLECTIONS += "<layernick>". This results in the "collections" variable in LayerBranch being set to "core <layernick>". This results in a failure to determine LAYERDEPENDS [1] (or LAYERRECOMMENDS... etc), for example [2]: ERROR: Issues found on branch master: meta-sourcery: Failed to add since LAYERDEPENDS is not satisfied meta-aarch64: Failed to add since LAYERDEPENDS is not satisfied meta-openstack: Failed to add since LAYERDEPENDS is not satisfied meta-openstack-compute-deploy: Failed to add since LAYERDEPENDS is not satisfied meta-openstack-controller-deploy: Failed to add since LAYERDEPENDS is not satisfied meta-openstack-qemu: Failed to add since LAYERDEPENDS is not satisfied meta-sdr: Failed to add since LAYERDEPENDS is not satisfied meta-cgl: Failed to add since LAYERDEPENDS is not satisfied <snip> This could probably be "fixed" with a proper manage.py shell script, but that is a bandage and not a robust fix. [1] https://git.yoctoproject.org/layerindex-web/tree/layerindex/update.py#n519 [2] https://layers.openembedded.org/layerindex/updates/35825/ 94561 1 tim.orling 2022-12-21 02:50:52 +0000 As a concrete example, 'meta-games' added 'meta-qt5' as a layer dependency. Since 'meta-qt5' had "core qt5-layer" in LayerBranch.collections, this caused a failure [1]: WARNING: meta-games: LAYERDEPENDS: OrderedDict([('qt5-layer', [])]) LAYERRECOMMENDS: OrderedDict() WARNING: Known collections on branch master: {('core meta-amateurradio', None), ('aspnet', None), ('rubygems', '1'), ('raspberrypi', None), ('phytec', '2'), ('ivibsp', None), ('rslsync-layer', None), ('minnow', None), ('core scanners-layer', None), ('intel-iot-mw', None), ('core efi-secure-boot', None), ('yoctobsp', '4'), ('core meta-arago-distro', None), ('core updatehub-ti', None), ('core meta-96boards', None) <snip> ERROR: Issues found on branch master: meta-games: Failed to add since LAYERDEPENDS is not satisfied Once the "Layer Collection" value of 'meta-qt5: master" LayerBranch was manually edited in the admin interface to be only 'qt5-layer' ("core " removed), the update could proceed [2]. [1] https://layers.openembedded.org/layerindex/updates/35826/ [2] https://layers.openembedded.org/layerindex/updates/35827/ 94562 2 liezhi.yang 2022-12-21 03:27:56 +0000 Hi Tim, I think that it has been fixed by: ==== commit edb574ae7e076cb2f6b239ac7fc67073256bf23c Author: Robert Yang <liezhi.yang@windriver.com> Date: Fri Dec 16 08:42:51 2022 -0800 utils.py: Use get_layer_var() to get BBFILE_COLLECTIONS ==== You've already merged the patch, but the database was wrong, there should be only one BBFLIE_COLLECTIONS for each layerbranch, if you can find multiple ones in the admin UI, then I think that you can try to fix the issue by: $ ./update.py -b master --fullreload 94575 3 randy.macleod 2022-12-22 15:41:09 +0000 The YP layerindex DB still needs to be cleaned up. Robert, we need someone to take time to maintain the DB, hopefully we can get someone signed up early in the new year. 96076 4 randy.macleod 2023-07-26 21:22:02 +0000 Bulk move of 64 bugs to 4.3 M3 after a quick review. If a bug is actually fixed, please add a commit link and resolve it. -- Randy for YP bug team. 96492 5 tgamblin 2023-09-14 16:53:30 +0000 Moved to M4. 96659 6 4979 tim.orling 2023-10-07 17:52:18 +0000 Created attachment 4979 fixup_collections.py Doing a fullreload on all layers is a valid option, but a bit of a heavy hammer. Instead we can use a script to update the database. New layers added after: https://git.yoctoproject.org/layerindex-web/commit/layerindex?id=d8ab1beebeb5b49693b522d8618393620495bd74 did not have this problem. The attached fixup_collections.py script is based on https://git.yoctoproject.org/layerindex-web/tree/layerindex/tools/fixup_duplicates.py 96660 7 tim.orling 2023-10-07 17:56:10 +0000 Fixed by running the fixup_collections.py script on the layers.openembedded.org instance on Sept 30, 2023. The script was copied into and run from a container used for this purpose. 4979 2023-10-07 17:52:18 +0000 2023-10-07 17:52:18 +0000 fixup_collections.py fixup_collections.py text/x-python-script 2289 tim.orling IyEvdXNyL2Jpbi9lbnYgcHl0aG9uMwoKIyBGaXggY29sbGVjdGlvbnMgdGhhdCBpbmNsdWRlICJj b3JlICIKIwojIENvcHlyaWdodCAoQykgMjAxNyBJbnRlbCBDb3Jwb3JhdGlvbgojIEF1dGhvcjog UGF1bCBFZ2dsZXRvbiA8cGF1bC5lZ2dsZXRvbkBsaW51eC5pbnRlbC5jb20+CiMgQ29weXJpZ2h0 IChDKSAyMDIzIEtvbnN1bGtvIEdyb3VwCiMgQXV0aG9yOiBUaW0gT3JsaW5nIDx0aW0ub3JsaW5n QGtvbnN1bGtvLmNvbT4KIwojIExpY2Vuc2VkIHVuZGVyIHRoZSBNSVQgbGljZW5zZSwgc2VlIENP UFlJTkcuTUlUIGZvciBkZXRhaWxzCiMKIyBTUERYLUxpY2Vuc2UtSWRlbnRpZmllcjogTUlUCgoj IS91c3IvYmluL2VudiBweXRob24zCgppbXBvcnQgc3lzCmltcG9ydCBvcwoKc3lzLnBhdGguaW5z ZXJ0KDAsIG9zLnBhdGgucmVhbHBhdGgoJy9vcHQvbGF5ZXJpbmRleCcpKQoKaW1wb3J0IG9wdHBh cnNlCmZyb20gbGF5ZXJpbmRleCBpbXBvcnQgdXRpbHMKaW1wb3J0IGxvZ2dpbmcKCmNsYXNzIERy eVJ1blJvbGxiYWNrRXhjZXB0aW9uKEV4Y2VwdGlvbik6CiAgICBwYXNzCgpsb2dnZXIgPSB1dGls cy5sb2dnZXJfY3JlYXRlKCdMYXllckluZGV4Rml4dXAnKQoKCgpkZWYgbWFpbigpOgogICAgcGFy c2VyID0gb3B0cGFyc2UuT3B0aW9uUGFyc2VyKAogICAgICAgIHVzYWdlID0gIiIiCiAgICAlcHJv ZyBbb3B0aW9ucyIiIikKCiAgICBwYXJzZXIuYWRkX29wdGlvbigiLW4iLCAiLS1kcnktcnVuIiwK ICAgICAgICAgICAgaGVscCA9ICJEb24ndCB3cml0ZSBhbnkgZGF0YSBiYWNrIHRvIHRoZSBkYXRh YmFzZSIsCiAgICAgICAgICAgIGFjdGlvbj0ic3RvcmVfdHJ1ZSIsIGRlc3Q9ImRyeXJ1biIpCiAg ICBwYXJzZXIuYWRkX29wdGlvbigiLWQiLCAiLS1kZWJ1ZyIsCiAgICAgICAgICAgIGhlbHAgPSAi RW5hYmxlIGRlYnVnIG91dHB1dCIsCiAgICAgICAgICAgIGFjdGlvbj0ic3RvcmVfY29uc3QiLCBj b25zdD1sb2dnaW5nLkRFQlVHLCBkZXN0PSJsb2dsZXZlbCIsIGRlZmF1bHQ9bG9nZ2luZy5JTkZP KQogICAgcGFyc2VyLmFkZF9vcHRpb24oIi1xIiwgIi0tcXVpZXQiLAogICAgICAgICAgICBoZWxw ID0gIkhpZGUgYWxsIG91dHB1dCBleGNlcHQgZXJyb3IgbWVzc2FnZXMiLAogICAgICAgICAgICBh Y3Rpb249InN0b3JlX2NvbnN0IiwgY29uc3Q9bG9nZ2luZy5FUlJPUiwgZGVzdD0ibG9nbGV2ZWwi KQoKICAgIG9wdGlvbnMsIGFyZ3MgPSBwYXJzZXIucGFyc2VfYXJncyhzeXMuYXJndikKCiAgICB1 dGlscy5zZXR1cF9kamFuZ28oKQogICAgaW1wb3J0IHNldHRpbmdzCiAgICBmcm9tIGxheWVyaW5k ZXgubW9kZWxzIGltcG9ydCBMYXllckJyYW5jaAogICAgZnJvbSBkamFuZ28uZGIgaW1wb3J0IHRy YW5zYWN0aW9uCgogICAgbG9nZ2VyLnNldExldmVsKG9wdGlvbnMubG9nbGV2ZWwpCgogICAgdHJ5 OgogICAgICAgIHdpdGggdHJhbnNhY3Rpb24uYXRvbWljKCk6CiAgICAgICAgICAgIGZvciBsYXll cmJyYW5jaCBpbiBMYXllckJyYW5jaC5vYmplY3RzLmZpbHRlcihjb2xsZWN0aW9uX19zdGFydHN3 aXRoPSdjb3JlICcpOgogICAgICAgICAgICAgICAgcHJpbnQoJ1N0cmlwcGluZyBzdXBlcmZsdW91 cyBjb2xsZWN0aW9uICJjb3JlICIgZnJvbSAlczolcycgJSAobGF5ZXJicmFuY2gubGF5ZXIsIGxh eWVyYnJhbmNoLmJyYW5jaCkpCiAgICAgICAgICAgICAgICBjb2xsZWN0aW9ucyA9IGxheWVyYnJh bmNoLmNvbGxlY3Rpb24uc3BsaXQoJyAnKQogICAgICAgICAgICAgICAgaWYgY29sbGVjdGlvbnNb MF09PSdjb3JlJyBhbmQgbGVuKGNvbGxlY3Rpb25zKSA+IDE6CiAgICAgICAgICAgICAgICAgICAg b3JpZ2luYWwgPSBsYXllcmJyYW5jaC5jb2xsZWN0aW9uCiAgICAgICAgICAgICAgICAgICAgcHJp bnQoJyAgT3JpZ2luYWw6ICVzJyAlIChvcmlnaW5hbCkpCiAgICAgICAgICAgICAgICAgICAgZml4 dXAgPSBjb2xsZWN0aW9uc1sxXQogICAgICAgICAgICAgICAgICAgIHByaW50KCcgIEZpbmFsOiAl cycgJSAoIGZpeHVwICkpCiAgICAgICAgICAgICAgICAgICAgTGF5ZXJCcmFuY2gub2JqZWN0cy5m aWx0ZXIoaWQ9bGF5ZXJicmFuY2guaWQpLnVwZGF0ZShjb2xsZWN0aW9uPWZpeHVwKQoKICAgICAg ICAgICAgaWYgb3B0aW9ucy5kcnlydW46CiAgICAgICAgICAgICAgICByYWlzZSBEcnlSdW5Sb2xs YmFja0V4Y2VwdGlvbigpCiAgICBleGNlcHQgRHJ5UnVuUm9sbGJhY2tFeGNlcHRpb246CiAgICAg ICAgcGFzcwoKICAgIHN5cy5leGl0KDApCgoKaWYgX19uYW1lX18gPT0gIl9fbWFpbl9fIjoKICAg IG1haW4oKQoK