| Summary: | mutlilib results in, if package has incompatible license, pulled in rootfs or if package has compatible license, blacklisted from RDEPENDS/rootfs | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] OE-Core | Reporter: | Quentin Schulz <foss+yocto> | ||||||||
| Component: | oe-core other | Assignee: | Unassigned <unassigned> | ||||||||
| Status: | RESOLVED FIXED | QA Contact: | |||||||||
| Severity: | critical | ||||||||||
| Priority: | Medium+ | CC: | meta.mr.watcher, meta.watcher, mhalstead, randy.macleod, raveendrapadala, richard.purdie | ||||||||
| Version: | unspecified | ||||||||||
| Target Milestone: | 4.99 | ||||||||||
| Hardware: | All | ||||||||||
| OS: | Multiple | ||||||||||
| Whiteboard: | possible backport 3.1 | ||||||||||
| OS type for building Yocto: | --- | Type of Regression: | --- | ||||||||
| Verified: | Documentation change: | Don't know | |||||||||
| Attachments: |
|
||||||||||
And before I forget, if we work on ditching LICENSE_EXCLUSION and move the blacklisting logic to package.bbclass but I think (not tested) there is also some work to do on https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/classes/base.bbclass#n577 Because if the we still create the skipped_pkgs and unskipped_pkgs lists the same way as today, there would be a problem in the following case: foo_0.1.bb: LICENSE = "GPLv3" LICENSE_foo-bar = "GPLv2" PACKAGES += "${PN}-bar" and INCOMPATIBLE_LICENSE = "GPLv3" in local.conf, the recipe will be skipped while it could actually be built because at least one package isn't GPLv3. Same for https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/classes/base.bbclass#n587 though not critical (only a warning): foo_0.1.bb: LICENSE = "GPLv3" LICENSE_foo-bar = "CustomLic" LICENSE_lib32-foo-bar = "GPLv3" PACKAGES += "foo-bar" Results in a warning stating lib32-foo-bar has a LICENSE CustomLic which is not in LICENSE. I think something like checking all LICENSE_* against LICENSE without having knowledge of PACKAGES would make sense? It's a recipe issue so it is actually wrong even if the package is not created in the current build configuration to have some license in LICENSE_<pkg> which is not in LICENSE. FWIW the package_rpm issue is caused by SECTION="" and a non-empty value avoids that. Created attachment 4667 [details]
Patch to attempt a fix part 1
Created attachment 4668 [details]
Patch to attempt a fix part 2
It looks like there is a race between the anonymous python base.bbclass and the multilib.bbclass anonymous python code. The multilib code needs to run *after* key expansion takes place in the data store. The easiest fix I could come up with was to add a new event to get the ordering right. Could you test to see if this helps please? Hi Richard, I tested on poky 5d47cdf448b6cff5bb7cc5b0ba0426b8235ec478. I needed your two diffs and to revert d492a70c72063c0b8ee3c8ed9f1d24556c7159c9 "base.bbclass: considering multilib when setting LICENSE_EXCLUSION" otherwise we have LICENSE_EXCLUSION-lib32-lib32-pkg for multilib packages. I have tested non multilib and multilib variants of the example, it works in that case. However, it breaks other packages which are inheriting update-alternatives bbclass. If you take the same example recipe in the bug description and add inherit update-alternatives you'll have the errors available in the attached log named li32-foo-debug.log (-D enabled). I checked as much as I could in this bbclass but I couldn't find what's inducing this. Let me know if there's anything I can do. Created attachment 4669 [details]
Bitbake log (with -D) for lib32-foo which inherits update-alternatives
(In reply to comment #7) > Created attachment 4669 [details] > Bitbake log (with -D) for lib32-foo which inherits update-alternatives These errors look "correct" in that for example, if you exclude GPLv3, gdbm and sed are excluded, that stops perl building and without perl, you don't get update-alternatives. I'm not sure that is a bug so much as an invalid configuration? There is an issue. If you add meta-gplv2 to the mix, the fix becomes clearer:
diff --git a/meta/recipes-devtools/perl/perl_5.30.1.bb b/meta/recipes-devtools/perl/perl_5.30.1.bb
index 9d747aa5b42..5f26738df56 100644
--- a/meta/recipes-devtools/perl/perl_5.30.1.bb
+++ b/meta/recipes-devtools/perl/perl_5.30.1.bb
@@ -332,7 +332,7 @@ python split_perl_packages () {
python() {
if d.getVar('CLASSOVERRIDE') == "class-target":
- d.setVar("PACKAGES_DYNAMIC", "^perl-module-.*(?<!native)$")
+ d.setVar("PACKAGES_DYNAMIC", "^${MLPREFIX}perl-module-.*(?<!native)$")
elif d.getVar('CLASSOVERRIDE') == "class-native":
d.setVar("PACKAGES_DYNAMIC", "^perl-module-.*-native$")
elif d.getVar('CLASSOVERRIDE') == "class-nativesdk":
Does that address the issues seen?
You were right, I completely forgot most errors were expected because I was missing meta-gplv2 layer. Also right that perl was missing the suggested patch. After building lib32-foo with inherit update-alternatives (and adding meta-gplv2), there are a few WARNING messages though: WARNING: lib32-glibc-2.31+gitAUTOINC+71f2b249a2-r0 do_package: QA Issue: lib32-glibc package lib32-glibc - suspicious values 'ldconfig' in RRECOMMENDS [multilib] WARNING: lib32-python3-3.8.2-r0 do_package: QA Issue: lib32-python3 package lib32-python3-tests - suspicious values 'bash' in RDEPENDS [multilib] WARNING: lib32-python3-3.8.2-r0 do_package: QA Issue: lib32-python3 package lib32-python3-crypt - suspicious values 'openssl ca-certificates' in RRECOMMENDS [multilib] WARNING: lib32-python3-3.8.2-r0 do_package: QA Issue: lib32-python3 package lib32-python3-modules - suspicious values 'python3-misc' in RDEPENDS [multilib] WARNING: lib32-python3-3.8.2-r0 do_package: QA Issue: lib32-python3 package lib32-python3-venv - suspicious values 'python3-pyvenv' in RPROVIDES [multilib] WARNING: lib32-python3-3.8.2-r0 do_package: QA Issue: lib32-python3 package lib32-python3-dev - suspicious values 'bash-dev python3-misc-dev' in RRECOMMENDS [multilib] Good progress :) Team, openssl recipe provides openssl binary and two more libraries (libssl and libcrypto) Other modules linking to these libraries are not able to do it because the current recipe doesn't have "PROVIDES=libssl libcrypto". Details of libssl and libcrypto: --> libcrypto is the portion of OpenSSL for performing general-purpose cryptography, which can be used without libssl. --> libssl is the portion of OpenSSL which supports TLS, and depends on libcrypto. So requesting the team to update the same. infact yocto documentation has also cited the same and i quote A list of aliases by which a particular recipe can be known. By default, a recipe's own PN is implicitly already in its PROVIDES list and therefore does not need to mention that it provides itself. If a recipe uses PROVIDES, the additional aliases are synonyms for the recipe and can be useful for satisfying dependencies of other recipes during the build as specified by DEPENDS. Team, OpenSSL bitbake provides openssl binary and two more libraries libcrypto and libssl but the bitbake is not mentioning the same in PROVIDES=libssl libcrypto Details of libssl and libcrypto: --> libssl is the portion of OpenSSL which supports TLS, and depends on libcrypto. --> libcrypto is the portion of OpenSSL for performing general-purpose cryptography, which can be used without libssl. So requesting the team to do the needfull so that components depending on libcrypto and libssl libraries will link successfully. The same was cited in the yocto documentation. I am citing the same for reference: A list of aliases by which a particular recipe can be known. By default, a recipe's own PN is implicitly already in its PROVIDES list and therefore does not need to mention that it provides itself. If a recipe uses PROVIDES, the additional aliases are synonyms for the recipe and can be useful for satisfying dependencies of other recipes during the build as specified by DEPENDS. Reverting back to the correct summary. Somehow this was updated with information unrelated to the bug. Perhaps a faulty automated tool. Reverting back to the correct summary. Somehow this was updated with information unrelated to the bug. Perhaps a faulty automated tool. This was fixed in master with the changes related to https://git.yoctoproject.org/cgit.cgi/poky/commit/meta/classes/base.bbclass?h=master&id=edbf8d3999126e3c271a47dada54d1724f59f573 |
How to reproduce: poky 5d47cdf448b6cff5bb7cc5b0ba0426b8235ec478 Create a new machine in meta/conf/machine/qemuarm64-multilib.conf: ``` require conf/multilib.conf MULTILIBS = "multilib:lib32" DEFAULTTUNE_virtclass-multilib-lib32 = "armv7athf-neon-vfpv4" MACHINEOVERRIDES =. "qemuarm64:" require qemuarm64.conf ``` Create dummy recipe in meta/recipes-bsp/acpid/foo_0.1.bb: ``` SUMMARY = "" HOMEPAGE = "" LICENSE = "GPLv2 & GPLv3" SECTION = "" DEPENDS = "" LICENSE_${PN} = "GPLv2" LICENSE_${PN}-bar = "GPLv2" LICENSE_foo-baz = "GPLv3" do_install() { echo "foo" > ${D}/foo echo "foo-bar" > ${D}/foo-bar echo "foo-baz" > ${D}/foo-baz } PACKAGES += "foo-bar ${PN}-baz" FILES_${PN} = "/foo" FILES_${PN}-bar = "/foo-bar" FILES_${PN}-baz = "/foo-baz" ``` $ source oe-init-build-env $ echo 'INCOMPATIBLE_LICENSE = "GPLv3"' >> conf/local.conf $ echo 'PACKAGE_CLASSES = "package_ipk"' >> conf/local.conf #Only needed because I'm too lazy to figure out why rpm fails with the example recipe $ MACHINE=qemuarm64-multilib bitbake lib32-foo -D Interesting snippets: DEBUG: Skipping the package foo-bar at do_rootfs because of incompatible license(s): GPL-3.0 DEBUG: Skipping the package lib32-foo-baz at do_rootfs because of incompatible license(s): GPL-3.0 WARNING: lib32-foo-0.1-r0 do_package: QA Issue: Excluding lib32-foo-bar from packaging as it has incompatible license(s): GPL-3.0 [incompatible-license] So, Yocto *thinks* lib32-foo-bar shouldn't be packaged because it has GPL-3.0 license, which is incorrect (note that the license in pkgdata is correct though). Moreover, Yocto *thinks* lib32-foo-baz shouldn't be put in the rootfs but actually still packages it (present in /tmp/deploy/ipk) which is also incorrect. I don't know how critical it is since I don't know if there's an additional check before doing the rootfs from the deployed packages, but this could potentially results in GPLv3 packages in a rootfs which shouldn't contain any. So now, I've my finger on the code responsible of this but can't think of the proper way to fix it. The error is with LICENSE_EXCLUSION which is defined in base.bbclass but used in package.bbclass. Code snippet starts from: https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/classes/base.bbclass#n561 pkgs is a list of strings in PACKAGES. When there are strings in there using ${PN}, since multilib sets this value to lib32-${PN} before this code is executed, you get a mix of PACKAGES with prepended lib32- and some without. From the example above, the resulting pkgs is: ['foo-bar','lib32-foo-baz','lib32-foo','lib32-foo-dev',...]. Line 565, there is a check for incompatible licenses on pkg value. We try to find an incompatible license for 'foo-bar' pkg but we only have LICENSE_lib32-foo-bar defined because PN is lib32-foo-bar. Thus, it defaults to LICENSE. We try to find an incompatible license for 'lib32-foo-baz' pkg but we only have LICENSE_foo-baz defined because PN wasn't used here. Thus, it defaults to LICENSE. So far, already two issues. Then, line 574, we set LICENSE_EXCLUSION for skipped package (both foo-bar and lib32-foo-baz because they both were "matched" against LICENSE and not their own). Here, in case of multilib packages, MLPREFIX is added to the name of the package (even though the debug message just above claims otherwise). Thus, we have: LICENSE_EXCLUSION-lib32-foo-bar and LICENSE_EXCLUSION-lib32-lib32-foo-baz set. Now, let's check in package.bbclass where this is used: https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/classes/package.bbclass#n1325 Here, PACKAGES have all strings prepended with lib32- when in multilib case. We iterated over the PACKAGES of the recipe and check for LICENSE_EXCLUSION-<pkg>. For lib32-foo-bar, we have one LICENSE_EXCLUSION-lib32-foo-bar (but that is incorrect since it should be GPLv2 and not incompatible with GPLv3 from local.conf). For lib32-foo-baz, we do not have any because it's LICENSE_EXCLUSION-lib32-lib32-foo-baz and not LICENSE_EXCLUSION-lib32-foo-baz even though there should be one because it's GPLv3. Outcome: lib32-foo-bar is not packaged and cannot be put into rootfs but should. lib32-foo-baz is packaged and can be put into rootfs but shouldn't. I was tempted to just add the MLPREFIX in base.bbclass when checking incompatible packages but that is a mistake. Many packages actually rely on this side-effect. e.g. https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/recipes-extended/hdparm/hdparm_9.58.bb Here you can see PACKAGES not using ${PN}, in that case wiper. If we were to build lib32-hdparm with the MLPREFIX added in base,bbclass or prepend all PACKAGES with lib32- in multilib_virtclass_handler before PACKAGES is evaluated in base.bbclass, LICENSE_wiper would still be LICENSE_wiper but we would try to match "wiper" pkg from base.bbclass with LICENSE_lib32-wiper which does not exist and resulting in LICENSE being used which is not what is expected with the current recipe. Now, we could check for LICENSE_lib32-wiper AND LICENSE_wiper but that is a mistake IMO since we could technically have two different licenses, one for wiper and one for lib32-wiper (some proprietary code or something, added for one variant, I don't know) and then instead of using the default LICENSE as expected for lib32-wiper, we would use LICENSE_wiper which is incorrect. Finally, I'm questioning the use of LICENSE_EXCLUSION. I think, we could technically get rid of it and do this whole licensing thing directly in package.bbclass. But considering that additional logic is needed in base.bbclass wrt licenses (completely excluding a recipe for example), this would mean duplicated code. This has critical severity because it breaks INCOMPATIBLE_LICENSE behaviour on multilib builds but feel free to lower it down because tmp/deploy/licenses/*/license.manifest contains appropriate license (that is, incompatible, and with the example, GPLv3) for the package that shouldn't have been in the image.