11427 2017-04-27 10:44:43 +0000 opkg: libresolv backend fails to find depenednecies on x86/core2-32 platforms (BAD_RECOMMENDS issue/docs/better debug output?) 2019-02-08 15:43:33 +0000 1 1 7 Build System, Metadata & Runtime OE-Core core 2.3 x86 Multiple RESOLVED FIXED Medium normal 4.99 1 holger alejandro.delcastillo alejandro.delcastillo meta.mr.watcher meta.watcher mikko.ylinen randy.macleod richard.purdie --- --- No (bug/feature does not impact docs) oldest_to_newest 72888 0 holger 2017-04-27 10:44:43 +0000 Trying to build for my "sysmocom-bsc-v2" machine (tuning for core2-32) will fail to build a rootfs on missing dependency. Using PACKAGECONFIG_pn-opkg... = "" to disable the libresolv backend is resolving the issue. Error output: NOTE: Installing the following packages: rauc dropbear task-core-boot opkg NOTE: /home/hfreyther/images/poky-resurrection/new/tmp/work/bsc_v2-poky-linux/image-rauc-rescue-initramfs/1.0-r0/recipe-sysroot-native/usr/bin/opkg --volatile-cache -f /home/hfreyther/images/poky-resurrection/new/tmp/work/bsc_v2-poky-linux/image-rauc-rescue-initramfs/1.0-r0/opkg.conf -t /home/hfreyther/images/poky-resurrection/new/tmp/work/bsc_v2-poky-linux/image-rauc-rescue-initramfs/1.0-r0/temp/ipktemp/ -o /home/hfreyther/images/poky-resurrection/new/tmp/work/bsc_v2-poky-linux/image-rauc-rescue-initramfs/1.0-r0/rootfs --force_postinstall --prefer-arch-to-version install rauc dropbear task-core-boot opkg ERROR: Unable to install packages. Command '/home/hfreyther/images/poky-resurrection/new/tmp/work/bsc_v2-poky-linux/image-rauc-rescue-initramfs/1.0-r0/recipe-sysroot-native/usr/bin/opkg --volatile-cache -f /home/hfreyther/images/poky-resurrection/new/tmp/work/bsc_v2-poky-linux/image-rauc-rescue-initramfs/1.0-r0/opkg.conf -t /home/hfreyther/images/poky-resurrection/new/tmp/work/bsc_v2-poky-linux/image-rauc-rescue-initramfs/1.0-r0/temp/ipktemp/ -o /home/hfreyther/images/poky-resurrection/new/tmp/work/bsc_v2-poky-linux/image-rauc-rescue-initramfs/1.0-r0/rootfs --force_postinstall --prefer-arch-to-version install rauc dropbear task-core-boot opkg' returned 1: Collected errors: * opkg_finalize_intercepts: Failed to open dir /home/hfreyther/images/poky-resurrection/new/tmp/work/bsc_v2-poky-linux/image-rauc-rescue-initramfs/1.0-r0/temp/ipktemp//opkg-vA5gmJ/opkg-intercept-P0Esvt: No such file or directory. * rm_r: Failed to open dir /home/hfreyther/images/poky-resurrection/new/tmp/work/bsc_v2-poky-linux/image-rauc-rescue-initramfs/1.0-r0/temp/ipktemp//opkg-vA5gmJ/opkg-intercept-P0Esvt: No such file or directory. * rm_r: Failed to open dir /home/hfreyther/images/poky-resurrection/new/tmp/work/bsc_v2-poky-linux/image-rauc-rescue-initramfs/1.0-r0/temp/ipktemp//opkg-vA5gmJ: No such file or directory. Solver encountered 1 problem(s): Problem 1/1: - nothing provides kbd needed by keymaps-1.0-r31.0.bsc_v2 Solution 1: - do not ask to install a package providing task-core-boot Package is available: $ cat opkg.conf arch all 1 arch any 6 arch noarch 11 arch x86 16 arch i586 21 arch core2-32 26 arch bsc_v2 31 src oe file:/home/hfreyther/images/poky-resurrection/new/tmp/deploy/ipk src oe-all file:/home/hfreyther/images/poky-resurrection/new/tmp/deploy/ipk/all src oe-core2-32 file:/home/hfreyther/images/poky-resurrection/new/tmp/deploy/ipk/core2-32 src oe-bsc_v2 file:/home/hfreyther/images/poky-resurrection/new/tmp/deploy/ipk/bsc_v2 $ cat rootfs/var/lib/opkg/lists/oe-core2-32 | egrep -C 18 "Package: kbd$" ... Package: kbd Version: 2.0.4-r0.0 Depends: libc6 (>= 2.25), update-alternatives-opkg Provides: console-tools Replaces: console-tools Conflicts: console-tools Section: base Architecture: core2-32 Maintainer: Poky <poky@yoctoproject.org> MD5Sum: ee695f031f2c524929318d0f0738e453 Size: 167626 Filename: kbd_2.0.4-r0.0_core2-32.ipk Source: http://cdn.kernel.org/pub/linux/utils/kbd/kbd-2.0.4.tar.xz file://run-ptest file://set-proper-path-of-resources.patch Description: Keytable files and keyboard utilities Keytable files and keyboard utilities. OE: kbd License: GPLv2+ Priority: optional Debug output of -V4 and kbd (and kbd is only in the core2-32 Packages file so not sure why there is dup) $ grep "kbd " temp/log.do_rootfs pkg_vec_insert_merge: Adding new pkg=kbd version=2.0.4 arch=core2-32. pkg_vec_insert_merge: kbd 2.0.4 arch=core2-32 vs. kbd 2.0.4 arch=core2-32. pkg_vec_insert_merge: Duplicate for pkg=kbd version=2.0.4 arch=core2-32. pkg_vec_insert_merge: Merging kbd 2.0.4 arch=core2-32, set_status=1. pkg2solvable: kbd-dev depends kbd = 2.0.4-r0.0 - nothing provides kbd needed by keymaps-1.0-r31.0.bsc_v2 72910 1 alejandro.delcastillo 2017-04-27 18:51:49 +0000 Nothing stands out as being clearly of...but I am suspecting the root cause may be a behavior change between internalsolv and libsolv: Provides dependencies that have version constrains have to be real packages. The libsolv behavior is more restrictive, but is the correct one by Debian policy [1]. Seems like opkg is failing when installing task-core-boot. A way to get more info would be to try to manually run the opkg install command on task-core-boot [2]. I would assume that will fail on keymaps. Then the idea would be to try to install keymaps manually, so we can get down the tree chain (and so on). [1] https://www.debian.org/doc/debian-policy/ch-relationships.html (Provides section) [2] /home/hfreyther/images/poky-resurrection/new/tmp/work/bsc_v2-poky-linux/image-rauc-rescue-initramfs/1.0-r0/recipe-sysroot-native/usr/bin/opkg --volatile-cache -f /home/hfreyther/images/poky-resurrection/new/tmp/work/bsc_v2-poky-linux/image-rauc-rescue-initramfs/1.0-r0/opkg.conf -t /home/hfreyther/images/poky-resurrection/new/tmp/work/bsc_v2-poky-linux/image-rauc-rescue-initramfs/1.0-r0/temp/ipktemp/ -o /home/hfreyther/images/poky-resurrection/new/tmp/work/bsc_v2-poky-linux/image-rauc-rescue-initramfs/1.0-r0/rootfs --force_postinstall --prefer-arch-to-version install task-core-boot 72911 2 holger 2017-04-27 18:53:40 +0000 so you would like to have "opkg -V4 install keymaps"? What are you looking for? 72912 3 alejandro.delcastillo 2017-04-27 18:59:35 +0000 yes, if you can go down the chain would be great. So, try to install task-core-boot first. If that fails because keymaps can't be installed, then try to install keymaps. If keymaps fail because X , then try X. I am basically trying to see what's the dependency that can't be satisified. 72913 4 alejandro.delcastillo 2017-04-27 18:59:54 +0000 yes, if you can go down the chain would be great. So, try to install task-core-boot first. If that fails because keymaps can't be installed, then try to install keymaps. If keymaps fail because X , then try X. I am basically trying to see what's the dependency that can't be satisified. 72914 5 holger 2017-04-27 19:00:44 +0000 Am I missing something but isn't it printed? task-core-boot -> keymaps -> kbd (kbd can't as it seems to have been eliminated, merged with itself?)? 72915 6 alejandro.delcastillo 2017-04-27 19:23:55 +0000 Nope, you are not missing anything. I mainly wanted to see at which node the dependency fails, but I guess it's clear it's on keymaps. Can you paste the control file of keymaps then? 72958 7 alejandro.delcastillo 2017-05-01 21:08:23 +0000 also, instructions on how to reproduce the problem can go a long way. If you could provide me with those, it will be simpler for me to root cause what's going on. 72959 8 holger 2017-05-01 21:12:19 +0000 Sorry. I didn't have the time to try the rootfs by hand. 1.) Get your pyro poky 2.) Get the BSP git clone git://git.sysmocom.de/poky/meta-sysmocom-bsp/ 3.) Use "sysmocom-bsc-v2" as MACHINE type 4.) build "image-rauc-rescue-initramfs" (there are other layers like meta-telephony but they should not be required here 72984 9 alejandro.delcastillo 2017-05-02 22:29:37 +0000 I was able to reproduce the problem using both dnf and opkg. After digging a bit more: adelcast@delcastillo2 ~/upstream/poky/meta-sysmocom-bsp $ grep -R BAD_RECOMMENDATIONS | grep kbd recipes-core/images/image-rauc-rescue-initramfs.bb:BAD_RECOMMENDATIONS_append = " busybox-syslog kbd kbd-consolefonts kbd-keymaps" removing the BAD_RECOMMENDATIONS fixes the issue. According to the yocto manual [1]: "It is important to realize that if you choose to not install packages using this variable and some other packages are dependent on them (i.e. listed in a recipe's RDEPENDS variable), the OpenEmbedded build system ignores your request and will install the packages to avoid dependency errors." which confused me a bit, since packagegroup-core-boot depends on keymaps, which depends on kbd. However, that seems to be a documentation/bug issue with OE, not with opkg, so I am closing this ticket as not a bug. Feel free to re-open if you experience something different that points to opkg directly. [1] http://www.yoctoproject.org/docs/2.3/ref-manual/ref-manual.html 72986 10 holger 2017-05-03 07:16:30 +0000 (In reply to comment #9) > I was able to reproduce the problem using both dnf and opkg. After digging a > bit more: > > adelcast@delcastillo2 ~/upstream/poky/meta-sysmocom-bsp $ grep -R > BAD_RECOMMENDATIONS | grep kbd > recipes-core/images/image-rauc-rescue-initramfs.bb: > BAD_RECOMMENDATIONS_append = " busybox-syslog kbd kbd-consolefonts > kbd-keymaps" Oh! How did you debug that? Maybe this is something for opkg to ease? E.g. I didn't notice anything related to BAD_RECOMMENDATIONS in the output. > "It is important to realize that if you choose to not install packages using > this variable and some other packages are dependent on them (i.e. listed in > a recipe's RDEPENDS variable), the OpenEmbedded build system ignores your > request and will install the packages to avoid dependency errors." > > which confused me a bit, since packagegroup-core-boot depends on keymaps, > which depends on kbd. However, that seems to be a documentation/bug issue > with OE, not with opkg, so I am closing this ticket as not a bug. Feel free > to re-open if you experience something different that points to opkg > directly. I disagree here. Te above says "ignores your request" but with libresolv it will fail. I don't mind it failing (it is probably an improvement) but it should lead to an error that is easier to understand? 73003 11 alejandro.delcastillo 2017-05-03 15:26:23 +0000 (In reply to comment #10) > (In reply to comment #9) > > I was able to reproduce the problem using both dnf and opkg. After digging a > > bit more: > > > > adelcast@delcastillo2 ~/upstream/poky/meta-sysmocom-bsp $ grep -R > > BAD_RECOMMENDATIONS | grep kbd > > recipes-core/images/image-rauc-rescue-initramfs.bb: > > BAD_RECOMMENDATIONS_append = " busybox-syslog kbd kbd-consolefonts > > kbd-keymaps" > > > Oh! How did you debug that? Maybe this is something for opkg to ease? E.g. I > didn't notice anything related to BAD_RECOMMENDATIONS in the output. The installation succeeded if I manually deleted the file that keep package status (/var/lib/opkg/status) and manually tried to install keymaps. Looking at that file, I noticed that kbd status was set to "deinstall hold not-installed", which is not a common status. Grepping on oe-core found that in oe-core at lib/oe/package_manager.py OE is manually modifying the status file in the case of BAD_RECOMMENDATIONS. Opkg (and dnf) are correctly identifying the conflict, so I don't think there is anything for opkg to do. As I mentioned, if this worked on the internal solver, then this probably means OE needs to change it's BAD_RECOMMENDATIONS code to have it worked as before. > > "It is important to realize that if you choose to not install packages using > > this variable and some other packages are dependent on them (i.e. listed in > > a recipe's RDEPENDS variable), the OpenEmbedded build system ignores your > > request and will install the packages to avoid dependency errors." > > > > which confused me a bit, since packagegroup-core-boot depends on keymaps, > > which depends on kbd. However, that seems to be a documentation/bug issue > > with OE, not with opkg, so I am closing this ticket as not a bug. Feel free > > to re-open if you experience something different that points to opkg > > directly. > > I disagree here. Te above says "ignores your request" but with libresolv it > will fail. I don't mind it failing (it is probably an improvement) but it > should lead to an error that is easier to understand? I do believe BAD_RECOMMENDS is not working as advertised, but that is not an opkg issue (it also fails with dnf), but an OE issue. I am not saying that the request should be ignored, what I am saying is that it shouldn't be tagged as an opkg issue but as an OE issue. 73036 12 richard.purdie 2017-05-04 10:35:39 +0000 Reopen against OE-Core as there is likely an issue here. 73079 13 mikko.ylinen 2017-05-05 11:06:43 +0000 (In reply to comment #12) > Reopen against OE-Core as there is likely an issue here. FWIW, we're seeing this too. A package is listed in BAD_RECOMMENDATIONS and when I add it in a packagegroup (the packagegroup RDEPENDS on that package), I get the same failure. 73435 14 alejandro.delcastillo 2017-05-22 22:49:29 +0000 I took a look at the code again, and looks to me like the current implementation for opkg is a bit hackish (it manually pins the bad recommends package to be deinstalled). Looking at dnf, it uses an --exclude flag to signal BAD_RECOMMENDS packages. I haven't tried this myself, but I am guessing those excludes will be ignored if there is a hard dependency. I would be open to implement an --excludes flag in opkg (or at least give it a shot, if libsolv provides a way to signal that a package should be excluded, unless it's a hard dependency). If that pans out, we could keep the solution that changes the status file directly for opkg internalsolver, but have the --exclude flag solution for libsolv. Ross, let me know your thoughts to see if I show move ahead and explore adding the --exclude option. 73693 15 ross.burton 2017-06-05 20:07:36 +0000 --exclude sounds like a much better solution than hacking the status file. Did you look at what dpkg/apt does? I believe you can do something like: $ apt-get install foo thing-that-foo-recommends- To install foo but not thing-that-foo-recommends (the trailing hyphen being the trick). 77319 16 alejandro.delcastillo 2017-10-03 22:08:21 +0000 Today, I finally decided to take another stab at this. Looking into opkg, I realized that it already have a -add-exclude flag that does exactly the same thing as dnf --excludes. However, neither of those flags solve the BAD_RECOMMENDATIONS issue as they unconditionally exclude the listed packages. I ran a test where I included in an image package A and B, where A recommends C and B depends on C. I added a BAD_RECOMMENDATIONS=C. The expected behavior was to have package A, B and C installed (BAD_RECOMMENDATIONS would be ignored, since C depends on B). Instead, I got a solver "nothing provides C" error on both dnf and opkg. At this point I am not sure how to proceed forward as it looks like we may need a libsolv way to implement something similar to smart's ignore-recommends per-package flag. Maybe we can loop in the dnf mantainer to decide on the best way forward for both opkg and dnf, since this is a feature request for both package managers? any other ideas? 79296 17 alejandro.delcastillo 2018-02-01 17:11:50 +0000 I prototyped a solution that adds a new --add-ignore-recommends option to opkg. That option is based on the libsolv solver flag SOLVER_DISFAVOR. The problem is that SOLVER_DISFAVOR doesn't work in conjunction with SOLVER_FLAG_STRONG_RECOMMENDS, which opkg uses to enforce recommended packages upgrades. I opened a bug ticket to libsolv [1]. This ticket is blocked until the libsolv ticket is solved. [1] https://github.com/openSUSE/libsolv/issues/254 82557 18 alejandro.delcastillo 2018-12-20 20:32:50 +0000 The libsolv fix just landed on master. I tested with my prototype of --add-ignore-recommends and it worked, so we have a path forward. I am going to wait until libsolv releases (0.7.3) to submit the new opkg feature (--add-ignore-recommends flag). Once that is in, we need to put the patches on OE, then modify OE BAD_RECOMMENDATIONS code to use the new flag. 82974 19 alejandro.delcastillo 2019-02-08 15:43:33 +0000 Commit 0d11e813ba9b4e8de9e6e5099ff85f5d914243bc replaces the current BAD_RECOMMENDATIONS implementation with the one based on the new opkg flag --add-ignore-recommends. This should fix the issue on this ticket.