Bug 6816 - BAD_RECOMMENDATIONS not ignoring packages with opkg backend
Summary: BAD_RECOMMENDATIONS not ignoring packages with opkg backend
Status: RESOLVED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: deployment (show other bugs)
Version: unspecified
Hardware: All Multiple
: Medium normal
Target Milestone: 1.7 M4
Assignee: Paul Barker
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2014-10-07 14:02 UTC by chris.carr
Modified: 2014-11-11 11:04 UTC (History)
2 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments
opkg: Upgrade to v0.2.3 (8.29 KB, patch)
2014-10-11 10:16 UTC, Paul Barker
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description chris.carr 2014-10-07 14:02:58 UTC
When using the opkg backend as the package manager, not all the entries in the BAD_RECOMMENDATIONS are being "ignored". When the same configuration is used with the rpm backend, then the packages are correctly "ignored".


BAD_RECOMMENDATIONS=" \
    busybox-udhcpc \
    e2fsprogs-e2fsck \
    udev-hwdb \
    update-rc.d \
    vim-help \
    vim-syntax \
    vim-tutor \
    "

All of which are specified as RRECOMMENDS in various recipes.

The BAD_RECOMMENDATIONS all seem to write the correct entries to the
/var/lib/opkg/status file (desinstall hold not-installed), which I've
confirmed with some debug in the package_manager.py. However when the
packages are being installed in the rootfs stage, then only a couple of
them are processed and ignored.

The strange thing is, that depending on the verbosity of the opkg-cl
command, the number of packages successfully processed changes. The higher
the verbosity the more packages from BAD_RECOMMENDATIONS are processed.

I include the output of the rootfs creation below, with the "ignoring
recommendation" lines pulled out.


-bash-4.2$
/srv/nfs/chris/owc_up/poky/build/tmp/sysroots/x86_64-linux/usr/bin/opkg-cl
-v
opkg version 0.2.2

Default Run:

NOTE:
/srv/nfs/chris/owc_up/poky/build/tmp/sysroots/x86_64-linux/usr/bin/opkg-cl
-f
/srv/nfs/chris/owc_up/poky/build/tmp/work/da923rc-owc_cc-linux-gnuspe/open
ware-core/1.0-r0/opkg.conf -o
/srv/nfs/chris/owc_up/poky/build/tmp/work/da923rc-owc_cc-linux-gnuspe/open
ware-core/1.0-r0/rootfs --force_postinstall --prefer-arch-to-version
install opkg-collateral packagegroup-owc-system packagegroup-owc-boot
packagegroup-owc-net opkg packagegroup-owc-ruby

vim: ignoring recommendation for vim-syntax at user request
systemd: ignoring recommendation for e2fsprogs-e2fsck at user request

Debug V1:

NOTE:
/srv/nfs/chris/owc_up/poky/build/tmp/sysroots/x86_64-linux/usr/bin/opkg-cl
-V1 -f
/srv/nfs/chris/owc_up/poky/build/tmp/work/da923rc-owc_cc-linux-gnuspe/open
ware-core/1.0-r0/opkg.conf -o
/srv/nfs/chris/owc_up/poky/build/tmp/work/da923rc-owc_cc-linux-gnuspe/open
ware-core/1.0-r0/rootfs --force_postinstall --prefer-arch-to-version
install opkg-collateral packagegroup-owc-system packagegroup-owc-boot
packagegroup-owc-net opkg packagegroup-owc-ruby

vim: ignoring recommendation for vim-syntax at user request
systemd: ignoring recommendation for e2fsprogs-e2fsck at user request

Debug V2:

NOTE:
/srv/nfs/chris/owc_up/poky/build/tmp/sysroots/x86_64-linux/usr/bin/opkg-cl
-V2 -f
/srv/nfs/chris/owc_up/poky/build/tmp/work/da923rc-owc_cc-linux-gnuspe/open
ware-core/1.0-r0/opkg.conf -o
/srv/nfs/chris/owc_up/poky/build/tmp/work/da923rc-owc_cc-linux-gnuspe/open
ware-core/1.0-r0/rootfs --force_postinstall --prefer-arch-to-version
install opkg-collateral packagegroup-owc-system packagegroup-owc-boot
packagegroup-owc-net opkg packagegroup-owc-ruby

vim: ignoring recommendation for vim-syntax at user request
systemd: ignoring recommendation for udev-hwdb at user request
systemd: ignoring recommendation for e2fsprogs-e2fsck at user request

Debug V3:

NOTE:
/srv/nfs/chris/owc_up/poky/build/tmp/sysroots/x86_64-linux/usr/bin/opkg-cl
-V3 -f
/srv/nfs/chris/owc_up/poky/build/tmp/work/da923rc-owc_cc-linux-gnuspe/open
ware-core/1.0-r0/opkg.conf -o
/srv/nfs/chris/owc_up/poky/build/tmp/work/da923rc-owc_cc-linux-gnuspe/open
ware-core/1.0-r0/rootfs --force_postinstall --prefer-arch-to-version
install opkg-collateral packagegroup-owc-system packagegroup-owc-boot
packagegroup-owc-net opkg packagegroup-owc-ruby

vim: ignoring recommendation for vim-help at user request
vim: ignoring recommendation for vim-syntax at user request
vim: ignoring recommendation for vim-tutor at user request
rsyslog: ignoring recommendation for update-rc.d at user request
busybox: ignoring recommendation for update-rc.d at user request
busybox: ignoring recommendation for busybox-udhcpc at user request
dbus-1: ignoring recommendation for update-rc.d at user request
systemd: ignoring recommendation for update-rc.d at user request
systemd: ignoring recommendation for udev-hwdb at user request
systemd: ignoring recommendation for e2fsprogs-e2fsck at user request
run-postinsts: ignoring recommendation for update-rc.d at user request
ntp: ignoring recommendation for update-rc.d at user request
quagga: ignoring recommendation for update-rc.d at user request
dropbear: ignoring recommendation for update-rc.d at user request
rarpd: ignoring recommendation for update-rc.d at user request

Debug V4:

NOTE:
/srv/nfs/chris/owc_up/poky/build/tmp/sysroots/x86_64-linux/usr/bin/opkg-cl
-V4 -f
/srv/nfs/chris/owc_up/poky/build/tmp/work/da923rc-owc_cc-linux-gnuspe/open
ware-core/1.0-r0/opkg.conf -o
/srv/nfs/chris/owc_up/poky/build/tmp/work/da923rc-owc_cc-linux-gnuspe/open
ware-core/1.0-r0/rootfs --force_postinstall --prefer-arch-to-version
install opkg-collateral packagegroup-owc-system packagegroup-owc-boot
packagegroup-owc-net opkg packagegroup-owc-ruby

vim: ignoring recommendation for vim-help at user request
vim: ignoring recommendation for vim-syntax at user request
vim: ignoring recommendation for vim-tutor at user request
rsyslog: ignoring recommendation for update-rc.d at user request
busybox: ignoring recommendation for update-rc.d at user request
busybox: ignoring recommendation for busybox-udhcpc at user request
dbus-1: ignoring recommendation for update-rc.d at user request
systemd: ignoring recommendation for update-rc.d at user request
systemd: ignoring recommendation for udev-hwdb at user request
systemd: ignoring recommendation for e2fsprogs-e2fsck at user request
run-postinsts: ignoring recommendation for update-rc.d at user request
ntp: ignoring recommendation for update-rc.d at user request
quagga: ignoring recommendation for update-rc.d at user request
dropbear: ignoring recommendation for update-rc.d at user request
rarpd: ignoring recommendation for update-rc.d at user request
Comment 1 Paul Barker 2014-10-11 09:38:39 UTC
If the opkg status file is being written correctly this does sound like an opkg bug rather than a bug in package_manager.py in OpenEmbedded. I'll have a look into it and see what I can find.
Comment 2 Paul Barker 2014-10-11 10:15:52 UTC
I've had a look at this and I've spotted a potential cause. Unfortunately this hits the most complex, legacy code paths in opkg (the functions pkg_hash_fetch_unsatisfied_dependencies and pkg_hash_fetch_best_installation_candidate). I made several changes to this code path in v0.2.3, I haven't sent an update to OpenEmbedded yet as I understand that we've passed the feature freeze for the 1.7 release.

I'll attach a patch to this bug which updates opkg to v0.2.3 in openembedded-core. Could you check if the problem is already solved in this release?

Thanks
Comment 3 Paul Barker 2014-10-11 10:16:49 UTC
Created attachment 2179 [details]
opkg: Upgrade to v0.2.3
Comment 4 chris.carr 2014-10-13 10:48:55 UTC
I've ran the build again with you upgrade to V0.2.3. 
Unfortunately the problem still exists.

I've added some debug to dump the status file after package_manager.py processes the BAD_RECOMMENDATIONS.

Package: update-rc.d
Version: 0.7-r5
Status: deinstall hold not-installed
...
Package: vim-tutor
Version: 7.4.373-r0
Status: deinstall hold not-installed
...
Package: vim-help
Version: 7.4.373-r0
Status: deinstall hold not-installed
...
Package: busybox-udhcpc
Version: 1.22.1-r32
Status: deinstall hold not-installed
...
Package: udev-hwdb
Version: 1:216+git0+5d0ae62c66-r0
Depends: udev
Status: deinstall hold not-installed
...
Package: vim-syntax
Version: 7.4.373-r0
Status: deinstall hold not-installed
...
Package: e2fsprogs-e2fsck
Version: 1.42.9-r0
Depends: libblkid1 (>= 2.24.2), libcom-err2 (>= 1.42.9), libe2p2 (>= 1.42.9), libc6 (>= 2.20), libext2fs2 (>= 1.42.9), libuuid1 (>= 2.24.2)
Status: deinstall hold not-installed
...

Build Output

-bash-4.2$ /srv/nfs/chris/owc_up/poky/build/tmp/sysroots/x86_64-linux/usr/bin/opkg-cl -v
opkg version 0.2.3

Default
NOTE: /srv/nfs/chris/owc_up/poky/build/tmp/sysroots/x86_64-linux/usr/bin/opkg-cl -f /srv/nfs/chris/owc_up/poky/build/tmp/work/da923rc-owc_cc-linux-gnuspe/openware-core/1.0-r0/opkg.conf -o /srv/nfs/chris/owc_up/poky/build/tmp/work/da923rc-owc_cc-linux-gnuspe/openware-core/1.0-r0/rootfs --force_postinstall --prefer-arch-to-version   install packagegroup-owc-system packagegroup-owc-boot packagegroup-owc-net opkg-collateral opkg

vim: ignoring recommendation for vim-syntax at user request
systemd: ignoring recommendation for e2fsprogs-e2fsck at user request


V1
NOTE: /srv/nfs/chris/owc_up/poky/build/tmp/sysroots/x86_64-linux/usr/bin/opkg-cl -V1 -f /srv/nfs/chris/owc_up/poky/build/tmp/work/da923rc-owc_cc-linux-gnuspe/openware-core/1.0-r0/opkg.conf -o /srv/nfs/chris/owc_up/poky/build/tmp/work/da923rc-owc_cc-linux-gnuspe/openware-core/1.0-r0/rootfs --force_postinstall --prefer-arch-to-version   install packagegroup-owc-system packagegroup-owc-boot packagegroup-owc-net opkg-collateral opkg

vim: ignoring recommendation for vim-syntax at user request
systemd: ignoring recommendation for e2fsprogs-e2fsck at user request

V2
NOTE: /srv/nfs/chris/owc_up/poky/build/tmp/sysroots/x86_64-linux/usr/bin/opkg-cl -V2 -f /srv/nfs/chris/owc_up/poky/build/tmp/work/da923rc-owc_cc-linux-gnuspe/openware-core/1.0-r0/opkg.conf -o /srv/nfs/chris/owc_up/poky/build/tmp/work/da923rc-owc_cc-linux-gnuspe/openware-core/1.0-r0/rootfs --force_postinstall --prefer-arch-to-version   install packagegroup-owc-system packagegroup-owc-boot packagegroup-owc-net opkg-collateral opkg

vim: ignoring recommendation for vim-syntax at user request
systemd: ignoring recommendation for udev-hwdb at user request
systemd: ignoring recommendation for e2fsprogs-e2fsck at user request


V3
NOTE: /srv/nfs/chris/owc_up/poky/build/tmp/sysroots/x86_64-linux/usr/bin/opkg-cl -V3 -f /srv/nfs/chris/owc_up/poky/build/tmp/work/da923rc-owc_cc-linux-gnuspe/openware-core/1.0-r0/opkg.conf -o /srv/nfs/chris/owc_up/poky/build/tmp/work/da923rc-owc_cc-linux-gnuspe/openware-core/1.0-r0/rootfs --force_postinstall --prefer-arch-to-version   install packagegroup-owc-system packagegroup-owc-boot packagegroup-owc-net opkg-collateral opkg

vim: ignoring recommendation for vim-help at user request
vim: ignoring recommendation for vim-syntax at user request
vim: ignoring recommendation for vim-tutor at user request
rsyslog: ignoring recommendation for update-rc.d at user request
busybox: ignoring recommendation for update-rc.d at user request
busybox: ignoring recommendation for busybox-udhcpc at user request
dbus-1: ignoring recommendation for update-rc.d at user request
systemd: ignoring recommendation for update-rc.d at user request
systemd: ignoring recommendation for udev-hwdb at user request
systemd: ignoring recommendation for e2fsprogs-e2fsck at user request
run-postinsts: ignoring recommendation for update-rc.d at user request
quagga: ignoring recommendation for update-rc.d at user request
ntp: ignoring recommendation for update-rc.d at user request
dropbear: ignoring recommendation for update-rc.d at user request
dnsmasq: ignoring recommendation for update-rc.d at user request
rarpd: ignoring recommendation for update-rc.d at user request


V4
NOTE: /srv/nfs/chris/owc_up/poky/build/tmp/sysroots/x86_64-linux/usr/bin/opkg-cl -V4 -f /srv/nfs/chris/owc_up/poky/build/tmp/work/da923rc-owc_cc-linux-gnuspe/openware-core/1.0-r0/opkg.conf -o /srv/nfs/chris/owc_up/poky/build/tmp/work/da923rc-owc_cc-linux-gnuspe/openware-core/1.0-r0/rootfs --force_postinstall --prefer-arch-to-version   install packagegroup-owc-system packagegroup-owc-boot packagegroup-owc-net opkg-collateral opkg

vim: ignoring recommendation for vim-help at user request
vim: ignoring recommendation for vim-syntax at user request
vim: ignoring recommendation for vim-tutor at user request
rsyslog: ignoring recommendation for update-rc.d at user request
busybox: ignoring recommendation for update-rc.d at user request
busybox: ignoring recommendation for busybox-udhcpc at user request
dbus-1: ignoring recommendation for update-rc.d at user request
systemd: ignoring recommendation for update-rc.d at user request
systemd: ignoring recommendation for udev-hwdb at user request
systemd: ignoring recommendation for e2fsprogs-e2fsck at user request
run-postinsts: ignoring recommendation for update-rc.d at user request
quagga: ignoring recommendation for update-rc.d at user request
ntp: ignoring recommendation for update-rc.d at user request
dropbear: ignoring recommendation for update-rc.d at user request
dnsmasq: ignoring recommendation for update-rc.d at user request
rarpd: ignoring recommendation for update-rc.d at user request
Comment 5 Paul Barker 2014-10-13 11:04:37 UTC
Ok, this is very strange, particularly the difference in result depending on the verbosity level.

I've got a very busy week ahead so it may be next week before I can look at this. When I get time to though I'll try to reproduce this locally by using your BAD_RECOMMENDATIONS value and building an image with vim, rsyslog, busybox, dbus-1, systemd, run-postinsts, ntp, quagga, dropbear and rarpd.

Once I've confirmed it's reproducible I'll see what I can do to track down the root cause.
Comment 6 Paul Barker 2014-10-21 20:07:42 UTC
I almost don't believe it as it's a very weird error, but I've managed to replicate this. In local.conf I have added the following to my standard setup:

CORE_IMAGE_EXTRA_INSTALL += "opkg-dbg gdb vim rsyslog busybox dbus systemd run-postinsts ntp quagga dropbear"
DISTRO_FEATURES_append = " systemd"

BAD_RECOMMENDATIONS=" \
    busybox-udhcpc \
    e2fsprogs-e2fsck \
    udev-hwdb \
    update-rc.d \
    vim-help \
    vim-syntax \
    vim-tutor \
    "

(opkg-dbg and gdb are so that I can track down the bug on the target if needed and I couldn't find any recipe to provide 'rarpd')

Building without further modification gives the two ignored packages you saw (vim-syntax and e2fsprogs-e2fsck). Modifying package_manager.py to call opkg with '-V4' shows the rest of the expected packages being ignored.

I don't have the time to dig further into this today but at least I've replicated it. Hopefully I'll find some more time later in the week.
Comment 7 Paul Barker 2014-10-26 19:27:20 UTC
TL,DR: Fix incoming, I'll send the patch to the mailing list shortly.

This took a lot of tracking down and then the penny finally dropped. In package_manager.py, the command 'opkg <args> info <pkg>' is called to get information about each pkg in BAD_RECOMMENDATIONS in a format that can be written to the status file. The 'Status: ...' line is modified and all other lines are passed through. Changing the verbosity level argument for this command will change what it written into the status file. Crucially, with the default verbosity level, no blank lines are being printed by the opkg command and so no blank lines are being written to the status file to separate each package entry.

The package parsing code in opkg expects package entries in the status file to be separated by at least one blank line. If no blank line is seen, the next package entry is interpreted as a continuation of the last package entry, but the new values overwrite the old values.

So with the default verbosity level, a blank line follows vim-syntax and e2fsprogs-e2fsck for some reason and these are parsed. The others are dropped due to the lack of blank lines. As the verbosity increases, more debugging messages add blank lines and more packages are parsed.

The solution to ensure that this works correctly regardless of the verbosity level is simply add a blank line after the output of 'opkg info' is written to the status file, ensuring that the next package is separated from the current package.
Comment 8 Paul Barker 2014-11-11 11:04:15 UTC
Fixed in commit 3fa24ee to openembedded-core.