Bug 6734

Summary: do_rootfs: The postinstall intercept hook 'update_pixbuf_cache' failed
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Jackie Huang <jackie.huang>
Component: virtual machinesAssignee: Cristian Iorga <cristian.iorga>
Status: RESOLVED FIXED QA Contact: Alexandru Georgescu <alexandru.c.georgescu>
Severity: normal    
Priority: Medium+ CC: alexandru.c.georgescu, ntl, richard.purdie
Version: 1.7   
Target Milestone: 1.8 M3   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
strace of qemu-arm update_pixbuf_cache none

Description Jackie Huang 2014-09-18 09:06:50 UTC 
Build core-image-sato successfully, but the qemu-executed post-install scripts update_pixbuf_cache failed sometimes:

NOTE: Running intercept scripts:
NOTE: > Executing update_pixbuf_cache intercept ...
/buildarea/jhuang0/yp/y_x64_140612/tmp/sysroots/qemux86-64/usr/bin/crossscripts/qemuwrapper: line 2: 11161 Killed                  qemu-x86_64 "$@"
WARNING: The postinstall intercept hook 'update_pixbuf_cache' failed (exit code: 137)! See log for details!
WARNING: The postinstalls for the following packages will be postponed for first boot: gdk-pixbuf-loader-jpeg gdk-pixbuf-loader-png gdk-pixbuf-loader-gif gdk-pixbuf-loader-xpm gdk-pixbuf-loader-xbm gdk-pixbuf-loader-qtif gdk-pixbuf-loader-ani gdk-pixbuf-loader-ico gdk-pixbuf-loader-tga gdk-pixbuf-loader-pnm gdk-pixbuf-loader-bmp gdk-pixbuf-loader-wbmp gdk-pixbuf-loader-pcx gdk-pixbuf-loader-ras gdk-pixbuf-loader-icns


This not an error and they'll just be executed on the target on first boot.
But it takes a long time to run the script by qemu and if it fails, the first boot also takes a long time.

Seems like there is a timeout mechanism or something like that to kill the qemu process, since it doesn't fail every time:

$ pwd
/buildarea/jhuang0/yp/y_x64_140612/tmp/work/qemux86_64-poky-linux/core-image-sato/1.0-r0/temp
$ grep update_pixbuf_cache log.do_rootfs.*
log.do_rootfs.2708:NOTE: > Executing update_pixbuf_cache intercept ...
log.do_rootfs.2708:WARNING: The postinstall intercept hook 'update_pixbuf_cache' failed (exit code: 137)! See log for details!
log.do_rootfs.27878:NOTE: > Executing update_pixbuf_cache intercept ...
log.do_rootfs.28072:NOTE: > Executing update_pixbuf_cache intercept ...
log.do_rootfs.28424:NOTE: > Executing update_pixbuf_cache intercept ...
log.do_rootfs.2876:NOTE: > Executing update_pixbuf_cache intercept ...
log.do_rootfs.32169:NOTE: > Executing update_pixbuf_cache intercept ...
log.do_rootfs.32169:WARNING: The postinstall intercept hook 'update_pixbuf_cache' failed (exit code: 137)! See log for details!
Comment 2 Cristian Iorga 2014-12-09 15:15:06 UTC 
We need more data to investigate this issue.

Can you please observe the host system log (via dmesg) when this issue happens and attach to this bug report any suspicious messages?
Maybe at some point during the run of "update_pixbuf_cache" postinstall the system is out of memory?
Also, as I was unable to reproduce the issue, can you please refine more the conditions needed for this issue to reproduce?

Can the specs of the host build machine in any way impact the reproducibility of this issue? Limited amount of RAM, maybe?

Regards,
Cristian
Comment 3 Cristian Iorga 2014-12-09 15:16:00 UTC 
More info is needed in order to investigate this issue.
Comment 4 Jackie Huang 2014-12-10 07:13:17 UTC 
(In reply to comment #3)
> More info is needed in order to investigate this issue.

I tried several builds but was unable to reproduce the issue yet, maybe it has been fixed somewhere?

Anyway, I will keep and eye on it for some days, if it can be reproduced again, I will close the issue.
Comment 5 Nathan Lynch 2014-12-18 19:41:01 UTC 
I think I'm seeing the same issue.  In my case the qemu-arm process grows without bound; I've seen it get to ~7GB before manually killing it.  Presumably in the submitter's case it gets killed by the kernel eventually.

I can recreate it by enabling ptest, i.e. adding to local.conf:
     DISTRO_FEATURES_append = " ptest"
     EXTRA_IMAGE_FEATURES += "ptest-pkgs"

$ ps -o pid,vsize,rss,%mem,size,args $(pidof qemu-arm)
  PID    VSZ   RSS %MEM  SIZE COMMAND
 2708 1874472 1817896 11.0 1848884 qemu-arm -r 2.6.32 -L /home/nathanl/build/poky/tmp/work/beaglebone-poky-linux-gnueabi/core-image-minimal/1.0-r0/rootfs -E LD_LIBRARY_PATH=/home/nathanl/build/poky/tmp/work/beaglebone-poky-linux-gnueabi/core-image-minimal/1.0-r0/rootfs//usr/lib:/home/nathanl/build/poky/tmp/work/beaglebone-poky-linux-gnueabi/core-image-minimal/1.0-r0/rootfs//lib /home/nathanl/build/poky/tmp/work/beaglebone-poky-linux-gnueabi/core-image-minimal/1.0-r0/rootfs/usr/bin/gdk-pixbuf-query-loaders

I have an strace log of the qemu-arm process which I will attach.  It appears to be something to do with the udev ptest support?

I tested with b813bdebb36501500e86fea5f7e15b4b15ea0902 (qemu/libc-package: Fix qemu option handling) applied but this did not seem to help.
Comment 6 Nathan Lynch 2014-12-18 19:44:30 UTC 
Created attachment 2302 [details]
strace of qemu-arm update_pixbuf_cache
Comment 7 Stephen K Jolley 2014-12-29 19:16:27 UTC 
Is this still needinfo?
Comment 8 Cristian Iorga 2015-01-15 10:41:41 UTC 
Based on new info added to bug report, investigation restarted.
Comment 9 Cristian Iorga 2015-01-15 18:37:25 UTC 
Qemu user mode emulation that is used during the postinstall phase of image creation will fail due to looping infinitely on udev ptest.
See strace of qemu-arm update_pixbuf_cache attachment.
I am able to reproduce it by adding:
DISTRO_FEATURES_append = " ptest"
EXTRA_IMAGE_FEATURES += "ptest-pkgs"
to local.conf.

A related open issue seems to be open in qemu bugs report tool:

LD_PREFIX option reads directories recursively in an endless loop
https://bugs.launchpad.net/qemu/+bug/1245703

"If I run qemu user emulation with -L /path/to/my/sysroot/ in which also the proc and dev filesystem is mounted QEMU eats my memory until it gets killed by the kernel.

According to the strace output it follows the symbolic links in the proc filesystem running forever in a recursive loop."

Quote:
"Yeah, this -L code is just busted. It's really only intended to work
with extremely simple sysroot directories which don't have weird
stuff like proc mounts or symlinks and aren't very big."

"extremely simple sysroot" directorie is not our case, as such, the issue manifests in our case.

Please note that this QEMU issue has been opened on 2013-10-28, confirmed, but no resolution has been given.