Created attachment 1962 [details] liveboot print screen When trying to boot a core-image-sato-sdk for crownbay-noemgd image - liveboot or after image is installed, X crases in both cases (see screenshots attached).
Created attachment 1963 [details] normal boot
Forgot to add the build details:meta-intel 626f88de5b5555d26f84f876c55a3fd34cb9032c meta-qt3 3016129d90b7ac8517a5227d819f10ad417b5b45 poky 98ad3cb2c0f5975a0df4cebc06775aaae657700d
I did not had any issue with sato image. And my machine is not able to boot with the SDK image. I am working on getting the BIOS updated on the system to reproduce this issue. The process of updating the BIOS is taking long, as the BIOS image is not easily available, and flashing the EEPROM with the new BIOS need hardware programer.
Hi Alex, Per the screenshot there are some next clear next steps that would help move this along (as we cannot yet reproduce): Matchbox: error parsing /usr/share/themes/Sato/matchbox/theme.xml Incorrect Params in <font id='titlefont' def='Sans bold 32px'/> Does this message repeat in a regular Sato image (which appears to boot successfully for Nitin)? Nitin, comparing the builds, do you see a difference between the two rootfs images with respect to Matchbox - I'd expect not, but lets pursue the low hanging fruit.
FWIW core-image-sato on qemux86 doesn't produce that warning, which seems to imply that no fonts are installed. Building -sdk now.
As a reference point, core-image-sato-sdk runs on qemux86 fine. Grabbing the full syslog from a working and a broken machine would be useful.
Fiannly I got BIOS upgraded on the crownbay system. It is able to boot with the SDK image now. And I can see the error happening as mentioned on this bug.
BTW I am not seeing this issue with the SDK image I built myself.
Created attachment 1975 [details] sato-sdk image for crownbay-noemgd - liveboot Hi Nitin, This is what I get for sato-sdk image when doing liveboot. Image downloaded from: http://yocto-ab-master.jf.intel.com/pub/nightly-meta-intel/20140522-4/machines/crownbay-noemgd/crownbay-noemgd/
Created attachment 1976 [details] kernel panic for sato image on crownbay-noemgd I downloaded the sato only image too and tried to boot it after install. I have a kernel panic now. Image downloaded from here http://yocto-ab-master.jf.intel.com/pub/nightly-meta-intel/20140522-4/machines/crownbay-noemgd/crownbay-noemgd/
This is strange for me. I built the images with same commits, and they all worked fine. I will try AB image and try reproduce the issue here.
Adding Beth, I just tried the AB images on my crownbay, and they are failing. At the same time I also tried the images I built myself and they are working fine. Something is messed up somewhere. Beth, I am suspecting that AB build maybe causing some issue here. lets find out what is going on here. Alex, I can give you my images for you to test. But from the data so far I am certain that they will work fine for you. Nitin
I also note that the AB sdk image for crownbay has lot of fat fs errors. Beth has it to do anything with the space issue AB had yesterday?
nitin: No, that space issue was on the external builders.
Here is the auto.conf for the crownbay-noemgd build: DISTRO = "poky" PACKAGE_CLASSES = "package_rpm package_deb package_ipk" BB_NUMBER_THREADS = "16" PARALLEL_MAKE = "-j 16" SDKMACHINE ?= "x86_64" DL_DIR = "/srv/www/vhosts/autobuilder.yoctoproject.org/current_sources" SSTATE_DIR ?= "/srv/www/vhosts/autobuilder.yoctoproject.org/sstate/" MACHINE = "crownbay-noemgd" PREMIRRORS = "" BB_GENERATE_MIRROR_TARBALLS = "1" ADTREPO = "http://adtrepo.yoctoproject.org//1.6" Seems pretty straight forward. This build was done with a non-clean sstate.
I rebuilt the sato image with same configuration on autobuilder, and the image is working fine. Now, that narrows down the issue on autobuilder.
(In reply to comment #16) > I rebuilt the sato image with same configuration on autobuilder, and the > image is working fine. Now, that narrows down the issue on autobuilder. Nitin, I can you try on your side with the AB images? Thanks, Alex.
Hi Nitin, On RC2, we have the same issue as one in https://bugzilla.yoctoproject.org/attachment.cgi?id=1975 with emenlow. The image used is this one: http://yocto-ab-master.jf.intel.com/pub/nightly-meta-intel/20140522-4/machines/emenlow/emenlow/core-image-sato-sdk-emenlow-20140522154910.hddimg
Hello, I tried the image from autobuilder and no success with starting X (yocto-ab-master.jf.intel.com/pub/nightly-meta-intel/20140522-4/machines/crownbay-noemgd/crownbay-noemgd/core-image-sato-sdk-crownbay-noemgd.hddimg). However if I build the image myself (bitbake core-image-sato) X is fine Costin
Folks, If possible, can you get logged into via a serial console port and provide the full **/var/log/Xorg.0.log**, I think that Ross is on the right track with the missing fonts, but it would be helpful to get the full logs if possible. The last image attached is a kernel panic, this is not related to X not starting, so let's not confuse the 2 issues, if there is a kernel panic, it's a different issue.
Created attachment 1983 [details] boot after sdk image install on hdd Costin's latest update over email: I downloaded the .hddimg file from: http://yocto-ab-master.jf.intel.com/pub/nightly-meta-intel/20140527-1/machines/crownbay-noemgd/crownbay-noemgd/core-image-sato-sdk-crownbay-noemgd.hddimg LIVE: boots well INSTALLED: It installs the image on hdd, but after reboot gets stuck at a point without access to shell. I attached a snapshot. and after downloaded : http://yocto-ab-master.jf.intel.com/pub/nightly-meta-intel/20140527-1/machines/crownbay/crownbay/core-image-sato-sdk-crownbay.hddimg LIVE + INSTALLED: it gets to a point where it stuck. At this point the shell cannot be accessed. When trying to install it gets to the same point as on LIVE without possibility to do anything further. Costin
I tried the same thing what Costin' tried in the previous comment on this bug. And the core-image-sato-sdk-crownbay-noemgd.hddimg is working fine for both liveboot and install on my system. Next I am going to try the core-image-sato-sdk-crownbay.hddimg . Constin, can you get your system's BIOS updated? I have given all the information to Alex about it.
This images is giving udev permissions issues: http://yocto-ab-master.jf.intel.com/pub/nightly-meta-intel/20140527-1/machines/crownbay/crownbay/core-image-sato-sdk-crownbay.hddimg Something is still going weird in the autobuilder environment.
For further information this issue does not happen on these build systems: 1. Nitin's build machine: Open Suse 12.3 64bit, both with 1.5 & 1.6 buildtools. 2. Ada's build system: Both could build images which worked without issues on the hardware. Ada, can you provide distro details of your build system, for reference here? Thanks, Nitin
Beth started a new build with clean sstate and host buildtools. I tried this image from that build: http://yocto-ab-master.jf.intel.com/pub/crownbay-noemgd/20140528-1/machines/crownbay-noemgd/crownbay-noemgd/core-image-sato-sdk-crownbay-noemgd-20140528141458.hddimg It gives same error as seen before here: https://bugzilla.yoctoproject.org/attachment.cgi?id=1975
The sdk image built on the master AB is failing to boot as seen below. The error is showing that the ramdisk image created on AB was corrupted. RAMDISK: gzip image found at block 0 RAMDISK: incomplete write (13457 != 31743) write error List of all partitions: 0800 15278080 sda driver: sd No filesystem could mount root, tried: ext3 ext2 ext4 vfat msdos iso9660 btrfs VFS: Unable to mount root fs on unknown-block(1,0) User configuration error - no valid root filesystem found Kernel panic - not syncing: Invalid configuration from end user prevents continuing CPU: 1 PID: 1 Comm: swapper/0 Not tainted 3.14.4-yocto-standard #1 Hardware name: To be filled by O.E.M. To be filled by O.E.M./To be filled by O.E.M., BIOS 4.6.3 06/05/2012 00000000 00000000 f643be70 c1827727 f643bebc f643be90 c182366c c1a512c4 c1c9f9c0 00000000 f643bebc 00008001 f5642028 f643bee8 c1bd6f9e c1a40c18 f643bebc f5642022 00008001 00000000 f564213f 00000000 f745e840 c1a404ca Call Trace: [<c1827727>] dump_stack+0x4b/0x75 [<c182366c>] panic+0x82/0x166 [<c1bd6f9e>] mount_block_root+0x1ca/0x1da [<c1002930>] ? do_coprocessor_segment_overrun+0x50/0x80 [<c1bd71a0>] mount_root+0xf3/0xfb [<c1bd72f6>] prepare_namespace+0x14e/0x192 [<c1bd6c89>] kernel_init_freeable+0x1f3/0x200 [<c1bd64df>] ? do_early_param+0x78/0x78 [<c181fa80>] kernel_init+0x10/0xe0 [<c1835277>] ret_from_kernel_thread+0x1b/0x28 [<c181fa70>] ? rest_init+0x80/0x80 Kernel Offset: 0x0 from 0xc1000000 (relocation range: 0xc0000000-0xf83fdfff)
adding "ramdisk_size=32768" parameter to kernel command line helps with the RAMDISK error, but still kernel is failing to boot. RAMDISK: gzip image found at block 0 List of all partitions: 0800 15278080 sda driver: sd No filesystem could mount root, tried: ext3 ext2 ext4 vfat msdos iso9660 btrfs VFS: Unable to mount root fs on unknown-block(1,0) User configuration error - no valid root filesystem found Kernel panic - not syncing: Invalid configuration from end user prevents continuing CPU: 1 PID: 1 Comm: swapper/0 Not tainted 3.14.4-yocto-standard #1 Hardware name: To be filled by O.E.M. To be filled by O.E.M./To be filled by O.E.M., BIOS 4.6.3 06/05/2012 00000000 00000000 f643be70 c1827727 f643bebc f643be90 c182366c c1a512c4 c1c9f9c0 00000000 f643bebc 00008001 f480d028 f643bee8 c1bd6f9e c1a40c18 f643bebc f480d022 00008001 00000000 f480d13f 00000000 f74421a0 c1a404ca Call Trace: [<c1827727>] dump_stack+0x4b/0x75 [<c182366c>] panic+0x82/0x166 [<c1bd6f9e>] mount_block_root+0x1ca/0x1da [<c1002930>] ? do_coprocessor_segment_overrun+0x50/0x80 [<c1bd71a0>] mount_root+0xf3/0xfb [<c1bd72f6>] prepare_namespace+0x14e/0x192 [<c1bd6c89>] kernel_init_freeable+0x1f3/0x200 [<c1bd64df>] ? do_early_param+0x78/0x78 [<c181fa80>] kernel_init+0x10/0xe0 [<c1835277>] ret_from_kernel_thread+0x1b/0x28 [<c181fa70>] ? rest_init+0x80/0x80 Kernel Offset: 0x0 from 0xc1000000 (relocation range: 0xc0000000-0xf83fdfff)
So: This one build with OpenSuse 13.1 does NOT boot after install or at liveboot: http://yocto-ab-master.jf.intel.com//pub/crownbay-noemgd/20140528-1 This one built with DEBIAN: http://autobuilder.yoctoproject.org/pub/crownbay-noemgd/20140529-1/ does boot after install or at liveboot
Has anyone confirmed that the file systems are identical, or if they're not what the differences are?
(In reply to comment #29) > Has anyone confirmed that the file systems are identical, or if they're not > what the differences are? FWIW, sizes are different 2067611648 for Opensuse build one vs 2097725440 the debian one. Also, I managed to create an image and boot it under a local OpenSuse 13.1 machine here. It may be some strange combination of meta-tlk and meta-q3 layers, but I didn't get to reproduce the issue. Building w/ or witout tlk and/or qt3 layers does give me an error, but they are different from what I get from the AB ones.
(In reply to comment #28) > So: > > This one build with OpenSuse 13.1 does NOT boot after install or at > liveboot: > http://yocto-ab-master.jf.intel.com//pub/crownbay-noemgd/20140528-1 > > > This one built with DEBIAN: > http://autobuilder.yoctoproject.org/pub/crownbay-noemgd/20140529-1/ does > boot after install or at liveboot Alex, If you see my comment # 26 here, the debian generated image is not booting on crownbay here. I wonder why you and I are seeing different results on our systems. Which version of BIOS are you running with? Nitin
Beth, Did you try building an qemu-sdk image on the ab01? This issue will be much easier to debug with qemu. Nitin
For this image: http://autobuilder.yoctoproject.org/pub/crownbay-noemgd/20140529-1/machines/crownbay-noemgd/crownbay-noemgd/core-image-sato-sdk-crownbay-noemgd-20140529000856.hddimg I am seeing same failure as seen in comment: #26 & #27 For this image: http://yocto-ab-master.jf.intel.com//pub/crownbay-test/crown-emgd-meta-qt3-nometatlk-prepopulatedsstate/crownbay-noemgd/core-image-sato-sdk-crownbay-noemgd-20140529114618.hddimg This image is working properly. Also note that I always use the tlk layer in my builds and I have not seen this issue on my build system. It means the meta-tlk + suse 13.1 combination is causing this issue.
FWIW, this is the combinations that we tried today: The ab images were downloaded and dd-ed to the USB. The local built ones, were done with #bitbake -C rootfs core-image-sato-sdk There are also some sato images, but we mostly focused on the SDK ones. https://docs.google.com/spreadsheets/d/1JeoUrIWB1Kso872Op5KLIjwWsBjamyrjIwbgqLRlbl4/edit#gid=0
Created attachment 1987 [details] qemu boot err running QEMU http://yocto-ab-master.jf.intel.com/pub/crownbay-test/qemux86-nometaqt3-metatlk-prepopsstate/ fails (see attachment)
Created attachment 1988 [details] qemu serial log
Created attachment 1989 [details] matchbox font failure with qemu For the qemu image which Beth built on ab01, I am seeing the matchbox font failure. Attached is the screen. For the images here: http://yocto-ab-master.jf.intel.com/pub/crownbay-test/qemux86-nometaqt3-metatlk-prepopsstate/
Created attachment 1990 [details] diff-internal-doesntwork-external-works
There are many eyebrow raising differences in the Comment 38 I am noting few here: Why this is different? Are there mercurial related changes in these systems? diff -r test-external/usr/lib/python2.7/config/Makefile test-internal/usr/lib/python2.7/config/Makefile 39,41c39,41 < HGVERSION= hg id -i $(srcdir) < HGTAG= hg id -t $(srcdir) < HGBRANCH= hg id -b $(srcdir) --- > HGVERSION= > HGTAG= > HGBRANCH= Looks like valgrind recipe does not have right dependency. diff -r test-external/usr/include/valgrind/config.h test-internal/usr/include/valgrind/config.h 32c32 < #define GDB_PATH "/usr/bin/gdb" --- > #define GDB_PATH "/no/gdb/was/found/at/configure/time" The sequencing of parameters are different below. This may cause undesired libraries getting picked up instead of desired ones. Is this sequencing change the reason for the matchbox failures. Is this expected? Ross? diff -r test-external/usr/bin/qt4/demos/shared/libdemo_shared.prl test-internal/usr/bin/qt4/demos/shared/libdemo_shared.prl 1c1 < QMAKE_PRL_BUILD_DIR = /home/pokybuild/poky/build/tmp/work/i586-poky-linux/qt4-x11-free/4.8.5-r0/qt-everywhere-opensource-src-4.8.5/demos/shared/ --- > QMAKE_PRL_BUILD_DIR = /home/pokybuild/poky/build/tmp/work/i586-poky-linux/qt4-x11-free/4.8.5-r0/qt-everywhere-opensource-src-4.8.5/./demos/shared 6c6 < QMAKE_PRL_LIBS = -L/home/pokybuild/poky/build/tmp/sysroots/qemux86/usr/lib -L/home/pokybuild/poky/build/tmp/work/i586-poky-linux/qt4-x11-free/4.8.5-r0/qt-everywhere-opensource-src-4.8.5/lib -L/home/pokybuild/poky/build/tmp/sysroots/qemux86/usr/lib -lQtOpenGL -L/home/pokybuild/poky/build/tmp/work/i586-poky-linux/qt4-x11-free/4.8.5-r0/qt-everywhere-opensource-src-4.8.5/lib -lQtGui -lQtCore -lGL -lpthread --- > QMAKE_PRL_LIBS = -L/home/pokybuild/poky/build/tmp/sysroots/qemux86/usr/lib -L/home/pokybuild/poky/build/tmp/work/i586-poky-linux/qt4-x11-free/4.8.5-r0/qt-everywhere-opensource-src-4.8.5/lib -lQtOpenGL -L/home/pokybuild/poky/build/tmp/sysroots/qemux86/usr/lib -L/home/pokybuild/poky/build/tmp/work/i586-poky-linux/qt4-x11-free/4.8.5-r0/qt-everywhere-opensource-src-4.8.5/lib -lQtGui -lQtCore -lGL -lpthread
changing the bug header.
So, some additional digging. The matchbox issue I've been able to get on sato on an OpenSuse 13.1 server (note. not a desktop build of openSuse 13.1). This is independent of any layers/machine. It's happening on qemx86 with a clean sstate, no qt3 and no tlk. So, we can rule out those layers for this issue. The kernel issue I'm still tracking down. More info soon.
Ok, one major difference between the two autobuilders is the one that everything is failing on is using XFS as it's file system.
(In reply to comment #39) > diff -r test-external/usr/lib/python2.7/config/Makefile > test-internal/usr/lib/python2.7/config/Makefile > 39,41c39,41 > < HGVERSION= hg id -i $(srcdir) > < HGTAG= hg id -t $(srcdir) > < HGBRANCH= hg id -b $(srcdir) > --- > > HGVERSION= > > HGTAG= > > HGBRANCH= I suspect python-before-mercurial in one, and python-after-mercurial in another: a floating dependency in Python. > Looks like valgrind recipe does not have right dependency. > > diff -r test-external/usr/include/valgrind/config.h > test-internal/usr/include/valgrind/config.h > 32c32 > < #define GDB_PATH "/usr/bin/gdb" > --- > > #define GDB_PATH "/no/gdb/was/found/at/configure/time" Agreed. > The sequencing of parameters are different below. This may cause undesired > libraries getting picked up instead of desired ones. Is this sequencing > change the reason for the matchbox failures. Is this expected? > Ross? > > diff -r test-external/usr/bin/qt4/demos/shared/libdemo_shared.prl > test-internal/usr/bin/qt4/demos/shared/libdemo_shared.prl > 1c1 > < QMAKE_PRL_BUILD_DIR = > /home/pokybuild/poky/build/tmp/work/i586-poky-linux/qt4-x11-free/4.8.5-r0/qt- > everywhere-opensource-src-4.8.5/demos/shared/ > --- > > QMAKE_PRL_BUILD_DIR = /home/pokybuild/poky/build/tmp/work/i586-poky-linux/qt4-x11-free/4.8.5-r0/qt-everywhere-opensource-src-4.8.5/./demos/shared > 6c6 > < QMAKE_PRL_LIBS = -L/home/pokybuild/poky/build/tmp/sysroots/qemux86/usr/lib > -L/home/pokybuild/poky/build/tmp/work/i586-poky-linux/qt4-x11-free/4.8.5-r0/ > qt-everywhere-opensource-src-4.8.5/lib > -L/home/pokybuild/poky/build/tmp/sysroots/qemux86/usr/lib -lQtOpenGL > -L/home/pokybuild/poky/build/tmp/work/i586-poky-linux/qt4-x11-free/4.8.5-r0/ > qt-everywhere-opensource-src-4.8.5/lib -lQtGui -lQtCore -lGL -lpthread > --- > > QMAKE_PRL_LIBS = -L/home/pokybuild/poky/build/tmp/sysroots/qemux86/usr/lib -L/home/pokybuild/poky/build/tmp/work/i586-poky-linux/qt4-x11-free/4.8.5-r0/qt-everywhere-opensource-src-4.8.5/lib -lQtOpenGL -L/home/pokybuild/poky/build/tmp/sysroots/qemux86/usr/lib -L/home/pokybuild/poky/build/tmp/work/i586-poky-linux/qt4-x11-free/4.8.5-r0/qt-everywhere-opensource-src-4.8.5/lib -lQtGui -lQtCore -lGL -lpthread That's Qt which isn't used in Sato, so is unrelated. The ordering of the gdk-pixbuf loaders doesn't impact library loading order.
So if you take a broken image, and delete the fontconfig cache in /var/cache/fontconfig, it works. Comparing the broken cache and the freshly generated cache on a good boot reveals that the caches are mostly empty, specifically the fonts in /usr/share/fonts/ttf should be in 4322f4d3cdf1bd54794b691285ca062-le32d4.cache-4 but they are not. This implies the font cache generation at rootfs time broke on the build host. The next step would be to look at the rootfs log to see if anything odd is happening, the cache is generated in a scriptlet after the packages have been installed.
The evidence that the fontconfig cache in broken images claims that there are no fonts installed (but there is) and that ab01 is running a niche filesystem (XFS), I took a look at fontconfig's git repo and discovered that there's a patch which looks relevant (lstat failing on file systems that don't support dirent->d_type) and we don't have in daisy. http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=ross/daisy&id=5de504b29dda8d099dc2e9fa64814233b4a95fa4 is a commit based on daisy to cherry-pick that commit to fontconfig. If someone with access to the Suse/XFS build machine can attempt a built that would be much appreciated as I'm now going to bed.
So the XFX may be a red-herring since the fontcache.bbclass uses the qemuwrapper to call do the fc-cache inside qemu itself. I can reproduce this on yocto-ab01.jf.intel.com pretty readily. Ross you should have access to that machine, I tried building with with and without the updated version of fontconfig and it failed for both (I did verify fc-cache version) and also was able to confirm moving the cache and rebuilding on the target corrected the problem. As far as I can tell there are no errors in the rootfs NOTE: Running intercept scripts: NOTE: > Executing update_font_cache intercept ... NOTE: > Executing update_icon_cache intercept ... No Errors
(In reply to comment #46) > So the XFX may be a red-herring since the fontcache.bbclass uses the > qemuwrapper to call do the fc-cache inside qemu itself. I can reproduce > this > on yocto-ab01.jf.intel.com pretty readily. Ross you should have access to > that > machine, I tried building with with and without the updated version of > fontconfig and it failed for both (I did verify fc-cache version) and also > was > able to confirm moving the cache and rebuilding on the target corrected the > problem. As far as I can tell there are no errors in the rootfs > > NOTE: Running intercept scripts: > NOTE: > Executing update_font_cache intercept ... > NOTE: > Executing update_icon_cache intercept ... > > No Errors The fontconfig issue won't cause errors because it's behaving as designed (incorrect design). The qemu used to run fc-cache isn't a system-level qemu running with a disk image but a single-binary qemu running against the rootfs directory before the disk images are creates, so any bad interaction should still happen. I'll have a play this morning.
RE: Kernel panics. So today I tried to reproduce the kernel panic issue. I ended up with 5 builds all of which had the matchbox issue. None hit the kernel issue. So I attempted one of the earlier reported images that did have the issue. Fresh download. No panic. The interesting thing may be my setup. SSH with X11 forwarding to my Fedora 20 machine from my windows box. Not entirely sure if this has anything to do with it, but the fact that an image that did panic earlier doesn't now, is a bit odd.
Last night we finally root-caused this. The file system the builds were happening on is a many TB XFS file system and the fc-cache (more accurately, libfreetype) was running as a 32-bit process without large files. More often than not, stating a font file resulted in an inode number that was too big to fit into a 32-bit stat, so fstat() errors and return EOVERFLOW. The solution is to build with large files. Saul has a patch to enable it for fontconfig/freetype for 1.6, and I've a patch series in testing to globally enable it.
The patch to freetype was merged, closing.
This is verified since we did retest with the release BSPs after the fix was merged.