Bug 8452 - LSB builds fail in kernel sanity test with xmalloc failure
Summary: LSB builds fail in kernel sanity test with xmalloc failure
Status: VERIFIED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: core (show other bugs)
Version: 1.9
Hardware: All Multiple
: High normal
Target Milestone: 2.0
Assignee: Aníbal Limón
QA Contact: Lucian Musat
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2015-10-07 10:05 UTC by Ross Burton
Modified: 2015-11-10 12:35 UTC (History)
8 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: Regression (Used to work)
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Ross Burton 2015-10-07 10:05:46 UTC
Specifically the LSB builds are failing the kernel sanity testing:

| ======================================================================
| FAIL: test_kernel_module (oeqa.runtime.kernelmodule.KernelModuleTest)
| ----------------------------------------------------------------------
| Traceback (most recent call last):
|   File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/meta/lib/oeqa/utils/decorators.py", line 106, in wrapped_f
|     return func(*args)
|   File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/meta/lib/oeqa/utils/decorators.py", line 94, in wrapped_f
|     return f(*args)
|   File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/meta/lib/oeqa/utils/decorators.py", line 94, in wrapped_f
|     return f(*args)
|   File "/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/meta/lib/oeqa/runtime/kernelmodule.py", line 31, in test_kernel_module
|     self.assertEqual(status, 0, msg="\n".join([cmd, output]))
| AssertionError: 2 != 0 : cd /usr/src/kernel && make scripts
|   HOSTCC  scripts/basic/fixdep
|   HOSTCC  scripts/kconfig/conf.o
|   SHIPPED scripts/kconfig/zconf.tab.c
|   SHIPPED scripts/kconfig/zconf.lex.c
|   SHIPPED scripts/kconfig/zconf.hash.c
|   HOSTCC  scripts/kconfig/zconf.tab.o
|   HOSTLD  scripts/kconfig/conf
| scripts/kconfig/conf --silentoldconfig Kconfig
| bash: xmalloc: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/armv5e-poky-linux-gnueabi/bash/4.3.30-r0/bash-4.3.30/locale.c:81: cannot allocate 2 bytes (0 bytes allocated)
| bash: xmalloc: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/armv5e-poky-linux-gnueabi/bash/4.3.30-r0/bash-4.3.30/locale.c:81: cannot allocate 2 bytes (0 bytes allocated)
| bash: xmalloc: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/armv5e-poky-linux-gnueabi/bash/4.3.30-r0/bash-4.3.30/locale.c:81: cannot allocate 2 bytes (0 bytes allocated)
| Makefile:588: /usr/src/kernel/arch//Makefile: No such file or directory
| bash: xmalloc: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/armv5e-poky-linux-gnueabi/bash/4.3.30-r0/bash-4.3.30/locale.c:81: cannot allocate 2 bytes (0 bytes allocated)
| bash: xmalloc: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/armv5e-poky-linux-gnueabi/bash/4.3.30-r0/bash-4.3.30/locale.c:81: cannot allocate 2 bytes (0 bytes allocated)
| bash: xmalloc: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/armv5e-poky-linux-gnueabi/bash/4.3.30-r0/bash-4.3.30/locale.c:81: cannot allocate 2 bytes (0 bytes allocated)
| bash: xmalloc: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/armv5e-poky-linux-gnueabi/bash/4.3.30-r0/bash-4.3.30/locale.c:81: cannot allocate 2 bytes (0 bytes allocated)
| bash: xmalloc: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/armv5e-poky-linux-gnueabi/bash/4.3.30-r0/bash-4.3.30/locale.c:81: cannot allocate 2 bytes (0 bytes allocated)
| bash: xmalloc: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/armv5e-poky-linux-gnueabi/bash/4.3.30-r0/bash-4.3.30/locale.c:81: cannot allocate 2 bytes (0 bytes allocated)
| bash: xmalloc: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/armv5e-poky-linux-gnueabi/bash/4.3.30-r0/bash-4.3.30/locale.c:81: cannot allocate 2 bytes (0 bytes allocated)
| bash: xmalloc: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/armv5e-poky-linux-gnueabi/bash/4.3.30-r0/bash-4.3.30/locale.c:81: cannot allocate 2 bytes (0 bytes allocated)
| bash: xmalloc: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/armv5e-poky-linux-gnueabi/bash/4.3.30-r0/bash-4.3.30/locale.c:81: cannot allocate 2 bytes (0 bytes allocated)
| bash: xmalloc: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/armv5e-poky-linux-gnueabi/bash/4.3.30-r0/bash-4.3.30/locale.c:81: cannot allocate 2 bytes (0 bytes allocated)
| bash: xmalloc: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/armv5e-poky-linux-gnueabi/bash/4.3.30-r0/bash-4.3.30/locale.c:81: cannot allocate 2 bytes (0 bytes allocated)
| bash: xmalloc: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/armv5e-poky-linux-gnueabi/bash/4.3.30-r0/bash-4.3.30/locale.c:81: cannot allocate 2 bytes (0 bytes allocated)
| make: execvp: /usr/src/kernel/scripts/gcc-goto.sh: Permission denied
| bash: xmalloc: /home/pokybuild/yocto-autobuilder/yocto-worker/nightly-arm-lsb/build/build/tmp/work/armv5e-poky-linux-gnueabi/bash/4.3.30-r0/bash-4.3.30/locale.c:81: cannot allocate 2 bytes (0 bytes allocated)
| make: *** No rule to make target '/usr/src/kernel/arch//Makefile'.  Stop.
|
Comment 1 Benjamin Esquivel 2015-10-07 20:28:54 UTC
One of the things that look strange here is that the test is not triggering the logic for gathering logs to see the host/target context (mem, proc, dmesg, etc). Mariano did an intentional failure of the test and it *should* be presenting such logs.

The first thing that occurs to us is to check what is going on with the host memory at the time of failure so we'll send a forced build with a patch that writes periodically the memory status to a file in the disc so we can come later an analyze it.

One of the theories is that if the Ram is not being sufficient then the process for gathering the logs cannot start as well.
Comment 2 Richard Purdie 2015-10-08 06:22:41 UTC
As I've mentioned elsewhere, if you download one of the broken images and boot it locally, you can reproduce this the first time you run "make scripts" inside an image. I've run top in an image whilst showing the bug and I didn't see any significant resource problem.

If you google the error, there are comments about PAM. We enable PAM in poky-lsb. We're also seeing this bug on x86, x86-64, arm and ppc so it looks like its something strucutural in our images, not something kernel/arch specific.
Comment 3 Aníbal Limón 2015-10-09 19:54:26 UTC
I tested with arm lsb sdk image that fails and the problem is caused by bash custom malloc.

I found it because i disabled bash custom malloc passing --without-bash-malloc to configure and all works good.
Comment 4 Ross Burton 2015-10-09 20:29:26 UTC
Also of note is that the latest master runs on the AB don't fail like this...
Comment 5 Aníbal Limón 2015-10-09 21:58:01 UTC
I found that all the distr(In reply to comment #3)
> I tested with arm lsb sdk image that fails and the problem is caused by bash
> custom malloc.
> 
> I found it because i disabled bash custom malloc passing
> --without-bash-malloc to configure and all works good.

I found that common distros don't use it, for example:

Debian & Ubuntu: http://bazaar.launchpad.net/~doko/+junk/pkg-bash-debian/view/head:/rules#L89

Fedora 20 (extract and see spec): ftp://rpmfind.net/linux/fedora-secondary/development/rawhide/source/SRPMS/b/bash-4.2.45-4.fc20.src.rpm

LFS: http://www.linuxfromscratch.org/lfs/view/development/chapter06/bash.html

So there is no reason for maintain enabled.
Comment 7 Lucian Musat 2015-11-10 12:35:40 UTC
Having been seeing this of late so setting this as verified (on commit fc45deac89ef63ca1c44e763c38ced7dfd72cbe1)