Bug 14310

Summary:	xz / liblzma memory allocation failure during reproducible build testing
Product:	[QA/Testing] Build Testing	Reporter:	Saul Wold <saul.wold>
Component:	general	Assignee:	Changqing Li <changqing.li>
Status:	RESOLVED FIXED	QA Contact:
Severity:	normal
Priority:	Medium+	CC:	changqing.li, chee.yang.lee, oobitots, peter, randy.macleod
Version:	unspecified
Target Milestone:	3.4 M1
Hardware:	x86
OS:	Multiple
Whiteboard:	AB-INT
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	No (bug/feature does not impact docs)

Description Saul Wold 2021-03-19 16:26:29 UTC

During the reproducible build testing the following failure occured on the debian10-ty-1 machine (56cpu/128G) ram. There was a patch submitted about a year ago for a similar issue: http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=27ff81bfd880280607c79dce2f724c8bfafce02d




RPM build errors:
    Deprecated external dependency generator is used!
    Deprecated external dependency generator is used!
    Deprecated external dependency generator is used!
    Deprecated external dependency generator is used!
    Deprecated external dependency generator is used!
    liblzma: Memory allocation failed    liblzma: Memory allocation failed    liblzma: Memory allocation failed    liblzma: Memory allocation failed    liblzma: Memory allocation failedWARNING: exit code 1 from a shell command.


And


NOTE: Running task 10969 of 14696 (/home/pokybuild/yocto-worker/reproducible/build/meta/recipes-graphics/xorg-lib/libxv_1.0.11.bb:do_package_write_rpm)
ERROR: libxkbfile-1_1.1.0-r0 do_package_write_ipk: Fatal errors occurred in subprocesses:
Command 'PATH="/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/pokybuild/yocto-worker/reproducible/build/scripts:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/usr/bin/x86_64-poky-linux:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot/usr/bin/crossscripts:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/usr/sbin:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/usr/bin:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/sbin:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/bin:/home/pokybuild/yocto-worker/reproducible/build/bitbake/bin:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/hosttools" opkg-build -Z xz -a "--memlimit=50% --threads=56" libxkbfile-dbg /home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/deploy-ipks/core2-64' returned non-zero exit status 1.
Subprocess output:xz: (stdin): Cannot allocate memory
Command 'PATH="/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/pokybuild/yocto-worker/reproducible/build/scripts:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/usr/bin/x86_64-poky-linux:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot/usr/bin/crossscripts:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/usr/sbin:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/usr/bin:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/sbin:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/bin:/home/pokybuild/yocto-worker/reproducible/build/bitbake/bin:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/hosttools" opkg-build -Z xz -a "--memlimit=50% --threads=56" libxkbfile-src /home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/deploy-ipks/core2-64' returned non-zero exit status 1.
Subprocess output:xz: (stdin): Cannot allocate memory
Command 'PATH="/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/pokybuild/yocto-worker/reproducible/build/scripts:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/usr/bin/x86_64-poky-linux:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot/usr/bin/crossscripts:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/usr/sbin:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/usr/bin:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/sbin:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/bin:/home/pokybuild/yocto-worker/reproducible/build/bitbake/bin:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/hosttools" opkg-build -Z xz -a "--memlimit=50% --threads=56" libxkbfile /home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/deploy-ipks/core2-64' returned non-zero exit status 1.
Subprocess output:xz: (stdin): Cannot allocate memory
Command 'PATH="/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/pokybuild/yocto-worker/reproducible/build/scripts:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/usr/bin/x86_64-poky-linux:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot/usr/bin/crossscripts:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/usr/sbin:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/usr/bin:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/sbin:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/bin:/home/pokybuild/yocto-worker/reproducible/build/bitbake/bin:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/hosttools" opkg-build -Z xz -a "--memlimit=50% --threads=56" libxkbfile-dev /home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/deploy-ipks/core2-64' returned non-zero exit status 1.
Subprocess output:xz: (stdin): Cannot allocate memory

Comment 1 Randy MacLeod 2021-03-25 14:41:26 UTC

We have tunes xz down in the past to lower memory usage so perhaps we need to reduce it again.

Comment 2 Randy MacLeod 2021-03-25 14:57:20 UTC

*** Bug 14319 has been marked as a duplicate of this bug. ***

Comment 3 Peter Bergin 2021-04-07 06:46:56 UTC

FWIW I also see similarities with another issue I had which I fixed in http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=6d14fe119c6e7d1d26ef33dfaf3b44ceff113546

Comment 4 Chee Yang 2021-05-12 11:18:02 UTC

happen during build 

qemumips64 at ubuntu1804-ty-1
https://autobuilder.yoctoproject.org/typhoon/#/builders/74/builds/3412/steps/12/logs/stdio


qemux86-world at ubuntu1804-ty-1
https://autobuilder.yoctoproject.org/typhoon/#/builders/52/builds/3401/steps/11/logs/stdio

Comment 5 Chee Yang 2021-05-13 09:57:53 UTC

https://autobuilder.yoctoproject.org/typhoon/#/builders/63/builds/3407/steps/12/logs/stdio
qemuppc
ubuntu1804-ty-1

Comment 6 Randy MacLeod 2021-05-20 15:06:27 UTC

http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder-helper/commit/?id=19b4e74b3960174238acc79fd112f55706bc7385

config.json: Set XZ limits to more reasonable values on autobuilderHEADmaster
The autobuilders have 128GB memory, we don't want them using 50% which is
the default, 5% should be enough. Also limit the number of threads down
from 48 to something reasonable. This may be partly causing some of our
performance issues?

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>