14310 – xz / liblzma memory allocation failure during reproducible build testing

Bug 14310 - xz / liblzma memory allocation failure during reproducible build testing

Summary: xz / liblzma memory allocation failure during reproducible build testing

Status:	RESOLVED FIXED

Alias:	None

Product:	Build Testing
Classification:	QA/Testing
Component:	general (show other bugs)
Version:	unspecified
Hardware:	x86 Multiple

Importance:	Medium+ normal
Target Milestone:	3.4 M1
Assignee:	Changqing Li
QA Contact:

URL:
Whiteboard:	AB-INT

Duplicates (1):	14319 (view as bug list)
Depends on:
Blocks:

Reported:	2021-03-19 16:26 UTC by Saul Wold
Modified:	2021-05-20 15:06 UTC (History)
CC List:	5 users (show)

See Also:
OS type for building Yocto:	---
Type of Regression:	---
Verified:
Documentation change:	No (bug/feature does not impact docs)

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.

Description Saul Wold 2021-03-19 16:26:29 UTC

During the reproducible build testing the following failure occured on the debian10-ty-1 machine (56cpu/128G) ram. There was a patch submitted about a year ago for a similar issue: http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=27ff81bfd880280607c79dce2f724c8bfafce02d




RPM build errors:
    Deprecated external dependency generator is used!
    Deprecated external dependency generator is used!
    Deprecated external dependency generator is used!
    Deprecated external dependency generator is used!
    Deprecated external dependency generator is used!
    liblzma: Memory allocation failed    liblzma: Memory allocation failed    liblzma: Memory allocation failed    liblzma: Memory allocation failed    liblzma: Memory allocation failedWARNING: exit code 1 from a shell command.


And


NOTE: Running task 10969 of 14696 (/home/pokybuild/yocto-worker/reproducible/build/meta/recipes-graphics/xorg-lib/libxv_1.0.11.bb:do_package_write_rpm)
ERROR: libxkbfile-1_1.1.0-r0 do_package_write_ipk: Fatal errors occurred in subprocesses:
Command 'PATH="/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/pokybuild/yocto-worker/reproducible/build/scripts:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/usr/bin/x86_64-poky-linux:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot/usr/bin/crossscripts:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/usr/sbin:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/usr/bin:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/sbin:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/bin:/home/pokybuild/yocto-worker/reproducible/build/bitbake/bin:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/hosttools" opkg-build -Z xz -a "--memlimit=50% --threads=56" libxkbfile-dbg /home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/deploy-ipks/core2-64' returned non-zero exit status 1.
Subprocess output:xz: (stdin): Cannot allocate memory
Command 'PATH="/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/pokybuild/yocto-worker/reproducible/build/scripts:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/usr/bin/x86_64-poky-linux:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot/usr/bin/crossscripts:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/usr/sbin:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/usr/bin:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/sbin:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/bin:/home/pokybuild/yocto-worker/reproducible/build/bitbake/bin:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/hosttools" opkg-build -Z xz -a "--memlimit=50% --threads=56" libxkbfile-src /home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/deploy-ipks/core2-64' returned non-zero exit status 1.
Subprocess output:xz: (stdin): Cannot allocate memory
Command 'PATH="/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/pokybuild/yocto-worker/reproducible/build/scripts:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/usr/bin/x86_64-poky-linux:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot/usr/bin/crossscripts:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/usr/sbin:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/usr/bin:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/sbin:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/bin:/home/pokybuild/yocto-worker/reproducible/build/bitbake/bin:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/hosttools" opkg-build -Z xz -a "--memlimit=50% --threads=56" libxkbfile /home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/deploy-ipks/core2-64' returned non-zero exit status 1.
Subprocess output:xz: (stdin): Cannot allocate memory
Command 'PATH="/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/sysroots-uninative/x86_64-linux/usr/bin:/home/pokybuild/yocto-worker/reproducible/build/scripts:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/usr/bin/x86_64-poky-linux:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot/usr/bin/crossscripts:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/usr/sbin:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/usr/bin:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/sbin:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/recipe-sysroot-native/bin:/home/pokybuild/yocto-worker/reproducible/build/bitbake/bin:/home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/hosttools" opkg-build -Z xz -a "--memlimit=50% --threads=56" libxkbfile-dev /home/pokybuild/yocto-worker/reproducible/build/build-st-13724/reproducibleB/tmp/work/core2-64-poky-linux/libxkbfile/1_1.1.0-r0/deploy-ipks/core2-64' returned non-zero exit status 1.
Subprocess output:xz: (stdin): Cannot allocate memory

Comment 1 Randy MacLeod 2021-03-25 14:41:26 UTC

We have tunes xz down in the past to lower memory usage so perhaps we need to reduce it again.

Comment 2 Randy MacLeod 2021-03-25 14:57:20 UTC

*** Bug 14319 has been marked as a duplicate of this bug. ***

Comment 3 Peter Bergin 2021-04-07 06:46:56 UTC

FWIW I also see similarities with another issue I had which I fixed in http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=6d14fe119c6e7d1d26ef33dfaf3b44ceff113546

Comment 4 Chee Yang 2021-05-12 11:18:02 UTC

happen during build 

qemumips64 at ubuntu1804-ty-1
https://autobuilder.yoctoproject.org/typhoon/#/builders/74/builds/3412/steps/12/logs/stdio


qemux86-world at ubuntu1804-ty-1
https://autobuilder.yoctoproject.org/typhoon/#/builders/52/builds/3401/steps/11/logs/stdio

Comment 5 Chee Yang 2021-05-13 09:57:53 UTC

https://autobuilder.yoctoproject.org/typhoon/#/builders/63/builds/3407/steps/12/logs/stdio
qemuppc
ubuntu1804-ty-1

Comment 6 Randy MacLeod 2021-05-20 15:06:27 UTC

http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder-helper/commit/?id=19b4e74b3960174238acc79fd112f55706bc7385

config.json: Set XZ limits to more reasonable values on autobuilderHEADmaster
The autobuilders have 128GB memory, we don't want them using 50% which is
the default, 5% should be enough. Also limit the number of threads down
from 48 to something reasonable. This may be partly causing some of our
performance issues?

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>