14331 – Memory issues during oe-selftest

Bug 14331 - Memory issues during oe-selftest

Summary: Memory issues during oe-selftest

Status:	RESOLVED FIXED

Alias:	None

Product:	Functional (self) Testing
Classification:	QA/Testing
Component:	oe-selftest (show other bugs)
Version:	unspecified
Hardware:	x86 Multiple

Importance:	Medium+ normal
Target Milestone:	3.4 M1
Assignee:	Trevor Gamblin
QA Contact:

URL:
Whiteboard:	AB-INT

Depends on:
Blocks:

Reported:	2021-03-29 03:34 UTC by Paul Eggleton
Modified:	2021-05-20 15:08 UTC (History)
CC List:	2 users (show)

See Also:
OS type for building Yocto:	---
Type of Regression:	---
Verified:
Documentation change:	No (bug/feature does not impact docs)

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.

Description Paul Eggleton 2021-03-29 03:34:07 UTC

https://autobuilder.yoctoproject.org/typhoon/#builders/59/builds/3238

-------------- snip --------------
NOTE: recipe glibc-testsuite-2.33-r0: task do_check: Started
ERROR: glibc-testsuite-2.33-r0 do_check: Execution of event handler 'defaultrun_buildstats' failed
Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/qemux86/build/meta/classes/buildstats.bbclass", line 224, in defaultrun_buildstats(e=<bb.build.TaskSucceeded object at 0x7fc608caed00>):
         elif isinstance(e, bb.build.TaskSucceeded):
    >        write_task_data("passed", os.path.join(taskdir, e.task), e, d)
             if e.task == "do_rootfs":
  File "/home/pokybuild/yocto-worker/qemux86/build/meta/classes/buildstats.bbclass", line 88, in write_task_data(status='passed', logfile='/home/pokybuild/yocto-worker/qemux86/build/build-st-12650/tmp/buildstats/20210328015616/glibc-testsuite-2.33-r0/do_check', e=<bb.build.TaskSucceeded object at 0x7fc608caed00>, d=<bb.data_smart.DataSmart object at 0x7fc60bb6b5e0>):
                 f.write(d.expand("Elapsed time: %0.2f seconds\n" % elapsedtime))
    >            cpu, iostats, resources, childres = get_process_cputime(os.getpid())
                 if cpu:
  File "/home/pokybuild/yocto-worker/qemux86/build/meta/classes/buildstats.bbclass", line 18, in get_process_cputime(pid=5887):
     def get_process_cputime(pid):
    >    import resource
         with open("/proc/%d/stat" % pid, "r") as f:
ImportError: /home/pokybuild/yocto-worker/qemux86/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.9/lib-dynload/resource.cpython-39-x86_64-linux-gnu.so: failed to map segment from shared object
ERROR: glibc-testsuite-2.33-r0 do_check: Build of do_check failed
ERROR: glibc-testsuite-2.33-r0 do_check: Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/build.py", line 752, in exec_task
    return _exec_task(fn, task, d, quieterr)
  File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/build.py", line 726, in _exec_task
    event.fire(TaskSucceeded(task, fn, logfn, localdata), localdata)
  File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/event.py", line 216, in fire
    fire_class_handlers(event, d)
  File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/event.py", line 123, in fire_class_handlers
    execute_handler(name, handler, event, d)
  File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/event.py", line 93, in execute_handler
    ret = handler(event)
  File "/home/pokybuild/yocto-worker/qemux86/build/meta/classes/buildstats.bbclass", line 224, in defaultrun_buildstats
    write_task_data("passed", os.path.join(taskdir, e.task), e, d)
  File "/home/pokybuild/yocto-worker/qemux86/build/meta/classes/buildstats.bbclass", line 88, in write_task_data
    cpu, iostats, resources, childres = get_process_cputime(os.getpid())
  File "/home/pokybuild/yocto-worker/qemux86/build/meta/classes/buildstats.bbclass", line 18, in get_process_cputime
    import resource
ImportError: /home/pokybuild/yocto-worker/qemux86/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.9/lib-dynload/resource.cpython-39-x86_64-linux-gnu.so: failed to map segment from shared object
ERROR: Task (/home/pokybuild/yocto-worker/qemux86/build/meta/recipes-core/glibc/glibc-testsuite_2.33.bb:do_check) failed with exit code '1'
NOTE: Tasks Summary: Attempted 340 tasks of which 333 didn't need to be rerun and 1 failed.
Summary: 1 task failed:
  /home/pokybuild/yocto-worker/qemux86/build/meta/recipes-core/glibc/glibc-testsuite_2.33.bb:do_check
Summary: There were 3 ERROR messages shown, returning a non-zero exit code.
2021-03-28 03:03:42,969 - oe-selftest - INFO - ======================================================================
2021-03-28 03:03:42,969 - oe-selftest - INFO - FAIL: glibc.GlibcSelfTestSystemEmulated.test_glibc (subunit.RemotedTestCase)
2021-03-28 03:03:42,969 - oe-selftest - INFO - ----------------------------------------------------------------------
2021-03-28 03:03:42,969 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/qemux86/build/meta/lib/oeqa/selftest/cases/glibc.py", line 88, in test_glibc
    self.run_check_emulated()
  File "/home/pokybuild/yocto-worker/qemux86/build/meta/lib/oeqa/selftest/cases/glibc.py", line 78, in run_check_emulated
    self.run_check(ssh = qemu.ip)
  File "/home/pokybuild/yocto-worker/qemux86/build/meta/lib/oeqa/selftest/cases/glibc.py", line 30, in run_check
    bitbake("glibc-testsuite -c check")
  File "/home/pokybuild/yocto-worker/qemux86/build/meta/lib/oeqa/utils/commands.py", line 232, in bitbake
    return runCmd(cmd, ignore_status, timeout, output_log=output_log, **options)
  File "/home/pokybuild/yocto-worker/qemux86/build/meta/lib/oeqa/utils/commands.py", line 210, in runCmd
    raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))
AssertionError: Command 'bitbake  glibc-testsuite -c check' returned non-zero exit status 1:
Loading cache...done.
Loaded 0 entries from dependency cache.
Parsing recipes...done.
Parsing of 855 .bb files complete (0 cached, 855 parsed). 1486 targets, 46 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies
Build Configuration:
BB_VERSION           = "1.49.2"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "i686-poky-linux"
MACHINE              = "qemux86"
DISTRO               = "poky"
DISTRO_VERSION       = "3.2+snapshot-ea455ca8671d3bc2a1097989bfaabe92f3ca37ab"
TUNE_FEATURES        = "m32 core2"
TARGET_FPU           = ""
meta                 
meta-poky            
meta-yocto-bsp       = "master:ea455ca8671d3bc2a1097989bfaabe92f3ca37ab"
meta-selftest        = "master:81398c7e463d5e149913864f71c1c55afef62e93"
Initialising tasks...done.
Sstate summary: Wanted 5 Local 5 Network 0 Missed 0 Current 72 (100% match, 100% complete)
NOTE: Executing Tasks
NOTE: Running setscene task 73 of 77 (/home/pokybuild/yocto-worker/qemux86/build/meta/recipes-core/glibc/glibc-testsuite_2.33.bb:do_deploy_source_date_epoch_setscene)
NOTE: recipe glibc-testsuite-2.33-r0: task do_deploy_source_date_epoch_setscene: Started
NOTE: recipe glibc-testsuite-2.33-r0: task do_deploy_source_date_epoch_setscene: Succeeded
NOTE: Setscene tasks completed
NOTE: Running task 333 of 340 (/home/pokybuild/yocto-worker/qemux86/build/meta/recipes-core/glibc/glibc-testsuite_2.33.bb:do_fetch)
NOTE: recipe glibc-testsuite-2.33-r0: task do_fetch: Started
NOTE: recipe glibc-testsuite-2.33-r0: task do_fetch: Succeeded
NOTE: Running task 334 of 340 (/home/pokybuild/yocto-worker/qemux86/build/meta/recipes-core/glibc/glibc-testsuite_2.33.bb:do_unpack)
NOTE: Running task 335 of 340 (/home/pokybuild/yocto-worker/qemux86/build/meta/recipes-core/glibc/glibc-testsuite_2.33.bb:do_prepare_recipe_sysroot)
NOTE: recipe glibc-testsuite-2.33-r0: task do_prepare_recipe_sysroot: Started
NOTE: recipe glibc-testsuite-2.33-r0: task do_unpack: Started
NOTE: recipe glibc-testsuite-2.33-r0: task do_prepare_recipe_sysroot: Succeeded
NOTE: recipe glibc-testsuite-2.33-r0: task do_unpack: Succeeded
NOTE: Running task 336 of 340 (/home/pokybuild/yocto-worker/qemux86/build/meta/recipes-core/glibc/glibc-testsuite_2.33.bb:do_patch)
NOTE: recipe glibc-testsuite-2.33-r0: task do_patch: Started
NOTE: recipe glibc-testsuite-2.33-r0: task do_patch: Succeeded
NOTE: Running task 338 of 340 (/home/pokybuild/yocto-worker/qemux86/build/meta/recipes-core/glibc/glibc-testsuite_2.33.bb:do_configure)
NOTE: recipe glibc-testsuite-2.33-r0: task do_configure: Started
NOTE: recipe glibc-testsuite-2.33-r0: task do_configure: Succeeded
NOTE: Running task 339 of 340 (/home/pokybuild/yocto-worker/qemux86/build/meta/recipes-core/glibc/glibc-testsuite_2.33.bb:do_compile)
NOTE: recipe glibc-testsuite-2.33-r0: task do_compile: Started
NOTE: recipe glibc-testsuite-2.33-r0: task do_compile: Succeeded
NOTE: Running task 340 of 340 (/home/pokybuild/yocto-worker/qemux86/build/meta/recipes-core/glibc/glibc-testsuite_2.33.bb:do_check)
NOTE: recipe glibc-testsuite-2.33-r0: task do_check: Started
ERROR: glibc-testsuite-2.33-r0 do_check: Execution of event handler 'defaultrun_buildstats' failed
Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/qemux86/build/meta/classes/buildstats.bbclass", line 224, in defaultrun_buildstats(e=<bb.build.TaskSucceeded object at 0x7fc608caed00>):
         elif isinstance(e, bb.build.TaskSucceeded):
    >        write_task_data("passed", os.path.join(taskdir, e.task), e, d)
             if e.task == "do_rootfs":
  File "/home/pokybuild/yocto-worker/qemux86/build/meta/classes/buildstats.bbclass", line 88, in write_task_data(status='passed', logfile='/home/pokybuild/yocto-worker/qemux86/build/build-st-12650/tmp/buildstats/20210328015616/glibc-testsuite-2.33-r0/do_check', e=<bb.build.TaskSucceeded object at 0x7fc608caed00>, d=<bb.data_smart.DataSmart object at 0x7fc60bb6b5e0>):
                 f.write(d.expand("Elapsed time: %0.2f seconds\n" % elapsedtime))
    >            cpu, iostats, resources, childres = get_process_cputime(os.getpid())
                 if cpu:
  File "/home/pokybuild/yocto-worker/qemux86/build/meta/classes/buildstats.bbclass", line 18, in get_process_cputime(pid=5887):
     def get_process_cputime(pid):
    >    import resource
         with open("/proc/%d/stat" % pid, "r") as f:
ImportError: /home/pokybuild/yocto-worker/qemux86/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.9/lib-dynload/resource.cpython-39-x86_64-linux-gnu.so: failed to map segment from shared object
ERROR: glibc-testsuite-2.33-r0 do_check: Build of do_check failed
ERROR: glibc-testsuite-2.33-r0 do_check: Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/build.py", line 752, in exec_task
    return _exec_task(fn, task, d, quieterr)
  File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/build.py", line 726, in _exec_task
    event.fire(TaskSucceeded(task, fn, logfn, localdata), localdata)
  File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/event.py", line 216, in fire
    fire_class_handlers(event, d)
  File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/event.py", line 123, in fire_class_handlers
    execute_handler(name, handler, event, d)
  File "/home/pokybuild/yocto-worker/qemux86/build/bitbake/lib/bb/event.py", line 93, in execute_handler
    ret = handler(event)
  File "/home/pokybuild/yocto-worker/qemux86/build/meta/classes/buildstats.bbclass", line 224, in defaultrun_buildstats
    write_task_data("passed", os.path.join(taskdir, e.task), e, d)
  File "/home/pokybuild/yocto-worker/qemux86/build/meta/classes/buildstats.bbclass", line 88, in write_task_data
    cpu, iostats, resources, childres = get_process_cputime(os.getpid())
  File "/home/pokybuild/yocto-worker/qemux86/build/meta/classes/buildstats.bbclass", line 18, in get_process_cputime
    import resource
ImportError: /home/pokybuild/yocto-worker/qemux86/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.9/lib-dynload/resource.cpython-39-x86_64-linux-gnu.so: failed to map segment from shared object
ERROR: Task (/home/pokybuild/yocto-worker/qemux86/build/meta/recipes-core/glibc/glibc-testsuite_2.33.bb:do_check) failed with exit code '1'
NOTE: Tasks Summary: Attempted 340 tasks of which 333 didn't need to be rerun and 1 failed.
Summary: 1 task failed:
  /home/pokybuild/yocto-worker/qemux86/build/meta/recipes-core/glibc/glibc-testsuite_2.33.bb:do_check
Summary: There were 3 ERROR messages shown, returning a non-zero exit code.
----------------------------------------------------------------------
2021-03-28 03:03:42,970 - oe-selftest - INFO - Ran 15 tests in 4465.873s
2021-03-28 03:03:42,970 - oe-selftest - INFO - FAILED
2021-03-28 03:03:42,970 - oe-selftest - INFO -  (failures=1, skipped=1)
Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/qemux86/build/scripts/oe-selftest", line 60, in <module>
    ret = main()
  File "/home/pokybuild/yocto-worker/qemux86/build/scripts/oe-selftest", line 47, in main
    results = args.func(logger, args)
  File "/home/pokybuild/yocto-worker/qemux86/build/meta/lib/oeqa/selftest/context.py", line 400, in run
    rc = self._internal_run(logger, args)
  File "/home/pokybuild/yocto-worker/qemux86/build/meta/lib/oeqa/selftest/context.py", line 364, in _internal_run
    configuration = self.get_configuration(args)
  File "/home/pokybuild/yocto-worker/qemux86/build/meta/lib/oeqa/selftest/context.py", line 336, in get_configuration
    metadata = metadata_from_bb()
  File "/home/pokybuild/yocto-worker/qemux86/build/meta/lib/oeqa/utils/metadata.py", line 42, in metadata_from_bb
    info_dict['layers'] = get_layers(data_dict['BBLAYERS'])
  File "/home/pokybuild/yocto-worker/qemux86/build/meta/lib/oeqa/utils/metadata.py", line 97, in get_layers
    layer_dict[layer_name] = git_rev_info(layer)
  File "/home/pokybuild/yocto-worker/qemux86/build/meta/lib/oeqa/utils/metadata.py", line 84, in git_rev_info
    info['commit'] = repo.head.commit.hexsha
  File "/home/pokybuild/yocto-worker/qemux86/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.9/site-packages/git/refs/symbolic.py", line 197, in _get_commit
    obj = self._get_object()
  File "/home/pokybuild/yocto-worker/qemux86/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.9/site-packages/git/refs/symbolic.py", line 190, in _get_object
    return Object.new_from_sha(self.repo, hex_to_bin(self.dereference_recursive(self.repo, self.path)))
  File "/home/pokybuild/yocto-worker/qemux86/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.9/site-packages/git/objects/base.py", line 64, in new_from_sha
    oinfo = repo.odb.info(sha1)
  File "/home/pokybuild/yocto-worker/qemux86/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.9/site-packages/git/db.py", line 37, in info
    hexsha, typename, size = self._git.get_object_header(bin_to_hex(sha))
  File "/home/pokybuild/yocto-worker/qemux86/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.9/site-packages/git/cmd.py", line 1068, in get_object_header
    cmd = self._get_persistent_cmd("cat_file_header", "cat_file", batch_check=True)
  File "/home/pokybuild/yocto-worker/qemux86/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.9/site-packages/git/cmd.py", line 1051, in _get_persistent_cmd
    cmd = self._call_process(cmd_name, *args, **options)
  File "/home/pokybuild/yocto-worker/qemux86/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.9/site-packages/git/cmd.py", line 1006, in _call_process
    return self.execute(call, **exec_kwargs)
  File "/home/pokybuild/yocto-worker/qemux86/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.9/site-packages/git/cmd.py", line 721, in execute
    proc = Popen(command,
  File "/home/pokybuild/yocto-worker/qemux86/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.9/subprocess.py", line 947, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/home/pokybuild/yocto-worker/qemux86/build/buildtools/sysroots/x86_64-pokysdk-linux/usr/lib/python3.9/subprocess.py", line 1752, in _execute_child
    self.pid = _posixsubprocess.fork_exec(
OSError: [Errno 12] Cannot allocate memory
-------------- snip --------------

Comment 1 Randy MacLeod 2021-04-01 14:42:25 UTC

May be related to 14289.

Comment 2 Alexandre Belloni 2021-05-20 15:08:00 UTC

Probably fixed by c124290d90040806103b8c273948c4863e7bc1b4 "glibc: Add 8GB VM usage cap for usermode test suite"