Bug 13675

Summary: runqemu pid handling race
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Richard Purdie <richard.purdie>
Component: Scripts and ToolsAssignee: Trevor Gamblin <tvgamblin>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium+ CC: randy.macleod, ross.burton
Version: 3.0   
Target Milestone: 3.1 M3   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Richard Purdie 2019-12-04 13:09:09 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/524

2019-12-04 01:07:03,746 - oe-selftest - INFO - ======================================================================
2019-12-04 01:07:03,747 - oe-selftest - INFO - FAIL: runtime_test.TestImage.test_testimage_dnf (subunit.RemotedTestCase)
2019-12-04 01:07:03,747 - oe-selftest - INFO - ----------------------------------------------------------------------
2019-12-04 01:07:03,747 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/selftest/cases/runtime_test.py", line 167, in test_testimage_dnf
    bitbake('-c testimage core-image-full-cmdline')
  File "/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/utils/commands.py", line 221, in bitbake
    return runCmd(cmd, ignore_status, timeout, output_log=output_log, **options)
  File "/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/utils/commands.py", line 199, in runCmd
    raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))
AssertionError: Command 'bitbake  -c testimage core-image-full-cmdline' returned non-zero exit status 1:
NOTE: Bitbake server didn't start within 5 seconds, waiting for 90
Loading cache...done.
Loaded 1334 entries from dependency cache.
NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION           = "1.44.0"
BUILD_SYS            = "x86_64-linux"
NATIVELSBSTRING      = "universal"
TARGET_SYS           = "x86_64-poky-linux"
MACHINE              = "qemux86-64"
DISTRO               = "poky"
DISTRO_VERSION       = "3.0"
TUNE_FEATURES        = "m64 core2"
TARGET_FPU           = ""
meta                 
meta-poky            
meta-yocto-bsp       = "master-next:f05b8c72a2641f079b43ca64f7d1d5789829e44a"
meta-selftest        = "master:03e930f209e97ce57f9f8ddca626418fb1c11424"

Initialising tasks...done.
Sstate summary: Wanted 2 Found 2 Missed 0 Current 178 (100% match, 100% complete)
NOTE: Executing Tasks
NOTE: Running setscene task 179 of 180 (virtual:native:/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/recipes-extended/cpio/cpio_2.12.bb:do_populate_sysroot_setscene)
NOTE: recipe cpio-native-2.12-r0: task do_populate_sysroot_setscene: Started
NOTE: recipe cpio-native-2.12-r0: task do_populate_sysroot_setscene: Succeeded
NOTE: Running setscene task 180 of 180 (virtual:native:/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/recipes-extended/cpio/cpio_2.12.bb:do_deploy_source_date_epoch_setscene)
NOTE: recipe cpio-native-2.12-r0: task do_deploy_source_date_epoch_setscene: Started
NOTE: recipe cpio-native-2.12-r0: task do_deploy_source_date_epoch_setscene: Succeeded
NOTE: Setscene tasks completed
NOTE: Running task 821 of 821 (/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/recipes-extended/images/core-image-full-cmdline.bb:do_testimage)
NOTE: recipe core-image-full-cmdline-1.0-r0: task do_testimage: Started
ERROR: core-image-full-cmdline-1.0-r0 do_testimage: Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:do_testimage(d)
     0003:
File: '/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/classes/testimage.bbclass', lineno: 114, function: do_testimage
     0110:    netstat -an
     0111:}
     0112:
     0113:python do_testimage() {
 *** 0114:    testimage_main(d)
     0115:}
     0116:
     0117:addtask testimage
     0118:do_testimage[nostamp] = "1"
File: '/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/classes/testimage.bbclass', lineno: 333, function: testimage_main
     0329:            results.stop()
     0330:            results = None
     0331:    finally:
     0332:        signal.signal(signal.SIGTERM, orig_sigterm_handler)
 *** 0333:        tc.target.stop()
     0334:
     0335:    # Show results (if we have them)
     0336:    if not results:
     0337:        bb.fatal('%s - FAILED - tests were interrupted during execution' % pn, forcelog=True)
File: '/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/core/target/qemu.py', lineno: 63, function: stop
     0059:            self.stop()
     0060:            raise RuntimeError("FAILED to start qemu - check the task log and the boot log")
     0061:
     0062:    def stop(self):
 *** 0063:        self.runner.stop()
File: '/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/utils/qemurunner.py', lineno: 409, function: stop
     0405:            self.threadsock = None
     0406:        self.qemupid = None
     0407:        self.ip = None
     0408:        if os.path.exists(self.qemu_pidfile):
 *** 0409:            os.remove(self.qemu_pidfile)
     0410:        if self.monitorpipe:
     0411:            self.monitorpipe.close()
     0412:
     0413:    def stop_qemu_system(self):
Exception: FileNotFoundError: [Errno 2] No such file or directory: 'pidfile_52857'

ERROR: Logfile of failure stored in: /home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-36211/tmp/work/qemux86_64-poky-linux/core-image-full-cmdline/1.0-r0/temp/log.do_testimage.52857
NOTE: recipe core-image-full-cmdline-1.0-r0: task do_testimage: Failed
ERROR: Task (/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/recipes-extended/images/core-image-full-cmdline.bb:do_testimage) failed with exit code '1'
NOTE: Tasks Summary: Attempted 821 tasks of which 820 didn't need to be rerun and 1 failed.
NOTE: The errors for this build are stored in /home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-36211/tmp/log/error-report/error_report_20191203212129.txt
You can send the errors to a reports server by running:
  send-error-report /home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/build-st-36211/tmp/log/error-report/error_report_20191203212129.txt [-s server]
NOTE: The contents of these logs will be posted in public if you use the above command with the default server. Please ensure you remove any identifying or proprietary information when prompted before sending.

Summary: 1 task failed:
  /home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/recipes-extended/images/core-image-full-cmdline.bb:do_testimage
Summary: There was 1 ERROR message shown, returning a non-zero exit code.
Comment 1 Richard Purdie 2020-02-06 08:51:38 UTC
Happened again: https://autobuilder.yoctoproject.org/typhoon/#/builders/72/builds/1562
Comment 2 Richard Purdie 2020-02-06 08:52:27 UTC
Trevor: This should be a simple fix (add a try/except exception handler) if you fancy an easier bug for a change?! :)
Comment 3 Richard Purdie 2020-02-07 13:39:04 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/59/builds/1531 (master-next)
https://autobuilder.yoctoproject.org/typhoon/#/builders/44/builds/1553 (zeus)

I'd suggest rather than reproduce we just write a fix for the issue?
Comment 4 Trevor Gamblin 2020-02-07 13:47:33 UTC
Was writing it as you posted. Sent to the mailing list.