Bug 14209

Summary: Intermittent core-image-sato-1.0-r0 do_testimage fails with: systemd-journald.service: Failed with result 'watchdog'
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Steve Sakoman <steve>
Component: coreAssignee: Unassigned <unassigned>
Status: RESOLVED WORKSFORME QA Contact:
Severity: normal    
Priority: Medium CC: flowergom, meta.mr.watcher, meta.watcher, randy.macleod
Version: 3.1.5   
Target Milestone: 3.1.8   
Hardware: x86   
OS: Multiple   
Whiteboard: AB-INT
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Steve Sakoman 2021-02-01 16:22:14 UTC 
qemuppc-alt on opensuse151-ty-1:

https://autobuilder.yoctoproject.org/typhoon/#/builders/107/builds/1318

NOTE:  ... FAIL
Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/qemuppc-alt/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/qemuppc-alt/build/meta/lib/oeqa/runtime/cases/parselogs.py", line 375, in test_parselogs
    self.assertEqual(errcount, 0, msg=self.msg)
AssertionError: 2 != 0 : Log: TOPDIR/tmp/work/qemuppc-poky-linux/core-image-sato/1.0-r0/target_logs/dmesg_output.log
-----------------------
Central error: [  328.406236] systemd[1]: systemd-journald.service: Failed with result 'watchdog'.
***********************
[  328.138400] [d716de08] [c023adc0] __kmalloc+0x234/0x3cc
[  328.138550] [d716de38] [c01e9b70] kvmalloc_node+0x60/0xdc
[  328.138695] [d716de58] [c02905f8] seq_read+0x190/0x628
[  328.138865] [d716deb8] [c02f8ff8] proc_reg_read+0x70/0xc0
[  328.139048] [d716ded8] [c0258f64] vfs_read+0x114/0x1d8
[  328.139221] [d716df08] [c0259124] ksys_read+0x84/0x140
[  328.139370] [d716df38] [c00181b8] ret_from_syscall+0x0/0x34
[  328.139521] --- interrupt: c01 at 0xfeed4f0
[  328.139521]     LR = 0xfeed4d0
[  328.397439] systemd[1]: systemd-journald.service: Main process exited, code=killed, status=6/ABRT
[  328.406236] systemd[1]: systemd-journald.service: Failed with result 'watchdog'.
[  328.477021] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1.
[  328.582160] systemd[1]: Stopping Flush Journal to Persistent Storage...
[  328.597881] systemd[1]: dropbear@0-192.168.7.6:22-192.168.7.5:43936.service: Succeeded.
[  328.701126] systemd[1]: systemd-journal-flush.service: Control process exited, code=exited, status=1/FAILURE
[  328.707075] systemd[1]: systemd-journal-flush.service: Failed with result 'exit-code'.
[  328.722050] systemd[1]: Stopped Flush Journal to Persistent Storage.
[  328.735729] systemd[1]: Stopped Journal Service.
[  328.741850] systemd[1]: Condition check resulted in Journal Audit Socket being skipped.
[  328.803818] systemd[1]: Starting Journal Service...
[  328.907499] systemd-journald[321]: File /run/log/journal/d8c602a4618e440f8b44cd767cc0e5b4/system.journal corrupted or uncleanly shut down, renaming and replacing.

***********************
Central error: [  328.707075] systemd[1]: systemd-journal-flush.service: Failed with result 'exit-code'.
***********************
Comment 1 Minjae Kim 2021-02-08 05:02:37 UTC 
Happened again:
https://autobuilder.yoctoproject.org/typhoon/#/builders/107/builds/1341/steps/13/logs/stdio

Central error: [  207.227279] systemd[1]: systemd-journald.service: Failed with result 'watchdog'.
***********************
[  206.885353] [ddf8dd78] [c00b0d40] rcu_sched_clock_irq+0x870/0xbf4
[  206.885562] [ddf8dde8] [c00bad8c] update_process_times+0x58/0x98
[  206.885764] [ddf8de08] [c00d1ac4] tick_sched_timer+0x88/0x2d4
[  206.885982] [ddf8de48] [c00bb934] __hrtimer_run_queues.constprop.0+0x1f0/0x46c
[  206.886222] [ddf8dea8] [c00bcb74] hrtimer_interrupt+0x164/0x41c
[  206.886418] [ddf8def8] [c001151c] timer_interrupt+0x18c/0x33c
[  206.886612] [ddf8df38] [c001853c] ret_from_except+0x0/0x14
[  206.886817] --- interrupt: 901 at 0xffd310c
                   LR = 0xffd2e74
[  207.165354] systemd[1]: systemd-journald.service: Main process exited, code=killed, status=6/ABRT
[  207.227279] systemd[1]: systemd-journald.service: Failed with result 'watchdog'.
[  207.546099] systemd[1]: systemd-hostnamed.service: Succeeded.
[  208.200694] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1.
[  208.377879] systemd[1]: Created slice system-sshd.slice.
[  208.822107] systemd[1]: Started OpenSSH Per-Connection Daemon (192.168.7.1:33426).
[  209.203906] systemd[1]: Stopping Flush Journal to Persistent Storage...
[  216.235225] systemd[1]: systemd-journal-flush.service: Control process exited, code=exited, status=1/FAILURE
[  216.260759] systemd[1]: systemd-journal-flush.service: Failed with result 'exit-code'.
[  216.396379] systemd[1]: Stopped Flush Journal to Persistent Storage.
[  216.473919] systemd[1]: Stopped Journal Service.
[  216.476361] systemd[1]: Condition check resulted in Journal Audit Socket being skipped.
***********************
Central error: [  216.260759] systemd[1]: systemd-journal-flush.service: Failed with result 'exit-code'.
***********************
[  206.886817] --- interrupt: 901 at 0xffd310c
                   LR = 0xffd2e74
[  207.165354] systemd[1]: systemd-journald.service: Main process exited, code=killed, status=6/ABRT
[  207.227279] systemd[1]: systemd-journald.service: Failed with result 'watchdog'.
[  207.546099] systemd[1]: systemd-hostnamed.service: Succeeded.
[  208.200694] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1.
[  208.377879] systemd[1]: Created slice system-sshd.slice.
[  208.822107] systemd[1]: Started OpenSSH Per-Connection Daemon (192.168.7.1:33426).
[  209.203906] systemd[1]: Stopping Flush Journal to Persistent Storage...
[  216.235225] systemd[1]: systemd-journal-flush.service: Control process exited, code=exited, status=1/FAILURE
[  216.260759] systemd[1]: systemd-journal-flush.service: Failed with result 'exit-code'.
[  216.396379] systemd[1]: Stopped Flush Journal to Persistent Storage.
[  216.473919] systemd[1]: Stopped Journal Service.
[  216.476361] systemd[1]: Condition check resulted in Journal Audit Socket being skipped.
[  219.083481] irq 36: nobody cared (try booting with the "irqpoll" option)
[  219.083850] CPU: 0 PID: 273 Comm: resize Not tainted 5.4.94-yocto-standard #1
[  219.084098] Call Trace:
[  219.084201] [dfff5f00] [c00a19f4] __report_bad_irq+0x50/0x138 (unreliable)
[  219.084347] [dfff5f30] [c00a190c] note_interrupt+0x324/0x378
[  219.084459] [dfff5f70] [c009dbf4] handle_irq_event+0xe8/0x104
[  219.084570] [dfff5fa0] [c00a29c4] handle_fasteoi_irq+0xc0/0x29c
Comment 2 Randy MacLeod 2021-05-27 15:13:37 UTC 
Serial issues tracked in other bugs.
The other problem hasn't happened recently.
At the time we were seeing some systemd problems so this may have been fixed.