Bug 13844

Summary: Logrotate failure in systemd
Product: [QA/Testing] Runtime Testing Reporter: Richard Purdie <richard.purdie>
Component: oeqa/runtimeAssignee: Trevor Gamblin <tvgamblin>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium+ CC: randy.macleod, tvgamblin
Version: unspecified   
Target Milestone: 3.2 M2   
Hardware: x86   
OS: Multiple   
Whiteboard: AB-INT
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
Log from failing task none

Description Richard Purdie 2020-03-26 09:45:01 UTC 
There were two similar failures with systemd's logrotate service failing:

qemuarm:
https://autobuilder.yoctoproject.org/typhoon/#/builders/110/builds/629
qemumips:
https://autobuilder.yoctoproject.org/typhoon/#/builders/102/builds/396

Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/qemuarm-alt/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/qemuarm-alt/build/meta/lib/oeqa/runtime/cases/systemd.py", line 98, in test_systemd_failed
    self.assertTrue(match, msg='Some systemd units failed:\n%s' % output)
AssertionError: None is not true : Some systemd units failed:
  UNIT              LOAD   ACTIVE SUB    DESCRIPTION     
● logrotate.service loaded failed failed Rotate log files
Comment 1 Richard Purdie 2020-03-26 09:53:26 UTC 
The qemuarm builds tmpdir was gone but qemumips was there. From the logs:

Mar 25 23:59:08 qemumips systemd[1]: Started Session c179 of user root.
Mar 25 23:59:10 qemumips systemd[1]: Started OpenSSH Per-Connection Daemon (192.168.7.1:45286).
Mar 25 23:59:11 qemumips systemd[1]: sshd@177-192.168.7.2:22-192.168.7.1:45284.service: Succeeded.
Mar 25 23:59:11 qemumips systemd[1]: session-c179.scope: Succeeded.
Mar 25 23:59:11 qemumips systemd-logind[200]: Session c179 logged out. Waiting for processes to exit.
Mar 25 23:59:11 qemumips systemd-logind[200]: Removed session c179.
Mar 25 23:59:13 qemumips systemd-logind[200]: New session c180 of user root.
Mar 25 23:59:14 qemumips systemd[1]: Started Session c180 of user root.
Mar 25 23:59:37 qemumips systemd-journald[128]: Forwarding to syslog missed 14 messages.
Mar 26 00:00:02 qemumips systemd[1]: Starting Rotate log files...
Mar 26 00:00:06 qemumips logrotate[8831]: error: logrotate_test:1 lines must begin with a keyword or a filename (possibly in double quotes)
Mar 26 00:00:06 qemumips logrotate[8831]: error: stat of /var/log/logrotate_test failed: No such file or directory
Mar 26 00:00:06 qemumips systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
Mar 26 00:00:06 qemumips systemd[1]: logrotate.service: Failed with result 'exit-code'.
Mar 26 00:00:07 qemumips systemd[1]: Failed to start Rotate log files.
Mar 26 00:01:27 qemumips systemd-journald[128]: Forwarding to syslog missed 6 messages.
Mar 26 00:48:32 qemumips kernel: hrtimer: interrupt took 14266360 ns
Mar 26 00:48:32 qemumips kernel[180]: [ 7303.564172] hrtimer: interrupt took 14266360 ns
Mar 26 00:48:32 qemumips systemd-journald[128]: Forwarding to syslog missed 1 messages.
Mar 26 00:51:05 qemumips kernel: stap_9488 (hello.stp): systemtap: 4.2/0.178, base: d72dde3c, memory: 17data/30text/0ctx/2063net/46alloc kb, probes: 1
Mar 26 00:51:05 qemumips kernel[180]: [ 7457.107427] stap_9488 (hello.stp): systemtap: 4.2/0.178, base: d72dde3c, memory: 17data/30text/0ctx/2063net/46alloc kb, probes: 1
Mar 26 00:51:05 qemumips systemd-journald[128]: Forwarding to syslog missed 1 messages.
Mar 26 00:51:18 qemumips systemd[1]: sshd@178-192.168.7.2:22-192.168.7.1:45286.service: Succeeded.
Mar 26 00:51:18 qemumips systemd[1]: session-c180.scope: Succeeded.
Mar 26 00:51:18 qemumips systemd-logind[200]: Session c180 logged out. Waiting for processes to exit.
Mar 26 00:51:19 qemumips systemd[1]: Started OpenSSH Per-Connection Daemon (192.168.7.1:45642).
Mar 26 00:51:19 qemumips systemd-logind[200]: Removed session c180.
Mar 26 00:51:20 qemumips systemd-logind[200]: New session c181 of user root.
Comment 2 Richard Purdie 2020-03-26 09:54:56 UTC 
Created attachment 4653 [details]
Log from failing task
Comment 4 Trevor Gamblin 2020-04-22 11:35:10 UTC 
Has this been seen lately? Still trying to recreate.
Comment 5 Richard Purdie 2020-05-22 08:24:57 UTC 
Another systemd logrotate failure

https://autobuilder.yoctoproject.org/typhoon/#/builders/102/builds/537/steps/8/logs/step1c
Comment 9 Trevor Gamblin 2020-07-06 16:10:19 UTC 
I think we should give it a week or two and see if we still see the issue. In the meantime I'll see if I can test it using the info given in the commit message.
Comment 10 Trevor Gamblin 2020-07-23 07:54:20 UTC 
Closing this since we haven't seen it since commit http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=893db513d720e8100b3dff005c392434657e446a was merged.