Bug 14474

Summary: qemuppc ping test failure
Product: [QA/Testing] Runtime Testing Reporter: Richard Purdie <richard.purdie>
Component: oeqa/runtimeAssignee: Unassigned <unassigned>
Status: RESOLVED WORKSFORME QA Contact:
Severity: normal    
Priority: Medium CC: alexandre.belloni, bruce.ashfield, paulg, randy.macleod, sakib.sajal, steve
Version: unspecified   
Target Milestone: 4.0   
Hardware: x86   
OS: Multiple   
Whiteboard: AB-INT
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
Failure dmesg
none
Success dmesg none

Description Richard Purdie 2021-07-07 07:12:02 UTC 
https://autobuilder.yoctoproject.org/typhoon/#/builders/63/builds/3626

qemuppc ping test failure on debian10-ty-3

Note the "Configuring network interfaces... ip: RTNETLINK answers: File exists" below as that is likely related to the issue.


pokybuild@debian10-ty-3:~/yocto-worker/qemuppc/build/build-renamed/tmp/work/qemuppc-poky-linux/core-image-minimal/1.0-r0/testimage$ cat qemu_boot_log.20210707012539 
C>> annot manage 'undefined' PCI device type '<NULL>':
>>  1af4 1005 (0 ff 0)

>> =============================================================
>> OpenBIOS 1.1 [Mar 16 2021 08:16]
>> Configuration device id QEMU version 1 machine id 1
>> CPUs: 1
>> Memory: 256M
>> UUID: 00000000-0000-0000-0000-000000000000
>> CPU type PowerPC,G4
milliseconds isn't unique.
>> [ppc] Kernel already loaded (0x01000000 + 0x012739c0) (initrd 0x00000000 + 0x00000000)
>> [ppc] Kernel command line: root=/dev/vda rw  mem=256M ip=192.168.7.2::192.168.7.1:255.255.255.0 console=ttyS0 console=ttyS1 console=tty console=ttyS0  printk.time=1
>> switching to new context:
hwclock: can't open '/dev/misc/rtc': No such file or directory
chmod: /var/log/wtmp: No such file or directory
Failed to set mode -0664- for -/var/log/wtmp-.
INIT: Entering runlevel: 5
Configuring network interfaces... ip: RTNETLINK answers: File exists
hwclock: can't open '/dev/misc/rtc': No such file or directory
Starting syslogd/klogd: done

Poky (Yocto Project Reference Distro) 3.3+snapshot-8a3f38f2908b9c431cb0577f98ac955a9100319f qemuppc /dev/ttyS0

qemuppc login:
Comment 1 Richard Purdie 2021-07-07 07:15:59 UTC 
Looking at other successful qemuppc logs, th RTNETLINK message appears normal but the "chmod: /var/log/wtmp: No such file or directory
Failed to set mode -0664- for -/var/log/wtmp-" is not.
Comment 2 Richard Purdie 2021-07-07 09:22:05 UTC 
I reran the test manually in the same build directory with the same image. The most interesting thing was on this run:

QMP connected to QEMU at 07/07/21 07:26:56 and took 3.456681966781616 seconds
QMP released QEMU at 07/07/21 07:26:57 and took 1.1139729022979736 seconds from connect
runqemu - INFO - Copy done in 0.14536833763122559 seconds

whilst on the failed run:

QMP connected to QEMU at 07/07/21 01:26:34 and took 24.899512767791748 seconds
QMP released QEMU at 07/07/21 01:27:18 and took 43.95131969451904 seconds from connect
runqemu - INFO - Copy done in 0.26656150817871094 seconds

In some tests the wtmp error is present, in some it is not, it doesn't affect the ping success/failure. The RTNETLINK message is always there.

In one test run it took a long time, 

core-image-minimal () - Ran 66 tests in 17.148s
 vs
core-image-minimal () - Ran 66 tests in 2642.934s

and in the latter, the time seemed to be spent getting single characters over serial very slowly with periodic timeouts.
Comment 3 Richard Purdie 2021-07-07 09:31:01 UTC 
dmesg for the failing case shows boot in 13.4s, the success case was 9.2s. dmesg is pretty much identical except the usb stack init is later compared to the rest of the system in the success case. top and ps output show the same processes running with no significant looking differences.
Comment 4 Richard Purdie 2021-07-07 09:31:29 UTC 
Created attachment 4810 [details]
Failure dmesg
Comment 5 Richard Purdie 2021-07-07 09:31:43 UTC 
Created attachment 4811 [details]
Success dmesg
Comment 6 Randy MacLeod 2021-07-08 15:00:06 UTC 
Netconsole might be an option.
Comment 7 Randy MacLeod 2021-10-21 18:19:10 UTC 
I think that Paul has found a worlflow that might help with qemuppc hangs...
Comment 8 Randy MacLeod 2022-03-03 15:43:25 UTC 
Not seen in months.
Comment 9 Steve Sakoman 2024-03-01 22:27:31 UTC 
Occurred again in the dunfell 3.1.32 release build:

https://autobuilder.yoctoproject.org/typhoon/#/builders/63/builds/8596/steps/14/logs/stdio

Not reopening though since it happens less than once a year.