Bug 11895

Summary: Nightly-arm: do_testimage failing on Fedora26
Product: [QA/Testing] Runtime Testing Reporter: Sonia Leon <sonia.leon.bautista>
Component: generalAssignee: brian avery <brian.avery>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium+ CC: joshuagloe, richard.purdie
Version: unspecified   
Target Milestone: 2.4 M4   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
core image-minimal log
none
core image-sato log
none
core image-sato-sdk log none

Description Sonia Leon 2017-08-03 23:36:39 UTC 
While testing fedora26, nightly-arm Sanity test is failing with following error: 


NOTE: core-image-minimal-1.0-r0 do_testimage: Check full boot log: /home/ab/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-minimal/1.0-r0/testimage/qemu_boot_log.20170802002313
WARNING: core-image-minimal-1.0-r0 do_testimage: Qemu ended unexpectedly, dump data from host is in /tmp/oe-saved-tests/201708012024_qemu
WARNING: core-image-minimal-1.0-r0 do_testimage: qemu-system ended unexpectedly
NOTE: core-image-minimal-1.0-r0 do_testimage: Sending SIGTERM to runqemu
ERROR: core-image-minimal-1.0-r0 do_testimage: testimage received SIGTERM, shutting down...
ERROR: core-image-minimal-1.0-r0 do_testimage: core-image-minimal - FAILED - tests were interrupted during execution
ERROR: core-image-minimal-1.0-r0 do_testimage: Function failed: do_testimage
ERROR: Logfile of failure stored in: /home/ab/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-minimal/1.0-r0/temp/log.do_testimage.2269
NOTE: recipe core-image-minimal-1.0-r0: task do_testimage: Failed
ERROR: Task (/home/ab/yocto-autobuilder/yocto-worker/nightly-arm/build/meta/recipes-core/images/core-image-minimal.bb:do_testimage) failed with exit code '1'
NOTE: core-image-sato-sdk-1.0-r0 do_testimage: Qemu pid didn't appeared in 60 seconds
WARNING: core-image-sato-sdk-1.0-r0 do_testimage: Qemu ended unexpectedly, dump data from host is in /tmp/oe-saved-tests/201708012025_qemu
NOTE: core-image-sato-1.0-r0 do_testimage: Qemu pid didn't appeared in 60 seconds
WARNING: core-image-sato-1.0-r0 do_testimage: Qemu ended unexpectedly, dump data from host is in /tmp/oe-saved-tests/201708012025_qemu
NOTE: core-image-sato-sdk-1.0-r0 do_testimage: Sending SIGTERM to runqemu
NOTE: core-image-sato-sdk-1.0-r0 do_testimage: Output from runqemu:

ERROR: core-image-sato-sdk-1.0-r0 do_testimage: testimage received SIGTERM, shutting down...
ERROR: core-image-sato-sdk-1.0-r0 do_testimage: core-image-sato-sdk - FAILED - tests were interrupted during execution
ERROR: core-image-sato-sdk-1.0-r0 do_testimage: Function failed: do_testimage
NOTE: core-image-sato-1.0-r0 do_testimage: Sending SIGTERM to runqemu
ERROR: Logfile of failure stored in: /home/ab/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-sato-sdk/1.0-r0/temp/log.do_testimage.2267
NOTE: recipe core-image-sato-sdk-1.0-r0: task do_testimage: Failed
NOTE: core-image-sato-1.0-r0 do_testimage: Output from runqemu:

ERROR: core-image-sato-1.0-r0 do_testimage: testimage received SIGTERM, shutting down...
ERROR: core-image-sato-1.0-r0 do_testimage: core-image-sato - FAILED - tests were interrupted during execution
ERROR: core-image-sato-1.0-r0 do_testimage: Function failed: do_testimage
ERROR: Task (/home/ab/yocto-autobuilder/yocto-worker/nightly-arm/build/meta/recipes-sato/images/core-image-sato-sdk.bb:do_testimage) failed with exit code '1'
ERROR: Logfile of failure stored in: /home/ab/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/work/qemuarm-poky-linux-gnueabi/core-image-sato/1.0-r0/temp/log.do_testimage.2268
NOTE: recipe core-image-sato-1.0-r0: task do_testimage: Failed
ERROR: Task (/home/ab/yocto-autobuilder/yocto-worker/nightly-arm/build/meta/recipes-sato/images/core-image-sato.bb:do_testimage) failed with exit code '1'
NOTE: Tasks Summary: Attempted 565 tasks of which 554 didn't need to be rerun and 3 failed.
NOTE: The errors for this build are stored in /home/ab/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/log/error-report/error_report_20170802002313.txt
You can send the errors to a reports server by running:
  send-error-report /home/ab/yocto-autobuilder/yocto-worker/nightly-arm/build/build/tmp/log/error-report/error_report_20170802002313.txt [-s server]
NOTE: The contents of these logs will be posted in public if you use the above command with the default server. Please ensure you remove any identifying or proprietary information when prompted before sending.
NOTE: Writing buildhistory
+ buildhistory_commit
+ '[' '!' -d /tmp/yocto-autobuilder/buildhistory/poky/master/nightly-arm ']'
+ cat

+ cd /tmp/yocto-autobuilder/buildhistory/poky/master/nightly-arm/

+ '[' '!' -e .git ']'
+ git tag -f build-minus-3 build-minus-2

+ true
+ git tag -f build-minus-2 build-minus-1

+ git tag -f build-minus-1

+ check_git_config

+ git config user.email

+ git config user.name

++ git status --porcelain

++ grep -v ' metadata-revs$'

+ repostatus=

++ hostname

+ HOSTNAME=fedora26
+ CMDLINE='bitbake -v core-image-minimal core-image-sato core-image-sato-sdk -c testimage'
+ '[' '' '!=' '' ']'
+ buildhistory_single_commit 'bitbake -v core-image-minimal core-image-sato core-image-sato-sdk -c testimage' fedora26

+ '[' '' = '' ']'
+ commitopts='/tmp/yocto-autobuilder/buildhistory/poky/master/nightly-arm/ --allow-empty'
+ shortlogprefix='No changes: '
+ '[' 3 = 0 ']'
+ result=failed
+ case 0 in
++ mktemp

+ commitmsgfile=/tmp/tmp.l5lKzB4xUj
+ cat

+ cat /tmp/yocto-autobuilder/buildhistory/poky/master/nightly-arm/metadata-revs

+ git commit /tmp/yocto-autobuilder/buildhistory/poky/master/nightly-arm/ --allow-empty -F /tmp/tmp.l5lKzB4xUj --author 'buildhistory <buildhistory@poky>'

+ rm /tmp/tmp.l5lKzB4xUj

+ '[' 'file:////tmp/yocto-autobuilder/buildhistory-repo poky/master/nightly-arm:poky/master/nightly-arm' '!=' '' ']'
+ git push -q file:////tmp/yocto-autobuilder/buildhistory-repo poky/master/nightly-arm:poky/master/nightly-arm

fatal: '//tmp/yocto-autobuilder/buildhistory-repo' does not appear to be a git repository

fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

+ true
+ ret=0
+ trap '' 0
+ exit 0


Summary: 3 tasks failed:
  /home/ab/yocto-autobuilder/yocto-worker/nightly-arm/build/meta/recipes-core/images/core-image-minimal.bb:do_testimage
  /home/ab/yocto-autobuilder/yocto-worker/nightly-arm/build/meta/recipes-sato/images/core-image-sato-sdk.bb:do_testimage
  /home/ab/yocto-autobuilder/yocto-worker/nightly-arm/build/meta/recipes-sato/images/core-image-sato.bb:do_testimage
Summary: There were 4 WARNING messages shown.
Summary: There were 9 ERROR messages shown, returning a non-zero exit code.
program finished with exit code 1
elapsedTime=178.194673
Comment 1 Joshua Lock 2017-08-07 18:48:52 UTC 
This doesn't appear to be a code defect in yocto-autobuilder.

It looks like qemu is failing to start, could you please attach the qemu_boot_log.* and log.do_testimage.* files referenced in the output?

Is this error only seen with nightly-arm buildset on fedora26?

Also, it looks like the yocto-autobuilder instance is configured to push buildhistory but the target repository doesn't exist.
Comment 2 Joshua Lock 2017-08-10 15:17:59 UTC 
Sonia, could you provide the requested logs? We're running fedora26 on the public AB and haven't seen this issue.
Comment 3 Sonia Leon 2017-08-29 18:12:42 UTC 
Created attachment 3958 [details]
core image-minimal log
Comment 4 Sonia Leon 2017-08-29 18:13:24 UTC 
Created attachment 3959 [details]
core image-sato log
Comment 5 Sonia Leon 2017-08-29 18:14:02 UTC 
Created attachment 3960 [details]
core image-sato-sdk log
Comment 6 Sonia Leon 2017-08-29 18:22:02 UTC 
(In reply to comment #2)
> Sonia, could you provide the requested logs? We're running fedora26 on the
> public AB and haven't seen this issue.

I have included log.do_testimage.* files, the qemu_boot_log.* is not present, even it is menioned in main log. 

First test i ran was with 6964efddd31c479386d1643c1025bc102710392f and yesterday i ran a newest review cd26fc143bbfd948795c2e65a577f89627a4f841 and error is exactly the same.

As additional info, these are the qemu packages and versions installed in my fedora26, in case it could helps. 

ipxe-roms-qemu.noarch                  20161108-2.gitb991c67.fc26
libvirt-daemon-driver-qemu.x86_64      3.2.1-5.fc26             @updates        
qemu-block-curl.x86_64                 2:2.9.0-5.fc26           @updates        
qemu-block-dmg.x86_64                  2:2.9.0-5.fc26           @updates        
qemu-block-gluster.x86_64              2:2.9.0-5.fc26           @updates        
qemu-block-iscsi.x86_64                2:2.9.0-5.fc26           @updates        
qemu-block-nfs.x86_64                  2:2.9.0-5.fc26           @updates        
qemu-block-rbd.x86_64                  2:2.9.0-5.fc26           @updates        
qemu-block-ssh.x86_64                  2:2.9.0-5.fc26           @updates        
qemu-common.x86_64                     2:2.9.0-5.fc26           @updates        
qemu-guest-agent.x86_64                2:2.9.0-5.fc26           @updates        
qemu-img.x86_64                        2:2.9.0-5.fc26           @updates        
qemu-kvm.x86_64                        2:2.9.0-5.fc26           @updates        
qemu-system-x86.x86_64                 2:2.9.0-5.fc26           @updates        
qemu-system-x86-core.x86_64            2:2.9.0-5.fc26           @updates
Comment 7 Sonia Leon 2017-08-29 18:23:05 UTC 
info provided
Comment 8 Richard Purdie 2017-08-31 15:34:20 UTC 
Note " Qemu pid didn't appeared in 60 seconds"
Comment 9 brian avery 2017-09-19 16:43:04 UTC 
closing based on ed9d0fa5a92f837b14ff8f4e8d433da92d7dc968 and 03aa1ef733032e1acae4d291580968301679117f commits.  Should the issue recur, the additional logging info that was also added as part of these commits should help.