Bug 11869

Summary: FAILED runqemu.RunqemuTests.test_boot_deploy - Testcase 2007: Ubuntu1604
Product: [Build System, Metadata & Runtime] OE-Core Reporter: oscar lopez arandas <oscarx.lopez.arandas>
Component: configurationAssignee: Ed Bartosh <eduard.bartosh>
Status: RESOLVED INVALID QA Contact:
Severity: normal    
Priority: Medium+    
Version: 2.4   
Target Milestone: 2.4 M3   
Hardware: Other   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
log
none
NATIVELSBSTRING = "universal"
none
NATIVELSBSTRING = "ubuntu-16.04"
none
.log none

Description oscar lopez arandas 2017-07-28 21:49:51 UTC
Build Configuration:
BB_VERSION        = "1.35.0"
BUILD_SYS         = "x86_64-linux"
NATIVELSBSTRING   = "universal"
TARGET_SYS        = "x86_64-poky-linux"
MACHINE           = "qemux86-64"
DISTRO            = "poky"
DISTRO_VERSION    = "2.3"
TUNE_FEATURES     = "m64 core2"
BUILD: 2.4 M2: rc2-8e15e9b6e478f6368034519b2a8fd3c7ea71d23b
ENVIRONMENT: ubuntu 16.04 x86_64

======================================================================
2017-07-26 16:14:03,213 - oe-selftest - INFO - FAIL [61.647s]: test_boot_deploy (runqemu.RunqemuTests)
2017-07-26 16:14:03,213 - oe-selftest - INFO - ----------------------------------------------------------------------
2017-07-26 16:14:03,213 - oe-selftest - INFO - Traceback (most recent call last):
  File "/home/ab/autobuilder/yocto-worker/nightly-oe-selftest-ubuntu1604/build/meta/lib/oeqa/core/case.py", line 41, in _oeSetUp
    self.setUpMethod()
  File "/home/ab/autobuilder/yocto-worker/nightly-oe-selftest-ubuntu1604/build/meta/lib/oeqa/selftest/case.py", line 150, in setUp
    self.setUpLocal()
  File "/home/ab/autobuilder/yocto-worker/nightly-oe-selftest-ubuntu1604/build/meta/lib/oeqa/selftest/cases/runqemu.py", line 41, in setUpLocal
    bitbake(self.recipe)
  File "/home/ab/autobuilder/yocto-worker/nightly-oe-selftest-ubuntu1604/build/meta/lib/oeqa/utils/commands.py", line 213, in bitbake
    return runCmd(cmd, ignore_status, timeout, output_log=output_log, **options)
  File "/home/ab/autobuilder/yocto-worker/nightly-oe-selftest-ubuntu1604/build/meta/lib/oeqa/utils/commands.py", line 191, in runCmd
    raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))
AssertionError: Command 'bitbake  core-image-minimal' returned non-zero exit status 1:
NOTE: Starting bitbake server...
Parsing recipes...done.
Parsing of 821 .bb files complete (0 cached, 821 parsed). 1261 targets, 45 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies
Comment 1 Richard Purdie 2017-07-30 10:36:42 UTC
The log looks truncated. Please provide the full failure.
Comment 2 oscar lopez arandas 2017-07-31 15:02:46 UTC
(In reply to comment #1)
> The log looks truncated. Please provide the full failure.

We provide the complete failure attached in attachment
Comment 3 oscar lopez arandas 2017-07-31 15:03:29 UTC
Created attachment 3905 [details]
log
Comment 4 oscar lopez arandas 2017-07-31 16:14:57 UTC
We provide the complete failure attached in attachment
Comment 5 Ed Bartosh 2017-08-01 13:13:44 UTC
Looks like missing dependency to mkisofs to me:
tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/temp/run.build_iso.21623: 169: tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/temp/run.build_iso.21623: mkisofs: not found
Comment 6 Ed Bartosh 2017-08-01 16:29:05 UTC
I can't reproduce it in Ubuntu 16.04 docker container. Tested commit 8e15e9b6e478f6368034519b2a8fd3c7ea71d23b

$ oe-selftest -r runqemu.RunqemuTests.test_boot_deploy
2017-08-01 16:00:05,095 - oe-selftest - ERROR - Please unset SANITY_TESTED_DISTROS in order to run oe-selftest
[ed@797444d870c0 build-11869 (test)]$ vi conf/local.conf 
[ed@797444d870c0 build-11869 (test)]$ 
[ed@797444d870c0 build-11869 (test)]$ 
[ed@797444d870c0 build-11869 (test)]$ vi conf/local.conf 
[ed@797444d870c0 build-11869 (test)]$ oe-selftest -r runqemu.RunqemuTests.test_boot_deploy
2017-08-01 16:01:03,593 - oe-selftest - INFO - Adding layer libraries:
2017-08-01 16:01:03,594 - oe-selftest - INFO - 	/home/ed/git/yocto/poky/meta/lib
2017-08-01 16:01:03,594 - oe-selftest - INFO - 	/home/ed/git/yocto/poky/meta-yocto-bsp/lib
2017-08-01 16:01:03,594 - oe-selftest - INFO - 	/home/ed/git/yocto/poky/meta-selftest/lib
2017-08-01 16:01:03,595 - oe-selftest - INFO - Running bitbake -p
2017-08-01 16:01:14,021 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/ed/git/yocto/poky/build-11869/conf/local.conf
2017-08-01 16:01:14,022 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
NOTE: Reconnecting to bitbake server...
NOTE: Retrying server connection...
NOTE: Starting bitbake server...
Loading cache: 100% |#################################################################################################################################################| Time: 0:00:00
rootfs file: 
Qemu log file: /home/ed/git/yocto/poky/build-11869/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testimage/qemu_boot_log.20170801160223
SSH log file: /home/ed/git/yocto/poky/build-11869/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/testimage/ssh_target_log.20170801160223
Created listening socket for qemu serial console on: 127.0.0.1:56425
Created listening socket for qemu serial console on: 127.0.0.1:36359
launchcmd=runqemu nographic /home/ed/git/yocto/poky/build-11869/tmp/deploy/images/qemux86-64 tcpserial=36359 bootparams="console=tty1 console=ttyS0,115200n8 printk.time=1" qemuparams="-serial tcp:127.0.0.1:56425"
runqemu started, pid is 7446
waiting at most 60 seconds for qemu pid
qemu started - qemu procces pid is 7487
Starting logging thread
Starting thread event loop
Connection request received
Setting connection established event
Output from runqemu:
runqemu - INFO - DEPLOY_DIR_IMAGE: /home/ed/git/yocto/poky/build-11869/tmp/deploy/images/qemux86-64
runqemu - INFO - Running ls -t /home/ed/git/yocto/poky/build-11869/tmp/deploy/images/qemux86-64/*.qemuboot.conf...
runqemu - INFO - CONFFILE: /home/ed/git/yocto/poky/build-11869/tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64-20170801160125.qemuboot.conf
runqemu - INFO - Continuing with the following parameters:

runqemu - INFO - Running /sbin/ip link...
runqemu - INFO - Setting up tap interface under sudo
runqemu - INFO - Acquiring lockfile /tmp/qemu-tap-locks/tap0.lock...
runqemu - INFO - Created tap: tap0
runqemu - INFO - Network configuration: 192.168.7.2::192.168.7.1:255.255.255.0
runqemu - INFO - Running ldd tmp/work/x86_64-linux/qemu-helper-native/1.0-r1/recipe-sysroot-native/usr/bin//qemu-system-x86_64...
runqemu - INFO - Running tmp/work/x86_64-linux/qemu-helper-native/1.0-r1/recipe-sysroot-native/usr/bin//qemu-system-x86_64 -device virtio-net-pci,netdev=net0,mac=52:54:00:12:34:02 -netdev tap,id=net0,ifname=tap0,script=no,downscript=no -drive file=tmp/deploy/images/qemux86-64/core-image-minimal-qemux86-64-20170801160125.rootfs.ext4,if=virtio,format=raw -vga vmware -show-cursor -usb -usbdevice tablet -device virtio-rng-pci  -nographic -serial tcp:127.0.0.1:56425  -cpu core2duo -m 256 -serial tcp:127.0.0.1:36359 -kernel tmp/deploy/images/qemux86-64/bzImage--4.10.17+git0+e92bd55409_6648a34e00-r0-qemux86-64-20170801135531.bin -append 'root=/dev/vda rw highres=off  console=ttyS0 mem=256M ip=192.168.7.2::192.168.7.1:255.255.255.0 vga=0 uvesafb.mode_option=640x480-32 oprofile.timer=1 uvesafb.task_timeout=-1 console=tty1 console=ttyS0,115200n8 printk.time=1'

QEMU 2.8.1.1 monitor - type 'help' for more information
(qemu) 
Waiting at most 1000 seconds for login banner
Connection from 127.0.0.1:44596
Reached login banner
Logged as root in serial console
Stopping logging thread
Stop event received
Tearing down logging thread
Sending SIGTERM to runqemu
2017-08-01 16:02:41,764 - oe-selftest - INFO - test_boot_deploy (runqemu.RunqemuTests) ... ok
2017-08-01 16:02:41,766 - oe-selftest - INFO - 
2017-08-01 16:02:41,766 - oe-selftest - INFO - ----------------------------------------------------------------------
2017-08-01 16:02:41,766 - oe-selftest - INFO - Ran 1 test in 87.745s
2017-08-01 16:02:41,766 - oe-selftest - INFO - 
2017-08-01 16:02:41,766 - oe-selftest - INFO - OK
2017-08-01 16:02:41,766 - oe-selftest - INFO - RESULTS:
2017-08-01 16:02:41,767 - oe-selftest - INFO - RESULTS - runqemu.RunqemuTests.test_boot_deploy - Testcase 2007: PASSED
2017-08-01 16:02:41,767 - oe-selftest - INFO - SUMMARY:
2017-08-01 16:02:41,767 - oe-selftest - INFO - oe-selftest () - Ran 1 test in 87.746s
2017-08-01 16:02:41,767 - oe-selftest - INFO - oe-selftest - OK - All required tests passed


However, I've noticed that NATIVELSBSTRING is different in my environment:
$ bitbake core-image-minimal
NOTE: Starting bitbake server...
Parsing recipes: 100% |###############################################################################################################################################| Time: 0:00:09
Parsing of 823 .bb files complete (0 cached, 823 parsed). 1263 targets, 45 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION        = "1.35.0"
BUILD_SYS         = "x86_64-linux"
NATIVELSBSTRING   = "ubuntu-16.04"
TARGET_SYS        = "x86_64-poky-linux"
MACHINE           = "qemux86-64"
DISTRO            = "poky"
DISTRO_VERSION    = "2.3"
TUNE_FEATURES     = "m64 core2"
TARGET_FPU        = ""
meta              
meta-poky         
meta-yocto-bsp    
meta-selftest     = "test:8e15e9b6e478f6368034519b2a8fd3c7ea71d23b"

Initialising tasks: 100% |############################################################################################################################################| Time: 0:00:04
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
NOTE: Tasks Summary: Attempted 2398 tasks of which 2190 didn't need to be rerun and all succeeded.


Any idea why that could be? Can you try to reproduce this in a clean environment?
Comment 7 oscar lopez arandas 2017-08-02 15:23:54 UTC
> Any idea why that could be? Can you try to reproduce this in a clean
> environment?

1.reproducing with new environment and with NATIVELSBSTRING   = "universal" the failed is present. 
2.reproducing whith NATIVELSBSTRING   = "ubuntu-16.04" the failed is present 
 
3.  run  $bitbake core-image-minimal  and have failed 
===============================================================================
home/oscar/11869/build/tmp-glibc/hosttools/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status
Makefile.shared:171: recipe for target 'link_a.gnu' failed
make[4]: *** [link_a.gnu] Error 1
make[4]: Leaving directory '/home/oscar/11869/build/tmp-glibc/work/x86_64-linux/openssl-native/1.0.2l-r0/openssl-1.0.2l'
ERROR: oe_runmake failed
Makefile:358: recipe for target 'do_linux-shared' failed
make[3]: *** [do_linux-shared] Error 2
make[3]: Leaving directory '/home/oscar/11869/build/tmp-glibc/work/x86_64-linux/openssl-native/1.0.2l-r0/openssl-1.0.2l'
Makefile:311: recipe for target 'libcrypto.so.1.0.2' failed
===============================================================================
add log's in attachment
Comment 8 oscar lopez arandas 2017-08-02 15:25:22 UTC
Created attachment 3914 [details]
NATIVELSBSTRING   = "universal"
Comment 9 oscar lopez arandas 2017-08-02 15:28:07 UTC
Created attachment 3915 [details]
NATIVELSBSTRING   = "ubuntu-16.04"
Comment 10 oscar lopez arandas 2017-08-02 15:30:40 UTC
The titles of the records are changed an apology I confused them
Comment 11 Ed Bartosh 2017-08-03 08:14:03 UTC
The build failure is different in the last 2 attached logs. This one is linker failure while building openssl:

 /home/oscar/11869/build/tmp-glibc/hosttools/ld: libcrypto.a(sha1-x86_64.o): relocation R_X86_64_PC32 against undefined symbol `OPENSSL_ia32cap_P' can not be used when making a shared object; recompile with -fPIC
| /home/oscar/11869/build/tmp-glibc/hosttools/ld: final link failed: Bad value
| collect2: error: ld returned 1 exit status
| Makefile.shared:171: recipe for target 'link_a.gnu' failed
| make[4]: *** [link_a.gnu] Error 1

However, in the first log the failed recipe was core-image-minimal. It was failing as it couldn't find mkisofs binary:

/home/ab/autobuilder/yocto-worker/nightly-oe-selftest-ubuntu1604/build/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/temp/run.build_iso.21623: 169: /home/ab/autobuilder/yocto-worker/nightly-oe-selftest-ubuntu1604/build/build/tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/temp/run.build_iso.21623: mkisofs: not found

I'd suggest to create separate bug for the second failure as it's doesn't seem to be related to the first one.

Can you reproduce first one in the clean environment preferably on different machine?
Comment 12 oscar lopez arandas 2017-08-03 14:54:45 UTC
(In reply to comment #11)

> Can you reproduce first one in the clean environment preferably on different
> machine?

In which type of machine would be preferable to try?
Comment 13 Ed Bartosh 2017-08-03 15:24:53 UTC
Anything with Ubuntu 16.04 installed would work I guess.

The reason I'm asking this is that the issue just doesn't make sense to me and I suspect it only happens on autobuilder. If you have Ubuntu machine around and will be able to reproduce it there that would help a lot.
Comment 14 Ed Bartosh 2017-08-04 09:28:21 UTC
If you have access to autobuilder logs it would be great if you grep them for 'mkisofs: not found' to see how often it happens and on which hosts.

I tried to use http://errors.yoctoproject.org/, but couldn't find anything there yet. Search functiona doesn't work there, the site is extremely slow, so I had to search manually on every page. I looked for 'iso' as failing task is 'build_iso', but didn't find anything there yet.
Comment 15 oscar lopez arandas 2017-08-04 15:02:59 UTC
Autobuilder was reconfigured and the tests were redeemed again and the fault disappeared, it was a poor autobuilder configuration.
=========================================================================
 RESULTS - runqemu.RunqemuTests.test_boot_deploy - Testcase 2007: PASSED
2017-08-03 19:03:42,634 - oe-selftest - INFO - RESULTS - runqemu.RunqemuTests.test_boot_deploy_hddimg - Testcase 2008: PASSED
2017-08-03 19:03:42,634 - oe-selftest - INFO - RESULTS - runqemu.RunqemuTests.test_boot_machine - Testcase 2001: PASSED
2017-08-03 19:03:42,634 - oe-selftest - INFO - RESULTS - runqemu.RunqemuTests.test_boot_machine_ext4 - Testcase 2002: PASSED
2017-08-03 19:03:42,635 - oe-selftest - INFO - RESULTS - runqemu.RunqemuTests.test_boot_machine_iso - Testcase 2003: PASSED
2017-08-03 19:03:42,635 - oe-selftest - INFO - RESULTS - runqemu.RunqemuTests.test_boot_machine_slirp - Testcase 2009: PASSED
======================================================================
.log in attachments
Comment 16 oscar lopez arandas 2017-08-04 16:18:18 UTC
Created attachment 3918 [details]
.log
Comment 17 Ed Bartosh 2017-08-07 09:31:36 UTC
(In reply to comment #15)
> Autobuilder was reconfigured and the tests were redeemed again and the fault
> disappeared, it was a poor autobuilder configuration.
> =========================================================================
>  RESULTS - runqemu.RunqemuTests.test_boot_deploy - Testcase 2007: PASSED
> 2017-08-03 19:03:42,634 - oe-selftest - INFO - RESULTS -
> runqemu.RunqemuTests.test_boot_deploy_hddimg - Testcase 2008: PASSED
> 2017-08-03 19:03:42,634 - oe-selftest - INFO - RESULTS -
> runqemu.RunqemuTests.test_boot_machine - Testcase 2001: PASSED
> 2017-08-03 19:03:42,634 - oe-selftest - INFO - RESULTS -
> runqemu.RunqemuTests.test_boot_machine_ext4 - Testcase 2002: PASSED
> 2017-08-03 19:03:42,635 - oe-selftest - INFO - RESULTS -
> runqemu.RunqemuTests.test_boot_machine_iso - Testcase 2003: PASSED
> 2017-08-03 19:03:42,635 - oe-selftest - INFO - RESULTS -
> runqemu.RunqemuTests.test_boot_machine_slirp - Testcase 2009: PASSED
> ======================================================================
> .log in attachments

OK, thanks. Setting bug status to Resolved/Invalid then.