Bug 611 - Sanity Test Boot test returns invalid results.
Summary: Sanity Test Boot test returns invalid results.
Status: RESOLVED FIXED
Alias: None
Product: Meta-yocto
Classification: Build System, Metadata & Runtime
Component: meta-yocto (show other bugs)
Version: 1.0
Hardware: x86 x86
: Medium normal
Target Milestone: 1.0 M3
Assignee: Jiajun Xu
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2010-12-22 15:46 UTC by Beth Flanagan
Modified: 2010-12-23 17:25 UTC (History)
5 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Beth Flanagan 2010-12-22 15:46:34 UTC
Fresh checkout of poky, running bitbake poky-image-sato -c qemuimagetest_standalone (which does not create an image whereas qemuimagetest does) results in the following:

NOTE: 	Test Result for qemux86
NOTE: 	Testcase       PASS           FAIL           NORESULT       
NOTE: 	Boot           1              0              0              
NOTE: 	SSH            0              1              0              
NOTE: 	SCP            0              1              0              
NOTE: 	dmesg          0              1              0              
NOTE: 	shutdown       0              1              0         

Looking at build/tmp/

ls build/tmp/   
abi_version     log/            sstate-control/ sysroots/       
cache/          saved_tmpdir    stamps/         work/       

Obviously, no image exists, so, this couldn't have booted.
Comment 1 Jiajun Xu 2010-12-22 16:35:23 UTC
If there is no image existing, sanity test should exist and return warning. I will add such check.
Comment 2 Beth Flanagan 2010-12-22 16:47:01 UTC
What concerns me is the false positive. The boot passed. It shouldn't have. I think that's the part of this defect that is more important.
Comment 3 Jiajun Xu 2010-12-22 16:50:13 UTC
(In reply to comment #2)
> What concerns me is the false positive. The boot passed. It shouldn't have. I
> think that's the part of this defect that is more important.

Thanks for pointing this out, Beth. Yes, boot test case should return fail when image is created failed. This is casued by a ip address check issue in boot test case.
Comment 4 Beth Flanagan 2010-12-22 17:06:43 UTC
ah... here is the catch though. qemuimagetest_standalone doesn't build an image. So relying on an image being created won't help in this use case.
Comment 5 Jiajun Xu 2010-12-22 23:02:00 UTC
(In reply to comment #4)
> ah... here is the catch though. qemuimagetest_standalone doesn't build an
> image. So relying on an image being created won't help in this use case.

Hi Beth,
Could you help to review following patch? I suppose it should fix the issue you met.

diff --git a/scripts/qemuimage-testlib b/scripts/qemuimage-testlib
index c2c394d..2325237 100644
--- a/scripts/qemuimage-testlib
+++ b/scripts/qemuimage-testlib
@@ -365,8 +365,21 @@ Test_Create_Qemu()
        elif [ "$QEMUARCH" = "qemumips" ]; then
                KERNEL=$(Test_Find_Image -l ${DEPLOY_DIR}/images -k vmlinux -a ${QEMUARCH})
        fi
+
+       # If there is kernel image found, return failed directly
+       if [ $? -eq 1 ]; then
+               Test_Info "No kernel image file found under ${DEPLOY_DIR}/images for ${QEMUARCH}, pls. have a check"
+               return $ret
+       fi

        ROOTFS_IMAGE=$(Test_Find_Image -l ${DEPLOY_DIR}/images -t ${QEMUTARGET} -a ${QEMUARCH})
+
+       # If there is rootfs image found, return failed directly
+       if [ $? -eq 1 ]; then
+               Test_Info "No ${QEMUTARGET} rootfs image file found under ${DEPLOY_DIR}/images for ${QEMUARCH}, pls. have a check"
+               return $ret
+       fi
+
        TEST_ROOTFS_IMAGE="${TEST_TMP}/${QEMUTARGET}-${QEMUARCH}-test.ext3"

        CP=`which cp`
@@ -375,6 +388,11 @@ Test_Create_Qemu()
        fi
        $CP $ROOTFS_IMAGE $TEST_ROOTFS_IMAGE

+       if [ $? -ne 0 ]; then
+               Test_Info "Image ${ROOTFS_IMAGE} copy to ${TEST_ROOTFS_IMAGE} failed, return fail"
+               return $ret
+       fi
+
        export MACHINE=$QEMUARCH

        # Create Qemu in localhost VNC Port 1
@@ -402,6 +420,11 @@ Test_Create_Qemu()
        if [ ${up_time} -lt ${timeout} ]; then
                sleep 5
                TARGET_IPADDR=`Test_Fetch_Target_IP $PID`
+               # If IP address is 0, means there is no qemu process found
+               if [ ${TARGET_IPADDR} -eq 0 ]; then
+                       Test_Info "There is no qemu process or qemu ip address found, return failed"
+                       return $ret
+               fi
        fi
Comment 6 Jiajun Xu 2010-12-23 17:25:56 UTC
Fixing patch in master now, commit 236faced1d907b025a3136c5949a311104d3a661.