Bug 595 - Sanity tests do not always kill QEMU once timeout is reached
Summary: Sanity tests do not always kill QEMU once timeout is reached
Status: RESOLVED FIXED
Alias: None
Product: Test Plans/Suite
Classification: QA/Testing
Component: test-plans/suite (show other bugs)
Version: 1.0
Hardware: x86 Multiple
: Medium normal
Target Milestone: 1.0 M4
Assignee: Jiajun Xu
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2010-12-14 13:33 UTC by Scott Garman
Modified: 2011-01-16 17:43 UTC (History)
9 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Scott Garman 2010-12-14 13:33:05 UTC
A recent run of poky-incremental shows sanitytests taking unusually long time to complete (or failure due to the autobuilder's 30-minute task timeout):

http://autobuilder.pokylinux.org:8010/builders/poky-incremental/builds/29

This shouldn't happen - the much shorter sanitytest timeouts (of around 60 seconds) should kill QEMU if the test has not succeeded yet and report the test as failed.
Comment 1 Jiajun Xu 2011-01-12 08:08:24 UTC
The issue is clear to me now. It's because the timeout value of each case is 360 seconds but now some qemu targets(qemuarm and qemupp) took more than 360s for first time booting. We have a bug to track the booting slowness issue, http://bugzilla.pokylinux.org/show_bug.cgi?id=646. When a previous sanity test is still running, a next build is started. That caused the series failure you saw.
Since only the first time booting takes much time, I wrote a patch to fix the timeout issue on autobuilder. I move test case dmesg to the first one and save the image used for dmesg test. The image then will be used in following test cases, which can save a lot of time. And I also set timeout value to 200s, which means 10 cases will cost 200*10=2000s at maximum, while the timeout value of each sanity test task for each target is 2400s. It can ensure above failure not happen again I think.

Following path is supposed to fix the issue.

diff --git a/meta/classes/imagetest-qemu.bbclass b/meta/classes/imagetest-qemu.bbclass
index bcae10d..014fd16 100644
--- a/meta/classes/imagetest-qemu.bbclass
+++ b/meta/classes/imagetest-qemu.bbclass
@@ -5,6 +5,7 @@ TEST_LOG ?= "${LOG_DIR}/qemuimagetests"
 TEST_RESULT ?= "${TEST_DIR}/result"
 TEST_TMP ?= "${TEST_DIR}/tmp"
 TEST_SCEN ?= "sanity"
+SHARE_IMAGE ?= "1"
 
 python do_qemuimagetest() {
     qemuimagetest_main(d)
@@ -37,7 +38,7 @@ def qemuimagetest_main(d):
     """funtion to run each case under scenario"""
     def runtest(scen, case, fulltestpath):
         resultpath = bb.data.getVar('TEST_RESULT', d, 1)
-        testpath = bb.data.getVar('TEST_DIR', d, 1)
+        tmppath = bb.data.getVar('TEST_TMP', d, 1)
 
         """initialize log file for testcase"""
         logpath = bb.data.getVar('TEST_LOG', d, 1)
@@ -45,10 +46,9 @@ def qemuimagetest_main(d):
         caselog = os.path.join(logpath, "%s/log_%s.%s" % (scen, case, bb.data.getVar('DATETIME', d, 1)))
         os.system("touch %s" % caselog)
         
-
         """export TEST_TMP, TEST_RESULT, DEPLOY_DIR and QEMUARCH"""
         os.environ["PATH"] = bb.data.getVar("PATH", d, True)
-        os.environ["TEST_TMP"] = testpath
+        os.environ["TEST_TMP"] = tmppath
         os.environ["TEST_RESULT"] = resultpath
         os.environ["DEPLOY_DIR"] = bb.data.getVar("DEPLOY_DIR", d, True)
         os.environ["QEMUARCH"] = machine
@@ -56,6 +56,7 @@ def qemuimagetest_main(d):
         os.environ["DISPLAY"] = bb.data.getVar("DISPLAY", d, True)
         os.environ["POKYBASE"] = bb.data.getVar("POKYBASE", d, True)
         os.environ["TOPDIR"] = bb.data.getVar("TOPDIR", d, True)
+        os.environ["SHARE_IMAGE"] = bb.data.getVar("SHARE_IMAGE", d, True)
 
         """run Test Case"""
         bb.note("Run %s test in scenario %s" % (case, scen))
@@ -97,6 +98,15 @@ def qemuimagetest_main(d):
                        list.append((item, casefile, fulltestcase))
         return list
 
+    """Clean tmp folder for testing"""
+    def clean_tmp():
+        tmppath = bb.data.getVar('TEST_TMP', d, 1)
+
+        if os.path.isdir(tmppath):
+            for f in os.listdir(tmppath):
+                tmpfile = os.path.join(tmppath, f)
+                os.remove(tmpfile)
+
     """check testcase folder and create test log folder"""
     testpath = bb.data.getVar('TEST_DIR', d, 1)
     bb.utils.mkdirhier(testpath)
@@ -150,6 +160,9 @@ def qemuimagetest_main(d):
             bb.note(line)
     f.close()
 
+    """Clean temp files for testing"""
+    clean_tmp()
+
     if ret != 0:
         raise bb.build.FuncFailed("Some testcases fail, pls. check test result and test log!!!")
 
diff --git a/meta/conf/local.conf.sample b/meta/conf/local.conf.sample
index 9c58e6b..33a50dc 100644
--- a/meta/conf/local.conf.sample
+++ b/meta/conf/local.conf.sample
@@ -156,6 +156,15 @@ ENABLE_BINARY_LOCALE_GENERATION = "1"
 # list them like following.
 #TEST_SCEN = "sanity bat sanity:boot"
 
+# Because of the QEMU booting slowness issue(see bug #646 and #618), we need a 
+# workaround for sanity test running on autobuilder without timeout problem.
+# So we introduce variable SHARE_IMAGE here. It is by default set to 1. Poky
+# will use the latest built-out image for first test case running. Then the 
+# image will be still used in following cases and be removed after testing is end.
+# If it is set to 0, latest built-out image will be copied and tested for each
+# case, which will take much time.
+#SHARE_IMAGE = "1"
+
 # Set GLIBC_GENERATE_LOCALES to the locales you wish to generate should you not
 # wish to perform the time-consuming step of generating all LIBC locales.
 # NOTE: If removing en_US.UTF-8 you will also need to uncomment, and set
diff --git a/scripts/qemuimage-testlib b/scripts/qemuimage-testlib
index e2c2d24..b9afcf5 100644
--- a/scripts/qemuimage-testlib
+++ b/scripts/qemuimage-testlib
@@ -383,9 +383,16 @@ Test_Create_Qemu()
 	TEST_ROOTFS_IMAGE="${TEST_TMP}/${QEMUTARGET}-${QEMUARCH}-test.ext3"
 	
 	CP=`which cp`
+
+	# When SHARE_IMAGE is set, we use the existing image under tmp folder
 	if [ -e "$TEST_ROOTFS_IMAGE" ]; then
+		if [ ${SHARE_IMAGE} -eq 1 ]; then
+			ROOTFS_IMAGE="$TEST_ROOTFS_IMAGE"
+			TEST_ROOTFS_IMAGE="${TEST_TMP}/${QEMUTARGET}-${QEMUARCH}-shared-test.ext3"
+		fi
 		rm -rf $TEST_ROOTFS_IMAGE
 	fi
+
 	$CP $ROOTFS_IMAGE $TEST_ROOTFS_IMAGE
 
 	if [ $? -ne 0 ]; then
diff --git a/scripts/qemuimage-tests/sanity/compiler b/scripts/qemuimage-tests/sanity/compiler
index 0290540..d3168a9 100755
--- a/scripts/qemuimage-tests/sanity/compiler
+++ b/scripts/qemuimage-tests/sanity/compiler
@@ -11,7 +11,7 @@
 
 . $POKYBASE/scripts/qemuimage-testlib
 
-TIMEOUT=360
+TIMEOUT=200
 RET=1
 
 # Start qemu and check its network
diff --git a/scripts/qemuimage-tests/sanity/connman b/scripts/qemuimage-tests/sanity/connman
index 8c5b7c5..6e19b81 100755
--- a/scripts/qemuimage-tests/sanity/connman
+++ b/scripts/qemuimage-tests/sanity/connman
@@ -11,7 +11,7 @@
 
 . $POKYBASE/scripts/qemuimage-testlib
 
-TIMEOUT=360
+TIMEOUT=200
 RET=1
 
 # Start qemu and check its network
diff --git a/scripts/qemuimage-tests/sanity/dmesg b/scripts/qemuimage-tests/sanity/dmesg
index 36813dc..d98d7af 100755
--- a/scripts/qemuimage-tests/sanity/dmesg
+++ b/scripts/qemuimage-tests/sanity/dmesg
@@ -11,7 +11,7 @@
 
 . $POKYBASE/scripts/qemuimage-testlib
 
-TIMEOUT=360
+TIMEOUT=200
 RET=1
 
 # Start qemu and check its network
diff --git a/scripts/qemuimage-tests/sanity/rpm_query b/scripts/qemuimage-tests/sanity/rpm_query
index 9962115..7663901 100755
--- a/scripts/qemuimage-tests/sanity/rpm_query
+++ b/scripts/qemuimage-tests/sanity/rpm_query
@@ -11,7 +11,7 @@
 
 . $POKYBASE/scripts/qemuimage-testlib
 
-TIMEOUT=360
+TIMEOUT=200
 RET=1
 
 # Start qemu and check its network
diff --git a/scripts/qemuimage-tests/sanity/scp b/scripts/qemuimage-tests/sanity/scp
index ce3489d..ca19857 100755
--- a/scripts/qemuimage-tests/sanity/scp
+++ b/scripts/qemuimage-tests/sanity/scp
@@ -11,7 +11,7 @@
 
 . $POKYBASE/scripts/qemuimage-testlib
 
-TIMEOUT=360
+TIMEOUT=200
 RET=1
 SPID=0
 i=0
diff --git a/scripts/qemuimage-tests/sanity/shutdown b/scripts/qemuimage-tests/sanity/shutdown
index e36b4a9..54d7861 100755
--- a/scripts/qemuimage-tests/sanity/shutdown
+++ b/scripts/qemuimage-tests/sanity/shutdown
@@ -13,7 +13,7 @@
 
 . $POKYBASE/scripts/qemuimage-testlib
 
-TIMEOUT=360
+TIMEOUT=200
 RET=1
 i=0
 
diff --git a/scripts/qemuimage-tests/sanity/ssh b/scripts/qemuimage-tests/sanity/ssh
index 084530a..e0ade72 100755
--- a/scripts/qemuimage-tests/sanity/ssh
+++ b/scripts/qemuimage-tests/sanity/ssh
@@ -11,7 +11,7 @@
 
 . $POKYBASE/scripts/qemuimage-testlib
 
-TIMEOUT=360
+TIMEOUT=200
 RET=1
 
 # Start qemu and check its network
diff --git a/scripts/qemuimage-tests/sanity/zypper_help b/scripts/qemuimage-tests/sanity/zypper_help
index e052501..48e121c 100755
--- a/scripts/qemuimage-tests/sanity/zypper_help
+++ b/scripts/qemuimage-tests/sanity/zypper_help
@@ -11,7 +11,7 @@
 
 . $POKYBASE/scripts/qemuimage-testlib
 
-TIMEOUT=360
+TIMEOUT=200
 RET=1
 
 # Start qemu and check its network
diff --git a/scripts/qemuimage-tests/sanity/zypper_search b/scripts/qemuimage-tests/sanity/zypper_search
index b756db8..9ae69eb 100755
--- a/scripts/qemuimage-tests/sanity/zypper_search
+++ b/scripts/qemuimage-tests/sanity/zypper_search
@@ -11,7 +11,7 @@
 
 . $POKYBASE/scripts/qemuimage-testlib
 
-TIMEOUT=360
+TIMEOUT=200
 RET=1
 
 # Start qemu and check its network
diff --git a/scripts/qemuimage-tests/scenario/qemuarm/poky-image-lsb b/scripts/qemuimage-tests/scenario/qemuarm/poky-image-lsb
index 32d2bd1..0eb1926 100644
--- a/scripts/qemuimage-tests/scenario/qemuarm/poky-image-lsb
+++ b/scripts/qemuimage-tests/scenario/qemuarm/poky-image-lsb
@@ -1,8 +1,8 @@
+sanity shutdown
 sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
-sanity shutdown
 sanity zypper_help
 sanity zypper_search
 sanity rpm_query
diff --git a/scripts/qemuimage-tests/scenario/qemuarm/poky-image-sato b/scripts/qemuimage-tests/scenario/qemuarm/poky-image-sato
index 7b949a8..b60a89a 100644
--- a/scripts/qemuimage-tests/scenario/qemuarm/poky-image-sato
+++ b/scripts/qemuimage-tests/scenario/qemuarm/poky-image-sato
@@ -1,8 +1,8 @@
+sanity shutdown
 sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
-sanity shutdown
 sanity zypper_help
 sanity zypper_search
 sanity rpm_query
diff --git a/scripts/qemuimage-tests/scenario/qemuarm/poky-image-sdk b/scripts/qemuimage-tests/scenario/qemuarm/poky-image-sdk
index 3a8b129..53c9ad5 100644
--- a/scripts/qemuimage-tests/scenario/qemuarm/poky-image-sdk
+++ b/scripts/qemuimage-tests/scenario/qemuarm/poky-image-sdk
@@ -1,8 +1,8 @@
+sanity shutdown
 sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
-sanity shutdown
 sanity zypper_help
 sanity zypper_search
 sanity rpm_query
diff --git a/scripts/qemuimage-tests/scenario/qemumips/poky-image-lsb b/scripts/qemuimage-tests/scenario/qemumips/poky-image-lsb
index 32d2bd1..0eb1926 100644
--- a/scripts/qemuimage-tests/scenario/qemumips/poky-image-lsb
+++ b/scripts/qemuimage-tests/scenario/qemumips/poky-image-lsb
@@ -1,8 +1,8 @@
+sanity shutdown
 sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
-sanity shutdown
 sanity zypper_help
 sanity zypper_search
 sanity rpm_query
diff --git a/scripts/qemuimage-tests/scenario/qemumips/poky-image-sato b/scripts/qemuimage-tests/scenario/qemumips/poky-image-sato
index 7b949a8..b60a89a 100644
--- a/scripts/qemuimage-tests/scenario/qemumips/poky-image-sato
+++ b/scripts/qemuimage-tests/scenario/qemumips/poky-image-sato
@@ -1,8 +1,8 @@
+sanity shutdown
 sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
-sanity shutdown
 sanity zypper_help
 sanity zypper_search
 sanity rpm_query
diff --git a/scripts/qemuimage-tests/scenario/qemumips/poky-image-sdk b/scripts/qemuimage-tests/scenario/qemumips/poky-image-sdk
index 3a8b129..53c9ad5 100644
--- a/scripts/qemuimage-tests/scenario/qemumips/poky-image-sdk
+++ b/scripts/qemuimage-tests/scenario/qemumips/poky-image-sdk
@@ -1,8 +1,8 @@
+sanity shutdown
 sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
-sanity shutdown
 sanity zypper_help
 sanity zypper_search
 sanity rpm_query
diff --git a/scripts/qemuimage-tests/scenario/qemuppc/poky-image-lsb b/scripts/qemuimage-tests/scenario/qemuppc/poky-image-lsb
index 32d2bd1..0eb1926 100644
--- a/scripts/qemuimage-tests/scenario/qemuppc/poky-image-lsb
+++ b/scripts/qemuimage-tests/scenario/qemuppc/poky-image-lsb
@@ -1,8 +1,8 @@
+sanity shutdown
 sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
-sanity shutdown
 sanity zypper_help
 sanity zypper_search
 sanity rpm_query
diff --git a/scripts/qemuimage-tests/scenario/qemuppc/poky-image-sato b/scripts/qemuimage-tests/scenario/qemuppc/poky-image-sato
index 7b949a8..b60a89a 100644
--- a/scripts/qemuimage-tests/scenario/qemuppc/poky-image-sato
+++ b/scripts/qemuimage-tests/scenario/qemuppc/poky-image-sato
@@ -1,8 +1,8 @@
+sanity shutdown
 sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
-sanity shutdown
 sanity zypper_help
 sanity zypper_search
 sanity rpm_query
diff --git a/scripts/qemuimage-tests/scenario/qemuppc/poky-image-sdk b/scripts/qemuimage-tests/scenario/qemuppc/poky-image-sdk
index 3a8b129..53c9ad5 100644
--- a/scripts/qemuimage-tests/scenario/qemuppc/poky-image-sdk
+++ b/scripts/qemuimage-tests/scenario/qemuppc/poky-image-sdk
@@ -1,8 +1,8 @@
+sanity shutdown
 sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
-sanity shutdown
 sanity zypper_help
 sanity zypper_search
 sanity rpm_query
diff --git a/scripts/qemuimage-tests/scenario/qemux86-64/poky-image-lsb b/scripts/qemuimage-tests/scenario/qemux86-64/poky-image-lsb
index 32d2bd1..0eb1926 100644
--- a/scripts/qemuimage-tests/scenario/qemux86-64/poky-image-lsb
+++ b/scripts/qemuimage-tests/scenario/qemux86-64/poky-image-lsb
@@ -1,8 +1,8 @@
+sanity shutdown
 sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
-sanity shutdown
 sanity zypper_help
 sanity zypper_search
 sanity rpm_query
diff --git a/scripts/qemuimage-tests/scenario/qemux86-64/poky-image-sato b/scripts/qemuimage-tests/scenario/qemux86-64/poky-image-sato
index 7b949a8..b60a89a 100644
--- a/scripts/qemuimage-tests/scenario/qemux86-64/poky-image-sato
+++ b/scripts/qemuimage-tests/scenario/qemux86-64/poky-image-sato
@@ -1,8 +1,8 @@
+sanity shutdown
 sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
-sanity shutdown
 sanity zypper_help
 sanity zypper_search
 sanity rpm_query
diff --git a/scripts/qemuimage-tests/scenario/qemux86-64/poky-image-sdk b/scripts/qemuimage-tests/scenario/qemux86-64/poky-image-sdk
index 3a8b129..53c9ad5 100644
--- a/scripts/qemuimage-tests/scenario/qemux86-64/poky-image-sdk
+++ b/scripts/qemuimage-tests/scenario/qemux86-64/poky-image-sdk
@@ -1,8 +1,8 @@
+sanity shutdown
 sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
-sanity shutdown
 sanity zypper_help
 sanity zypper_search
 sanity rpm_query
diff --git a/scripts/qemuimage-tests/scenario/qemux86/poky-image-lsb b/scripts/qemuimage-tests/scenario/qemux86/poky-image-lsb
index 32d2bd1..0eb1926 100644
--- a/scripts/qemuimage-tests/scenario/qemux86/poky-image-lsb
+++ b/scripts/qemuimage-tests/scenario/qemux86/poky-image-lsb
@@ -1,8 +1,8 @@
+sanity shutdown
 sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
-sanity shutdown
 sanity zypper_help
 sanity zypper_search
 sanity rpm_query
diff --git a/scripts/qemuimage-tests/scenario/qemux86/poky-image-sato b/scripts/qemuimage-tests/scenario/qemux86/poky-image-sato
index 7b949a8..b60a89a 100644
--- a/scripts/qemuimage-tests/scenario/qemux86/poky-image-sato
+++ b/scripts/qemuimage-tests/scenario/qemux86/poky-image-sato
@@ -1,8 +1,8 @@
+sanity shutdown
 sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
-sanity shutdown
 sanity zypper_help
 sanity zypper_search
 sanity rpm_query
diff --git a/scripts/qemuimage-tests/scenario/qemux86/poky-image-sdk b/scripts/qemuimage-tests/scenario/qemux86/poky-image-sdk
index 3a8b129..53c9ad5 100644
--- a/scripts/qemuimage-tests/scenario/qemux86/poky-image-sdk
+++ b/scripts/qemuimage-tests/scenario/qemux86/poky-image-sdk
@@ -1,8 +1,8 @@
+sanity shutdown
 sanity boot
 sanity ssh
 sanity scp
 sanity dmesg
-sanity shutdown
 sanity zypper_help
 sanity zypper_search
 sanity rpm_query
Comment 2 Jiajun Xu 2011-01-16 17:43:54 UTC
Fixed by commit 2119274314f49678de78dc1c5aa2971676fa6774.