Bug 5337 - [QA] Optimize test case for better use of time and resources
Summary: [QA] Optimize test case for better use of time and resources
Status: VERIFIED FIXED
Alias: None
Product: Test Plans/Suite
Classification: QA/Testing
Component: test-plans/suite (show other bugs)
Version: 1.6
Hardware: x86 Multiple
: Medium enhancement
Target Milestone: 1.6
Assignee: Ionut Chisanovici
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2013-10-11 09:33 UTC by Stoicescu Cornel
Modified: 2013-12-10 09:39 UTC (History)
4 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments
Optimizations table (12.24 KB, application/vnd.openxmlformats-officedocument.spreadsheetml.sheet)
2013-10-28 10:27 UTC, Stoicescu Cornel
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Stoicescu Cornel 2013-10-11 09:33:38 UTC
Looking over many of our test cases for oe-core, meta-yocto and bitbake, we can identify many tests that can be optimized by smartly selecting the build targets.

An example is using 'core-image-minimal' instead of 'core-image-sato' as the build target. Another example is using a package with few build time dependencies instead of one with many build time dependencies(example: 'binutils-cross' instead of 'man')

Another thing we can do is use buildhistory to analyze image content instead of booting the image.

This is a proposal and I believe we can save both man/hour and autobuilder resources.
Comment 1 Stoicescu Cornel 2013-10-21 14:53:59 UTC
Started work on feature.
Comment 2 Stoicescu Cornel 2013-10-28 10:26:28 UTC
The modifications are done. I have sent them to QA team for review.

Affected test components: bitbake, oe-core, meta-yocto, HOB.
The optimizations made consist of changing the build target for the test case. Also in some cases I had to modify the steps a little bit in order to accomodate the new targets (like replacing "1. build core-image-sato" with "1. add ssh-server-openssh to EXTRA_IMAGE_FEATURES in local.conf; 2. build core-image-minimal").

Except for HOB, I re-run all modified test cases in order to assure results integrity. For HOB, the changes made are not that complex and the result is theoretically the same. 

The new build targets have been chosen as follow:

* m4-native
- build time: 1 min
- chosen as the fastest build-able recipe/target
- used in test cases that don't require a rpm/deb/ipk output package

* xcursor-transparent-theme
- build time: 8 min
- chosen as the fastest build-able recipe/target that has a rpm/deb/ipk package output
- is architecture-independent

* m4
-build time: 17 min
- chosen as the fastest build-able recipe/target that has a pm/deb/ipk package output and is architecture-dependent

* core-image-minimal
- build time: 40 min
- chosen as the fastest build-able image type
- in tests that require booting I added the 'ssh-server-openssh" image feature to facilitate automated runtime testing

* core-image-basic
- build time: 53 min
- takes longer to build than core-image-minimal; has more packages included
- contains open-ssh by default
- used in tests that require removing packages from the base image or additional package functionality (syslogd)

* core-image-sato
- build time: 80 min
- smallest image that contains an x server and a desktop environment
- used in tests that have graphical requirements (ex: emgd-driver-bin)

* core-image-sato-sdk
- build time: 148 min
- I did not use this image in my optimizations
- this image is still used in some unmodified test cases that require its -sdk functionality

There are a few other chosen build targets for the special context of the test case.

After I correct any issues signaled by my team upon review I will set this to RESOLVED.
Comment 3 Stoicescu Cornel 2013-10-28 10:27:58 UTC
Created attachment 1583 [details]
Optimizations table

Here is the optimizations table.
Comment 4 Stoicescu Cornel 2013-11-01 16:57:01 UTC
The changes are made and review complete.
Comment 5 Alexandru Georgescu 2013-11-08 08:42:41 UTC
Hi Ionut,
Please verify this!
Comment 6 Ionut Chisanovici 2013-11-25 10:19:24 UTC
Till now the oe-core weekly TC's were verified and they are ok.
We still need to verify the full pass TC's for oe-core and bsp's but this will be done in the next full pass run.

Ionut C
Comment 7 Ionut Chisanovici 2013-12-10 09:39:24 UTC
All the test cases were verified.
The spent time on a full pass has decreased considerable.

Cheers,
Ionut C