Bug 10794

Summary: Update sanity tests to cope with minimal eSDK installer
Product: [QA/Testing] Runtime Testing Reporter: Joshua Lock <joshuagloe>
Component: generalAssignee: Ross Burton <ross.burton>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium+ CC: anibal.limon, bluelightning, francisco.j.pedraza.gonzalez, leonardo.sandoval.gonzalez, ross.burton
Version: 2.3   
Target Milestone: 2.3 M3   
Hardware: x86   
OS: Multiple   
URL: http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=aca3afcb592ef4ff070760ab2aafba03152a57fb
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Bug Depends on: 10958    
Bug Blocks: 9596    

Description Joshua Lock 2016-12-12 21:43:21 UTC
After making changes to the autobuilder to build a minimal eSDK we observed unexpected eSDK sanity test failures. 

This is because the tests implicitly rely on a full eSDK installer, rather than installing any required components into the eSDK before testing.

In order to enable minimal eSDK installers on the autobuilder, and therefore for publishing with project releases, we need the eSDK sanity tests to handle running against a minimal eSDK installer.
Comment 1 Francisco Pedraza 2017-01-16 17:09:21 UTC
I just ran:
$ . oe-init-build-env
$ echo 'SDK_EXT_TYPE = "minimal' >> conf/local.conf
$ bitbake core-image-minimal core-image-sato -c populate_sdk_ext
$ echo 'INHERIT += "testsdk"' >> conf/local.conf
$ DISPLAY=:1 bitbake core-image-minimal core-image-sato -c testsdkext

and I successfully could reproduce the error.
Comment 2 Aníbal Limón 2017-01-16 21:07:51 UTC
Hi,

Currently there is no way to see what is installed in the extensible sdk see bug [1], so i suggest to add a checking like [2] but this will end in no tests runs on FULL eSDK because of the bug [1].

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=9038
[2] http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/sdk/gcc.py#n9
Comment 3 Aníbal Limón 2017-01-16 21:08:34 UTC
(In reply to comment #2)
> Hi,
> 
> Currently there is no way to see what is installed in the extensible sdk see
> bug [1], so i suggest to add a checking like [2] but this will end in no
> tests runs on FULL eSDK because of the bug [1].
> 
> [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=9038
> [2]
> http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/sdk/gcc.
> py#n9

I think the missing package is autotools, right Joshua?
Comment 4 Francisco Pedraza 2017-01-16 21:46:47 UTC
(In reply to comment #3)
> (In reply to comment #2)
> > Hi,
> > 
> > Currently there is no way to see what is installed in the extensible sdk see
> > bug [1], so i suggest to add a checking like [2] but this will end in no
> > tests runs on FULL eSDK because of the bug [1].
> > 
> > [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=9038
> > [2]
> > http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/sdk/gcc.
> > py#n9
> 
> I think the missing package is autotools, right Joshua?

Thanks Anibal, working on it.
Comment 5 Francisco Pedraza 2017-01-18 02:26:14 UTC
Implemented the same procedure as sdk/gcc.py
if not oeSDKTest.hasHostPackage(" " + machine):
        skipModule("SDK doesn't contain a cross-canadian toolchain").
Getting good skiping, however I got a regression for "test_extend_autotools_recipe_creation", will check this tomorrow.
Comment 6 Francisco Pedraza 2017-01-20 18:05:41 UTC
(In reply to comment #5)
> Implemented the same procedure as sdk/gcc.py
> if not oeSDKTest.hasHostPackage(" " + machine):
>         skipModule("SDK doesn't contain a cross-canadian toolchain").
> Getting good skiping, however I got a regression for
> "test_extend_autotools_recipe_creation", will check this tomorrow.

I just noticed that gcc is not installed on minimal and that could be the failure. Still working on it.
Comment 7 Paul Eggleton 2017-01-20 20:16:14 UTC
Currently that's by design, unless SDK_INCLUDE_TOOLCHAIN = "1" is set. The expectation is that you use devtool exclusively or you install the toolchain using devtool sdk-install.
Comment 8 Francisco Pedraza 2017-01-23 23:17:09 UTC
Created bug https://bugzilla.yoctoproject.org/show_bug.cgi?id=10958 due some issues after modified files at "~/oeqa/sdk/buildiptables.py" and "~/oeqa/sdk/buildcvs.py". That bug blockes this one.
Comment 9 Francisco Pedraza 2017-01-24 21:32:18 UTC
(In reply to comment #7)
> Currently that's by design, unless SDK_INCLUDE_TOOLCHAIN = "1" is set. The
> expectation is that you use devtool exclusively or you install the toolchain
> using devtool sdk-install.

Thank you Paul,
I just fixed this.
Regards
Comment 11 Francisco Pedraza 2017-01-25 17:09:47 UTC
Moved to "In Progress Review" http://lists.openembedded.org/pipermail/openembedded-core/2017-January/131773.html already sent the patch.
Comment 13 Ross Burton 2017-02-27 16:43:33 UTC
https://autobuilder.yoctoproject.org/main/builders/nightly-x86-64/builds/1132/steps/Running%20ESDK%20Sanity%20Tests/logs/stdio

The tests are not updated for minimal test cases, they just skip some of the tests but not all of them so the tests still fail.

In a minimal SDK the SDK has nothing in the sysroot.  I think the first thing the tests need to do is populate the SDK sysroot, which means running:

$ bitbake <IMAGENAME> -c rootfs-

(trailing hyphen)
Comment 14 Ross Burton 2017-03-08 17:24:39 UTC
oe-core 20c6e876ed6b8c8ac007848e2b36e9a8a56bb703 contains all the fixes.