Bug 10794 - Update sanity tests to cope with minimal eSDK installer
Summary: Update sanity tests to cope with minimal eSDK installer
Status: RESOLVED FIXED
Alias: None
Product: Runtime Testing
Classification: QA/Testing
Component: general (show other bugs)
Version: 2.3
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 2.3 M3
Assignee: Ross Burton
QA Contact:
URL: http://git.yoctoproject.org/cgit/cgit...
Whiteboard:
Depends on: 10958
Blocks: 9596
  Show dependency tree
 
Reported: 2016-12-12 21:43 UTC by Joshua Lock
Modified: 2017-12-06 17:20 UTC (History)
5 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Joshua Lock 2016-12-12 21:43:21 UTC
After making changes to the autobuilder to build a minimal eSDK we observed unexpected eSDK sanity test failures. 

This is because the tests implicitly rely on a full eSDK installer, rather than installing any required components into the eSDK before testing.

In order to enable minimal eSDK installers on the autobuilder, and therefore for publishing with project releases, we need the eSDK sanity tests to handle running against a minimal eSDK installer.
Comment 1 Francisco Pedraza 2017-01-16 17:09:21 UTC
I just ran:
$ . oe-init-build-env
$ echo 'SDK_EXT_TYPE = "minimal' >> conf/local.conf
$ bitbake core-image-minimal core-image-sato -c populate_sdk_ext
$ echo 'INHERIT += "testsdk"' >> conf/local.conf
$ DISPLAY=:1 bitbake core-image-minimal core-image-sato -c testsdkext

and I successfully could reproduce the error.
Comment 2 Aníbal Limón 2017-01-16 21:07:51 UTC
Hi,

Currently there is no way to see what is installed in the extensible sdk see bug [1], so i suggest to add a checking like [2] but this will end in no tests runs on FULL eSDK because of the bug [1].

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=9038
[2] http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/sdk/gcc.py#n9
Comment 3 Aníbal Limón 2017-01-16 21:08:34 UTC
(In reply to comment #2)
> Hi,
> 
> Currently there is no way to see what is installed in the extensible sdk see
> bug [1], so i suggest to add a checking like [2] but this will end in no
> tests runs on FULL eSDK because of the bug [1].
> 
> [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=9038
> [2]
> http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/sdk/gcc.
> py#n9

I think the missing package is autotools, right Joshua?
Comment 4 Francisco Pedraza 2017-01-16 21:46:47 UTC
(In reply to comment #3)
> (In reply to comment #2)
> > Hi,
> > 
> > Currently there is no way to see what is installed in the extensible sdk see
> > bug [1], so i suggest to add a checking like [2] but this will end in no
> > tests runs on FULL eSDK because of the bug [1].
> > 
> > [1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=9038
> > [2]
> > http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/meta/lib/oeqa/sdk/gcc.
> > py#n9
> 
> I think the missing package is autotools, right Joshua?

Thanks Anibal, working on it.
Comment 5 Francisco Pedraza 2017-01-18 02:26:14 UTC
Implemented the same procedure as sdk/gcc.py
if not oeSDKTest.hasHostPackage(" " + machine):
        skipModule("SDK doesn't contain a cross-canadian toolchain").
Getting good skiping, however I got a regression for "test_extend_autotools_recipe_creation", will check this tomorrow.
Comment 6 Francisco Pedraza 2017-01-20 18:05:41 UTC
(In reply to comment #5)
> Implemented the same procedure as sdk/gcc.py
> if not oeSDKTest.hasHostPackage(" " + machine):
>         skipModule("SDK doesn't contain a cross-canadian toolchain").
> Getting good skiping, however I got a regression for
> "test_extend_autotools_recipe_creation", will check this tomorrow.

I just noticed that gcc is not installed on minimal and that could be the failure. Still working on it.
Comment 7 Paul Eggleton 2017-01-20 20:16:14 UTC
Currently that's by design, unless SDK_INCLUDE_TOOLCHAIN = "1" is set. The expectation is that you use devtool exclusively or you install the toolchain using devtool sdk-install.
Comment 8 Francisco Pedraza 2017-01-23 23:17:09 UTC
Created bug https://bugzilla.yoctoproject.org/show_bug.cgi?id=10958 due some issues after modified files at "~/oeqa/sdk/buildiptables.py" and "~/oeqa/sdk/buildcvs.py". That bug blockes this one.
Comment 9 Francisco Pedraza 2017-01-24 21:32:18 UTC
(In reply to comment #7)
> Currently that's by design, unless SDK_INCLUDE_TOOLCHAIN = "1" is set. The
> expectation is that you use devtool exclusively or you install the toolchain
> using devtool sdk-install.

Thank you Paul,
I just fixed this.
Regards
Comment 11 Francisco Pedraza 2017-01-25 17:09:47 UTC
Moved to "In Progress Review" http://lists.openembedded.org/pipermail/openembedded-core/2017-January/131773.html already sent the patch.
Comment 13 Ross Burton 2017-02-27 16:43:33 UTC
https://autobuilder.yoctoproject.org/main/builders/nightly-x86-64/builds/1132/steps/Running%20ESDK%20Sanity%20Tests/logs/stdio

The tests are not updated for minimal test cases, they just skip some of the tests but not all of them so the tests still fail.

In a minimal SDK the SDK has nothing in the sysroot.  I think the first thing the tests need to do is populate the SDK sysroot, which means running:

$ bitbake <IMAGENAME> -c rootfs-

(trailing hyphen)
Comment 14 Ross Burton 2017-03-08 17:24:39 UTC
oe-core 20c6e876ed6b8c8ac007848e2b36e9a8a56bb703 contains all the fixes.