Bug 11300 - selftest: do_populate_sdk_ext task failing on eSDK test on GDC ABs
Summary: selftest: do_populate_sdk_ext task failing on eSDK test on GDC ABs
Status: VERIFIED FIXED
Alias: None
Product: eSDK
Classification: Yocto Project Subprojects
Component: eSDK (show other bugs)
Version: 2.3
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 2.4 M1
Assignee: Jose Perez C
QA Contact: oscar lopez arandas
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2017-04-05 21:42 UTC by Jose Perez C
Modified: 2017-06-15 14:31 UTC (History)
4 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments
esdk failure on AB worker (55.42 KB, text/plain)
2017-04-06 13:04 UTC, Jose Perez C
no flags Details
log.do_test_debian8 (166.99 KB, application/octet-stream)
2017-06-15 14:31 UTC, oscar lopez arandas
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Jose Perez C 2017-04-05 21:42:56 UTC
Commit: eff56e4f0d59b1d965a68e4f009b7f07717b7edd

Steps on AB:

1. Execute Selftest on GDC ABs 

Steps Manually:
1. poky/build/oe-selftest -r eSDK (only execute eSDK)
or
1. poky/build/oe-selftest --run-all (execute all selftest)

Expected Result:
All test cases should be pass 

Actual Result:
eSDK test cases are not being executed an error is displayed:

ERROR: core-image-minimal-1.0-r0 do_populate_sdk_ext: Failed to generate filtered task list for extensible SDK:


Note: I could not reproduce this on the server neither on my local machine

log : http://errors.yoctoproject.org/Errors/Details/139068/
Comment 1 Paul Eggleton 2017-04-05 22:00:32 UTC
Looking at the the logs, it determined it needed to run linux-yocto.do_populate_lic. Interestingly linux-yocto.do_populate_lic_setscene isn't shown as even being attempted within the internal eSDK task list run, which almost certainly means that the matching sstate package wasn't available in the sstate-cache. This suggests one of the following must have occurred:

1) The signature has changed e.g. because some variable value is different.

2) The sstate package was missing or otherwise inaccessible when the availability check went to find it.

Unfortunately we are still not logging enough to be able to tell which of these has occurred.

How reproducible is this?
Comment 2 Jose Perez C 2017-04-06 12:36:53 UTC
(In reply to comment #1)
> Looking at the the logs, it determined it needed to run
> linux-yocto.do_populate_lic. Interestingly
> linux-yocto.do_populate_lic_setscene isn't shown as even being attempted
> within the internal eSDK task list run, which almost certainly means that
> the matching sstate package wasn't available in the sstate-cache. This
> suggests one of the following must have occurred:
> 
> 1) The signature has changed e.g. because some variable value is different.
> 
> 2) The sstate package was missing or otherwise inaccessible when the
> availability check went to find it.
> 
> Unfortunately we are still not logging enough to be able to tell which of
> these has occurred.
> 
> How reproducible is this?

is 100% reproducible on all the workers of the ABs.
Comment 3 Jose Perez C 2017-04-06 13:04:04 UTC
Created attachment 3690 [details]
esdk failure on AB worker
Comment 4 Jose Perez C 2017-04-06 13:06:00 UTC
On manual execution of the task inside of the worker a different error is happening, is on the do_compile task for perl. 

- poky/build$ bitbake core-image-minimal -c populate_sdk_ext
.
.
.
6: linux-yocto-4.10.5+gitAUTOINC+01f18cba44_53be19cad6-r0 do_fetch (pidERROR: perl-native-5.24.1-r0 do_compile: oe_runmake failed


log attached.
Comment 5 Leonardo Sandoval Gonzalez 2017-04-14 18:17:18 UTC
I started a build on local AB centos7 machine but stop responding at some point, the latter is one of the machines that the error is coming. Assigning the bug to Anibal, he is more experienced in perl issues.
Comment 6 Aníbal Limón 2017-04-17 21:24:37 UTC
(In reply to comment #5)
> I started a build on local AB centos7 machine but stop responding at some
> point, the latter is one of the machines that the error is coming. Assigning
> the bug to Anibal, he is more experienced in perl issues.

I tested on local AB debian8 machine and the build was successful, the log about perl failure doesn't said anything only that fails to build docs. More investigation needed.
Comment 7 Aníbal Limón 2017-04-18 20:59:48 UTC
(In reply to comment #6)
> (In reply to comment #5)
> > I started a build on local AB centos7 machine but stop responding at some
> > point, the latter is one of the machines that the error is coming. Assigning
> > the bug to Anibal, he is more experienced in perl issues.
> 
> I tested on local AB debian8 machine and the build was successful, the log
> about perl failure doesn't said anything only that fails to build docs. More
> investigation needed.

I rebuild using the hash eff56e4f0d59b1d965a68e4f009b7f07717b7edd without failure.
Comment 8 Jose Perez C 2017-04-21 19:29:36 UTC
I failing again on commit : 55a53f53263ff4bf7bd6d0c454c464e595e11e4c

ERROR: Task linux-yocto.do_fetch attempted to execute unexpectedly
ERROR: Task linux-yocto.do_unpack attempted to execute unexpectedly
ERROR: Task linux-yocto.do_kernel_checkout attempted to execute unexpectedly
ERROR: Task linux-yocto.do_validate_branches attempted to execute unexpectedly
ERROR: Task linux-yocto.do_kernel_metadata attempted to execute unexpectedly
ERROR: Task linux-yocto.do_patch attempted to execute unexpectedly
ERROR: Task linux-yocto.do_populate_lic attempted to execute unexpectedly and should have been setscened
Comment 9 Jose Perez C 2017-04-26 14:03:00 UTC
Patch already sent to the list 

http://lists.openembedded.org/pipermail/openembedded-core/2017-April/136107.html
Comment 10 Jose Perez C 2017-04-27 15:08:58 UTC
Moving to 2.4 M1, this is not a release blocker and is only affecting GDC ABs
Comment 11 Richard Purdie 2017-04-28 08:07:33 UTC
I think the real issue here is that this test uses write_config instead of append_config and is hence overwriting the config from the setup method. We should likely change the test to use append_config too.
Comment 12 Aníbal Limón 2017-04-28 15:41:16 UTC
(In reply to comment #11)
> I think the real issue here is that this test uses write_config instead of
> append_config and is hence overwriting the config from the setup method. We
> should likely change the test to use append_config too.

I made only this change and fire a build into the GDC cluster.
Comment 13 Aníbal Limón 2017-05-01 17:12:33 UTC
(In reply to comment #12)
> (In reply to comment #11)
> > I think the real issue here is that this test uses write_config instead of
> > append_config and is hence overwriting the config from the setup method. We
> > should likely change the test to use append_config too.
> 
> I made only this change and fire a build into the GDC cluster.

The test passes, i sent other version of the patch to the ML,

http://lists.openembedded.org/pipermail/openembedded-core/2017-May/136205.html
Comment 14 Jose Perez C 2017-05-05 16:04:42 UTC
(In reply to comment #13)
> (In reply to comment #12)
> > (In reply to comment #11)
> > > I think the real issue here is that this test uses write_config instead of
> > > append_config and is hence overwriting the config from the setup method. We
> > > should likely change the test to use append_config too.
> > 
> > I made only this change and fire a build into the GDC cluster.
> 
> The test passes, i sent other version of the patch to the ML,
> 
> http://lists.openembedded.org/pipermail/openembedded-core/2017-May/136205.
> html

Just sent a patch that uses default configuration of oe-selftest (qemux86-64) 

http://lists.openembedded.org/pipermail/openembedded-core/2017-May/136359.html
Comment 16 oscar lopez arandas 2017-06-15 14:30:38 UTC
ENVIRONMENT:	Debian 8 x86_64
Merged in master
BUILD: 38c06ab45084904b28ed6588f147e6379cf8475c

Steps on AB:

1. Execute Selftest on GDC ABs 

Steps Manually:
1. poky/build/oe-selftest -r eSDK (only execute eSDK)
or
1. poky/build/oe-selftest --run-all (execute all selftest)

Expected Result:
All test cases should be pass 

Actual Result:
All test cases pass with success

full log in attachment
Comment 17 oscar lopez arandas 2017-06-15 14:31:21 UTC
Created attachment 3822 [details]
log.do_test_debian8