Bug 10929

Summary: oe-selftest corrupts sstate cache
Product: [QA/Testing] Build Testing Reporter: Richard Purdie <richard.purdie>
Component: generalAssignee: Mariano Lopez <mariano.lopez>
Status: VERIFIED FIXED QA Contact: Jose Perez C <jose.perez.carranza>
Severity: major    
Priority: High CC: benjamin.esquivel, joshuagloe, juan.fernandox.ramos.frayle, leonardo.sandoval.gonzalez, mariano.lopez
Version: unspecified   
Target Milestone: 2.3 M3   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Bug Depends on:    
Bug Blocks: 10035, 10874    
Attachments:
Description Flags
find-10929.sh none

Description Richard Purdie 2017-01-16 14:11:42 UTC
If you run oe-selftest, then try to "bitbake core-image-sato -c populate_sdk_ext", it fails with unexpected setscene execution errors. This is because things get cleaned from the sstate cache by oe-selftest but the stamp files are left meaning the eSDK can't find the sstate files, leading to obtuse build failures.

The appoximate list of broken (removed) sstate is:
groff-native openssl cross-localedef-native db glibc-locale cryptodev-linux ptest-runner base-files libffi sqlite3 ncurses expat opkg-utils bzip2
Comment 1 Leonardo Sandoval Gonzalez 2017-01-16 16:14:58 UTC
this is a more general bug 10035. However, lets not mark this as duplicated because the latter needs to be divided into several ones, like this one.
Comment 2 Benjamin Esquivel 2017-01-18 23:57:27 UTC
I am re-routing this issue to the main triage.
Comment 3 Benjamin Esquivel 2017-01-20 19:58:25 UTC
I can't seem to reproduce this issue, I will post my procedure here:

1. clone poky
2. init build environment (non memres)
3. execute a selftest --run-all-tests
4. execute a bitbake core-image-sato -c populate_sdk_ext
5. expect failure

I assume that the sstates that will get corrupted thus 5 would fail are the ones located at the build/sstate-cache but this is the point where my procedure might be wrong to reproduce the issue. RP, what am I missing?
Comment 4 Richard Purdie 2017-01-23 14:44:42 UTC
(In reply to comment #3)
> I can't seem to reproduce this issue, I will post my procedure here:
> 
> 1. clone poky
> 2. init build environment (non memres)
> 3. execute a selftest --run-all-tests
> 4. execute a bitbake core-image-sato -c populate_sdk_ext
> 5. expect failure
> 
> I assume that the sstates that will get corrupted thus 5 would fail are the
> ones located at the build/sstate-cache but this is the point where my
> procedure might be wrong to reproduce the issue. RP, what am I missing?

You're missing a "bitbake core-image-sato -c populate_sdk_ext" before running selftest.
Comment 5 Benjamin Esquivel 2017-01-23 14:52:32 UTC
(In reply to comment #4)
> (In reply to comment #3)
> > I can't seem to reproduce this issue, I will post my procedure here:
> > 
> > 1. clone poky
> > 2. init build environment (non memres)
> > 3. execute a selftest --run-all-tests
> > 4. execute a bitbake core-image-sato -c populate_sdk_ext
> > 5. expect failure
> > 
> > I assume that the sstates that will get corrupted thus 5 would fail are the
> > ones located at the build/sstate-cache but this is the point where my
> > procedure might be wrong to reproduce the issue. RP, what am I missing?
> 
> You're missing a "bitbake core-image-sato -c populate_sdk_ext" before
> running selftest.

thanks, retrying.
Comment 6 Benjamin Esquivel 2017-01-23 21:50:55 UTC
Created attachment 3601 [details]
find-10929.sh

I wrote this script and ran it on an opensuse system and did not show failures at the populate_sdk_ext task. I am running it on a fedora system but I suspect this can be seen at the autobuilder with more clarity so I'll try to run the test there as well.
Comment 7 Leonardo Sandoval Gonzalez 2017-01-24 16:25:20 UTC
*** Bug 10035 has been marked as a duplicate of this bug. ***
Comment 8 Mariano Lopez 2017-02-01 15:51:14 UTC
After a conversation with Benjamin and Richard, the problem is as follows:

1. Start simultaneous builds along with selftest.
2. One of the builds will check for share states for X recipe.
3. Build found share states for X recipe.
4. selftest runs cleansstate task on recipe X.
5. Build tries to fetch share states for X recipe and fails.

So, the only way to do this is to avoid using cleansstate in selftest, it is needed to check for this task in selftest's tests and change all of them for something that perform the required test without deleting sstate.
Comment 9 Mariano Lopez 2017-02-09 16:32:15 UTC
Sent patch to the mailing list:

https://patchwork.openembedded.org/series/5239/#
Comment 10 Mariano Lopez 2017-03-02 16:12:47 UTC
This has been solved in commit:

http://cgit.openembedded.org/openembedded-core/commit/?id=62c61087a10cc3b26fbff32c9e2efd1704a39724
Comment 11 Juan Ramos 2017-06-14 21:23:40 UTC
Verified, tested with commit b6a1004c73820cff833f2b6f09d2432706f5447a, no SSTATES were corrupted after selftest