Bug 10929 - oe-selftest corrupts sstate cache
Summary: oe-selftest corrupts sstate cache
Status: VERIFIED FIXED
Alias: None
Product: Build Testing
Classification: QA/Testing
Component: general (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: High major
Target Milestone: 2.3 M3
Assignee: Mariano Lopez
QA Contact: Jose Perez C
URL:
Whiteboard:
: 10035 (view as bug list)
Depends on:
Blocks: 10035 10874
  Show dependency tree
 
Reported: 2017-01-16 14:11 UTC by Richard Purdie
Modified: 2017-06-14 21:23 UTC (History)
5 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments
find-10929.sh (409 bytes, text/plain)
2017-01-23 21:50 UTC, Benjamin Esquivel
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Richard Purdie 2017-01-16 14:11:42 UTC
If you run oe-selftest, then try to "bitbake core-image-sato -c populate_sdk_ext", it fails with unexpected setscene execution errors. This is because things get cleaned from the sstate cache by oe-selftest but the stamp files are left meaning the eSDK can't find the sstate files, leading to obtuse build failures.

The appoximate list of broken (removed) sstate is:
groff-native openssl cross-localedef-native db glibc-locale cryptodev-linux ptest-runner base-files libffi sqlite3 ncurses expat opkg-utils bzip2
Comment 1 Leonardo Sandoval Gonzalez 2017-01-16 16:14:58 UTC
this is a more general bug 10035. However, lets not mark this as duplicated because the latter needs to be divided into several ones, like this one.
Comment 2 Benjamin Esquivel 2017-01-18 23:57:27 UTC
I am re-routing this issue to the main triage.
Comment 3 Benjamin Esquivel 2017-01-20 19:58:25 UTC
I can't seem to reproduce this issue, I will post my procedure here:

1. clone poky
2. init build environment (non memres)
3. execute a selftest --run-all-tests
4. execute a bitbake core-image-sato -c populate_sdk_ext
5. expect failure

I assume that the sstates that will get corrupted thus 5 would fail are the ones located at the build/sstate-cache but this is the point where my procedure might be wrong to reproduce the issue. RP, what am I missing?
Comment 4 Richard Purdie 2017-01-23 14:44:42 UTC
(In reply to comment #3)
> I can't seem to reproduce this issue, I will post my procedure here:
> 
> 1. clone poky
> 2. init build environment (non memres)
> 3. execute a selftest --run-all-tests
> 4. execute a bitbake core-image-sato -c populate_sdk_ext
> 5. expect failure
> 
> I assume that the sstates that will get corrupted thus 5 would fail are the
> ones located at the build/sstate-cache but this is the point where my
> procedure might be wrong to reproduce the issue. RP, what am I missing?

You're missing a "bitbake core-image-sato -c populate_sdk_ext" before running selftest.
Comment 5 Benjamin Esquivel 2017-01-23 14:52:32 UTC
(In reply to comment #4)
> (In reply to comment #3)
> > I can't seem to reproduce this issue, I will post my procedure here:
> > 
> > 1. clone poky
> > 2. init build environment (non memres)
> > 3. execute a selftest --run-all-tests
> > 4. execute a bitbake core-image-sato -c populate_sdk_ext
> > 5. expect failure
> > 
> > I assume that the sstates that will get corrupted thus 5 would fail are the
> > ones located at the build/sstate-cache but this is the point where my
> > procedure might be wrong to reproduce the issue. RP, what am I missing?
> 
> You're missing a "bitbake core-image-sato -c populate_sdk_ext" before
> running selftest.

thanks, retrying.
Comment 6 Benjamin Esquivel 2017-01-23 21:50:55 UTC
Created attachment 3601 [details]
find-10929.sh

I wrote this script and ran it on an opensuse system and did not show failures at the populate_sdk_ext task. I am running it on a fedora system but I suspect this can be seen at the autobuilder with more clarity so I'll try to run the test there as well.
Comment 7 Leonardo Sandoval Gonzalez 2017-01-24 16:25:20 UTC
*** Bug 10035 has been marked as a duplicate of this bug. ***
Comment 8 Mariano Lopez 2017-02-01 15:51:14 UTC
After a conversation with Benjamin and Richard, the problem is as follows:

1. Start simultaneous builds along with selftest.
2. One of the builds will check for share states for X recipe.
3. Build found share states for X recipe.
4. selftest runs cleansstate task on recipe X.
5. Build tries to fetch share states for X recipe and fails.

So, the only way to do this is to avoid using cleansstate in selftest, it is needed to check for this task in selftest's tests and change all of them for something that perform the required test without deleting sstate.
Comment 9 Mariano Lopez 2017-02-09 16:32:15 UTC
Sent patch to the mailing list:

https://patchwork.openembedded.org/series/5239/#
Comment 10 Mariano Lopez 2017-03-02 16:12:47 UTC
This has been solved in commit:

http://cgit.openembedded.org/openembedded-core/commit/?id=62c61087a10cc3b26fbff32c9e2efd1704a39724
Comment 11 Juan Ramos 2017-06-14 21:23:40 UTC
Verified, tested with commit b6a1004c73820cff833f2b6f09d2432706f5447a, no SSTATES were corrupted after selftest