Bug 522

Summary: enable sstate prebuilt for different build directory
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Kevin Tian <kevin.tian>
Component: coreAssignee: Richard Purdie <richard.purdie>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium CC: ke.yu, kevin.tian, meta.mr.watcher, meta.watcher, sgw, tom.rini
Version: 0.9   
Target Milestone: 1.0 M1   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: ---
Bug Depends on:    
Bug Blocks: 507    

Description Kevin Tian 2010-11-05 00:45:49 UTC
So far the prebuilt packages can be only reused if we use same build directory path as the one generating prebuilts. It's perhaps fine for delivering prebuilts to customer, but not good for developers who normally have various build directories which are removed and created occasionally. Having a prebuilt collection would ease those build-from-scratch tasks.

At least there're two factors blocking this purpose so far:
    PWD variable
    do_populate_sysroot encodes full path

For the first one we can easily add to base hash whitelist. But for 2nd one we may need more work. Perhaps we can split do_populate_sysroot into two stages, with the 1st stage installed to local sysroot directory, and the latter stage actually installed to target sysroot. This way 1st stage can be reused w/o need to rebuild from scratch
Comment 1 Richard Purdie 2010-11-10 14:25:06 UTC
The PWD part has now been fixed in master at least. Is the full path problem with do_populate_sysroot still present? Its possible this was already fixed close to the 0.9 release.
Comment 2 Tom Rini 2010-11-10 14:35:23 UTC
As of 2e0ab8c18f1869a6fd1f53140fed0b4f3a1c3274 it's still not right (mkdir .../build2, cp -a build/conf build2/, modify SSTATE_CACHE).
Comment 3 Kevin Tian 2010-11-10 16:49:06 UTC
I think do_populate_sysroot issue I observed last time did disappear now. So you're right that it has been fixed around 0.9.

I did a full sato build, there're still 31 packages which didn't reuse prebuilt. I haven't got time to look into the reason, but I'll do some work today and perhaps open a new bug for that.
Comment 4 Kevin Tian 2010-11-10 16:50:46 UTC
(In reply to comment #2)
> As of 2e0ab8c18f1869a6fd1f53140fed0b4f3a1c3274 it's still not right (mkdir
> .../build2, cp -a build/conf build2/, modify SSTATE_CACHE).

Tom, could you elaborate your problem? Is it related to the issues stated by this bug? I know there're still some problems with sstate health, but if yours is a new problem, then we may open a new bug to track the exact problem more closely...
Comment 5 Tom Rini 2010-11-11 06:02:51 UTC
(In reply to comment #4)
> (In reply to comment #2)
> > As of 2e0ab8c18f1869a6fd1f53140fed0b4f3a1c3274 it's still not right (mkdir
> > .../build2, cp -a build/conf build2/, modify SSTATE_CACHE).
> 
> Tom, could you elaborate your problem? Is it related to the issues stated by
> this bug? I know there're still some problems with sstate health, but if yours
> is a new problem, then we may open a new bug to track the exact problem more
> closely...

I think I hit the post button a tad too quickly.  What I have seen is that if I build say poky-image-base in /tmp/poky/build and then poky-image-basic in /tmp/poky/build, and then create a /tmp/poky/build2 which points at /tmp/poky/build/sstate-cache for SSTATE_CACHE, gcc-runtime is (a) attempted to be built rather than found in the cache and (b) fails with (from config.log):
configure:3605: checking for i586-poky-linux-gcc -march=i586 --sysroot=/home/trini/work/poky/build2/tmp/sysroots/i586-poky-linux option to accept ISO C89
configure:3669: i586-poky-linux-gcc -march=i586 --sysroot=/home/trini/work/poky/build2/tmp/sysroots/i586-poky-linux  -c -fexpensive-optimizations -fomit-frame-pointer -frename-registers -O2 -ggdb -feliminate-unused-debug-types  conftest.c >&5
conftest.c:12:19: fatal error: stdio.h: No such file or directory

This is with MACHINE=qemux86 (local.conf is otherwise stock except for -j6 and BB_NUMBER_THREADS=4) on ubuntu 9.10 x86_64 host.
Comment 6 Kevin Tian 2010-11-19 00:46:32 UTC
(In reply to comment #5)
> (In reply to comment #4)
> > (In reply to comment #2)
> > > As of 2e0ab8c18f1869a6fd1f53140fed0b4f3a1c3274 it's still not right (mkdir
> > > .../build2, cp -a build/conf build2/, modify SSTATE_CACHE).
> > 
> > Tom, could you elaborate your problem? Is it related to the issues stated by
> > this bug? I know there're still some problems with sstate health, but if yours
> > is a new problem, then we may open a new bug to track the exact problem more
> > closely...
> I think I hit the post button a tad too quickly.  What I have seen is that if I
> build say poky-image-base in /tmp/poky/build and then poky-image-basic in
> /tmp/poky/build, and then create a /tmp/poky/build2 which points at
> /tmp/poky/build/sstate-cache for SSTATE_CACHE, gcc-runtime is (a) attempted to
> be built rather than found in the cache and (b) fails with (from config.log):
> configure:3605: checking for i586-poky-linux-gcc -march=i586
> --sysroot=/home/trini/work/poky/build2/tmp/sysroots/i586-poky-linux option to
> accept ISO C89
> configure:3669: i586-poky-linux-gcc -march=i586
> --sysroot=/home/trini/work/poky/build2/tmp/sysroots/i586-poky-linux  -c
> -fexpensive-optimizations -fomit-frame-pointer -frename-registers -O2 -ggdb
> -feliminate-unused-debug-types  conftest.c >&5
> conftest.c:12:19: fatal error: stdio.h: No such file or directory
> This is with MACHINE=qemux86 (local.conf is otherwise stock except for -j6 and
> BB_NUMBER_THREADS=4) on ubuntu 9.10 x86_64 host.

So this is the similar issue that I'm seeing now. The bootstrap process is somewhat broken with sstate when some of gcc/libc stages change. gcc-runtime assumes gcc-cross-intermediate and thus eglibc-initial ready, however sstate.bbclass doesn't install anything for -intermediate and -initial prebuilts.

You suggested to use deptasks from pstage side, and I'll try that later.
Comment 7 Richard Purdie 2011-02-01 10:02:34 UTC
The toolchain bootstrap process has been changed so this set of problems no longer exists in master.