Bug 7019

Summary: Bitbake doesn't detect changes to SRC_URI files when using multiple bbappends
Product: [Build System, Metadata & Runtime] BitBake Reporter: Magnus Olsson <magnus>
Component: bitbakeAssignee: Richard Purdie <richard.purdie>
Status: VERIFIED FIXED QA Contact: Costin <costin.c.constantin>
Severity: normal    
Priority: Medium+ CC: bogdanx.a.voiculescu, poky.bs.watcher, poky.watcher
Version: unspecified   
Target Milestone: 1.8 M2   
Hardware: x86   
OS: Multiple   
Whiteboard: TC needed
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Don't know

Description Magnus Olsson 2014-12-03 09:58:26 UTC 
Bitbake doesn't detect all types of changes to SRC_URI if multiple layers provide different versions a SRC_URI file.

Consider the following case:

- meta-layer0 provides a "foo.bb" recipe. It's just a dummy do_build()-task.
- meta-layer1 extends foo.bb using "foo.bbappend", it adds a do_install()-task that installs FOO.txt. 
- meta-layer1 provides a default FOO.txt, but the idea is that another layer may replace it with "custom content".
- meta-layer2 is an example of such a customization. It also provides an "foo.bbappend" that inserts its own FOO.txt in front of meta-layer1 in FILESPATHEXTRA.

With all these layers enabled, "bitbake foo" will result in FOO.txt of meta-layer2 getting installed.
If FOO.txt of meta-layer2 is removed, "bitbake foo" should result in FOO.txt of meta-layer1 getting installed.
If FOO.txt of meta-layer2 is recreated, "bitbake foo" should again result in FOO.txt of meta-layer2 getting installed.
If FOO.txt of meta-layer1 is changed while FOO.txt of meta-layer2 is still present, Bitbake should not re-install FOO.txt.

I expect that Bitbake should detect revelant changes to FOO.txt, including if it disappears or reappears in any of the layers that provide it. This should be detected without invalidating sstate-cache.

However, the actual result is that the sstate-cache kicks in and Bitbake fails to detect the case when FOO.txt re-appears in meta-layer2.

To illustrate the issue, I have setup an example tree, get it using the following commands:

- git clone https://github.com/blastur/layer-append-test.git
- git submodule init
- git submodule update
- cd build
- source envsetup.sh

To test each of above test-cases, run "build/foo_test.sh". This script also contains some documentation on the different testcases, and what the expected result is. 

This may or may not be expected behaviour, but probably not the desired behavior. bluelightning / Paul Eggleton believes the problem is that the system does not re-scan paths for new files (FOO.txt), only deleted previously seen ones.
Comment 2 Costin 2015-04-13 10:44:27 UTC 
Hello,

Just tested on latest poky master hash: 84e8b1ba296c92ec00cfc5b5077ee253c4ed65ab.

Steps performed:
1) cloned fresh poky

2) created 3 new layers with necessary files in them:
cosscat@ubnt:/media/poky/3poky$ ls meta-layer?
meta-layer1:
conf  recipes-test

meta-layer2:
conf  recipes-test

meta-layer3:
conf  recipes-test

3) reproduced the original recipe files as per comment#1:
cosscat@ubnt:/media/poky/3poky$ ls meta-layer?/recipes-test
meta-layer1/recipes-test:
foo.bb

meta-layer2/recipes-test:
foo  foo.bbappend

meta-layer3/recipes-test:
foo  foo.bbappend

4) with both FOO.txt available in meta-layer2/recipes-test/foo and meta-layer3/recipes-test/foo
the result is:
cosscat@ubnt:/media/poky/3poky/build/tmp/work/i586-poky-linux/foo/1.0-r0$ cat FOO.txt:
This is from layer3 :)

5) changed FOO.txt from layer3:
cosscat@ubnt:/media/poky/3poky/meta-layer3/recipes-test/foo$ mv FOO.txt FOO1.txt
...(bitbake foo)
cosscat@ubnt:/media/poky/3poky/build/tmp/work/i586-poky-linux/foo/1.0-r0$ cat FOO.txt
from meta-layer2 ...

6) got FOO.txt back in layer3:
cosscat@ubnt:/media/poky/3poky/meta-layer3/recipes-test/foo$ mv FOO1.txt FOO.txt
...(bitbake foo)
cosscat@ubnt:/media/poky/3poky/build/tmp/work/i586-poky-linux/foo/1.0-r0$ cat FOO.txt
from meta-layer2 ...

The behavior should have been that after step 6, the FOO.txt from layer3 would be present in work dir. 

Costin
Comment 3 Costin 2015-04-13 10:46:41 UTC 
reopened as per last comment
Comment 4 Richard Purdie 2015-04-13 12:50:58 UTC 
This is not a correct reproducer of an problem. Why?

"""
cosscat@ubnt:/media/poky/3poky/build/tmp/work/i586-poky-linux/foo/1.0-r0$ cat FOO.txt
from meta-layer2 ...
"""

is expected. Why? The configuration had previously been built so it was installed from sstate, not rebuilt. This meant that WORKDIR was never really used and hence there is a stale file left there.

A better check is to check the tmp/deploy/xxx/foo_*.ipk file. You'll find that in each case this *does* have the correct contents.

Or as well as "mv FOO1.txt FOO.txt", add something to the file. In that case it will rebuild since there will not be a valid cache item from sstate to reuse.
Comment 5 Costin 2015-04-30 03:51:42 UTC 
Hello,

I checked as per comment#4 recommendations and indeed the expected results were present. 

I checked each time the content of the foo-1.0-r0.i586.rpm file from /media/poky/3poky/build/tmp/deploy/rpm/i586/ and found the expected result.

Steps performed: 

1) cd /media/poky/3poky/build/tmp/deploy/rpm/i586
2) mkdir test
3) bitbake foo # following comment#2 steps
4) cp foo-1.0-r0.i586.rpm test
5) rpm2cpio test/foo-1.0-r0.i586.rpm | cpio -idmv 
6) cat test/FOO.txt

Marking as verified.

Costin