Bug 3484 - do_package_write_* reexecuted on checksum change even with OEBasic
Summary: do_package_write_* reexecuted on checksum change even with OEBasic
Status: RESOLVED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: configuration (show other bugs)
Version: 1.4
Hardware: x86 Multiple
: Medium normal
Target Milestone: 1.4
Assignee: Richard Purdie
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2012-11-23 10:55 UTC by Martin Jansa
Modified: 2012-12-06 16:14 UTC (History)
1 user (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Martin Jansa 2012-11-23 10:55:29 UTC
I know that OEBasicHash is now preferred and probably will became mandatory later, but this behavior was bothering me for long time and maybe it would be worth fixing in older metadata or will reveal some other hidden issue.

2 very small recipes with dependency:
$ cat openembedded-core/meta/recipes-extended/package-test/a.bb 
inherit allarch
ALLOW_EMPTY_${PN} = "1"
LICENSE = "CLOSED"
$ cat openembedded-core/meta/recipes-extended/package-test/b.bb 
RDEPENDS_${PN} = "a"
inherit allarch
ALLOW_EMPTY_${PN} = "1"
LICENSE = "CLOSED"

First correct behavior with OEBasicHash:
$ bitbake b
properly builds them both

PR bump in a.bb
$ bitbake b | tee
properly builds "a" completely and reexecutes do_package_write_ipk in "b"

Switch to OEBasic
undo PR bump in a.bb
remove tmpdir and sstate-cache
$ bitbake b 
properly builds them both
a:
1.0-r0.do_compile                                             1.0-r0.do_package
1.0-r0.do_compile.sigdata.5571ce34d93789fdb026c44e75f10349    1.0-r0.do_package.sigdata.d1a900870369f1310631199dc895e729
1.0-r0.do_configure                                           1.0-r0.do_package_write
1.0-r0.do_configure.sigdata.dbd79124846df45e2491c6784f0458f0  1.0-r0.do_patch
1.0-r0.do_fetch                                               1.0-r0.do_patch.sigdata.ba7fc91148e89ae3e95e95f67dcc755c
1.0-r0.do_fetch.sigdata.336e0866ad2cce78308a14590563b9f5      1.0-r0.do_unpack
1.0-r0.do_install                                             1.0-r0.do_unpack.sigdata.d05875b69370fd4d7cb272ca00b15a28
1.0-r0.do_install.sigdata.2471ac4e82fe00be055afb01baf3faa6

b:
1.0-r0.do_build                                               1.0-r0.do_package.sigdata.0dfb7ce053e2d7752bdf8692495219a2
1.0-r0.do_compile                                             1.0-r0.do_package_write
1.0-r0.do_compile.sigdata.5571ce34d93789fdb026c44e75f10349    1.0-r0.do_patch
1.0-r0.do_configure                                           1.0-r0.do_patch.sigdata.ba7fc91148e89ae3e95e95f67dcc755c
1.0-r0.do_configure.sigdata.dbd79124846df45e2491c6784f0458f0  1.0-r0.do_populate_lic
1.0-r0.do_fetch                                               1.0-r0.do_populate_lic.sigdata.5c7ca26ebf17125436dfb70736a578f3
1.0-r0.do_fetch.sigdata.336e0866ad2cce78308a14590563b9f5      1.0-r0.do_populate_sysroot.qemux86-64
1.0-r0.do_install                                             1.0-r0.do_populate_sysroot.sigdata.5214a254a7e6e364908ac18597671161
1.0-r0.do_install.sigdata.2471ac4e82fe00be055afb01baf3faa6    1.0-r0.do_unpack
1.0-r0.do_package                                             1.0-r0.do_unpack.sigdata.d05875b69370fd4d7cb272ca00b15a28

PR bump in a.bb
$ bitbake b | tee
properly builds "a" completely, but reexecutes do_package_write_ipk in "b", which should be covered by 1.0-r0.do_package_write stamp IMHO

diff -rq all-oe-linux-before/ all-oe-linux
show only new "a" stamps

OE qemux86-64@ ~/oe-core/tmp-eglibc/stamps $ ls -lah all-oe-linux*/b/*package*
-rw-r--r-- 1 bitbake bitbake    0 Nov 23 11:36 all-oe-linux-before/b/1.0-r0.do_package
-rw-rw-r-- 1 bitbake bitbake 103K Nov 23 11:36 all-oe-linux-before/b/1.0-r0.do_package.sigdata.0dfb7ce053e2d7752bdf8692495219a2
-rw-r--r-- 1 bitbake bitbake    0 Nov 23 11:36 all-oe-linux-before/b/1.0-r0.do_package_write
-rw-r--r-- 1 bitbake bitbake    0 Nov 23 11:36 all-oe-linux/b/1.0-r0.do_package
-rw-rw-r-- 1 bitbake bitbake 103K Nov 23 11:36 all-oe-linux/b/1.0-r0.do_package.sigdata.0dfb7ce053e2d7752bdf8692495219a2
-rw-r--r-- 1 bitbake bitbake    0 Nov 23 11:37 all-oe-linux/b/1.0-r0.do_package_write

And it does not do this when the dependency is transitive, but a lot of do_package_write_ipk_setscene is called:
$ cat openembedded-core/meta/recipes-extended/package-test/c.bb 
RDEPENDS_${PN} = "b"
inherit allarch
ALLOW_EMPTY_${PN} = "1"
LICENSE = "CLOSED"

$ bitbake c
...
NOTE: Executing SetScene Tasks
NOTE: Running setscene task 106 of 107 (/OE/oe-core/openembedded-core/meta/recipes-extended/package-test/b.bb, do_package_write_ipk_setscene)
NOTE: Running setscene task 107 of 107 (/OE/oe-core/openembedded-core/meta/recipes-extended/package-test/a.bb, do_package_write_ipk_setscene)
NOTE: recipe b-1.0-r0: task do_package_write_ipk_setscene: Started
NOTE: recipe a-1.0-r1: task do_package_write_ipk_setscene: Started
NOTE: recipe b-1.0-r0: task do_package_write_ipk_setscene: Succeeded
NOTE: recipe a-1.0-r1: task do_package_write_ipk_setscene: Succeeded
NOTE: Executing RunQueue Tasks
NOTE: Running task 231 of 360 (ID: 4, /OE/oe-core/openembedded-core/meta/recipes-extended/package-test/c.bb, do_fetch)
NOTE: recipe c-1.0-r0: task do_fetch: Started
..
NOTE: Running noexec task 360 of 360 (ID: 10, /OE/oe-core/openembedded-core/meta/recipes-extended/package-test/c.bb, do_build)
NOTE: Tasks Summary: Attempted 360 tasks of which 348 didn't need to be rerun and all succeeded.

Another PR bump in a.bb
$ bitbake c
...
NOTE: Executing SetScene Tasks
NOTE: Running setscene task 107 of 107 (/OE/oe-core/openembedded-core/meta/recipes-extended/package-test/c.bb, do_package_write_ipk_setscene)
NOTE: recipe c-1.0-r0: task do_package_write_ipk_setscene: Started
NOTE: recipe c-1.0-r0: task do_package_write_ipk_setscene: Succeeded
NOTE: Executing RunQueue Tasks
NOTE: Running task 237 of 360 (ID: 294, /OE/oe-core/openembedded-core/meta/recipes-extended/package-test/a.bb, do_fetch)
NOTE: recipe a-1.0-r2: task do_fetch: Started
...
NOTE: recipe a-1.0-r2: task do_package_write_ipk: Succeeded
NOTE: Running noexec task 359 of 360 (ID: 297, /OE/oe-core/openembedded-core/meta/recipes-extended/package-test/a.bb, do_package_write)
NOTE: Tasks Summary: Attempted 360 tasks of which 351 didn't need to be rerun and all succeeded.

$ bitbake b
...
NOTE: Executing SetScene Tasks
NOTE: Running setscene task 105 of 105 (/OE/oe-core/openembedded-core/meta/recipes-extended/package-test/a.bb, do_package_write_ipk_setscene)
NOTE: recipe a-1.0-r2: task do_package_write_ipk_setscene: Started
NOTE: recipe a-1.0-r2: task do_package_write_ipk_setscene: Succeeded
NOTE: Executing RunQueue Tasks
NOTE: Tasks Summary: Attempted 351 tasks of which 351 didn't need to be rerun and all succeeded.

And this happens not only with allarch recipes.
Comment 1 Richard Purdie 2012-11-28 12:41:47 UTC
Before any given task runs, we clean out any stamps using the clean mask. I have strong evidence the problem is the build ends up executing the task:

package_write_ipk

then the task 

package_write

and before executing package_write, it wipes out package_write*, taking the stamp from the ipk task with it.

At the moment this is a suspicion, I don't know how widespread the problem is, or how we fix it.