12862 2018-07-25 13:29:35 +0000 do_package_setscene is unwillingly skipped 2026-03-11 21:25:01 +0000 1 1 7 Build System, Metadata & Runtime OE-Core core 2.5.1 x86 Multiple NEW Low normal Future 1 d-bazhenov richard.purdie mark.asselstine meta.mr.watcher meta.watcher randy.macleod richard.purdie --- New (Never tested) No (bug/feature does not impact docs) oldest_to_newest 81178 0 4350 d-bazhenov 2018-07-25 13:29:35 +0000 Created attachment 4350 bug-fix Hi everyone, do_package_setscene task can be unwillingly skipped in some situations. One such situation is to restore recipe to a previous state for which the recipe has built results stored in the shared state. When do_package_setscene is skipped, the recipe's packages-split directory is not updated accordingly to the stored state and its contents become out of sync. How to reproduce: 1. Make a recipe which installs files 'a' and 'b' and produces a package 'ab' which would contain both files. 2. Build the recipe. 3. Change the recipe to produce two packages 'a' and 'b' each containing a single file. 4. Build the recipe. 5. Restore original recipe (as in #1) 6. Build the recipe 7. Check ${workdir}/packages-split directory. There will still be two sub-directories for packages 'a' and 'b'. I was able to fix this bug by making a direct reference from do_build to do_package. This made do_package_setscene task 'unskippable' from the bitbake point of view. The attached patch fixes the bug. Please review. Best regards, Dmitry Bazhenov 81181 1 richard.purdie 2018-07-25 22:11:56 +0000 I'm not sure this is a bug since if anything actually depends on do_package, it would restore the correct artifact from sstate before allowing anything depending upon it to run. What problem is this causing other than some slightly stale data in a work directory? I suspect your patch will break assumptions made by rm_work.bbclass that the package task is removable and it will also cause builds from sstate to install much more data than they need to. 81182 2 d-bazhenov 2018-07-26 04:37:38 +0000 Hi Richard, As I understand package tasks are removed in rm_work.bbclass entirely. When removing, bitbake updates dependencies for the remaining tasks, so no harm is made with adding a direct reference in this regard. As for populating packages-split directory via do_package_setscene. It is the only task which is not run while do_packagedata_setscene and do_package_write_rpm are actually run restoring both the recipe-specific files and project-wise build artifacts. And the two latter tasks have more files to restore than do_package_setscene, so you shouldn't even notice much change. Especially because the typical usage scenario for this is when you restore a state of a single recipe or a group of recipes, but not the whole distribution. The stale data in the packages-split directory is misleading in understanding if everything is okay with the build, because there are discrepancies between split package data and generated packages. For example I can't be sure the configuration option which I introduced for my recipe to change the package generation (bundled package or a set of packages) works okay seeing that the packages-split directory contents do not coordinate with the option value. So even though the issue does not affect the finally built packages I still believe it is a bug. Regards, Dmitry 81189 3 richard.purdie 2018-07-26 14:49:21 +0000 We very nearly closed this as WONTFIX, instead I've left it as a low future, unassigned. The proposed fix will break rm_work and cause lots of extra sstate to be downloaded and unpacked even when it isn't used. I do agree there is a usability issue here. I'm not sure how we'd fix it though. 86562 4 mark.asselstine 2020-02-22 18:36:11 +0000 (In reply to comment #3) > We very nearly closed this as WONTFIX, instead I've left it as a low future, > unassigned. The proposed fix will break rm_work and cause lots of extra > sstate to be downloaded and unpacked even when it isn't used. > > I do agree there is a usability issue here. I'm not sure how we'd fix it > though. Rather than the approach of attempting to make the packages-split be properly sync'd with the state of the world to solve this, would it not be equally valid to assume it may be out of sync and thus just remove it. Thus, would removing the packages-split directory when SSTATETASKS doesn't include do_package_setscene be an option? 86563 5 richard.purdie 2020-02-23 11:15:15 +0000 (In reply to comment #4) > (In reply to comment #3) > > We very nearly closed this as WONTFIX, instead I've left it as a low future, > > unassigned. The proposed fix will break rm_work and cause lots of extra > > sstate to be downloaded and unpacked even when it isn't used. > > > > I do agree there is a usability issue here. I'm not sure how we'd fix it > > though. > > Rather than the approach of attempting to make the packages-split be > properly sync'd with the state of the world to solve this, would it not be > equally valid to assume it may be out of sync and thus just remove it. Yes, removing it would be better. The question is when to remove it and how to know when to do that. > Thus, would removing the packages-split directory when SSTATETASKS doesn't > include do_package_setscene be an option? SSTATETASKS is a fixed value, it doesn't dynamically change depending on system state unfortunately. My big worry with this bug is making things even more complex and fragile with the solution being worse than the original problem :(. There is code already there to remove conflicting/skipped recipes. We probably need to extend that to include sstate tasks which no longer match their hashes. 4350 2018-07-25 13:29:35 +0000 2018-07-25 13:29:35 +0000 bug-fix 0001-classes-package-add-direct-build-dependency-on-do_pa.patch text/plain 1483 d-bazhenov RnJvbSA0OWFiOTg3MTAyNDdiZWEwZjFlY2Y4YjU4MWNiODgzZmM4OGUzN2U1IE1vbiBTZXAgMTcg MDA6MDA6MDAgMjAwMQpGcm9tOiBEbWl0cnkgQmF6aGVub3YgPGRtaXRyeS5iYXpoZW5vdkBhdXJp Z2EuY29tPgpEYXRlOiBXZWQsIDI1IEp1bCAyMDE4IDE3OjI0OjU4ICswNTAwClN1YmplY3Q6IFtQ QVRDSF0gY2xhc3Nlcy9wYWNrYWdlOiBhZGQgZGlyZWN0IGJ1aWxkIGRlcGVuZGVuY3kgb24gZG9f cGFja2FnZQoKZG9fcGFja2FnZV9zZXRzY2VuZSB0YXNrIGNhbiBiZSB1bndpbGxpbmdseSBza2lw cGVkIGluIHNvbWUKc2l0dWF0aW9ucy4gQSBrbm93biBzaXR1YXRpb24gaXMgdG8gcmVzdG9yZSBy ZWNpcGUgdG8gYSBwcmV2aW91cwpzdGF0ZSBmb3Igd2hpY2ggdGhlIHJlY2lwZSBoYXMgYnVpbHQg cmVzdWx0cyBzdG9yZWQgaW4gdGhlCnNoYXJlZCBzdGF0ZS4KCldoZW4gZG9fcGFja2FnZV9zZXRz Y2VuZSBpcyBza2lwcGVkLCB0aGUgcmVjaXBlJ3MKcGFja2FnZXMtc3BsaXQgZGlyZWN0b3J5IGlz IG5vdCB1cGRhdGVkIGFjY29yZGluZ2x5IHRvIHRoZQpzdG9yZWQgc3RhdGUgYW5kIGl0cyBjb250 ZW50cyBiZWNvbWUgb3V0IG9mIHN5bmMuCgpUaGlzIHBhdGNoIGZpeGVzIHRoaXMgc2l0dWF0aW9u IGJ5IGFkZGluZyBhIGRpcmVjdCByZWZlcmVuY2UKZnJvbSBkb19idWlsZCB0YXNrIGJlY2F1c2Ug c2V0c2NlbmUgdGFza3MgdGhhdCBhcmUgZGlyZWN0bHkKcmVmZXJlbmNlZCBieSB0aGUgYnVpbGQg dGFzayBhcmUgdW5za2lwcGFibGUuCgpTaWduZWQtb2ZmLWJ5OiBEbWl0cnkgQmF6aGVub3YgPGRt aXRyeS5iYXpoZW5vdkBhdXJpZ2EuY29tPgotLS0KIG1ldGEvY2xhc3Nlcy9wYWNrYWdlLmJiY2xh c3MgfCAyICstCiAxIGZpbGUgY2hhbmdlZCwgMSBpbnNlcnRpb24oKyksIDEgZGVsZXRpb24oLSkK CmRpZmYgLS1naXQgYS9tZXRhL2NsYXNzZXMvcGFja2FnZS5iYmNsYXNzIGIvbWV0YS9jbGFzc2Vz L3BhY2thZ2UuYmJjbGFzcwppbmRleCBlZGVmZmE5NzhmLi41ODc5ZGQwMzRmIDEwMDY0NAotLS0g YS9tZXRhL2NsYXNzZXMvcGFja2FnZS5iYmNsYXNzCisrKyBiL21ldGEvY2xhc3Nlcy9wYWNrYWdl LmJiY2xhc3MKQEAgLTIyMzQsNyArMjIzNCw3IEBAIHB5dGhvbiBkb19wYWNrYWdlICgpIHsKIAog ZG9fcGFja2FnZVtkaXJzXSA9ICIke1NITElCU1dPUktESVJ9ICR7UEtHREVTVFdPUkt9ICR7RH0i CiBkb19wYWNrYWdlW3ZhcmRlcHNdICs9ICIke1BBQ0tBR0VCVUlMRFBLR0R9ICR7UEFDS0FHRVNQ TElURlVOQ1N9ICR7UEFDS0FHRUZVTkNTfSAke0BnZW5fcGFja2FnZXZhcihkKX0iCi1hZGR0YXNr IHBhY2thZ2UgYWZ0ZXIgZG9faW5zdGFsbAorYWRkdGFzayBwYWNrYWdlIGJlZm9yZSBkb19idWls ZCBhZnRlciBkb19pbnN0YWxsCiAKIFNTVEFURVRBU0tTICs9ICJkb19wYWNrYWdlIgogZG9fcGFj a2FnZVtjbGVhbmRpcnNdID0gIiR7UEtHREVTVH0gJHtQS0dERVNUV09SS30iCi0tIAoyLjE3LjEK Cg==