Bug 14342

Summary: Setscene tasks in both covered and notcovered errors at the end of build
Product: [Build System, Metadata & Runtime] BitBake Reporter: Nathan.Dunne
Component: bitbakeAssignee: Richard Purdie <richard.purdie>
Status: RESOLVED FIXED QA Contact:
Severity: minor    
Priority: Medium+ CC: alex.kanavin, anuj.mittal, diego.sueiro, JPEWhacker, poky.bs.watcher, poky.watcher, randy.macleod, stefan.herbrechtsmeier
Version: unspecified   
Target Milestone: 3.4 M3   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
combined config files for reproducing the errors
none
Failing-bitbake-cookerdaemon.log
none
Failing-console-latest.log none

Description Nathan.Dunne 2021-04-08 10:41:12 UTC
Created attachment 4785 [details]
combined config files for reproducing the errors

When building 3+ multiconfigs that share recipes, with no sstate cache, at the end of the build recipes from poky/meta/ give errors similar to:

ERROR: Setscene task mc:host:virtual:native:.../poky/meta/recipes-support/sqlite/sqlite3_3.35.0.bb:do_populate_lic in both covered and notcovered.

The images produced are still correct but a large number of errors appear at the end of the build. Re-running the same bitbake command immediately now that the sstate cache has been populated results in the build completing without errors.

Attached is the config files for reproducing the errors
Comment 1 Richard Purdie 2021-04-08 16:55:10 UTC
I did run this and can reproduce. A key bit of information is that before the ERROR messages about covered/notcovered, there are a ton of warnings like:

WARNING: Runqeueue deadlocked on deferred tasks, forcing task mc:host:/media/build1/poky/meta/recipes-core/base-passwd/base-passwd_3.5.29.bb:do_populate_sysroot

Items warned about then appear in the error list so I believe they're related and this should go a long way to uncovering why it is failing. Please do include warnings in future bug reports as they can hint at the issues! :)
Comment 2 Richard Purdie 2021-04-09 13:11:29 UTC
I believe I have a patch which addresses the issue:

From 504c5f6581949c0da3c66fd678e301bbc97fb0c3 Mon Sep 17 00:00:00 2001
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Date: Fri, 9 Apr 2021 14:03:52 +0100
Subject: runqueue: Fix deferred task issues

In a multiconfig situation there are circumstances where firstly, tasks
are deferred when they shouldn't be, then later, tasks can end up as
both covered and not covered.

This patch fixes two related issues. Firstly, the stamp validity checking
is done up front in the build and not reevaulated. When rebuilding the
deferred task list after scenequeue hash change updates, we need therefore
need to check if a task was in notcovered *or* covered when deciding to
defer it. This avoids strange logs like:

NOTE: Running setscene task X of Y (mc:initrfs_guest:/A/alsa-state.bb:do_deploy_source_date_epoch_setscene)
NOTE: Deferring mc:initrfs_guest:/A/alsa-state.bb:do_deploy_source_date_epoch after mc:host:/A/alsa-state.bb:do_deploy_source_date_epoch

where tasks have run but are then deferred.

Since we're recalculating the whole list, we also need to clear it before
iterating to rebuild it. By ensuring covered tasks aren't added to the
deferred queue, the covered + notcovered issue should also be avoided.
in the task deadlock forcing code.

[YOCTO #14342]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 bitbake/lib/bb/runqueue.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
index cd56a55472..feb42d3e6b 100644
--- a/bitbake/lib/bb/runqueue.py
+++ b/bitbake/lib/bb/runqueue.py
@@ -2845,6 +2845,7 @@ def update_scenequeue_data(tids, sqdata, rqdata, rq, cooker, stampcache, sqrq, s
     sqdata.valid |= rq.validate_hashes(tocheck, cooker.data, len(sqdata.stamppresent), False, summary=summary)
 
     sqdata.hashes = {}
+    sqrq.sq_deferred = {}
     for mc in sorted(sqdata.multiconfigs):
         for tid in sorted(sqdata.sq_revdeps):
             if mc_from_tid(tid) != mc:
@@ -2857,6 +2858,9 @@ def update_scenequeue_data(tids, sqdata, rqdata, rq, cooker, stampcache, sqrq, s
                 continue
             if tid in sqrq.scenequeue_notcovered:
                 continue
+            if tid in sqrq.scenequeue_covered:
+                continue
+
             sqdata.outrightfail.add(tid)
 
             h = pending_hash_index(tid, rqdata)
Comment 4 Nathan.Dunne 2021-04-19 13:06:51 UTC
Hi Richard,

I've tested with the latest master of poky and the errors are gone, but I'm still seeing the warnings like:

WARNING: Runqeueue deadlocked on deferred tasks, forcing task mc:docker_guest:virtual:native:/work1/natdun02/oss/ais/yocto/refstack/poky/meta/recipes-devtools/pseudo/pseudo_git.bb:do_populate_sysroot

Is this expected?
Comment 5 Richard Purdie 2021-04-19 15:58:26 UTC
(In reply to comment #4)
> Hi Richard,
> 
> I've tested with the latest master of poky and the errors are gone, but I'm
> still seeing the warnings like:
> 
> WARNING: Runqeueue deadlocked on deferred tasks, forcing task
> mc:docker_guest:virtual:native:/work1/natdun02/oss/ais/yocto/refstack/poky/
> meta/recipes-devtools/pseudo/pseudo_git.bb:do_populate_sysroot
> 
> Is this expected?

It is hard to tell without seeing the configuration. You can certainly create a set of configurations which can deadlock, in those cases it should figure out how to carefully advance out of it. Was that with the test case in this bug or a different one?
Comment 6 Nathan.Dunne 2021-04-20 08:00:46 UTC
Hi Richard, the same configuration as was triggering both the errors and the warnings before is now giving just the warnings. 

I was unsure if your patch was intending to resolve the warnings as well, or just the errors?
Comment 7 Richard Purdie 2021-04-21 21:32:35 UTC
(In reply to comment #6)
> Hi Richard, the same configuration as was triggering both the errors and the
> warnings before is now giving just the warnings. 
> 
> I was unsure if your patch was intending to resolve the warnings as well, or
> just the errors?

In my local tests I thought it had resolved both. I'd best recheck as it sounds like there is some further issue lurking here :(
Comment 8 Richard Purdie 2021-04-29 19:40:04 UTC
(In reply to comment #7)
> (In reply to comment #6)
> > Hi Richard, the same configuration as was triggering both the errors and the
> > warnings before is now giving just the warnings. 
> > 
> > I was unsure if your patch was intending to resolve the warnings as well, or
> > just the errors?
> 
> In my local tests I thought it had resolved both. I'd best recheck as it
> sounds like there is some further issue lurking here :(

I've tried several of these builds now with master and I can't get the warnings. 

Are you using master? 
Are you using hash equivalence?
Can you still reproduce the warnings? 
Do you have any simpler reproducer (e.g. with core-image-minimal instead of sato)?
Any pointers on how to reproduce welcome...

I have another fix to optimise some sstate reuse for deferred tasks which should also help but I think unrelated to the issue mentioned here.
Comment 9 Richard Purdie 2021-05-02 22:55:36 UTC
I merged a couple of extra patches related to other issues I spotted with this code, I'm going to need more info to more any further forward if there is still an issue though.
Comment 10 Randy MacLeod 2021-05-13 15:27:05 UTC
Nathan?
Comment 11 Diego Sueiro 2021-06-11 12:34:57 UTC
Created attachment 4805 [details]
Failing-bitbake-cookerdaemon.log
Comment 12 Diego Sueiro 2021-06-11 12:35:30 UTC
Created attachment 4806 [details]
Failing-console-latest.log
Comment 13 Diego Sueiro 2021-06-11 12:37:34 UTC
The issue was fixed with http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=7ea37b929153e112b8d843c05fab06354aa575fc, where we were only seeing the deadlock warning messages.

After http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/bitbake/lib/bb/runqueue.py?id=76891afd76679b00ac620070cd90c9d53c7f6262, we started having build issues with the logs attached (Failing-bitbake-cookerdaemon.log and Failing-console-latest.log).

This is how to reproduce the build environment:
Clean SSTATE_DIR

conf/bblayers.conf: with poky/meta, poky/meta-poky and poky/meta-yocto-bsp

conf/local.conf:
```
DISTRO ?= "poky"
PACKAGE_CLASSES ?= "package_rpm"
EXTRA_IMAGE_FEATURES ?= "debug-tweaks"
USER_CLASSES ?= "buildstats image-prelink"
PATCHRESOLVE = "noop"
BB_DISKMON_DIRS ??= "\
    STOPTASKS,${TMPDIR},1G,100K \
    STOPTASKS,${DL_DIR},1G,100K \
    STOPTASKS,${SSTATE_DIR},1G,100K \
    STOPTASKS,/tmp,100M,100K \
    ABORT,${TMPDIR},100M,1K \
    ABORT,${DL_DIR},100M,1K \
    ABORT,${SSTATE_DIR},100M,1K \
    ABORT,/tmp,10M,1K"
PACKAGECONFIG_append_pn-qemu-system-native = " sdl"
CONF_VERSION = "1"
MACHINE ?= "qemuarm64"
BBMULTICONFIG = "one two three"

do_image[mcdepends] += "mc:one:two:core-image-minimal:do_image_complete mc:one:three:virtual/kernel:do_deploy"
```

conf/multiconfig/one.conf:
```
TMPDIR = "${TOPDIR}/tmp_one"
```

conf/multiconfig/two.conf:
```
TMPDIR = "${TOPDIR}/tmp_two"
MACHINE = "qemuarm"
```

conf/multiconfig/three.conf:
```
TMPDIR = "${TOPDIR}/tmp_three"
MACHINE = "qemuarm"
```

Run the build with: bitbake mc:one:core-image-minimal

The second run does not produce any warnings or errors.

I could only reproduce this issue when multiple MACHNES are used.
Comment 14 Richard Purdie 2021-08-06 05:37:17 UTC
I wasn't able to reproduce this bug after I merged this patch:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a15eea5077506111e4167a2361da0306d23543f3
Comment 15 Diego Sueiro 2021-08-10 05:43:57 UTC
@RP,

Thanks for fixing this.
I validated using the setup I mentioned in https://bugzilla.yoctoproject.org/show_bug.cgi?id=14342#c13 and it is working as expected but only when building from scratch (empty sstate cache).
If I run the same bitbake multiconfig command again I get thousands of messages like:
```
Sstate summary: Wanted 336 Local 333 Network 0 Missed 3 Current 3228 (99% match, 99% complete)                                                                                                                     
NOTE: Executing Tasks
...
WARNING: Runqeueue deadlocked on deferred tasks, forcing task ...
...

ERROR: Setscene task ... in both covered and notcovered.
...
NOTE: Tasks Summary: Attempted 9959 tasks of which 9959 didn't need to be rerun and all succeeded.

Summary: There were 1529 WARNING messages shown.
Summary: There were 2152 ERROR messages shown, returning a non-zero exit code.
```

The issue is also reproducible on hardknott branch. Is it possible to have the final fix backported to it?
I've cherry-picked this patch locally and the behavior is the same.
Comment 16 Randy MacLeod 2021-08-12 14:56:03 UTC
Apparently still broken in master and needs a backport to hardknott.
Comment 17 Richard Purdie 2021-08-25 14:26:07 UTC
(In reply to comment #15)
> Thanks for fixing this.
> I validated using the setup I mentioned in
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=14342#c13 and it is
> working as expected but only when building from scratch (empty sstate cache).
> If I run the same bitbake multiconfig command again I get thousands of
> messages like:
> ```
> Sstate summary: Wanted 336 Local 333 Network 0 Missed 3 Current 3228 (99%
> match, 99% complete)                                                        
> 
> NOTE: Executing Tasks
> ...
> WARNING: Runqeueue deadlocked on deferred tasks, forcing task ...
> ...
> 
> ERROR: Setscene task ... in both covered and notcovered.
> ...
> NOTE: Tasks Summary: Attempted 9959 tasks of which 9959 didn't need to be
> rerun and all succeeded.
> 
> Summary: There were 1529 WARNING messages shown.
> Summary: There were 2152 ERROR messages shown, returning a non-zero exit
> code.
> ```
> 
> The issue is also reproducible on hardknott branch. Is it possible to have
> the final fix backported to it?
> I've cherry-picked this patch locally and the behavior is the same.

Sorry, I'm not clear on what you mean here. Are you saying the issue is resolved and working ok on master?

If so, are you asking these changes be backported to hardknott? Or are you saying there is still an issue on master?
Comment 18 Richard Purdie 2021-08-25 14:27:43 UTC
(In reply to comment #17)
> (In reply to comment #15)
> > Thanks for fixing this.
> > I validated using the setup I mentioned in
> > https://bugzilla.yoctoproject.org/show_bug.cgi?id=14342#c13 and it is
> > working as expected but only when building from scratch (empty sstate cache).
> > If I run the same bitbake multiconfig command again I get thousands of
> > messages like:
> > ```
> > Sstate summary: Wanted 336 Local 333 Network 0 Missed 3 Current 3228 (99%
> > match, 99% complete)                                                        
> > 
> > NOTE: Executing Tasks
> > ...
> > WARNING: Runqeueue deadlocked on deferred tasks, forcing task ...
> > ...
> > 
> > ERROR: Setscene task ... in both covered and notcovered.
> > ...
> > NOTE: Tasks Summary: Attempted 9959 tasks of which 9959 didn't need to be
> > rerun and all succeeded.
> > 
> > Summary: There were 1529 WARNING messages shown.
> > Summary: There were 2152 ERROR messages shown, returning a non-zero exit
> > code.
> > ```
> > 
> > The issue is also reproducible on hardknott branch. Is it possible to have
> > the final fix backported to it?
> > I've cherry-picked this patch locally and the behavior is the same.
> 
> Sorry, I'm not clear on what you mean here. Are you saying the issue is
> resolved and working ok on master?
> 
> If so, are you asking these changes be backported to hardknott? Or are you
> saying there is still an issue on master?

Rereading that again, I think you mean there is a different issue on master where if you rerun the command, you see warnings/errors? So hardknott isn't the issue here and the issue is master has a different problem?
Comment 19 Diego Sueiro 2021-08-25 16:16:13 UTC
> Rereading that again, I think you mean there is a different issue on master
> where if you rerun the command, you see warnings/errors? So hardknott isn't
> the issue here and the issue is master has a different problem?

With the configs that I described in comment #13, the multiconfig builds are broken on both master and hardknott branches.

On the master branch, the first bitbake build command (with empty sstate cache) finishes successfully, but then in the subsequent bitbake builds (with local sstate cache) it fails with thousands of "ERROR: Setscene task ... in both covered and notcovered." messages.

On the hardknott branch, the first bitbake build command fails straight away (with empty sstate cache) with thousands of "ERROR: mc... No suitable staging package found" messages.
Comment 20 stefan.herbrechtsmeier 2021-09-03 13:50:52 UTC
We see 'Deferring', 'Runqeueue deadlocked' and 'Setscene task' warnings and errors in our multiconfig setup with a populated tmp directory.

I could reproduce the failures with a poky setup by setting BBMULTICONFIG to "a b" in local.conf, add two empty multiconfig (a.conf, b.conf) and a mc-test.bb recipe with the following content:

LICENSE = "MIT"

MC_DEPLOY_DIR_IMAGE = "${TMPDIR}/deploy/images/qemux86-64"

do_install() {
    install -d ${D}${libdir}/${BPN}
    install ${MC_DEPLOY_DIR_IMAGE}/core-image-minimal-qemux86-64.ext4 ${D}${libdir}/${BPN}/${BPN}.ext4
}
do_install[mcdepends] += "multiconfig:a:b:core-image-minimal:do_image_complete"

Two calls of `bitbake mc:a:uc-test` triggers the errors. If not a `bitbake -c cleanall mc:b:core-image-minimal` in-between should triggers the error at the last step.
Comment 21 Alexander Kanavin 2021-09-07 09:35:55 UTC
This seems to be the offending commit - try reverting
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a15eea5077506111e4167a2361da0306d23543f3
Comment 22 Richard Purdie 2021-09-07 11:35:25 UTC
Hi Nathan/Diego/Stefan,

I tracked down a couple of issues with the multiconfig code and the two patches posted today onto bitbake-devel should fix the issues. I tested with Stefan's test case as that was easier to reproduce and debug with (thanks!).

Would you be able to see if those patches address the failures you were seeing?

Thanks,

Richard
Comment 23 Diego Sueiro 2021-09-07 13:37:12 UTC
(In reply to comment #22)
> Hi Nathan/Diego/Stefan,
> 
> I tracked down a couple of issues with the multiconfig code and the two
> patches posted today onto bitbake-devel should fix the issues. I tested with
> Stefan's test case as that was easier to reproduce and debug with (thanks!).
> 
> Would you be able to see if those patches address the failures you were
> seeing?
> 
> Thanks,
> 
> Richard


Unfortunately, it is now failing using the config setup from comment #13 for the first build (empty TMPDIR and sstate-cache) with thousands of messages like this:

```
ERROR: mc:two:libcomps-native-0.1.17-r0 do_deploy_source_date_epoch_setscene: No suitable staging package found
ERROR: Logfile of failure stored in: /.../tmp_two/work/x86_64-linux/libcomps-native/0.1.17-r0/temp/log.do_deploy_source_date_epoch_setscene.9377
WARNING: Setscene task (mc:two:virtual:native:/.../poky/meta/recipes-devtools/libcomps/libcomps_0.1.17.bb:do_deploy_source_date_epoch_setscene) failed with exit code '1' - real task will be run instead
```
Comment 24 Alexander Kanavin 2021-09-07 15:05:56 UTC
Unfortunately I have to confirm what Diego says - the fixes do regress the setup described in https://bugzilla.yoctoproject.org/show_bug.cgi?id=14342#c13 (adjusted for the new override syntax).
Comment 25 Richard Purdie 2021-09-07 17:23:58 UTC
Ok, I think I understand why it breaks the other test case. I've updated master-next with a new version of the patch.

The code is the same, it is just being placed into a different function task_completeoutright() rather than a setscene one. I'm trying various tests locally but appreciate any wider testing.
Comment 26 Richard Purdie 2021-09-07 18:36:16 UTC
I updated master-next again to add an update_scenequeue_data() call. This was after more local testing showed the case in comment 13 showed different errors later in the build.
Comment 27 Alexander Kanavin 2021-09-08 05:17:12 UTC
The latest patches fix the errors in #13 and #20 for me. I ran both builds to completion, with tasks to execute, and without.
Comment 28 Diego Sueiro 2021-09-09 07:23:44 UTC
(In reply to comment #26)
> I updated master-next again to add an update_scenequeue_data() call. This
> was after more local testing showed the case in comment 13 showed different
> errors later in the build.

I can confirm that the following patches fixed the test case from #13:
```
58cbdaecf7 bitbake: runqueue: Handle deferred task rehashing in multiconfig builds
8aad4c243a bitbake: runqueue: Fix multiconfig deferred task sstate validity caching issue
```

I also cherry-picked the following 3 patches and confirm that it also passed for the test case from #13:
```
58cbdaecf7 bitbake: runqueue: Handle deferred task rehashing in multiconfig builds
8aad4c243a bitbake: runqueue: Fix multiconfig deferred task sstate validity caching issue
a49e5c0f4f bitbake: runqueue: Fix deferred task issues
```

Is it possible to backport these patches to hardknott?
Comment 29 Richard Purdie 2021-09-09 11:24:59 UTC
I think I'm happier with the situation now with these fixes so yes, we could consider it for hardknott.