Bug 8440

For case 1, as Toaster aborts before the build, there isn't really a build log to show. Instead, we could default to showing the general console log for this situation, as we can get the path for that file (especially once there is only one console log file, after the patches for bug 8411).

Belen, could you please give some steps which will enable me to easily reproduce the issue, both for Toaster and CLI builds?

(In reply to comment #2)
> Belen, could you please give some steps which will enable me to easily
> reproduce the issue, both for Toaster and CLI builds?

I think you can just set a machine that doesn't exist or try to build a target that doesn't exist. Like 'xyz' or whatever.

Situations to cover:

1. In Toaster, run a build for a non-existent target "xyz" for a project. Note that for this situation, we are already capturing the failure, it's just that you can't see the build log.

2. On the command line, run the same build for a non-existent target "xyz". In this situation, the failure is not even noticed by Toaster.

Belen, I've covered the two situations in comment 4 above. Are there any other types of failure you'd like covered?

(In reply to comment #5)
> Belen, I've covered the two situations in comment 4 above. Are there any
> other types of failure you'd like covered?

Can't think of anything off the top of my head, but something else might come to mind when I look at the patches. Do you have a branch somewhere I could look at?

The branch is:
elliot/toaster/failed_builds-8440

Submitted for review to toaster mailing list:
https://lists.yoctoproject.org/pipermail/toaster/2016-February/003754.html

Need a version 4 rework to incorporate review comments from joshuagl:

* When creating dict objects, use
d = {'foo': 'bar', 'baz': 'bants', ...}
rather than
d = {}
d['foo'] = 'bar'
d['baz'] = 'bants'
...

* When declaring a function with optional arguments, don't put spaces around the '=', i.e. use:
def f(arg1, arg2=None)
rather than
def f(arg1, arg2 = None)

Also need to decide whether we want to capture builds earlier than the buildTargets() command. Here are the pros and cons of using an earlier event:

* Pros: (1) we don't need to add an event to every command called in bitbake: we just capture a really early event in the bitbake run, like SanityCheck; (2) we can capture nearly every type of error bitbake might throw (currently we don't create a build object until we see the buildTargets() command about to run on the server, so the bitbake run might fall over before we have our Build object).

* Cons: (1) if we use a really early event, this will happen before we know what the build targets are (buildTargets() is the first command to the server containing the target information, so if we use any earlier event, we won't have targets for the build). This means that we won't be able to show the targets for the build in the recent builds section or in the builds tables; which will in turn wreck the design (or require a different design).

Belen will need to make a decision on this.

> 
> * Cons: (1) if we use a really early event, this will happen before we know
> what the build targets are (buildTargets() is the first command to the
> server containing the target information, so if we use any earlier event, we
> won't have targets for the build). This means that we won't be able to show
> the targets for the build in the recent builds section or in the builds
> tables; which will in turn wreck the design (or require a different design).
> 
> Belen will need to make a decision on this.

We can definitely find a way of displaying builds with unknown targets in the UI. I'll come up with something.

After some discussion with Elliot, we will work towards getting target information for builds that fail due to:

* invalid machine
* invalid distro
* invalid target

So far, those 3 cases are the earliest build failures I've come across.

Created attachment 2967 [details]
Design - Showing builds without target information

Just in case we manage to show builds started from CLI that fail at early stages.

I've spent a while trying various things to record early-failing command-line builds, where MACHINE or DISTRO are the reason for the failure. (I haven't managed to resolve these two situations, but have resolved it for TARGET.)

As background, here's what's happening (as I understand it) when you run a command-line build from a Toaster environment (i.e. you've started Toaster and are using the same prompt where you started it from); most of this goes on in bitbake/lib/bb/main.py:

1. Toaster is running two processes: a server process and a client process. We use the client process to trigger builds from the Toaster UI. The server process runs the actual builds in the background.

2. When you run "bitbake <target>" from the command-line, bitbake creates a fake "server", which is in reality an XMLRPC client connecting to the existing server started by Toaster.

3. bitbake (on the command line) connects to the fake server (this really connects a client to the Toaster XMLRPC server); it gets a connection object back.

4. bitbake sets up the event queue on the returned connection. From this point, any events which occur during the build can now be monitored and collected by Toaster.

The problem we have is that the MACHINE and DISTRO failures we are trying to capture occur before step 3. I don't think we can monitor events at the time of those failures as they happen before bitbake has set up the event queue on the XMLRPC client. If we fire events which say that the build has failed, we can't hear them, because there's no channel for those events between bitbake (on the command line) and Toaster.

I don't think your comment from Feb 9 is correct in that we can't see any events from toasterland that occur very early (like a sanity failure due to a bad machine).  Take a look at poky-contrib/bavery/toaster/earlyBuildTargetNotification.

If you set your MACHINE ??= "BUNNY" in local.conf and run:
$>bitbake zlib
you should see something like:
WARNING: EARLY BUILD TARGET EVENT PRE SANITY TARGET=: ['zlib']
in toaster_ui.log. This means toasterland is seeing a very early event that is fired before the sanity check fails.  A similar event gets fired if the sanity check dies before the distro sanity check.

What I did to see this:
1) made a new event TargetsAcquired in event.py
2) added a line to fire the event if the command was buildTargets in xmlrpc.py
3) subscribed to the event in toasterui.py and then logged it when I saw it.

Thanks for this, Brian. I've used your code pretty much as is, but modified toasterui.py so that we create the Build object when the TargetsAcquired event occurs.

Submitted for review:
https://lists.yoctoproject.org/pipermail/toaster/2016-February/003837.html

Fix is now in master: f73bbc0cbb591c394a30cf5dc3b5628f997c46d7

feedback on http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/commit/?h=toaster-next&id=78443585bc92eca7140267126d53f749be3cdd6a

RP
elliot, bavery: I guess the other option would be to put in a specific event like BuildInit right at the start of the buildTarget and buildFile functions

RP
bavery: the patch I saw for TargetAcqured only worked for the xmlrpc backend and didn't work for buildFile

RP
I'm happy to have a BuildInit event at the start of those two command functions


Seems like a reasonable plan 2 me !

I've had a try at implementing this, which does work for the "bitbake zzzz" (invalid target) case.

However, this doesn't work for the bad MACHINE or bad DISTRO case, as those errors occur before buildTargets() (cooker.py) is called.

Is there any way we can fire an event before any of the sanity checking occurs, which effectively states that a bitbake "run" is starting? Where would be the best place to do that?

(NB it would be useful if this event could also contain the target info.)

After discussion with RP, it looks like there's not really a way round this.

In cases like a bad MACHINE, bitbake doesn't reach the ConfigParsed event because the base configuration can't be parsed and the targets being built aren't relevant (because they can't be built, regardless of what they are).

RP also pointed out that another problem is that the full target list "includes BBPKGS, yet we can't query BBPKGS until we have a data store...[and] we don't have a data store if we can't parse". So bitbake doesn't have a targets list at all if the configuration is bad.

If we can't get the information out of bitbake, there's no way we can "see" problems which occur before buildTargets() (like bad configuration), unless we wrap bitbake in our own tool which *can* see bitbake's command line. Even then, we would only be guessing at the targets and tasks, as we would still have no access to bitbake's configuration (which contains variables representing the default task if none is specified on the command line etc.).

The solution might have to be that we have to ignore command line builds which fail before buildTargets() due to bad configuration; as the user is on the command line anyway (running the build), I don't think this is a massive problem.

Note that in the previous Toaster version, where we always had an XMLRPC server running, we could get at the buildTargets() arguments as they were passed to the bitbake server, which occurred before the build proper started. We no longer have a permanent server which runs command line builds, so we can't do this any more.

It *is* possible to capture bad targets (e.g. "bitbake zzz") using a BuildInit event, as suggested by RP. I have some tentative code for this already.

If Brian and Belen don't have any opinion how to deal with this, I will add the BuildInit event to capture bad targets, and ignore bad MACHINE and DISTRO, as we are unable to capture these in environments without an XMLRPCServer.

Brian is ok with this. From my perspective, Bad MACHINE/DISTRO are really obvious from the command line. Bad Parse, however is something people (me?) have to iterate on a couple of times if they are trying out (or rolling their own) new iffy layers.

(In reply to comment #23)
> Brian is ok with this. From my perspective, Bad MACHINE/DISTRO are really
> obvious from the command line. Bad Parse, however is something people (me?)
> have to iterate on a couple of times if they are trying out (or rolling
> their own) new iffy layers.

FWIW I agree: catching some cases better than nothing. Thanks!

Submitted v6 to Toaster mailing list:
https://lists.yoctoproject.org/pipermail/toaster/2016-April/004497.html

Depends on bug 8556, as the changes required here cross over with fixes for it.

Submitted patches to mailing list:
https://lists.yoctoproject.org/pipermail/toaster/2016-July/004997.html