| Summary: | Toaster in analysis mode downloads the wrong cooker logs | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] Toaster | Reporter: | Belen Barros Pena <belen.barros.pena> | ||||
| Component: | toaster | Assignee: | Elliot Smith <elliot.smith> | ||||
| Status: | VERIFIED FIXED | QA Contact: | Alexandru Roman <alexandru.costinx.roman> | ||||
| Severity: | normal | ||||||
| Priority: | Medium+ | CC: | alexandru.costinx.roman, belen.barros.pena, brian.avery, jessica.zhang, stanciux.mihail | ||||
| Version: | 1.8 | ||||||
| Target Milestone: | 2.1 M1 | ||||||
| Hardware: | x86 | ||||||
| OS: | Multiple | ||||||
| Whiteboard: | |||||||
| OS type for building Yocto: | --- | Type of Regression: | --- | ||||
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |||||
| Bug Depends on: | 8411 | ||||||
| Bug Blocks: | |||||||
| Attachments: |
|
||||||
|
Description
Belen Barros Pena
2015-09-22 12:56:58 UTC
I tested this in analysis mode. The cooker_log_path field in the db orm_build table for each build isn't being updated correctly, so all the builds point to the same log file. I confirmed this by looking at the SQLite db. In managed/build mode, the log file paths are correct. I made some changes to pass the log file location as part of the BuildStarted event. I used BB_CONSOLELOG, which should contain the build's log file path. Using this approach, the build name and the output log file path correspond, e.g. build name|cooker_log_path 20150928123720|/home/ell/toaster-run/build/tmp/log/cooker/qemux86/20150928123720.log 20150928123726|/home/ell/toaster-run/build/tmp/log/cooker/qemux86/20150928123726.log But the actual logs generated have different paths, e.g. for the above two builds, the paths were: /home/ell/toaster-run/build/tmp/log/cooker/qemux86/20150928123716.log /home/ell/toaster-run/build/tmp/log/cooker/qemux86/20150928123720.log Note that the path of the log for the second build (/home/ell/toaster-run/build/tmp/log/cooker/qemux86/20150928123720.log) corresponds with the expected path for the first build (20150928123720). This suggests that when the BuildStarted event occurs, the timestamp for the build start has been reset and is no longer correct: the actual log file path has been set before BuildStarted occurs and doesn't seem to be readily accessible when that event is constructed. After speaking to Paul Eggleton, he suggested this is a bug, and that we could work around it by looking at the log file directory when the build starts, and find the latest-dated log file, then associating that with the build. Submitted to toaster mailing list for review. Brian has identified an additional situation not covered by this bug during review, so moving back to implementation. to reproduce the additional situation: new proj - master Add meta-iv layer from the layer index build ecryptfs-utils bitbake dies on parsing one of the recipe files in the layer. The "download build log" button for the failed build results in a "The build artifact you are trying to download no longer exists." Pretty sure this is because bb dies during parsing and b4 any events come in. Created attachment 2781 [details]
Design - Displaying failed build requests
The attached design document describes the build dashboard page for builds that fail at the build request state, for which we have no build log and no data.
Brian: for now, I've simply hidden the "Download build log" button for builds which fail before the BuildStarted event, and for which we have no cooker_log_path. Belen raised an issue for us to deal with builds which fail early: bug 8440. I've resubmitted this branch with the additional patch elliot/toaster/wrong_log-8373 to the toaster mailing list. I'm reopening this, as it didn't make it to bitbake master and needs reworking. The proper fix for this will be provided by bug 8411, which is slated for 2.1. RP has already posted a series of patches which should enable this to be fixed properly: http://lists.openembedded.org/pipermail/bitbake-devel/2015-October/006544.html http://lists.openembedded.org/pipermail/bitbake-devel/2015-October/006546.html http://lists.openembedded.org/pipermail/bitbake-devel/2015-October/006545.html I'll keep an eye on where they get to; hopefully we can get the proper fix in before 2.1. Note that the work-around in my original patch was not accepted by RP (http://lists.openembedded.org/pipermail/bitbake-devel/2015-October/006540.html). I agree with his conclusions, but it does mean that this bug is now effectively blocked by bug 8411. Sent an RFC to mailing list (2015-10-20): https://lists.yoctoproject.org/pipermail/toaster/2015-October/003179.html Note that my proposed solution (on branch poky-contrib, elliot/toaster/wrong_log-8373-v2) isn't blocked by bug 8411, as it creates a parallel log file owned by Toaster. RP was OK with how I did this, but wasn't keen on the way it duplicates writing the "Build Configuration" section rather than reusing the output from bitbake. I've investigated and can't find an easy way to capture bitbake's output of the build configuration in our log files, as it happens before the BuildStarted event. (We use the BuildStarted event as the trigger to start writing to a new build log file.) Here are some alternative solutions (there may be others): 1. Spend more time figuring out whether we can capture bitbake's output of the build configuration. 2. Use existing bitbake code to output the build configuration to our log. 3. Don't include the build configuration in our logs. With (1), I'm not sure how much work is involved; after my research, (2) looks difficult and painful (it doesn't look easy to reuse the code which writes the build configuration, as it's in bbclass files); (3) can be done immediately and painlessly, but means our build logs are less useful. Also note that once bug 8411 is fixed, log paths will be broken for analysis AND build modes. One other option I missed is for bitbake to create per-build logs. The BuildStarted event is probably not the correct event to use for starting our logs. For instance, I have routinely tried to build odd (i.e. not core-image-[minimal|sato] and had dependencies missing from the layers included. This pops out during parsing and should be part of our logs I would think. Would starting the logging earlier help with capturing the build configuration? It would help with some of the unexplained build errors w/o much/any info attached. It would also allow us to do things like have the parsing,setting up runquee, etc. be in the progress bar so that it doesn't just st at 0% for so long that the user begins to wonder if it crashed. Thanks Brian, using ParseStarted actually solved all the issues I was having and simplified things. Submitted to toaster mailing list for review: https://lists.yoctoproject.org/pipermail/toaster/2015-October/003196.html See branch: elliot/toaster/wrong_log-8373-v2 I submitted another version of this patch to the toaster mailing list: https://lists.yoctoproject.org/pipermail/toaster/2015-October/003238.html See branch: elliot/toaster/wrong_log-8373-v2 ParseStarted can't be used to initiate logs for cli builds, as we don't get that event for those builds. I'm not sure if this event is fired and we're missing it somehow, or if it just doesn't fire (needs some more investigation). I will open a bug to track that. I opened a separate build to investigate/fix the issue where we don't get "Build configuration" for cli builds: bug 8604. Waiting for bug 8411 to be closed. Verified on master: 36b43b233c841b3c09fc17e04686b4b758780963 |