| Summary: | Create facilty to collect build system statistics | ||
|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] OE-Core | Reporter: | Randy MacLeod <randy.macleod> |
| Component: | oe-core other | Assignee: | Sakib Sajal <sakib.sajal> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | enhancement | ||
| Priority: | Medium+ | CC: | tim.orling |
| Version: | 3.3 | ||
| Target Milestone: | 3.4 M2 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | Don't know | |
|
Description
Randy MacLeod
2020-08-13 10:48:57 UTC
Run top like this: top -b -n1 This shouldn't take more than a second to execute: # On a lightly loaded 192 'core' system: $ time top -b -n1 > /tmp/output real 0m0.258s user 0m0.047s sys 0m0.060s # when doing a build on the same system: $ time top -b -n1 > /tmp/output real 0m0.854s user 0m0.178s sys 0m0.521s and when the build was underway with a load average of: load average: 278.38, 171.74, 87.09 the top command still runs in under 1 second: real 0m0.782s user 0m0.058s sys 0m0.293s Perhaps there are other command line options that should be added. Hi Sakib, On Wed, 2020-09-02 at 17:57 -0400, Sakib Sajal wrote: > I have been trying to reproduce the failure: > https://bugzilla.yoctoproject.org/show_bug.cgi?id=13802 , for many > day > now, without success. > > Therefore I started working with Randy to collect system-wide > statistics. > > Randy and I have created an outline regarding the YP-AB issues. > > I think we should start collecting the data only after a failure has > occurred. Initially, I will only collect data when the LTP failure > occurs, unless you have a better idea of where in the code I should > start the data collection. > > If you have time, could you please review the outline below: > > Problem Statement (related to: > https://bugzilla.yoctoproject.org/show_bug.cgi?id=13802): > There's are many intermittent failures that are hard to > reproduce/reason > about and existing logs are not always helpful to debug the problem. I think you need to highlight that you're focusing on the system load aspect of the failure and would like to understand the state of the system at failure time. Generic logs are a different problem. > Proposed solution: > add monitoring system to bitbake: > 1) get logs all the time, store in a ring buffer, end when the > bitbake > server encounters an error > pros: > i) can log all the stats before and immediately after the time > of > failure > cons: > i) heavyweight, logging all the time > > 2) get the log only when bitbake server encounters an error > pros: > i) lightweight, one time logging > cons: > i) by the time the logs are created, the problematic > process/thread > may have already exited. > > along with the above, we can check for other bitbake servers and the > processes/threads that are running and log them after the failure. > We > may also be able to take advantage of the sstate-cache and the logs > present there. My suspicion is that the "load" on the system doesn't vary so fast that we wouldn't be able to catch the state upon failure. I think that a snapshot of the system stats at failure time should be fine and we'd get more people using such a thing. > Implementation details: > 1) > a) Use sar: Collect, report, or save system activity > information. > pros: > i) available on most systems > ii) designed for system monitoring > cons: > i) needs to be installed and maintained for each autobuilder. > ii) less information provided compared to top. > > b) Use nagios or other system monitoring tools. I'm afraid this simply won't work on the autobuilder, anything needing host system dependencies like this and sysadmin setup is extremely fragile and prone to failure. > 2) get logs, from top, iostat and any other sysstat tools, > In order to reduce system lag, we only run the tool only after the > failure has occurred. > For the data to be useful, it should be collected as soon as the > failure > occurred. > Each monitoring tool needs to run to completion as soon as possible, > possibly under a second. > We need to decide the order in which the tools need to be run so that > all the relevant information is captured. > > Tools considered: > 1) top > 2) iostat > 3) vmstat > 4) sar > 5) iotop (can include per process statistics, requires kernel module) > > Tools proposed: > 1) top -b -n 1 # batch output for one iteration > 2) iostat -x > 3) vmstat --active --disk --timestamp I much prefer the idea of 2). One question I have is what permissions are needed, i.e. I think you have to be root to get data from iotop? If that is the case we need to look at what options we have to allow the user to access this data with minimal sysadmin setup (and we can easily verify that access is enabled or not). In terms of implementation, I think this could all even be done from an event handler since those see failures and can act accordingly when they see one happen. I'd note that we have buildstats.bbclass which aims to track certain system usage stats of tasks. Sadly it doesn't log these periodically, although if it listened for a heartbeat event, it probably could. It also doesn't do anything upon a task failure, which would be the really interesting bit to add to it? I should also mention lib/bb/diskmonitor.py. This code monitors free space on the disks and takes action on builds if it gets low. Its probably not so relevant here but worth noting we do have such functionality. So between these bits we have code which checks/logs on a time basis (disk usage), we also have code which saves off resource usage. I'd also mention we have some problems with failures where some other system limits are reached, e.g. the number of open files, or other finite system resources. We have considered logging/monitoring those. Sorry to inject more thoughts here but its the right time to think about them. It would make most sense to start with a simple "upon failure" status logging, I think having a once every say X seconds "system stat" mode may also be useful. Once we have a mechanism, we can tweak the stats we need to collect and adapt as needed so I'm less worried about that than the mechanism initially. Cheers, Richard In addition to the top output, we'll likely collect dmesg, nfsiostat initially. Fixed by several commits, the latest commit is https://git.openembedded.org/openembedded-core/commit/?id=1a0fb3c0794f4e66086e567a297b4d9379c6b8f3 Additional work will be required but the feature is functional now. |