Bug 14006 - Create facilty to collect build system statistics
Summary: Create facilty to collect build system statistics
Status: RESOLVED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: oe-core other (show other bugs)
Version: 3.3
Hardware: x86 Multiple
: Medium+ enhancement
Target Milestone: 3.4 M2
Assignee: Sakib Sajal
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2020-08-13 10:48 UTC by Randy MacLeod
Modified: 2021-07-12 19:26 UTC (History)
1 user (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Don't know


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Randy MacLeod 2020-08-13 10:48:57 UTC
There are a number of timeout and hang defect where it would be useful to 
collect statistics about what is running on a build host when that condition occurs.

We should create a tool to dump top, iostat, and possibly other data to help debug such problems. This code can be called immediately after a such a problem is encountered since we believe that the load will be similar for 10s of seconds or more after such events.

One alternative is to continuously run a tool such as Nagios ( https://www.nagios.org/ ) on each build host but that makes more work and maintenance for those who support the YP AB. Let's start with this approach and see if we're able to learn enough to debug this list of timeout/hand in the AB-INT list ( https://bugzilla.yoctoproject.org/buglist.cgi?quicksearch=AB-INT&list_id=620110 )
Comment 1 Randy MacLeod 2020-08-13 10:49:53 UTC
Run top like this: top -b -n1
This shouldn't take more than a second to execute:

# On a lightly loaded 192 'core' system:
$ time top -b -n1 > /tmp/output

real    0m0.258s
user    0m0.047s
sys     0m0.060s

# when doing a build on the same system:
$ time top -b -n1 > /tmp/output

real    0m0.854s
user    0m0.178s
sys     0m0.521s

and when the build was underway with a load average of:
   load average: 278.38, 171.74, 87.09
the top command still runs in under 1 second:

real    0m0.782s
user    0m0.058s
sys     0m0.293s


Perhaps there are other command line options that should be added.
Comment 2 Sakib Sajal 2020-09-03 13:28:33 UTC
Hi Sakib,

On Wed, 2020-09-02 at 17:57 -0400, Sakib Sajal wrote:
> I have been trying to reproduce the failure: 
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=13802 , for many
> day 
> now, without success.
>
> Therefore I started working with Randy to collect system-wide
> statistics.
>
> Randy and I have created an outline regarding the YP-AB issues.
>
> I think we should start collecting the data only after a failure has 
> occurred. Initially, I will only collect data when the LTP failure 
> occurs, unless you have a better idea of where in the code I should 
> start the data collection.
>
> If you have time, could you please review the outline below:
>
> Problem Statement (related to: 
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=13802):
> There's are many intermittent failures that are hard to
> reproduce/reason 
> about and existing logs are not always helpful to debug the problem.

I think you need to highlight that you're focusing on the system load
aspect of the failure and would like to understand the state of the
system at failure time.

Generic logs are a different problem.

> Proposed solution:
> add monitoring system to bitbake:
> 1) get logs all the time, store in a ring buffer, end when the
> bitbake 
> server encounters an error
> pros:
>      i) can log all the stats before and immediately after the time
> of 
> failure
> cons:
>      i) heavyweight, logging all the time
>
> 2) get the log only when bitbake server encounters an error
> pros:
>      i) lightweight, one time logging
> cons:
>      i) by the time the logs are created, the problematic
> process/thread 
> may have already exited.
>
> along with the above, we can check for other bitbake servers and the 
> processes/threads that are running and log them after the failure.
> We 
> may also be able to take advantage of the sstate-cache and the logs 
> present there.

My suspicion is that the "load" on the system doesn't vary so fast that
we wouldn't be able to catch the state upon failure. I think that a
snapshot of the system stats at failure time should be fine and we'd
get more people using such a thing.

> Implementation details:
> 1)
>      a) Use sar: Collect, report, or save system activity
> information.
> pros:
>      i) available on most systems
>      ii) designed for system monitoring
> cons:
>      i) needs to be installed and maintained for each autobuilder.
>      ii) less information provided compared to top.
>
>      b) Use nagios or other system monitoring tools.

I'm afraid this simply won't work on the autobuilder, anything needing
host system dependencies like this and sysadmin setup is extremely
fragile and prone to failure.

> 2) get logs, from top, iostat and any other sysstat tools,
> In order to reduce system lag, we only run the tool only after the 
> failure has occurred.
> For the data to be useful, it should be collected as soon as the
> failure 
> occurred.
> Each monitoring tool needs to run to completion as soon as possible, 
> possibly under a second.
> We need to decide the order in which the tools need to be run so that
> all the relevant information is captured.
>
> Tools considered:
> 1) top
> 2) iostat
> 3) vmstat
> 4) sar
> 5) iotop (can include per process statistics, requires kernel module)
>
> Tools proposed:
> 1) top -b -n 1 # batch output for one iteration
> 2) iostat -x
> 3) vmstat --active --disk --timestamp

I much prefer the idea of 2). One question I have is what permissions
are needed, i.e. I think you have to be root to get data from iotop? If
that is the case we need to look at what options we have to allow the
user to access this data with minimal sysadmin setup (and we can easily
verify that access is enabled or not).

In terms of implementation, I think this could all even be done from an
event handler since those see failures and can act accordingly when
they see one happen.

I'd note that we have buildstats.bbclass which aims to track certain
system usage stats of tasks. Sadly it doesn't log these periodically,
although if it listened for a heartbeat event, it probably could. It
also doesn't do anything upon a task failure, which would be the really
interesting bit to add to it?

I should also mention lib/bb/diskmonitor.py. This code monitors free
space on the disks and takes action on builds if it gets low. Its
probably not so relevant here but worth noting we do have such
functionality.

So between these bits we have code which checks/logs on a time basis
(disk usage), we also have code which saves off resource usage.

I'd also mention we have some problems with failures where some other
system limits are reached, e.g. the number of open files, or other
finite system resources. We have considered logging/monitoring those.

Sorry to inject more thoughts here but its the right time to think
about them. It would make most sense to start with a simple "upon
failure" status logging, I think having a once every say X seconds
"system stat" mode may also be useful. Once we have a mechanism, we can
tweak the stats we need to collect and adapt as needed so I'm less
worried about that than the mechanism initially.

Cheers,

Richard
Comment 3 Randy MacLeod 2020-09-03 15:00:53 UTC
In addition to the top output, we'll likely collect dmesg, nfsiostat initially.
Comment 4 Sakib Sajal 2021-07-12 19:26:12 UTC
Fixed by several commits, the latest commit is
https://git.openembedded.org/openembedded-core/commit/?id=1a0fb3c0794f4e66086e567a297b4d9379c6b8f3

Additional work will be required but the feature is functional now.