Bug 10622

Summary: buildstats: use JSON format
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Markus Lehtonen <markus.lehtonen>
Component: devtools / tool chainAssignee: Unassigned <unassigned>
Status: NEW --- QA Contact:
Severity: enhancement    
Priority: Medium CC: bluelightning, joshuagloe, markus.lehtonen, meta.mr.watcher, meta.watcher, randy.macleod, richard.purdie, ross.burton, tim.orling
Version: 2.2   
Target Milestone: Future   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Don't know

Description Markus Lehtonen 2016-11-08 15:04:00 UTC 
It would be easier to consume buildstats if they were stored in some other format that per-task text files. I would propose using one (per-build) json-formatted file that would contain all tasks from all recipes.

That would make buildstats a lot faster and easier to read. It would also be nicer on the filesystem level to have one file instead of thousands of them per build. Oe-build-perf-test script is already converting existing buildstats into json so we could use that format.
Comment 1 Markus Lehtonen 2016-11-08 15:09:20 UTC 
The format could look like this

[
    {
        "name": "bc-native",
        "epoch": null,
        "version": "1.06"
        "revision": "r3",
        "tasks": {
            "do_compile": {
                "child_rusage": {
                    "ru_inblock": 0,
                    "ru_majflt": 0,
                    "ru_maxrss": 55212,
                    "ru_minflt": 80968,
                    "ru_nivcsw": 907,
                    "ru_nvcsw": 320,
                    "ru_oublock": 912,
                    "ru_stime": 0.21015299999999998,
                    "ru_utime": 1.9195280000000001
                },
                "elapsed_time": 2.120001,
                "iostat": {
                    "cancelled_write_bytes": 4096,
                    "rchar": 12604928,
                    "read_bytes": 0,
                    "syscr": 3553,
                    "syscw": 707,
                    "wchar": 885406,
                    "write_bytes": 503808
                },
                "rusage": {
                    "ru_inblock": 0,
                    "ru_majflt": 0,
                    "ru_maxrss": 58512,
                    "ru_minflt": 4503,
                    "ru_nivcsw": 19,
                    "ru_nvcsw": 83,
                    "ru_oublock": 72,
                    "ru_stime": 0.011193,
                    "ru_utime": 0.192331
                },
                "start_time": 1475630641.599999,
                "status": "PASSED"
            },
            "do_configure": {
                    ...
            },
            ...
    },
    {
        "name": "dbus-glib-native",
        ...
    },
    ...
]
Comment 2 Markus Lehtonen 2017-05-05 10:09:37 UTC 
Moving to 2.4
Comment 3 Richard Purdie 2019-03-07 15:55:36 UTC 
One concern with single or small numbers of files is concurrency and locking. The existing multiple files were done individually to avoid locking issues.