Bug 13696 - Ptest log parsing taking hours on the autobuilder
Summary: Ptest log parsing taking hours on the autobuilder
Status: RESOLVED FIXED
Alias: None
Product: Package Testing (ptest)
Classification: QA/Testing
Component: ptest (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: High normal
Target Milestone: 3.1 M3
Assignee: Richard Purdie
QA Contact:
URL:
Whiteboard: with backport
Depends on:
Blocks:
 
Reported: 2019-12-10 13:18 UTC by Richard Purdie
Modified: 2020-02-21 16:01 UTC (History)
3 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)

Attachments
Log file its trying to parse (3.20 MB, application/gzip)
2019-12-10 13:21 UTC, Richard Purdie 		no flags Details
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Richard Purdie 2019-12-10 13:18:53 UTC 
ptest log parsing seems to be taking hours on the autobuilder. E.g: 18 hours (so far) for:

https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/556

Traceback from the build whilst running:

Traceback (most recent call first):
  File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/utils/logparser.py", line 64, in parse
    current_section['log'] = current_section['log'] + line
  File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 53, in test_ptestrunner
    results, sections = parser.parse(ptest_runner_log)
  File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/usr/lib/python3.5/unittest/case.py", line 601, in run
    testMethod()
  File "/usr/lib/python3.5/unittest/case.py", line 649, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python3.5/unittest/suite.py", line 122, in run
    test(result)
  File "/usr/lib/python3.5/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python3.5/unittest/runner.py", line 176, in run
    test(result)
  File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/context.py", line 89, in runTests
    result = self.runner.run(self.suites)
  File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/classes/testimage.bbclass", line 322, in testimage_main
    results = tc.runTests()
  File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/classes/testimage.bbclass", line 114, in do_testimage
    testimage_main(d)

Logfile is 71MB
Comment 1 Richard Purdie 2019-12-10 13:21:56 UTC 
Created attachment 4598 [details]
Log file its trying to parse
Comment 2 Richard Purdie 2019-12-10 13:29:29 UTC 
This the bluez5 upgrade increased the logs from 5 -> 65 MB and this broke the parser which isn't scaling.
Comment 3 akuster 2020-01-16 15:58:51 UTC 
would creating a ptest log per package help ?
Comment 4 Richard Purdie 2020-02-20 14:01:43 UTC 
(In reply to comment #3)
> would creating a ptest log per package help ?

Well, it could but we should just fix the stupid code ;-)
Comment 5 Richard Purdie 2020-02-20 14:13:14 UTC 
Patch on the mailing list and queued in -next