Bug 13696

Summary: Ptest log parsing taking hours on the autobuilder
Product: [QA/Testing] Package Testing (ptest) Reporter: Richard Purdie <richard.purdie>
Component: ptestAssignee: Richard Purdie <richard.purdie>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: High CC: akuster808, akuster, randy.macleod
Version: unspecified   
Target Milestone: 3.1 M3   
Hardware: x86   
OS: Multiple   
Whiteboard: with backport
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
Log file its trying to parse none

Description Richard Purdie 2019-12-10 13:18:53 UTC 
ptest log parsing seems to be taking hours on the autobuilder. E.g: 18 hours (so far) for:

https://autobuilder.yoctoproject.org/typhoon/#/builders/81/builds/556

Traceback from the build whilst running:

Traceback (most recent call first):
  File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/utils/logparser.py", line 64, in parse
    current_section['log'] = current_section['log'] + line
  File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/runtime/cases/ptest.py", line 53, in test_ptestrunner
    results, sections = parser.parse(ptest_runner_log)
  File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f
    return func(*args, **kwargs)
  File "/usr/lib/python3.5/unittest/case.py", line 601, in run
    testMethod()
  File "/usr/lib/python3.5/unittest/case.py", line 649, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python3.5/unittest/suite.py", line 122, in run
    test(result)
  File "/usr/lib/python3.5/unittest/suite.py", line 84, in __call__
    return self.run(*args, **kwds)
  File "/usr/lib/python3.5/unittest/runner.py", line 176, in run
    test(result)
  File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/lib/oeqa/core/context.py", line 89, in runTests
    result = self.runner.run(self.suites)
  File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/classes/testimage.bbclass", line 322, in testimage_main
    results = tc.runTests()
  File "/home/pokybuild/yocto-worker/qemux86-64-ptest/build/meta/classes/testimage.bbclass", line 114, in do_testimage
    testimage_main(d)

Logfile is 71MB
Comment 1 Richard Purdie 2019-12-10 13:21:56 UTC 
Created attachment 4598 [details]
Log file its trying to parse
Comment 2 Richard Purdie 2019-12-10 13:29:29 UTC 
This the bluez5 upgrade increased the logs from 5 -> 65 MB and this broke the parser which isn't scaling.
Comment 3 akuster 2020-01-16 15:58:51 UTC 
would creating a ptest log per package help ?
Comment 4 Richard Purdie 2020-02-20 14:01:43 UTC 
(In reply to comment #3)
> would creating a ptest log per package help ?

Well, it could but we should just fix the stupid code ;-)
Comment 5 Richard Purdie 2020-02-20 14:13:14 UTC 
Patch on the mailing list and queued in -next