Bug 16337 - AB-INT: send_qa_email: Processing regression report is too long
Summary: AB-INT: send_qa_email: Processing regression report is too long
Status: NEW
Alias: None
Product: AutoBuilder
Classification: Infrastructure
Component: autobuilder (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 6.2 M1
Assignee: Paul Barker
QA Contact:
URL:
Whiteboard: AB-INT
Depends on:
Blocks:
 
Reported: 2026-06-30 08:48 UTC by Mathieu Dubois-Briand
Modified: 2026-10-29 14:56 UTC (History)
4 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Mathieu Dubois-Briand 2026-06-30 08:48:52 UTC
We had this test time out on the autobuilder:

====================================================================================================
Processing regression report (1782618075.3)
====================================================================================================

INFO: Generating regression report between yocto-6.0_M3 and master
INFO: Comparing master to yocto-6.0_M3
INFO: Compute regression report between yocto-6.0_M3 and master
INFO: SHA-1 revision for yocto-6.0_M3 in /srv/pokybuild/yocto-worker/a-full/build/layers/openembedded-core is f235ffc43c7783a84e1ac0dce8489fcba8354e12
INFO: SHA-1 revision for master in /srv/pokybuild/yocto-worker/a-full/build/layers/openembedded-core is d595615c6a027b80d6f7ea80ab25e85e97b2436f
INFO: Fetching test results for f235ffc43c7783a84e1ac0dce8489fcba8354e12 in /tmp/sendqaemail.5l_6m4cf
INFO: Fetching matching revision: refs/tags/master/66750-gf235ffc43c7783a84e1ac0dce8489fcba8354e12/0
From push.yoctoproject.org:yocto-testresults
 * [new tag]         master/66750-gf235ffc43c7783a84e1ac0dce8489fcba8354e12/0 -> master/66750-gf235ffc43c7783a84e1ac0dce8489fcba8354e12/0
INFO: Fetching matching revision: refs/tags/master/66750-gf235ffc43c7783a84e1ac0dce8489fcba8354e12/1
From push.yoctoproject.org:yocto-testresults
 * [new tag]         master/66750-gf235ffc43c7783a84e1ac0dce8489fcba8354e12/1 -> master/66750-gf235ffc43c7783a84e1ac0dce8489fcba8354e12/1
INFO: Fetching matching revision: refs/tags/master/66750-gf235ffc43c7783a84e1ac0dce8489fcba8354e12/2
From push.yoctoproject.org:yocto-testresults
 * [new tag]         master/66750-gf235ffc43c7783a84e1ac0dce8489fcba8354e12/2 -> master/66750-gf235ffc43c7783a84e1ac0dce8489fcba8354e12/2
 * [new tag]         yocto-6.0_M3 -> yocto-6.0_M3
INFO: Fetching test results for d595615c6a027b80d6f7ea80ab25e85e97b2436f in /tmp/sendqaemail.5l_6m4cf
INFO: Fetching matching revision: refs/tags/master/67481-gd595615c6a027b80d6f7ea80ab25e85e97b2436f/0
From push.yoctoproject.org:yocto-testresults
 * [new tag]         master/67481-gd595615c6a027b80d6f7ea80ab25e85e97b2436f/0 -> master/67481-gd595615c6a027b80d6f7ea80ab25e85e97b2436f/0
INFO: Fetching matching revision: refs/tags/master/67481-gd595615c6a027b80d6f7ea80ab25e85e97b2436f/1
From push.yoctoproject.org:yocto-testresults
 * [new tag]         master/67481-gd595615c6a027b80d6f7ea80ab25e85e97b2436f/1 -> master/67481-gd595615c6a027b80d6f7ea80ab25e85e97b2436f/1
INFO: Fetching matching revision: refs/tags/master/67481-gd595615c6a027b80d6f7ea80ab25e85e97b2436f/2
From push.yoctoproject.org:yocto-testresults
 * [new tag]         master/67481-gd595615c6a027b80d6f7ea80ab25e85e97b2436f/2 -> master/67481-gd595615c6a027b80d6f7ea80ab25e85e97b2436f/2
INFO: Fetching matching revision: refs/tags/master/67481-gd595615c6a027b80d6f7ea80ab25e85e97b2436f/3
From push.yoctoproject.org:yocto-testresults
 * [new tag]         master/67481-gd595615c6a027b80d6f7ea80ab25e85e97b2436f/3 -> master/67481-gd595615c6a027b80d6f7ea80ab25e85e97b2436f/3
INFO: Fetching matching revision: refs/tags/master/67481-gd595615c6a027b80d6f7ea80ab25e85e97b2436f/4
From push.yoctoproject.org:yocto-testresults
 * [new tag]         master/67481-gd595615c6a027b80d6f7ea80ab25e85e97b2436f/4 -> master/67481-gd595615c6a027b80d6f7ea80ab25e85e97b2436f/4
INFO: Fetching matching revision: refs/tags/master/67481-gd595615c6a027b80d6f7ea80ab25e85e97b2436f/5
From push.yoctoproject.org:yocto-testresults
 * [new tag]         master/67481-gd595615c6a027b80d6f7ea80ab25e85e97b2436f/5 -> master/67481-gd595615c6a027b80d6f7ea80ab25e85e97b2436f/5
INFO: Fetching matching revision: refs/tags/master/67481-gd595615c6a027b80d6f7ea80ab25e85e97b2436f/6
From push.yoctoproject.org:yocto-testresults
 * [new tag]         master/67481-gd595615c6a027b80d6f7ea80ab25e85e97b2436f/6 -> master/67481-gd595615c6a027b80d6f7ea80ab25e85e97b2436f/6
INFO: Fetching matching revision: refs/tags/master/67481-gd595615c6a027b80d6f7ea80ab25e85e97b2436f/7
From push.yoctoproject.org:yocto-testresults
 * [new tag]         master/67481-gd595615c6a027b80d6f7ea80ab25e85e97b2436f/7 -> master/67481-gd595615c6a027b80d6f7ea80ab25e85e97b2436f/7
INFO: Fetching matching revision: refs/tags/master/67481-gd595615c6a027b80d6f7ea80ab25e85e97b2436f/8
INFO: Fetching matching revision: refs/tags/master/67481-gd595615c6a027b80d6f7ea80ab25e85e97b2436f/9
INFO: Running resulttool regression between SHA1 f235ffc43c7783a84e1ac0dce8489fcba8354e12 and d595615c6a027b80d6f7ea80ab25e85e97b2436f
INFO: Comparing:
TestedRev(commit='f235ffc43c7783a84e1ac0dce8489fcba8354e12', commit_number='66750', tags=['master/66750-gf235ffc43c7783a84e1ac0dce8489fcba8354e12/0', 'master/66750-gf235ffc43c7783a84e1ac0dce8489fcba8354e12/1', 'master/66750-gf235ffc43c7783a84e1ac0dce8489fcba8354e12/2'])
to
TestedRev(commit='d595615c6a027b80d6f7ea80ab25e85e97b2436f', commit_number='67481', tags=['master/67481-gd595615c6a027b80d6f7ea80ab25e85e97b2436f/0', 'master/67481-gd595615c6a027b80d6f7ea80ab25e85e97b2436f/1', 'master/67481-gd595615c6a027b80d6f7ea80ab25e85e97b2436f/2', 'master/67481-gd595615c6a027b80d6f7ea80ab25e85e97b2436f/3', 'master/67481-gd595615c6a027b80d6f7ea80ab25e85e97b2436f/4', 'master/67481-gd595615c6a027b80d6f7ea80ab25e85e97b2436f/5', 'master/67481-gd595615c6a027b80d6f7ea80ab25e85e97b2436f/6', 'master/67481-gd595615c6a027b80d6f7ea80ab25e85e97b2436f/7', 'master/67481-gd595615c6a027b80d6f7ea80ab25e85e97b2436f/8', 'master/67481-gd595615c6a027b80d6f7ea80ab25e85e97b2436f/9'])


command timed out: 1200 seconds without output running [b'/srv/pokybuild/yocto-worker/a-full/yocto-autobuilder-helper/scripts/send_qa_email.py', b'None', b'/srv/pokybuild/yocto-worker/a-full/layerinfo.json', b'/srv/autobuilder/repos/a-full-4096', b'--url', b'https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/4096', b'-p', b'', b'-r', b'20260628-8', b'-R', b'/srv/autobuilder/valkyrie.yocto.io/pub/non-release/20260628-8/testresults'], attempting to kill
process killed by signal 9
program finished with exit code -1
elapsedTime=1559.365461
Comment 1 Mathieu Dubois-Briand 2026-06-30 09:00:13 UTC
a-full alma9-vk-2 master completed at 2026-06-28 04:04:36+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/4096/steps/40/logs/stdio
Comment 2 Mathieu Dubois-Briand 2026-07-02 08:48:45 UTC
This is a different error, but in the same code, 2 days later. So far I can't
really believe it's unrelated.
ValueError: Unable to find 'commit' value 'f235ffc43c7783a84e1ac0dce8489fcba8354e12'

====================================================================================================
Processing regression report (1782839091.1)
====================================================================================================

INFO: Generating regression report between yocto-6.0_M3 and master
INFO: Comparing master to yocto-6.0_M3
INFO: Compute regression report between yocto-6.0_M3 and master
INFO: SHA-1 revision for yocto-6.0_M3 in /srv/pokybuild/yocto-worker/a-full/build/layers/openembedded-core is f235ffc43c7783a84e1ac0dce8489fcba8354e12
INFO: SHA-1 revision for master in /srv/pokybuild/yocto-worker/a-full/build/layers/openembedded-core is 9780e392300549f69e11e2f3bcb9bff77b21d572
INFO: Fetching test results for f235ffc43c7783a84e1ac0dce8489fcba8354e12 in /tmp/sendqaemail.si9uzwk_
INFO: Fetching matching revision: refs/tags/master/66750-gf235ffc43c7783a84e1ac0dce8489fcba8354e12/0
From push.yoctoproject.org:yocto-testresults
 * [new tag]         master/66750-gf235ffc43c7783a84e1ac0dce8489fcba8354e12/0 -> master/66750-gf235ffc43c7783a84e1ac0dce8489fcba8354e12/0
INFO: Fetching matching revision: refs/tags/master/66750-gf235ffc43c7783a84e1ac0dce8489fcba8354e12/1
From push.yoctoproject.org:yocto-testresults
 * [new tag]         master/66750-gf235ffc43c7783a84e1ac0dce8489fcba8354e12/1 -> master/66750-gf235ffc43c7783a84e1ac0dce8489fcba8354e12/1
INFO: Fetching matching revision: refs/tags/master/66750-gf235ffc43c7783a84e1ac0dce8489fcba8354e12/2
From push.yoctoproject.org:yocto-testresults
 * [new tag]         master/66750-gf235ffc43c7783a84e1ac0dce8489fcba8354e12/2 -> master/66750-gf235ffc43c7783a84e1ac0dce8489fcba8354e12/2
 * [new tag]         yocto-6.0_M3 -> yocto-6.0_M3
INFO: Fetching test results for 9780e392300549f69e11e2f3bcb9bff77b21d572 in /tmp/sendqaemail.si9uzwk_
INFO: Fetching matching revision: refs/tags/master/67586-g9780e392300549f69e11e2f3bcb9bff77b21d572/0
INFO: Running resulttool regression between SHA1 f235ffc43c7783a84e1ac0dce8489fcba8354e12 and 9780e392300549f69e11e2f3bcb9bff77b21d572
Traceback (most recent call last):
  File "/srv/pokybuild/yocto-worker/a-full/build/layers/openembedded-core/scripts/resulttool", line 83, in <module>
    sys.exit(main())
             ~~~~^^
  File "/srv/pokybuild/yocto-worker/a-full/build/layers/openembedded-core/scripts/resulttool", line 77, in main
    ret = args.func(args, logger)
  File "/srv/pokybuild/yocto-worker/a-full/build/layers/openembedded-core/scripts/lib/resulttool/regression.py", line 373, in regression_git
    index1 = gitarchive.rev_find(revs, 'commit', args.commit)
  File "/srv/pokybuild/yocto-worker/a-full/build/layers/openembedded-core/meta/lib/oeqa/utils/gitarchive.py", line 284, in rev_find
    raise ValueError("Unable to find '{}' value '{}'".format(attr, val))
ValueError: Unable to find 'commit' value 'f235ffc43c7783a84e1ac0dce8489fcba8354e12'
Traceback (most recent call last):
  File "/srv/pokybuild/yocto-worker/a-full/build/layers/openembedded-core/scripts/yocto_testresults_query.py", line 126, in <module>
    ret =  main()
  File "/srv/pokybuild/yocto-worker/a-full/build/layers/openembedded-core/scripts/yocto_testresults_query.py", line 122, in main
    args.func(args)
    ~~~~~~~~~^^^^^^
  File "/srv/pokybuild/yocto-worker/a-full/build/layers/openembedded-core/scripts/yocto_testresults_query.py", line 91, in regression
    report = compute_regression_report(workdir, basebranch, baserevision, targetbranch, targetrevision, args)
  File "/srv/pokybuild/yocto-worker/a-full/build/layers/openembedded-core/scripts/yocto_testresults_query.py", line 64, in compute_regression_report
    report = subprocess.check_output(command).decode("utf-8")
             ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/usr/lib64/python3.14/subprocess.py", line 473, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
           ~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
               **kwargs).stdout
               ^^^^^^^^^
  File "/usr/lib64/python3.14/subprocess.py", line 578, in run
    raise CalledProcessError(retcode, process.args,
                             output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/srv/pokybuild/yocto-worker/a-full/build/layers/openembedded-core/scripts/resulttool', 'regression-git', '--branch', 'master', '--commit', 'f235ffc43c7783a84e1ac0dce8489fcba8354e12', '--branch2', 'master', '--commit2', '9780e392300549f69e11e2f3bcb9bff77b21d572', '/tmp/sendqaemail.si9uzwk_']' returned non-zero exit status 1.
ERROR: Error while generating regression report: Command '['/srv/pokybuild/yocto-worker/a-full/build/layers/openembedded-core/scripts/yocto_testresults_query.py', 'regression-report', 'yocto-6.0_M3', 'master', '-t', '/tmp/sendqaemail.si9uzwk_']' returned non-zero exit status 1.

a-full fedora44-vk-1 master completed at 2026-06-30 17:07:09+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/4110/steps/38/logs/stdio
Comment 3 Tim Orling 2026-07-02 14:44:19 UTC
First failure is timeout, perhaps related to git server load issues at the time.

The second one might be something about the version of git in Fedora 44, we should re-run on a-full. Might be a problem with the new Fedora 44 worker. But, we do not have worker filtering for a-full. Probably have to pause all the workers except fedora44 and kick off a build.