15140 2023-06-08 21:10:06 +0000 AB-INT: yocto-testresults tagging failure, tag already exists 2023-10-12 07:05:26 +0000 1 1 5 Infrastructure AutoBuilder autobuilder unspecified x86 Multiple RESOLVED FIXED AB-INT Medium+ normal 4.3 M4 1 alexandre.belloni alexis.lothore alexandre.belloni infras.ab.watcher Infras.watcher pidge randy.macleod tgamblin --- --- No (bug/feature does not impact docs) oldest_to_newest 95680 0 alexandre.belloni 2023-06-08 21:10:06 +0000 Cloning into '/tmp/sendqaemail.pvwa2qqi'... 2d08981fb294a94aa1fff6f535372528b261e5fa INFO: Reading files from /srv/autobuilder/autobuilder.yocto.io/pub/non-release/20230525-2/testresults INFO: Found 1 revisions to store INFO: Storing test result into git repository /tmp/sendqaemail.pvwa2qqi INFO: Committing data into to branch master INFO: Updating /tmp/sendqaemail.pvwa2qqi HEAD to latest commit INFO: Creating tag master/70345-g9e3be6c905e717c3d6d00e75ff78fc2edd0e02e8/2 To push.yoctoproject.org:yocto-testresults 2d08981..baf0f04 master -> master To push.yoctoproject.org:yocto-testresults ! [rejected] master/70345-g9e3be6c905e717c3d6d00e75ff78fc2edd0e02e8/2 -> master/70345-g9e3be6c905e717c3d6d00e75ff78fc2edd0e02e8/2 (already exists) error: failed to push some refs to 'push.yoctoproject.org:yocto-testresults' hint: Updates were rejected because the tag already exists in the remote. Working off ssh://git@push.yoctoproject.org/poky:master Traceback (most recent call last): File "/home/pokybuild/yocto-worker/a-quick/yocto-autobuilder-helper/scripts/send-qa-email", line 216, in <module> send_qa_email() File "/home/pokybuild/yocto-worker/a-quick/yocto-autobuilder-helper/scripts/send-qa-email", line 156, in send_qa_email subprocess.check_call(["git", "push", "--tags"], cwd=tempdir) File "/usr/lib64/python3.11/subprocess.py", line 413, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['git', 'push', '--tags']' returned non-zero exit status 1. 95681 1 alexandre.belloni 2023-06-08 21:10:36 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2143/steps/29/logs/stdio 96224 2 alexandre.belloni 2023-08-10 14:29:20 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/85/builds/2209/steps/29/logs/stdio 96240 3 alexis.lothore 2023-08-10 18:56:41 +0000 I think I _theoretically_ found out what happens here. Diving into tag generation code in resulttool, I observe that the tag number (the "tag_number}" in "{branch}/{commit_count}-g{commit}/{tag_number}" is always computed based on existing tags. This tag number is here to allow to store multiple test results regarding a single poky revision. I eventually reach a "git tag" command to list existing tags. Passed repository is the test results repository cloned by send-qa-email. The twist is that this clone is in fact a shallow clone, so only recent tags are known. Since recently, my guess is that we had "luck" because if two builds targeting the same poky revision happened, they were timely close enough so the shallow clone was still aware of previous tags on the same poky revision, and tag number was incremented then pushed correctly. BUT: one of my recent development (2c0d9e19771a4fc8365a40d553b4be19c491785f in yocto-autobuilder-helper) made the clone even more shallow (depth=1). So whenever a build targets a revision which has already been tested, but other tests have been pushed since then, send-qa-email fails to detect the existing tag, so it uses 0 (as first tag number since it thinks it is running tests on this specific version for the first time), creates a tag that does not exist locally but exists remotely, and fails to push. The proper fix should be to use git ls-remote instead of git tag to make sure to get all remote tags. Quickly checking gitarchive.py (which is called to generate the tag), I see only two places that need the fix. I will come up with something and send a patch 96243 4 alexandre.belloni 2023-08-11 08:19:54 +0000 I guess you are right, I got this today: 2023-08-10 22:28:27,720 - oe-selftest - INFO - ... FAIL Stderr: 2023-08-10 21:02:03,493 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-3253880/conf/local.conf 2023-08-10 21:02:03,493 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf INFO: Shallow-cloning testresults in /tmp/yocto-testresults-query.pj8tcxbc Cloning into '/tmp/yocto-testresults-query.pj8tcxbc'... fatal: ambiguous argument 'yocto-4.0': unknown revision or path not in the working tree. Use '--' to separate paths from revisions, like this: 'git <command> [<revision>...] -- [<file>...]' ERROR: Can not find SHA-1 for yocto-4.0 in /home/pokybuild/yocto-worker/oe-selftest-centos/build 2023-08-10 22:28:27,721 - oe-selftest - INFO - 4: 45/46 388/533 (0.18s) (0 failed) (yoctotestresultsquerytests.TestResultsQueryTests.test_get_sha1) 2023-08-10 22:28:27,721 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/lib/oeqa/selftest/cases/yoctotestresultsquerytests.py", line 27, in test_get_sha1 self.assertEqual( File "/usr/lib64/python3.9/unittest/case.py", line 837, in assertEqual assertion_func(first, second, msg=msg) File "/usr/lib64/python3.9/unittest/case.py", line 830, in _baseAssertEqual raise self.failureException(msg) AssertionError: None != '00cfdde791a0176c134f31e5a09eff725e75b905' Stderr: 2023-08-10 21:02:03,493 - oe-selftest - INFO - Adding: "include selftest.inc" in /home/pokybuild/yocto-worker/oe-selftest-centos/build/build-st-3253880/conf/local.conf 2023-08-10 21:02:03,493 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf INFO: Shallow-cloning testresults in /tmp/yocto-testresults-query.pj8tcxbc Cloning into '/tmp/yocto-testresults-query.pj8tcxbc'... fatal: ambiguous argument 'yocto-4.0': unknown revision or path not in the working tree. Use '--' to separate paths from revisions, like this: 'git <command> [<revision>...] -- [<file>...]' ERROR: Can not find SHA-1 for yocto-4.0 in /home/pokybuild/yocto-worker/oe-selftest-centos/build 96417 5 tgamblin 2023-09-14 15:32:07 +0000 Moved to M4.