| Summary: | AB_INT: resulttool: File exists | ||
|---|---|---|---|
| Product: | [QA/Testing] Tooling | Reporter: | Mathieu Dubois-Briand <mathieu.dubois-briand> |
| Component: | tooling | Assignee: | Yoann Congal <yoann.congal> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | normal | ||
| Priority: | Medium | CC: | richard.purdie, tim.orling, yoann.congal |
| Version: | unspecified | ||
| Target Milestone: | 6.0 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | AB-INT | ||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | Done (doc changes complete) | |
a-full ubuntu2204-vk-2 mathieu/master-next completed at 2024-12-14T10:21:05Z https://valkyrie.yoctoproject.org/#/builders/29/builds/643/steps/29/logs/stdio a-full fedora40-vk-3 mathieu/master-next completed at 2024-12-17T19:59:03Z https://valkyrie.yoctoproject.org/#/builders/29/builds/661/steps/29/logs/stdio a-full ubuntu2204-vk-3 mathieu/master-next completed at 2024-12-20T12:04:25Z https://valkyrie.yoctoproject.org/#/builders/29/builds/685/steps/29/logs/stdio a-full fedora41-vk-1 mathieu/master-next completed at 2025-02-21T20:39:34Z https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/1064/steps/29/logs/stdio a-full opensuse156-vk-1 mathieu/master-next completed at 2025-02-27T21:00:33Z https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/1097/steps/31/logs/stdio a-full debian11-vk-3 mathieu/master-next completed at 2025-04-13T21:01:56Z https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/1391/steps/31/logs/stdio Bulk move of all unassigned 5.2 medium importance bugs to 5.3. a-full alma8-vk-2 mathieu/master-next completed at 2025-07-09 18:09:44+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/1986/steps/33/logs/stdio a-full rocky9-vk-3 mathieu/master-next completed at 2025-07-22 13:17:31+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2058/steps/31/logs/stdio a-quick debian12-vk-8 kraj/clang-merge-2 completed at 2025-04-20 06:58:02+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/62/builds/79/steps/31/logs/stdio a-quick debian11-vk-1 kraj/clang-merge-2 completed at 2025-05-02 10:53:37+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/62/builds/105/steps/31/logs/stdio a-full fedora41-vk-1 mathieu/master-next completed at 2025-09-26 18:04:01+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2446/steps/31/logs/stdio a-full ubuntu2404-vk-1 master completed at 2025-11-14 07:26:11+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2721/steps/37/logs/stdio a-full fedora42-vk-1 master completed at 2025-11-15 11:11:46+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2724/steps/37/logs/stdio a-full debian12-vk-1 master completed at 2025-11-16 05:50:23+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2728/steps/37/logs/stdio a-full ubuntu2404-vk-2 master completed at 2025-11-18 10:34:55+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2737/steps/37/logs/stdio a-full fedora39-vk-1 master completed at 2025-11-28 02:23:24+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2797/steps/37/logs/stdio a-full debian12-vk-1 master completed at 2025-11-29 04:13:24+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2803/steps/37/logs/stdio a-full fedora42-vk-1 master completed at 2025-11-30 04:11:30+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2806/steps/37/logs/stdio a-full rocky8-vk-1 master completed at 2025-12-01 04:09:28+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2807/steps/39/logs/stdio a-full rocky9-vk-2 master completed at 2025-12-02 04:54:34+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2810/steps/37/logs/stdio a-full ubuntu2204-vk-1 master completed at 2025-12-04 00:23:12+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2818/steps/37/logs/stdio a-full ubuntu2204-vk-1 mathieu/master-next completed at 2025-12-23 00:26:49+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2913/steps/37/logs/stdio a-full stream9-vk-1 mathieu/master-next completed at 2025-12-25 20:49:44+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2938/steps/37/logs/stdio a-full alma9-vk-2 mathieu/master-next completed at 2025-12-26 15:19:26+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2946/steps/37/logs/stdio So the issue here is in resulttool, when archiving logs: - we create a local tag, making sure it does not clash with a tag already existing on the remote. - we copy the logs in shared NFS, folder name does correspond to this tag, e.g. /srv/autobuilder/valkyrie.yocto.io/pub/non-release/20251129-12/testresults/../../testresult-logarchives/master/65319-gac7327583aef83fd400190506f3147623de38b68/0 - we push the tag But, if for any reason the tag is not pushed, the same tag name will be reused on the next build of the same oe-core version. And if the logs were already copied, the log copy of this next build will fail. I believe the solution here is to make sure to push the tag before copying any log file. Also, for some branches like mine, the tag is never pushed: rebuilding the same oe-core version on such a branch will always trigger this failure. I believe copying logs while not pushing the tag does not make sense, so we should just make sure to only copy the logs when we do push tags. I'm going to send a few patches trying to fix this. Also on whinlatter-nut: https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2997#bb-step-37 Patches sent by Mathieu: *[PATCH 0/2] oeqa/gitarchive: Push tag before copying log files https://lists.openembedded.org/g/openembedded-core/message/228687 * [PATCH 1/2] oeqa/gitarchive: Fix git push URL parameter https://lists.openembedded.org/g/openembedded-core/message/228688 * [PATCH 2/2] oeqa/gitarchive: Push tag before copying log files https://lists.openembedded.org/g/openembedded-core/message/228689 (In reply to Yoann Congal from comment #21) > Patches sent by Mathieu: > *[PATCH 0/2] oeqa/gitarchive: Push tag before copying log files > https://lists.openembedded.org/g/openembedded-core/message/228687 > * [PATCH 1/2] oeqa/gitarchive: Fix git push URL parameter > https://lists.openembedded.org/g/openembedded-core/message/228688 > * [PATCH 2/2] oeqa/gitarchive: Push tag before copying log files > https://lists.openembedded.org/g/openembedded-core/message/228689 Thanks, I forgot to list these here. Also, we have one on the autobuilder-helper side: [PATCH yocto-autobuilder-helper] send_qa_email: Only copy logs when we are pushing tags https://lists.yoctoproject.org/g/yocto-patches/message/2885 a-full ubuntu2404-vk-3 whinlatter-next&stable/whinlatter-nut&stable/2.16-nut completed at 2026-01-08 10:30:05+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3010/steps/37/logs/stdio Stable build hit: whinlatter a-full fedora40-vk-3 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/2995 OE-Core commits merged on master: oeqa/gitarchive: Fix git push URL parameter https://git.openembedded.org/openembedded-core/commit/?id=024f08629feeec8198d1e489633e475959754cfe oeqa/gitarchive: Push tag before copying log files https://git.openembedded.org/openembedded-core/commit/?id=7d869c9c5aaeeda9fa476bfe6b05ded6e225379d scripts/oe-git-archive: Ensure new push parameter is specified https://git.openembedded.org/openembedded-core/commit/?id=983cb2171e53564bc9dd188136439f3e2ad9e188 (but not the yocto-autobuilder-helper part, yet) a-full fedora40-vk-3 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3082 Note: This is a weird build: yocto-autobuilder-helper stayed on default (master?) but layers/bitbake where on stable/whinlatter-nut whinlatter a-full ubuntu2204-vk-4 https://autobuilder.yoctoproject.org/valkyrie/#/builders/29/builds/3083 Patch merged in yocto-autobuilder-helper/master: send_qa_email: Only copy logs when we are pushing tags https://git.yoctoproject.org/yocto-autobuilder-helper/commit/?id=904d5722ced282b58474e3b605ce87897994f4f7 Also, maybe related: scripts/send_qa_email: Show git processing commands used in logs - yocto-autobuilder-helper - Decoupled release build scripts https://git.yoctoproject.org/yocto-autobuilder-helper/commit/?id=612fb877fef9885b7793117556a5152e312de4da scripts/send_qa_email.py: Add missing log parameter - yocto-autobuilder-helper - Decoupled release build scripts https://git.yoctoproject.org/yocto-autobuilder-helper/commit/?id=e47be6614f777dd8f670618250645c2265c391d0 scripts/send_qa_email.py: Fix master branch comparision for regression report - yocto-autobuilder-helper - Decoupled release build scripts https://git.yoctoproject.org/yocto-autobuilder-helper/commit/?id=ec17abb0b2d759b800cefec4fcac61cee74dbefc whinlatter a-full rocky8-vk-1 https://autobuilder.yoctoproject.org/valkyrie/?#/builders/29/builds/3112 Last commit was merged in autobuilder-helper master branch: https://git.yoctoproject.org/yocto-autobuilder-helper/commit/?id=904d5722ced282b58474e3b605ce87897994f4f7 @Yoann: What about stable branches? Is the issue still present? (In reply to Mathieu Dubois-Briand from comment #30) > Last commit was merged in autobuilder-helper master branch: > https://git.yoctoproject.org/yocto-autobuilder-helper/commit/ > ?id=904d5722ced282b58474e3b605ce87897994f4f7 > > @Yoann: What about stable branches? Is the issue still present? I've not seen this one recently but not everything is merged. You can attribute this bug to me and I'll close it once everything is merged on their respective branches. For example: This is not merged: send_qa_email: Only copy logs when we are pushing tags https://git.yoctoproject.org/yocto-autobuilder-helper/commit/?h=contrib/ycongal/whinlatter-nut&id=b72ff5eb03a2296287b39c82443ef048d6a6db4b Yoann to close when everything backported openembedded-core/whinlatter: * oeqa/gitarchive: Fix git push URL parameter * https://git.openembedded.org/openembedded-core/commit/?h=whinlatter&id=97a510e319286cf5cd86d3a905bb1f8311d3db7e * oeqa/gitarchive: Push tag before copying log files * https://git.openembedded.org/openembedded-core/commit/?h=whinlatter&id=412c9c6388d449e9a455d1b1961fe17d81b8528f * scripts/oe-git-archive: Ensure new push parameter is specified * https://git.openembedded.org/openembedded-core/commit/?h=whinlatter&id=003b6951b58e732dd5e780c3624fd1966a3db5c5 yocto-autobuilder-helper/whinlatter: * send_qa_email: Only copy logs when we are pushing tags * https://git.yoctoproject.org/yocto-autobuilder-helper/commit/?h=whinlatter&id=be29ed1e75ef15b9bf395c576a3a0f519d2b2013 Everything merged, closing! |
Probable AB-INT We had two builds with a similar error: 277 INFO: Creating tag mathieu/master-next/76713-gd2218d353b5d7becad9610f997c57b8099ea3753/0 278 Traceback (most recent call last): 279 File "/srv/pokybuild/yocto-worker/a-full/build/scripts/resulttool", line 83, in <module> 280 sys.exit(main()) 281 ^^^^^^ 282 File "/srv/pokybuild/yocto-worker/a-full/build/scripts/resulttool", line 77, in main 283 ret = args.func(args, logger) 284 ^^^^^^^^^^^^^^^^^^^^^^^ 285 File "/srv/pokybuild/yocto-worker/a-full/build/scripts/lib/resulttool/store.py", line 89, in store 286 shutil.copytree(tempdir, logdir) 287 File "/usr/lib64/python3.12/shutil.py", line 600, in copytree 288 return _copytree(entries=entries, src=src, dst=dst, symlinks=symlinks, 289 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 290 File "/usr/lib64/python3.12/shutil.py", line 498, in _copytree 291 os.makedirs(dst, exist_ok=dirs_exist_ok) 292 File "<frozen os>", line 225, in makedirs 293 FileExistsError: [Errno 17] File exists: '/srv/autobuilder/valkyrie.yocto.io/pub/non-release/20241217-82/testresults/../../testresult-logarchives/mathieu/master-next/76713-gd2218d353b5d7becad9610f997c57b8099ea3753/0' 294 Traceback (most recent call last): 295 File "/srv/pokybuild/yocto-worker/a-full/yocto-autobuilder-helper/scripts/send_qa_email.py", line 281, in <module> 296 send_qa_email() 297 File "/srv/pokybuild/yocto-worker/a-full/yocto-autobuilder-helper/scripts/send_qa_email.py", line 191, in send_qa_email 298 subprocess.check_call([resulttool, "store", "--revision", revision, "-l", args.results_dir + "/../../testresult-logarchives" , args.results_dir, tempdir]) 299 File "/usr/lib64/python3.12/subprocess.py", line 413, in check_call 300 raise CalledProcessError(retcode, cmd) 301 subprocess.CalledProcessError: Command '['/srv/pokybuild/yocto-worker/a-full/build/scripts/resulttool', 'store', '--revision', 'd2218d353b5d7becad9610f997c57b8099ea3753', '-l', '/srv/autobuilder/valkyrie.yocto.io/pub/non-release/20241217-82/testresults/../../testresult-logarchives', '/srv/autobuilder/valkyrie.yocto.io/pub/non-release/20241217-82/testresults', '/tmp/sendqaem 301 ail.wzqr2sik']' returned non-zero exit status 1. https://valkyrie.yoctoproject.org/#/builders/29/builds/643/steps/29/logs/stdio https://valkyrie.yoctoproject.org/#/builders/29/builds/661/steps/29/logs/stdio