https://autobuilder.yoctoproject.org/valkyrie/#/builders/61/builds/3252/steps/13/logs/stdio Failed ptests: {'python3': ['test_call_count_thread_safe', 'python3']} qemuarm64-ptest ubuntu2204-vk-arm2 --- ====================================================================== FAIL: test_call_count_thread_safe (test.test_unittest.testmock.testthreadingmock.TestThreadingMock.test_call_count_thread_safe) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/lib/python3.14/test/test_unittest/testmock/testthreadingmock.py", line 219, in test_call_count_thread_safe self.assertEqual(m.call_count, LOOPS * THREADS) ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: 983 != 1000 ---------------------------------------------------------------------- Ran 1089 tests in 5.499s FAILED (failures=1, skipped=2) test test_unittest failed 0:04:23 load avg: 2.16 [452/492/1] test_userdict passed -- running (1): test_subprocess (39.3 sec) --- Full logs available at: https://valkyrie.yocto.io/pub/non-release/20260313-136/testresults/qemuarm64-ptest/python3.log
Likely a timing issue. We could increase the timeout.
Hi, I'd like to work on this bug. Based on the traceback and Randy's comment, this looks like a timing or race condition under load on qemuarm64. I'll start by reproducing it locally and explore whether adjusting the thread loop count or adding a small tolerance to the assertion would help. Will update here with findings.
Thanks Sai. CCed Trevor since I think he has worked on python recently so he might be interested and/or helpful. Any luck reproducing the issue ? What is you test system like? Try running stress -c 10 or so, depending on how many cores your test system has. This is a poor but sometimes helpful way to mimic the load seen in the YP autobuilder machines which run builds and tests alongside each other.
Hi Randy, Thanks for the tip about stress! Yes, I was able to reproduce the issue. I confirmed it's a race condition in CPython's ThreadingMock._increment_mock_call() where concurrent calls lose increments to call_count under load. Reproduction results: - x86 (native): 23/30 failures with 50 threads x 10000 calls - qemuarm64 (chroot via QEMU): 3/20 failures with just 10 threads x 100 calls (same scale as the original bug report) The fix overrides _increment_mock_call in ThreadingMixin and wraps it with the existing _mock_calls_events_lock to make the increment thread-safe. The fix has been merged into CPython main: - Issue: https://github.com/python/cpython/issues/150175 - PR: https://github.com/python/cpython/pull/150176 (merged by cjw296) It will be included in the next Python 3.14 release and should resolve this intermittent ptest failure once Yocto updates the python3 recipe. Thanks for the guidance! Sai Sneha
It's great to hear that you reproduced, fixed and upstreamed a fix already. Would you be able to cherry-pick the fix back to oe-core/master following: https://docs.yoctoproject.org/contributor-guide/index.html ?
Hi Randy, Thanks! I'd be happy to cherry-pick the fix to oe-core/master. I'll go through the contributor guide and submit a patch. Sai Sneha
Hi Randy, As requested, I've cherry-picked the fix to oe-core and submitted a patch to the mailing list: Subject: [PATCH] python3: Fix ThreadingMock call_count race condition To: openembedded-core@lists.openembedded.org Testing done: - bitbake python3 -c patch: applied cleanly - x86 (50 threads x 10000 calls x 30 runs): 0/30 failures - qemuarm64 (10 threads x 100 calls x 20 runs): 0/20 failures - All 19 existing ThreadingMock tests pass Upstream CPython fix: https://github.com/python/cpython/pull/150176 Sai Sneha
Super! It's being tested in: https://git.openembedded.org/openembedded-core/log/?h=master-next and if all goes well, it will show up in: https://git.openembedded.org/openembedded-core-contrib/log/?h=master-review and be reviewed Thursday, May 28th at 4 AM ET, but not by me! ;-) Later on Thursday, if there are no objections, it will be in oe-core master. Fun, eh!
Thanks Randy! Great to know it's being tested. I'll keep an eye on the autobuilder results and the Thursday review. Exciting to see it go through the full process!
Hi Randy, The oe-core patch has been successfully merged into master by Richard Purdie: https://git.openembedded.org/openembedded-core/commit/?id=6f7af3f76c8ce0a77ddc779a850071a714caff33 Thank you for the guidance throughout the process!
Sai, I'm happy to help and thankful for your contribution. Thanks for adding the link, I've confirmed that it's on master since I'm a little obsessive! ;-) ❯ git branch -a --contains 6f7af3f76c8ce0a77ddc779a850071a714caff33 * master remotes/origin/HEAD -> origin/master remotes/origin/master remotes/origin/master-next Resolving the bug for you but next time, you're free to do that yourself since this is a community of trust without a team to do individual patch QA. Thanks again!