https://autobuilder.yoctoproject.org/valkyrie/#/builders/61/builds/3463/steps/13/logs/stdio Failed ptests: {'gawk': ['randtest']} qemuarm64-ptest ubuntu2404-vk-arm1 ---- PASS: rand Tau= 305, Autocorr= -5.2484 sigma Test failed. FAIL: randtest (...) PASS: rand Tau= 305, Autocorr= -5.2484 sigma Test failed. FAIL: randtest (...) debug: [running]$ ssh -l root -o serveralivecountmax=2 -o serveraliveinterval=30 -o userknownhostsfile=/dev/null -o stricthostkeychecking=no -o loglevel=error 192.168.7.6 export path=/usr/sbin:/sbin:/usr/bin:/bin; dmesg | grep "killed process" debug: no output from ssh call debug: [command returned '1' after 0.18 seconds] debug: command: dmesg | grep "killed process" status: 1 output: warning: there were failing ptests. note: ... expected failure traceback (most recent call last): file "/srv/pokybuild/yocto-worker/qemuarm64-ptest/build/layers/openembedded-core/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ file "/srv/pokybuild/yocto-worker/qemuarm64-ptest/build/layers/openembedded-core/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ file "/srv/pokybuild/yocto-worker/qemuarm64-ptest/build/layers/openembedded-core/meta/lib/oeqa/core/decorator/__init__.py", line 35, in wrapped_f return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ file "/srv/pokybuild/yocto-worker/qemuarm64-ptest/build/layers/openembedded-core/meta/lib/oeqa/runtime/cases/ptest.py", line 27, in test_ptestrunner_expectfail self.do_ptestrunner() file "/srv/pokybuild/yocto-worker/qemuarm64-ptest/build/layers/openembedded-core/meta/lib/oeqa/runtime/cases/ptest.py", line 121, in do_ptestrunner self.fail(failmsg) assertionerror: failed ptests: {'gawk': ['randtest']} ---- Full logs at: https://valkyrie.yocto.io/pub/releases/yocto-6.0_M3.rc1/testresults/qemuarm64-ptest/core-image-ptest-gawk/log.do_testimage.2407178.20260410133524
I would like to work on this bug.
I have analyzed and worked on this issue. The randtest test is part of the gawk ptest suite and is executed through the Maketests framework. While it passes consistently in my local environment, it relies on deterministic random number sequences that are not guaranteed across different architectures, libc implementations, and runtime conditions. This explains the intermittent failures (AB-INT) observed in the Yocto autobuilder. I initially attempted to disable the test using PTEST_DISABLED_TESTS, but found that gawk’s test framework does not honor this variable. The fix removes randtest during do_install_ptest() and marks it as skipped, following the same pattern used for other nondeterministic tests such as time and timeout. After applying the fix: - randtest is reported as SKIP - no failures occur - ptest results are stable
Created attachment 5216 [details] Patch to disable randtest in gawk ptest due to nondeterministic behavior Attaching patch to disable randtest in gawk ptest. randtest relies on deterministic random number sequences which are not guaranteed across different environments, leading to intermittent failures (AB-INT). The fix removes randtest during do_install_ptest() and marks it as skipped, following the same approach used for other unstable tests. Tested locally and confirmed randtest is skipped and ptest results are stable.
Hi Aditya, Thanks for working on this bug. It is tempting to skip the test and we may eventually end up doing that but it might be better to identify the source of the non-deterministic randomness in our environment and fix that. Can you explain how you came to this conclusion ? Could we add code to the test to dump out what part of the test is failing to learn more about the problem ? I haven't even looked at the failing code yet so let me know if I should take the time do to that to be more effective at helping review the issue. I have however worked in the area of random number generation and entropy pool depth for yocto a bit so I do have some experience in this area! ;-) Also this issue happened once on qemuarm64, an emulated target that we have extensive experience with so I'm especially suspicious about the claimed incorrect behaviour of random numbers. Adding Ross from Arm as a sanity check! Also, now or once we finish this discussion, are you able to send the patch to the oe-core email list as documented in: https://docs.yoctoproject.org/contributor-guide/index.html ? This how we get more people to examine and hopefully review and reply to the patch, test it on the Yocto Autobuilders and get changes merged. Thanks again!
Hi, Thank you for the detailed feedback. My initial conclusion about nondeterministic behavior was based on the nature of the randtest test itself, which compares expected sequences of random numbers. Since such sequences can depend on libc implementations, floating point behavior, and execution environment, I assumed this could lead to intermittent failures as seen in the autobuilder logs. In my local testing (on qemux86-64), randtest consistently passed, which aligns with stable CPU timing and consistent libc behavior. However, I understand your point that this might indicate an underlying issue rather than the test itself. I agree that it would be better to investigate further before skipping the test. As a next step, I can: - Run the randtest script directly and try to identify which part of the output diverges - Add debugging output to capture the failing conditions Please let me know if you would recommend a specific approach for instrumenting the test or capturing the failure more effectively. Thanks again for your guidance! Also, if you want me to send this patch now to the oe-core email list as documented in: https://docs.yoctoproject.org/contributor-guide/index.html ? I can do it or if its ok I will send it after we finish our discussion on this.
Hi, Thanks for the suggestion. I investigated randtest further by enabling debug output of the autocorrelation values and running the test multiple times through the ptest framework. In my observations: - The test normally produces small correlation values close to zero (as expected for random sequences) - However, in some runs, the condition: abs(corr[tau] / sigma) > max_allowed_sigma is triggered, leading to a failure - I was able to reproduce a failing run where only randtest fails while all other tests pass This indicates that the failure is caused by sensitivity of the statistical threshold rather than a clear functional issue in rand(). The behavior appears to depend on minor variations in floating point calculations and runtime conditions (timing, entropy state, etc.), which aligns with the intermittent nature observed in autobuilder results. Given that: - The underlying randomness appears valid - The failure depends on subtle environmental factors - Similar tests (time/timeout) are already skipped for non-determinism it seems reasonable to consider randtest as inherently unstable in these environments. Please let me know your thoughts on whether we should proceed with skipping it or further adjust the test behavior. Thanks!
Hi, Good analysis! I'm tempted to look at the statistical tests but I'm buys and so I won't. :-( if you add a summary of your analysis to your commit log I suspect the patch will be merged. For bonus points, please report the issue to the upstream gawk devs by email or a bug tracking system and include a link to that in the commit log. Hmmm.. Rather than skipping the test, I wonder if upstream would accept a more generous limit. I guess the best approach is to skip for now and talk with upstream in parallel. Thanks!
Hi, Thanks for the feedback. I have updated the commit message to include a summary of the debugging analysis, describing how the failure was reproduced and the conditions under which it occurs. I also plan to report this behavior to the upstream gawk developers and will include a link to that discussion in the patch once available. Thanks again for your guidance — this was very helpful.
Created attachment 5217 [details] Updated patch with detailed analysis of randtest intermittent failure and root cause
Reported the issue to upstream gawk developers. Upstream clarified that the preferred approach would be to improve the randtest itself rather than disabling it in Yocto. They suggested proposing a change to test/randtest.sh. Currently, I am newcomer to yocto so I do not yet have sufficient confidence in the statistical behavior of the test to propose a safe upstream modification without affecting its validation criteria. However, based on investigation, the intermittent failures appear to be caused by sensitivity in the autocorrelation threshold condition: abs(corr[tau] / sigma) > max_allowed_sigma I will continue to investigate potential improvements to make the test less sensitive to minor environmental variations, and will follow up with an upstream proposal if I arrive at a well-justified approach. In the meantime, proceeding with the Yocto-side workaround to avoid intermittent failures.
Hi, I wanted to follow up on my patch, Kindly let me know if there are any feedback or changes required from my side, and whether there is an expected timeline for the review or merge of this patch. Thanks for your time and guidance. Regards, Aditya GS
Created attachment 5221 [details] Patch for randtest.sh Based on upstream feedback, I attempted to improve the randtest itself rather than disabling it in Yocto. The issue appears to be caused by sensitivity to small statistical fluctuations, where even a single autocorrelation value exceeding the sigma threshold results in intermittent failures. I implemented a change to allow a small number of exceedances before marking the test as failed. This reduces flaky behavior while preserving the intent of detecting statistically significant issues. Please find the patch attached for reference. The upstream submission has also been sent to bug-gawk@gnu.org for review. Looking forward to feedback.
Aditya, I think your approach makes sense and I'm glad to see that you've sent your change upstream to the gawk developers. Can you submit a patch to the oe-core list ( using git send-email as Yoann suggested ) ? The full contributor guide is here: https://docs.yoctoproject.org/dev/contributor-guide/index.html Occasionally people have trouble with git send-email and their emial system so if you're in that situation, you might be able to try using 'b4'. It's a new tool and I suspect that > 98% of contributions are still via git send-email so try that first. If you're stuck, see below and/or ask here or on the oe-core email list for guidance about using b4. We really should add that to the docs... ❯ git log .b4-config commit db5b855f175820918d6643439ccbfe3a70119279 Author: Quentin Schulz <quentin.schulz@cherry.de> Date: Wed Feb 5 11:16:00 2025 b4-config: Add basic b4 config file b4[1] is a very nice tool for mail-based contribution. A config[2] file exists to set up a few defaults. We can use it to set the Cc recipients to always add, in our case the mailing list. Because we do not have anything to check for now, disable needs-checking so patches can be sent without running b4 prep --check. The mid-term/long-term plan is to use patchtest locally. Because we do not have any auto-to-cc support (and the implicit one using scripts/get_maintainer.pl cannot work for us), also disable needs-auto-to-cc so patches can be sent without running b4 prep --auto-to-cc. [1] https://pypi.org/project/b4/ [2] https://b4.docs.kernel.org/en/latest/config.html Signed-off-by: Quentin Schulz <quentin.schulz@cherry.de> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Hi, Thanks for the guidance and for pointing me to the contributor guide. Currently, I am facing some issues with git send-email on my office laptop. I plan to switch to my personal laptop to proceed with sending the patch, but I first need to set up Ubuntu on that system, so it might take a bit of time. In the meantime, I will continue working on updating the patch based on the suggestions from the gawk upstream maintainers. Once that is ready, I will submit the patch to the oe-core mailing list as well. I will first try using git send-email, and if I encounter issues there, I will try using the b4 tool as suggested. Thanks again for the guidance. Regards, Aditya
Based on upstream feedback, I updated the approach for handling the randtest instability. Instead of allowing a small number of statistical outliers, the test now detects cases where the computed sigma is too small (indicating unstable normalization) and retries the computation from scratch. This ensures that the test remains strict while avoiding false failures caused by edge cases in statistical fluctuation. After implementing this change and testing across multiple runs, the behavior is stable with no intermittent failures observed. The updated patch has been sent upstream to the gawk maintainers for review. Once the upstream version is finalized, I will follow up with submission to oe-core as well. Looking forward to feedback.
Created attachment 5222 [details] Updated patch, created as suggested by gawk upstream
I have sent a patch to the OpenEmbedded-Core mailing list to address this issue. The patch implements a more robust handling of unstable sigma values in randtest, retrying computations instead of tolerating statistical outliers, as discussed. I have included the relevant maintainers and contributors in CC for visibility and review: joaomarcos.costa@bootlin.com ross.burton@arm.com randy.macleod@windriver.com richard.purdie@linuxfoundation.org Please let me know if there are any comments or required revisions.
Update: The issue has now been addressed upstream in gawk. The proposed fix to randtest.sh was reviewed by the upstream maintainer and has been integrated into the gawk repository, with the changes properly incorporated into the original file and test/ChangeLog. Since the root cause has been resolved upstream, this eliminates the need for a workaround (such as skipping the test) on the Yocto side. Next step would be to backport the upstream fix into oe-core once the corresponding commit is available. Please let me know if you would like me to prepare a backport patch. Thanks, Aditya
I am currently working on preparing a backport patch for oe-core based on the upstream changes, and am already in discussion with oe-core maintainers regarding this. The patch will be shared shortly for review. Thanks, Aditya
Created attachment 5223 [details] Backport patch The backport patch has been submitted to oe-core for review. Thanks, Aditya
Created attachment 5224 [details] updated backport patch Update: Addressed review comments and submitted updated (v2) backport patch to oe-core for review. Thanks, Aditya
Created attachment 5225 [details] Updated backport patch
Update: Addressed patchtest feedback by adding the missing Signed-off-by to the backported patch file and updated the patch accordingly. Resubmitted as v3 to oe-core. Thanks, Aditya
Created attachment 5226 [details] Updated backport patch Update: Reworked the patch based on review feedback to target the master branch as per OE-Core backport policy. Addressed the following: - Updated patch to apply against gawk 5.4.0 in master - Fixed formatting issues (Upstream-Status, whitespace) - Updated commit message to use [YOCTO #16254] - Ensured proper integration in SRC_URI The updated patch has been submitted as v4 to the oe-core mailing list for review. Thanks, Aditya
Created attachment 5227 [details] Updated backport patch Update: Addressed patch test issue by ensuring the correct patch was submitted and fixing the Upstream-Status formatting in the backported patch file. Resent updated patch for review. Thanks, Aditya
Created attachment 5228 [details] Updated backport patch Update: The issue has now been addressed and the fix has been accepted and merged into OE-Core. Upstream commit: https://git.openembedded.org/openembedded-core/commit/?id=ccabae3036a713cac8b08731b765ff1dc3ed93eb The fix resolves the intermittent failures in randtest.sh by improving the handling of cases where sigma becomes too small, avoiding false positives and ensuring stable test behavior. My earlier submissions went through multiple review iterations, and the maintainers finalized and merged an updated version of the patch to resolve the CI instability more quickly. I have reviewed the merged change and confirm that it addresses the root cause discussed in this bug. Thanks to the maintainers for reviewing and integrating the fix. Thanks, Aditya
Hi Randy, Could you please confirm if it is appropriate to mark this bug as RESOLVED/FIXED? Thanks, Aditya
Hi Aditya, It's your call: You could close it now or you can wait a week or two to see if it really resolves the intermittent errors seen on the Yocto autobuilders. If you resolve it as fixed, and it happens again, most likely it will be re-opened by someone who triages the YP AB issues but they might open a new bug. Flip a coin! ;-) Thanks for working on this bug and getting what is hopefully a fix, upstream into gawk. ../Randy
Hi Randy, Thanks for the clarification. Given that the fix has been merged and addresses the root cause, I will mark this bug as RESOLVED/FIXED. I’ll keep an eye on autobuilder results, and if the issue reappears, it can be reopened or tracked separately as needed. Thanks again for the guidance. Regards, Aditya
Hello, It happened again, so I am reopening this one: https://autobuilder.yoctoproject.org/valkyrie/#/builders/110/builds/747/steps/14/logs/stdio Failed ptests: {'gawk': ['randtest']} qemux86-64-musl-ptest alma8-vk-1 I don't see any helpful traces in the log file [1], though. [1] https://valkyrie.yocto.io/pub/non-release/20260721-60/testresults/qemux86-64-musl-ptest/gawk.log
Aditya, Any thoughts? It took 6 weeks to happen again, not a week or two. I haven't looked at the logs so please let us know if it's the same root cause and what you think should be done. Thanks.
Hi Randy, Thanks for the update. It is unfortunate to see that the failure has reoccurred after several weeks. The previous fix appears to have significantly reduced the occurrence rate, but it clearly did not eliminate the issue entirely. At the moment, I am occupied with other higher-priority work and will not be able to dedicate the time needed to properly investigate and debug this new occurrence. Because of that, I plan to unassign myself from this bug so that others can take it forward if needed. Based on the investigation I performed earlier, my understanding remains that randtest is particularly sensitive to statistical variations and has shown intermittent behavior across environments. Given that the test is still occasionally causing autobuilder failures even after the upstream fix was merged, one possible approach may be to skip or disable randtest in the Yocto ptest environment, similar to how a few other unstable gawk ptests are already handled. This would prevent a single intermittent test from affecting overall ptest results. However, I have not investigated this latest occurrence, so I cannot confirm whether it is the same root cause as before. Therefore, this should be considered a suggestion rather than a conclusion. Thanks to everyone for the guidance and reviews during the work on this issue. Regards, Aditya
Aditya thanks for your work and reply. Himani, please look at the failure and think about whether this test even makes sense in the YP AB context of overloaded systems.
Hi randy, I reviewed the latest failure (qemux86-64-musl-ptest, 2026-07-21). The log shows the same root cause: Tau= 162, Autocorr= 5.15181 sigma Test failed. FAIL: randtest This is the same class of statistical threshold failure as the original report (Tau=305, -5.2484 sigma on qemuarm64). The upstream fix merged in June (ccabae3036a7) reduced the frequency but did not eliminate it — it reoccurred after 6 weeks on a different architecture (x86-64) and libc (musl). I agree with your assessment that this test does not make sense in the YP autobuilder context. The randtest performs autocorrelation analysis on random number sequences with a tight sigma threshold, which is inherently sensitive to CPU scheduling variability and system load — conditions that are unavoidable on shared overloaded CI systems. I have submitted a patch to skip randtest in the ptest suite, following the same pattern already used for 'time' and 'timeout' tests (bug 14371). The patch was build-tested locally — confirmed that randtest is removed from the ptest image and properly listed in skipped.txt. Patch sent to oe-core mailing list: Message-ID: 20260727091430.81858-1-HimaniRamesh.Barde@windriver.com Thanks, Himani
Commit merged, so we can close this :) https://git.openembedded.org/openembedded-core/commit/?id=5266eed8f8c2096462da720093aa1556df726098
qemux86-64-ptest ubuntu2604-vk-4 wrynose completed at 2026-08-27 08:52:22+00:00 https://autobuilder.yoctoproject.org/valkyrie/?#/builders/73/builds/4414 https://valkyrie.yocto.io/pub/non-release/20260827-51/testresults/qemux86-64-ptest/core-image-ptest-gawk/log.do_testimage.3555993.20260827083852 | Tau= 171, Autocorr= 5.17177 sigma | Test failed. | FAIL: randtest I need to evaluate whether I can backport the fix master->wrynose.