Bug 16254

I would like to work on this bug.

I have analyzed and worked on this issue.

The randtest test is part of the gawk ptest suite and is
executed through the Maketests framework. While it passes
consistently in my local environment, it relies on deterministic
random number sequences that are not guaranteed across
different architectures, libc implementations, and runtime
conditions.

This explains the intermittent failures (AB-INT) observed in
the Yocto autobuilder.

I initially attempted to disable the test using
PTEST_DISABLED_TESTS, but found that gawk’s test framework
does not honor this variable.

The fix removes randtest during do_install_ptest() and marks
it as skipped, following the same pattern used for other
nondeterministic tests such as time and timeout.

After applying the fix:
- randtest is reported as SKIP
- no failures occur
- ptest results are stable

Created attachment 5216 [details]
Patch to disable randtest in gawk ptest due to nondeterministic behavior

Attaching patch to disable randtest in gawk ptest.

randtest relies on deterministic random number sequences which
are not guaranteed across different environments, leading to
intermittent failures (AB-INT).

The fix removes randtest during do_install_ptest() and marks
it as skipped, following the same approach used for other
unstable tests.

Tested locally and confirmed randtest is skipped and
ptest results are stable.

Hi Aditya,

Thanks for working on this bug.

It is tempting to skip the test and we may eventually end up doing that but
it might be better to identify the source of the non-deterministic
randomness in our environment and fix that. Can you explain how you 
came to this conclusion ? Could we add code to the test to dump out
what part of the test is failing to learn more about the problem ?

I haven't even looked at the failing code yet so let me know if 
I should take the time do to that to be more effective at helping review
the issue. I have however worked in the area of random number generation
and entropy pool depth for yocto a bit so I do have some experience in this
area! ;-) Also this issue happened once on qemuarm64, an emulated target that
we have extensive experience with so I'm especially suspicious about the
claimed incorrect behaviour of random numbers. Adding Ross from Arm as a
sanity check!

Also, now or once we finish this discussion, are you able to send the
patch to the oe-core email list  as documented in:
  https://docs.yoctoproject.org/contributor-guide/index.html ?
This how we get more people to examine and hopefully review and reply to
the patch,  test it on the Yocto Autobuilders and get changes merged.

Thanks again!

Hi,

Thank you for the detailed feedback.

My initial conclusion about nondeterministic behavior was based on the nature of the randtest test itself, which compares expected sequences of random numbers. Since such sequences can depend on libc implementations, floating point behavior, and execution environment, I assumed this could lead to intermittent failures as seen in the autobuilder logs.

In my local testing (on qemux86-64), randtest consistently passed, which aligns with stable CPU timing and consistent libc behavior. However, I understand your point that this might indicate an underlying issue rather than the test itself.

I agree that it would be better to investigate further before skipping the test.

As a next step, I can:
- Run the randtest script directly and try to identify which part of the output diverges
- Add debugging output to capture the failing conditions

Please let me know if you would recommend a specific approach for instrumenting the test or capturing the failure more effectively.

Thanks again for your guidance!

Also, if you want me to send this patch now to the oe-core email list  as documented in:
  https://docs.yoctoproject.org/contributor-guide/index.html ? I can do it or if its ok I will send it after we finish our discussion on this.

Hi,

Thanks for the suggestion.

I investigated randtest further by enabling debug output of the autocorrelation values and running the test multiple times through the ptest framework.

In my observations:
- The test normally produces small correlation values close to zero (as expected for random sequences)
- However, in some runs, the condition:
    abs(corr[tau] / sigma) > max_allowed_sigma
  is triggered, leading to a failure
- I was able to reproduce a failing run where only randtest fails while all other tests pass

This indicates that the failure is caused by sensitivity of the statistical threshold rather than a clear functional issue in rand().

The behavior appears to depend on minor variations in floating point calculations and runtime conditions (timing, entropy state, etc.), which aligns with the intermittent nature observed in autobuilder results.

Given that:
- The underlying randomness appears valid
- The failure depends on subtle environmental factors
- Similar tests (time/timeout) are already skipped for non-determinism

it seems reasonable to consider randtest as inherently unstable in these environments.

Please let me know your thoughts on whether we should proceed with skipping it or further adjust the test behavior.

Thanks!

Hi, 

Good analysis! 

I'm tempted to look at the statistical tests but I'm buys and so I won't. :-(

if you add a summary of your analysis to your commit log
I suspect the patch will be merged. For bonus points,
please report the issue to the upstream gawk devs by email
or a bug tracking system and include a link to that in
the commit log.

Hmmm..
Rather than skipping the test, I wonder if upstream would accept
a more generous limit.

I guess the best approach is to skip for now and 
talk with upstream in parallel.

Thanks!

Hi,

Thanks for the feedback.

I have updated the commit message to include a summary of the debugging
analysis, describing how the failure was reproduced and the conditions
under which it occurs.

I also plan to report this behavior to the upstream gawk developers and
will include a link to that discussion in the patch once available.

Thanks again for your guidance — this was very helpful.

Created attachment 5217 [details]
Updated patch with detailed analysis of randtest intermittent failure and root cause

Reported the issue to upstream gawk developers.

Upstream clarified that the preferred approach would be to improve the
randtest itself rather than disabling it in Yocto. They suggested
proposing a change to test/randtest.sh.

Currently, I am newcomer to yocto so I do not yet have
sufficient confidence in the statistical behavior of the test to propose
a safe upstream modification without affecting its validation criteria.

However, based on investigation, the intermittent failures appear to be
caused by sensitivity in the autocorrelation threshold condition:

    abs(corr[tau] / sigma) > max_allowed_sigma

I will continue to investigate potential improvements to make the test
less sensitive to minor environmental variations, and will follow up
with an upstream proposal if I arrive at a well-justified approach.

In the meantime, proceeding with the Yocto-side workaround to avoid
intermittent failures.

Hi,

I wanted to follow up on my patch,
Kindly let me know if there are any feedback or changes required from
my side, and whether there is an expected timeline for the review or
merge of this patch.

Thanks for your time and guidance.

Regards,
Aditya GS

Created attachment 5221 [details]
Patch for randtest.sh

Based on upstream feedback, I attempted to improve the randtest
itself rather than disabling it in Yocto.

The issue appears to be caused by sensitivity to small statistical
fluctuations, where even a single autocorrelation value exceeding
the sigma threshold results in intermittent failures.

I implemented a change to allow a small number of exceedances before
marking the test as failed. This reduces flaky behavior while preserving
the intent of detecting statistically significant issues.

Please find the patch attached for reference. The upstream submission
has also been sent to bug-gawk@gnu.org for review.

Looking forward to feedback.

Aditya,

I think your approach makes sense and I'm glad to see that you've sent
your change upstream to the gawk developers.

Can you submit a patch to the oe-core list ( using git send-email as Yoann
suggested ) ?

The full contributor guide is here:
   https://docs.yoctoproject.org/dev/contributor-guide/index.html
Occasionally people have trouble with git send-email and their emial system
so if you're in that situation, you might be able to try using 'b4'. 
It's a new tool and I suspect that > 98% of contributions are still via
git send-email so try that first.

If you're stuck, see below and/or ask here or on the oe-core email list
for guidance about using b4. We really should add that to the docs...
❯ git log .b4-config
commit db5b855f175820918d6643439ccbfe3a70119279
Author: Quentin Schulz <quentin.schulz@cherry.de>
Date:   Wed Feb 5 11:16:00 2025

    b4-config: Add basic b4 config file

    b4[1] is a very nice tool for mail-based contribution. A config[2] file
    exists to set up a few defaults. We can use it to set the Cc recipients
    to always add, in our case the mailing list.

    Because we do not have anything to check for now, disable needs-checking
    so patches can be sent without running b4 prep --check. The
    mid-term/long-term plan is to use patchtest locally.

    Because we do not have any auto-to-cc support (and the implicit one
    using scripts/get_maintainer.pl cannot work for us), also disable
    needs-auto-to-cc so patches can be sent without running b4 prep
    --auto-to-cc.

    [1] https://pypi.org/project/b4/
    [2] https://b4.docs.kernel.org/en/latest/config.html

    Signed-off-by: Quentin Schulz <quentin.schulz@cherry.de>
    Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

Hi,

Thanks for the guidance and for pointing me to the contributor guide.

Currently, I am facing some issues with git send-email on my office laptop.
I plan to switch to my personal laptop to proceed with sending the patch,
but I first need to set up Ubuntu on that system, so it might take a bit
of time.

In the meantime, I will continue working on updating the patch based on
the suggestions from the gawk upstream maintainers. Once that is ready,
I will submit the patch to the oe-core mailing list as well.

I will first try using git send-email, and if I encounter issues there,
I will try using the b4 tool as suggested.

Thanks again for the guidance.

Regards,  
Aditya

Based on upstream feedback, I updated the approach for handling the
randtest instability.

Instead of allowing a small number of statistical outliers, the test
now detects cases where the computed sigma is too small (indicating
unstable normalization) and retries the computation from scratch.

This ensures that the test remains strict while avoiding false failures
caused by edge cases in statistical fluctuation. After implementing this
change and testing across multiple runs, the behavior is stable with no
intermittent failures observed.

The updated patch has been sent upstream to the gawk maintainers for
review. Once the upstream version is finalized, I will follow up with
submission to oe-core as well.

Looking forward to feedback.

Created attachment 5222 [details]
Updated patch, created as suggested by gawk upstream

I have sent a patch to the OpenEmbedded-Core mailing list to address this issue.
The patch implements a more robust handling of unstable sigma values in randtest, retrying computations instead of tolerating statistical outliers, as discussed.
I have included the relevant maintainers and contributors in CC for visibility and review:

joaomarcos.costa@bootlin.com
ross.burton@arm.com
randy.macleod@windriver.com
richard.purdie@linuxfoundation.org

Please let me know if there are any comments or required revisions.

Update:

The issue has now been addressed upstream in gawk.

The proposed fix to randtest.sh was reviewed by the upstream maintainer
and has been integrated into the gawk repository, with the changes
properly incorporated into the original file and test/ChangeLog.

Since the root cause has been resolved upstream, this eliminates the
need for a workaround (such as skipping the test) on the Yocto side.

Next step would be to backport the upstream fix into oe-core once the
corresponding commit is available.

Please let me know if you would like me to prepare a backport patch.

Thanks,
Aditya

I am currently working on preparing a backport patch for oe-core
based on the upstream changes, and am already in discussion with
oe-core maintainers regarding this.

The patch will be shared shortly for review.

Thanks,
Aditya

Created attachment 5223 [details]
Backport patch

The backport patch has been submitted to oe-core for review.

Thanks,
Aditya

Created attachment 5224 [details]
updated backport patch

Update:

Addressed review comments and submitted updated (v2) backport patch
to oe-core for review.

Thanks,
Aditya

Created attachment 5225 [details]
Updated backport patch

Update:

Addressed patchtest feedback by adding the missing Signed-off-by
to the backported patch file and updated the patch accordingly.
Resubmitted as v3 to oe-core.

Thanks,
Aditya

Created attachment 5226 [details]
Updated backport patch

Update:

Reworked the patch based on review feedback to target the master branch
as per OE-Core backport policy.

Addressed the following:
- Updated patch to apply against gawk 5.4.0 in master
- Fixed formatting issues (Upstream-Status, whitespace)
- Updated commit message to use [YOCTO #16254]
- Ensured proper integration in SRC_URI

The updated patch has been submitted as v4 to the oe-core mailing list
for review.

Thanks,
Aditya

Created attachment 5227 [details]
Updated backport patch

Update: 

Addressed patch test issue by ensuring the correct patch was submitted and fixing the Upstream-Status formatting in the backported patch file. Resent updated patch for review. 

Thanks, 
Aditya

Created attachment 5228 [details]
Updated backport patch

Update:

The issue has now been addressed and the fix has been accepted and merged
into OE-Core.

Upstream commit:
https://git.openembedded.org/openembedded-core/commit/?id=ccabae3036a713cac8b08731b765ff1dc3ed93eb

The fix resolves the intermittent failures in randtest.sh by improving
the handling of cases where sigma becomes too small, avoiding false
positives and ensuring stable test behavior.

My earlier submissions went through multiple review iterations, and the
maintainers finalized and merged an updated version of the patch to
resolve the CI instability more quickly.

I have reviewed the merged change and confirm that it addresses the root
cause discussed in this bug.

Thanks to the maintainers for reviewing and integrating the fix.

Thanks,
Aditya

Hi Randy,

Could you please confirm if it is appropriate to mark this bug as
RESOLVED/FIXED?

Thanks,
Aditya

Hi Aditya,

It's your call:

You could close it now or 
you can wait a week or two to see  if it really resolves 
the intermittent errors seen on the Yocto autobuilders.

If you resolve it as fixed, and it happens again, 
most likely it will be re-opened by someone who triages the YP AB issues
but they might open a new bug.

Flip a coin! ;-)

Thanks for working on this bug and getting what is hopefully a fix, upstream
into gawk.

../Randy

Hi Randy,

Thanks for the clarification.

Given that the fix has been merged and addresses the root cause,
I will mark this bug as RESOLVED/FIXED.

I’ll keep an eye on autobuilder results, and if the issue reappears,
it can be reopened or tracked separately as needed.

Thanks again for the guidance.

Regards,
Aditya

Hello,

It happened again, so I am reopening this one:

https://autobuilder.yoctoproject.org/valkyrie/#/builders/110/builds/747/steps/14/logs/stdio

Failed ptests:
{'gawk': ['randtest']}

qemux86-64-musl-ptest alma8-vk-1

I don't see any helpful traces in the log file [1], though.

[1] https://valkyrie.yocto.io/pub/non-release/20260721-60/testresults/qemux86-64-musl-ptest/gawk.log

Aditya, 

Any thoughts? It took 6 weeks to happen again, not a week or two.
I haven't looked at the logs so please let us know if it's the same root
cause and what you think should be done. Thanks.

Hi Randy,

Thanks for the update.

It is unfortunate to see that the failure has reoccurred after several weeks. The previous fix appears to have significantly reduced the occurrence rate, but it clearly did not eliminate the issue entirely.

At the moment, I am occupied with other higher-priority work and will not be able to dedicate the time needed to properly investigate and debug this new occurrence. Because of that, I plan to unassign myself from this bug so that others can take it forward if needed.

Based on the investigation I performed earlier, my understanding remains that randtest is particularly sensitive to statistical variations and has shown intermittent behavior across environments. Given that the test is still occasionally causing autobuilder failures even after the upstream fix was merged, one possible approach may be to skip or disable randtest in the Yocto ptest environment, similar to how a few other unstable gawk ptests are already handled. This would prevent a single intermittent test from affecting overall ptest results.

However, I have not investigated this latest occurrence, so I cannot confirm whether it is the same root cause as before. Therefore, this should be considered a suggestion rather than a conclusion.

Thanks to everyone for the guidance and reviews during the work on this issue.

Regards,
Aditya

Aditya thanks for your work and reply.

Himani, please look at the failure and think about whether this test even makes sense in the YP AB context of overloaded systems.

Hi randy,

I reviewed the latest failure (qemux86-64-musl-ptest, 2026-07-21). The log shows the same root cause:

Tau= 162, Autocorr= 5.15181 sigma Test failed. FAIL: randtest

This is the same class of statistical threshold failure as the original report (Tau=305, -5.2484 sigma on qemuarm64). The upstream fix merged in June (ccabae3036a7) reduced the frequency but did not eliminate it — it reoccurred after 6 weeks on a different architecture (x86-64) and libc (musl).

I agree with your assessment that this test does not make sense in the YP autobuilder context. The randtest performs autocorrelation analysis on random number sequences with a tight sigma threshold, which is inherently sensitive to CPU scheduling variability and system load — conditions that are unavoidable on shared overloaded CI systems.

I have submitted a patch to skip randtest in the ptest suite, following the same pattern already used for 'time' and 'timeout' tests (bug 14371). The patch was build-tested locally — confirmed that randtest is removed from the ptest image and properly listed in skipped.txt.

Patch sent to oe-core mailing list: Message-ID: 20260727091430.81858-1-HimaniRamesh.Barde@windriver.com

Thanks, 
Himani

Commit merged, so we can close this :)

https://git.openembedded.org/openembedded-core/commit/?id=5266eed8f8c2096462da720093aa1556df726098

qemux86-64-ptest ubuntu2604-vk-4 wrynose completed at 2026-08-27 08:52:22+00:00
https://autobuilder.yoctoproject.org/valkyrie/?#/builders/73/builds/4414

https://valkyrie.yocto.io/pub/non-release/20260827-51/testresults/qemux86-64-ptest/core-image-ptest-gawk/log.do_testimage.3555993.20260827083852
| Tau= 171, Autocorr= 5.17177 sigma
| Test failed.
| FAIL: randtest

I need to evaluate whether I can backport the fix master->wrynose.