Bug 14259 - AB-INT: NFS mounting issue on qemu oe-selftest glibc.GlibcSelfTestSystemEmulated.test_glibc
Summary: AB-INT: NFS mounting issue on qemu oe-selftest glibc.GlibcSelfTestSystemEmula...
Status: ACCEPTED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: core (show other bugs)
Version: 3.3
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 5.0.23
Assignee: Hemanth Kumar M D
QA Contact:
URL:
Whiteboard: AB-INT
Depends on:
Blocks:
 
Reported: 2021-02-26 20:06 UTC by Leonardo Sandoval
Modified: 2026-11-10 15:12 UTC (History)
14 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Leonardo Sandoval 2021-02-26 20:06:52 UTC
Full log at https://autobuilder.yoctoproject.org/typhoon/#/builders/59/builds/3111/steps/19/logs/stdio

2021-02-26 11:55:57,797 - oe-selftest - INFO - ERROR: glibc.GlibcSelfTestSystemEmulated.test_glibc (subunit.RemotedTestCase)
2021-02-26 11:55:57,797 - oe-selftest - INFO - ----------------------------------------------------------------------
2021-02-26 11:55:57,797 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/qemux86/build/meta/lib/oeqa/selftest/cases/glibc.py", line 88, in test_glibc
    self.run_check_emulated()
  File "/home/pokybuild/yocto-worker/qemux86/build/meta/lib/oeqa/selftest/cases/glibc.py", line 76, in run_check_emulated
    raise Exception("Failed to setup NFS mount on target ({})".format(repr(output)))
Exception: Failed to setup NFS mount on target ('mount: mounting 192.168.7.9:/home/pokybuild/yocto-worker/qemux86/build/build-st-33972/tmp/work on /home/pokybuild/yocto-worker/qemux86/build/build-st-33972/tmp/work failed: Input/output error')
Comment 1 Randy MacLeod 2021-10-21 15:01:54 UTC
This may have been due to the RCU stalls that were fixed in June. Re-open if it happens again.
Comment 2 Alexandre Belloni 2022-03-11 16:17:16 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/59/builds/4836/steps/19/logs/stdio
qemux86 ubuntu2004-ty-1 1646942555
Comment 3 Alexandre Belloni 2022-06-09 13:06:57 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/73/builds/5127/steps/20/logs/stdio
qemux86-64 centos7-ty-4 non SSD
Comment 4 Alexandre Belloni 2022-08-11 14:47:23 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/59/builds/5523/steps/19/logs/stdio

qemux86 ubuntu2204-ty-3
Comment 5 Alexandre Belloni 2022-09-01 15:43:12 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/73/builds/5747/steps/18/logs/stdio

qemux86-64 debian11-ty-3
Comment 6 Randy MacLeod 2022-10-13 14:26:24 UTC
Naveen, Can we add anything to the glibc test to help us debug this should it happen again? Perhaps you should write out what you know about this bug here and that will help you or someone reading your analysis.
Comment 7 Yash Shinde 2022-10-17 12:10:29 UTC
(In reply to comment #6)
> Naveen, Can we add anything to the glibc test to help us debug this should
> it happen again? Perhaps you should write out what you know about this bug
> here and that will help you or someone reading your analysis.

Naveen and I tried many times to reproduce the issue as given in the description but were unable to do so. It occurs very rarely as reported in upstream as well.
Comment 8 Randy MacLeod 2022-10-17 13:24:11 UTC
Yash, Thanks for the comment on testing. Can we add anything to the glibc test to help us debug this should it happen again?
Comment 9 Yash Shinde 2022-10-18 07:02:53 UTC
(In reply to comment #8)
> Yash, Thanks for the comment on testing. Can we add anything to the glibc
> test to help us debug this should it happen again?

We haven't checked on that yet. Will need to check the same.
Comment 10 Alexandre Belloni 2022-12-22 22:06:23 UTC
https://autobuilder.yoctoproject.org/typhoon/#builders/59/builds/6335/steps/19/logs/stdio

qemux86 debian11-ty-2
Comment 11 Alexandre Belloni 2023-02-16 16:34:10 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/59/builds/6547/steps/19/logs/stdio

qemux86 fedora37-ty-2
Comment 12 Randy MacLeod 2023-02-16 18:18:01 UTC
Change Yash's email to WR.

Yash, Naveen, 
Should I give this defect to Yash or will Naveen resume working on Yocto eventually?
Comment 13 Alexandre Belloni 2023-02-23 17:29:03 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/59/builds/6656/steps/18/logs/stdio
qemux86 fedora36-ty-4
Comment 14 Alexandre Belloni 2023-03-30 14:21:38 UTC
https://autobuilder.yoctoproject.org/typhoon/#/builders/59/builds/6838/steps/19/logs/stdio

qemux86 ubuntu2204-ty-1

This one is a tiemout:

2023-03-29 12:18:55,055 - oe-selftest - INFO - ======================================================================
2023-03-29 12:18:55,055 - oe-selftest - INFO - ERROR: glibc.GlibcSelfTestSystemEmulated.test_glibc (subunit.RemotedTestCase)
2023-03-29 12:18:55,056 - oe-selftest - INFO - ----------------------------------------------------------------------
2023-03-29 12:18:55,056 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/pokybuild/yocto-worker/qemux86/build/meta/lib/oeqa/selftest/cases/glibc.py", line 93, in test_glibc
    self.run_check_emulated()
  File "/home/pokybuild/yocto-worker/qemux86/build/meta/lib/oeqa/selftest/cases/glibc.py", line 80, in run_check_emulated
    raise Exception("Failed to setup NFS mount on target ({})".format(repr(output)))
Exception: Failed to setup NFS mount on target ('mount: mounting 192.168.7.15:/home/pokybuild/yocto-worker/qemux86/build/build-st-2685844/tmp/work on /home/pokybuild/yocto-worker/qemux86/build/build-st-2685844/tmp/work failed: Connection timed out')
Comment 15 Randy MacLeod 2023-04-13 15:47:34 UTC
I don't think that Yash will get to this anytime soon so moving to unassigned.
Comment 16 Randy MacLeod 2023-10-30 15:37:25 UTC
Build move to 5.0 -- ../Randy
Comment 17 Randy MacLeod 2024-04-24 20:30:27 UTC
Bulk move from 5.0* to 5.99 to indicate that there isn't any one available to work on this issue.
Comment 18 Mathieu Dubois-Briand 2024-10-16 15:43:49 UTC
qemux86-64-tc rocky9-vk-3
https://valkyrie.yoctoproject.org/#/builders/66/builds/178/steps/12/logs/stdio
Comment 19 Mathieu Dubois-Briand 2025-03-17 15:33:00 UTC
qemux86-tc fedora40-vk-2 mathieu/master-next completed at 2025-03-13T21:01:21Z
https://autobuilder.yoctoproject.org/valkyrie/#/builders/28/builds/1155/steps/12/logs/stdio
Comment 20 Mathieu Dubois-Briand 2025-03-24 09:28:52 UTC
qemux86-64-tc ubuntu2004-vk-1 master-next completed at 2025-03-20T15:21:28Z
https://autobuilder.yoctoproject.org/valkyrie/#/builders/66/builds/1210/steps/13/logs/stdio
Comment 21 Mathieu Dubois-Briand 2025-03-26 17:26:59 UTC
qemux86-tc debian12-vk-4 master-next completed at 2025-03-24T12:19:24Z
https://autobuilder.yoctoproject.org/valkyrie/#/builders/28/builds/1220/steps/12/logs/stdio
Comment 22 Mathieu Dubois-Briand 2025-05-05 08:52:48 UTC
qemux86-64-tc ubuntu2404-vk-3 master-next completed at 2025-05-02T18:38:21Z
https://autobuilder.yoctoproject.org/valkyrie/#/builders/66/builds/1474/steps/12/logs/stdio
Comment 24 Mathieu Dubois-Briand 2025-09-02 06:45:45 UTC
qemux86-tc ubuntu2204-vk-2 master completed at 2025-09-02 02:59:36+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/28/builds/2218/steps/12/logs/stdio
Comment 25 Randy MacLeod 2025-09-04 14:50:07 UTC
This is still happening so Sundeep will have someone work on it again in the next month or two.
Comment 26 Randy MacLeod 2025-09-11 15:07:53 UTC
Sundeep, can you have a junior member of the team start to look at this since it happens ~10 times a year and is distracting.
Comment 27 Sundeep Kokkonda 2025-10-14 14:07:39 UTC
Sure Randy. We are looking into it, will post the update.
Comment 28 Hemanth Kumar 2025-11-05 05:42:29 UTC
We conducted multiple rounds of testing across different host environments, including Ubuntu, Debian, and Fedora. However, the NFS mount failure could not be reproduced in any of our local test setups. Is there any specific config causing this issue?
Comment 29 Randy MacLeod 2025-11-05 17:13:05 UTC
Leonardo please help Hemanth out.
Comment 30 Randy MacLeod 2025-11-06 15:56:47 UTC
Leonardo is not longer working on YP.
Comment 31 Paul Barker 2026-02-12 16:41:45 UTC
Closing this as obsolete as it hasn't been seen in a while and we've fixed at least one issue in unfs3.

Related bugs:
- #16029
- #16113

We can re-open if this reoccurs.
Comment 32 Yoann Congal 2026-02-16 13:19:04 UTC
scarthgap qemux86-64-tc debian12-vk-1 https://autobuilder.yoctoproject.org/valkyrie/?#/builders/66/builds/3219
> Exception: Failed to setup NFS mount on target ('mount: mounting 192.168.7.1:/srv/pokybuild/yocto-worker/qemux86-64-tc/build/build-st-2594343/tmp/work on /srv/pokybuild/yocto-worker/qemux86-64-tc/build/build-st-2594343/tmp/work failed: Connection refused')

That was during the 5.0.16rc1 release build.

Reopening :(
Comment 33 Paul Barker 2026-02-19 15:53:35 UTC
Perhaps we need to backport e51d5e19cb1b ("unfs3: Fix race issue causing a glibc test hang")
Comment 34 Randy MacLeod 2026-02-19 15:55:36 UTC
Yoann to backport as a first idea. Hopefully that's all that is needed.
Comment 35 Randy MacLeod 2026-03-31 14:49:37 UTC
Yoann, 

Sundeep would like take over this bug unless you have it underway.
He would backport the unfs3 patch (e51d5e19cb1b ) 
and in a separate commit, add some debug logs.

Okay?
Comment 36 Yoann Congal 2026-03-31 15:21:22 UTC
(In reply to Randy MacLeod from comment #35)
> Yoann, 
> 
> Sundeep would like take over this bug unless you have it underway.
> He would backport the unfs3 patch (e51d5e19cb1b ) 
> and in a separate commit, add some debug logs.
> 
> Okay?

Okay! I've not started anything with this one, so help is very welcome.
I've assigned the bug to sundeep.kokkonda@windriver.com.
Comment 37 Randy MacLeod 2026-04-21 19:30:18 UTC
Bumped to latest 5.0 release.
Sundeep, please assign to someone ASAP.
Comment 38 Paul Barker 2026-05-13 12:41:07 UTC
This hasn't been seen on master since commit e51d5e19cb1b ("unfs3: Fix race issue causing a glibc test hang"), or on stable since it was backported. Time to close it and re-open if the issue reoccurs?
Comment 39 Paul Barker 2026-05-14 15:13:07 UTC
Patch merged and backported.
Comment 40 Yoann Congal 2026-05-21 08:16:33 UTC
Note: On scarthgap it's commit 34f34512e5ee ("unfs3: Fix race issue causing a glibc test hang")
Comment 41 Yoann Congal 2026-10-03 21:26:11 UTC
It happened again, reopening:

qemux86-tc debian12-vk-9 scarthgap completed at 2026-10-02 17:09:37+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/28/builds/4745

Exception: Failed to setup NFS mount on target ('mount: mounting 192.168.7.9:/srv/pokybuild/yocto-worker/qemux86-tc/build/build-st-2702238/tmp/work on /srv/pokybuild/yocto-worker/qemux86-tc/build/build-st-2702238/tmp/work failed: Connection refused')
2026-10-02 17:09:30,187 - oe-selftest - INFO - ERROR: glibc.GlibcSelfTestSystemEmulated.test_glibc (subunit.RemotedTestCase)
Exception: Failed to setup NFS mount on target ('mount: mounting 192.168.7.9:/srv/pokybuild/yocto-worker/qemux86-tc/build/build-st-2702238/tmp/work on /srv/pokybuild/yocto-worker/qemux86-tc/build/build-st-2702238/tmp/work failed: Connection refused')
2026-10-02 17:09:30,187 - oe-selftest - INFO - FAILED
2026-10-02 17:09:34,793 - oe-selftest - INFO - RESULTS - glibc.GlibcSelfTestSystemEmulated.test_glibc: ERROR (2719.21s)
2026-10-02 17:09:36,106 - oe-selftest - INFO - oe-selftest - FAIL - Required tests failed (successes=16, skipped=1, failures=0, errors=1)
ERROR: Command . ./oe-init-build-env; DISPLAY=:1 oe-selftest -a -t machine -t toolchain-system -j 15 failed with exit code 1, see errors above. (1790960976.2: 3716.1)
Comment 42 Yoann Congal 2026-11-10 15:12:30 UTC
qemux86-tc debian12-vk-7 scarthgap completed at 2026-11-10 11:46:39+00:00
https://autobuilder.yoctoproject.org/valkyrie/#/builders/28/builds/4966

Exception: Failed to setup NFS mount on target ('mount: mounting 192.168.7.13:/srv/pokybuild/yocto-worker/qemux86-tc/build/build-st-2324850/tmp/work on /srv/pokybuild/yocto-worker/qemux86-tc/build/build-st-2324850/tmp/work failed: Connection refused')
2026-11-10 11:46:29,968 - oe-selftest - INFO - ERROR: glibc.GlibcSelfTestSystemEmulated.test_glibc (subunit.RemotedTestCase)