| Summary: | AB-INT: NFS mounting issue on qemu oe-selftest glibc.GlibcSelfTestSystemEmulated.test_glibc | ||
|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] OE-Core | Reporter: | Leonardo Sandoval <leonardo.sandoval> |
| Component: | core | Assignee: | Hemanth Kumar M D <Hemanth.KumarMD> |
| Status: | ACCEPTED --- | QA Contact: | |
| Severity: | normal | ||
| Priority: | Medium+ | CC: | alexandre.belloni, hemanth.250302, mathieu.dubois-briand, meta.mr.watcher, meta.watcher, paul, pavel, pgowda.cve, randy.macleod, richard.purdie, steve, sundeep.kokkonda, Yash.Shinde, yoann.congal |
| Version: | 3.3 | ||
| Target Milestone: | 5.0.23 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | AB-INT | ||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |
|
Description
Leonardo Sandoval
2021-02-26 20:06:52 UTC
This may have been due to the RCU stalls that were fixed in June. Re-open if it happens again. https://autobuilder.yoctoproject.org/typhoon/#/builders/59/builds/4836/steps/19/logs/stdio qemux86 ubuntu2004-ty-1 1646942555 https://autobuilder.yoctoproject.org/typhoon/#/builders/73/builds/5127/steps/20/logs/stdio qemux86-64 centos7-ty-4 non SSD https://autobuilder.yoctoproject.org/typhoon/#/builders/59/builds/5523/steps/19/logs/stdio qemux86 ubuntu2204-ty-3 https://autobuilder.yoctoproject.org/typhoon/#/builders/73/builds/5747/steps/18/logs/stdio qemux86-64 debian11-ty-3 Naveen, Can we add anything to the glibc test to help us debug this should it happen again? Perhaps you should write out what you know about this bug here and that will help you or someone reading your analysis. (In reply to comment #6) > Naveen, Can we add anything to the glibc test to help us debug this should > it happen again? Perhaps you should write out what you know about this bug > here and that will help you or someone reading your analysis. Naveen and I tried many times to reproduce the issue as given in the description but were unable to do so. It occurs very rarely as reported in upstream as well. Yash, Thanks for the comment on testing. Can we add anything to the glibc test to help us debug this should it happen again? (In reply to comment #8) > Yash, Thanks for the comment on testing. Can we add anything to the glibc > test to help us debug this should it happen again? We haven't checked on that yet. Will need to check the same. https://autobuilder.yoctoproject.org/typhoon/#builders/59/builds/6335/steps/19/logs/stdio qemux86 debian11-ty-2 https://autobuilder.yoctoproject.org/typhoon/#/builders/59/builds/6547/steps/19/logs/stdio qemux86 fedora37-ty-2 Change Yash's email to WR. Yash, Naveen, Should I give this defect to Yash or will Naveen resume working on Yocto eventually? https://autobuilder.yoctoproject.org/typhoon/#/builders/59/builds/6656/steps/18/logs/stdio qemux86 fedora36-ty-4 https://autobuilder.yoctoproject.org/typhoon/#/builders/59/builds/6838/steps/19/logs/stdio qemux86 ubuntu2204-ty-1 This one is a tiemout: 2023-03-29 12:18:55,055 - oe-selftest - INFO - ====================================================================== 2023-03-29 12:18:55,055 - oe-selftest - INFO - ERROR: glibc.GlibcSelfTestSystemEmulated.test_glibc (subunit.RemotedTestCase) 2023-03-29 12:18:55,056 - oe-selftest - INFO - ---------------------------------------------------------------------- 2023-03-29 12:18:55,056 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): File "/home/pokybuild/yocto-worker/qemux86/build/meta/lib/oeqa/selftest/cases/glibc.py", line 93, in test_glibc self.run_check_emulated() File "/home/pokybuild/yocto-worker/qemux86/build/meta/lib/oeqa/selftest/cases/glibc.py", line 80, in run_check_emulated raise Exception("Failed to setup NFS mount on target ({})".format(repr(output))) Exception: Failed to setup NFS mount on target ('mount: mounting 192.168.7.15:/home/pokybuild/yocto-worker/qemux86/build/build-st-2685844/tmp/work on /home/pokybuild/yocto-worker/qemux86/build/build-st-2685844/tmp/work failed: Connection timed out') I don't think that Yash will get to this anytime soon so moving to unassigned. Build move to 5.0 -- ../Randy Bulk move from 5.0* to 5.99 to indicate that there isn't any one available to work on this issue. qemux86-64-tc rocky9-vk-3 https://valkyrie.yoctoproject.org/#/builders/66/builds/178/steps/12/logs/stdio qemux86-tc fedora40-vk-2 mathieu/master-next completed at 2025-03-13T21:01:21Z https://autobuilder.yoctoproject.org/valkyrie/#/builders/28/builds/1155/steps/12/logs/stdio qemux86-64-tc ubuntu2004-vk-1 master-next completed at 2025-03-20T15:21:28Z https://autobuilder.yoctoproject.org/valkyrie/#/builders/66/builds/1210/steps/13/logs/stdio qemux86-tc debian12-vk-4 master-next completed at 2025-03-24T12:19:24Z https://autobuilder.yoctoproject.org/valkyrie/#/builders/28/builds/1220/steps/12/logs/stdio qemux86-64-tc ubuntu2404-vk-3 master-next completed at 2025-05-02T18:38:21Z https://autobuilder.yoctoproject.org/valkyrie/#/builders/66/builds/1474/steps/12/logs/stdio scarthgap qemux86-tc fedora40-vk-1 https://autobuilder.yoctoproject.org/valkyrie/?#/builders/28/builds/2195/steps/12/logs/stdio qemux86-64-tc debian12-vk-3 https://autobuilder.yoctoproject.org/valkyrie/?#/builders/66/builds/2197/steps/12/logs/stdio qemux86-tc ubuntu2204-vk-2 master completed at 2025-09-02 02:59:36+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/28/builds/2218/steps/12/logs/stdio This is still happening so Sundeep will have someone work on it again in the next month or two. Sundeep, can you have a junior member of the team start to look at this since it happens ~10 times a year and is distracting. Sure Randy. We are looking into it, will post the update. We conducted multiple rounds of testing across different host environments, including Ubuntu, Debian, and Fedora. However, the NFS mount failure could not be reproduced in any of our local test setups. Is there any specific config causing this issue? Leonardo please help Hemanth out. Leonardo is not longer working on YP. Closing this as obsolete as it hasn't been seen in a while and we've fixed at least one issue in unfs3. Related bugs: - #16029 - #16113 We can re-open if this reoccurs. scarthgap qemux86-64-tc debian12-vk-1 https://autobuilder.yoctoproject.org/valkyrie/?#/builders/66/builds/3219 > Exception: Failed to setup NFS mount on target ('mount: mounting 192.168.7.1:/srv/pokybuild/yocto-worker/qemux86-64-tc/build/build-st-2594343/tmp/work on /srv/pokybuild/yocto-worker/qemux86-64-tc/build/build-st-2594343/tmp/work failed: Connection refused') That was during the 5.0.16rc1 release build. Reopening :( Perhaps we need to backport e51d5e19cb1b ("unfs3: Fix race issue causing a glibc test hang")
Yoann to backport as a first idea. Hopefully that's all that is needed. Yoann, Sundeep would like take over this bug unless you have it underway. He would backport the unfs3 patch (e51d5e19cb1b ) and in a separate commit, add some debug logs. Okay? (In reply to Randy MacLeod from comment #35) > Yoann, > > Sundeep would like take over this bug unless you have it underway. > He would backport the unfs3 patch (e51d5e19cb1b ) > and in a separate commit, add some debug logs. > > Okay? Okay! I've not started anything with this one, so help is very welcome. I've assigned the bug to sundeep.kokkonda@windriver.com. Bumped to latest 5.0 release. Sundeep, please assign to someone ASAP. This hasn't been seen on master since commit e51d5e19cb1b ("unfs3: Fix race issue causing a glibc test hang"), or on stable since it was backported. Time to close it and re-open if the issue reoccurs?
Patch merged and backported. Note: On scarthgap it's commit 34f34512e5ee ("unfs3: Fix race issue causing a glibc test hang")
It happened again, reopening: qemux86-tc debian12-vk-9 scarthgap completed at 2026-10-02 17:09:37+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/28/builds/4745 Exception: Failed to setup NFS mount on target ('mount: mounting 192.168.7.9:/srv/pokybuild/yocto-worker/qemux86-tc/build/build-st-2702238/tmp/work on /srv/pokybuild/yocto-worker/qemux86-tc/build/build-st-2702238/tmp/work failed: Connection refused') 2026-10-02 17:09:30,187 - oe-selftest - INFO - ERROR: glibc.GlibcSelfTestSystemEmulated.test_glibc (subunit.RemotedTestCase) Exception: Failed to setup NFS mount on target ('mount: mounting 192.168.7.9:/srv/pokybuild/yocto-worker/qemux86-tc/build/build-st-2702238/tmp/work on /srv/pokybuild/yocto-worker/qemux86-tc/build/build-st-2702238/tmp/work failed: Connection refused') 2026-10-02 17:09:30,187 - oe-selftest - INFO - FAILED 2026-10-02 17:09:34,793 - oe-selftest - INFO - RESULTS - glibc.GlibcSelfTestSystemEmulated.test_glibc: ERROR (2719.21s) 2026-10-02 17:09:36,106 - oe-selftest - INFO - oe-selftest - FAIL - Required tests failed (successes=16, skipped=1, failures=0, errors=1) ERROR: Command . ./oe-init-build-env; DISPLAY=:1 oe-selftest -a -t machine -t toolchain-system -j 15 failed with exit code 1, see errors above. (1790960976.2: 3716.1) qemux86-tc debian12-vk-7 scarthgap completed at 2026-11-10 11:46:39+00:00 https://autobuilder.yoctoproject.org/valkyrie/#/builders/28/builds/4966 Exception: Failed to setup NFS mount on target ('mount: mounting 192.168.7.13:/srv/pokybuild/yocto-worker/qemux86-tc/build/build-st-2324850/tmp/work on /srv/pokybuild/yocto-worker/qemux86-tc/build/build-st-2324850/tmp/work failed: Connection refused') 2026-11-10 11:46:29,968 - oe-selftest - INFO - ERROR: glibc.GlibcSelfTestSystemEmulated.test_glibc (subunit.RemotedTestCase) |