13674 2019-12-04 13:06:51 +0000 master dnf failures on qemumips 2025-03-24 09:41:51 +0000 1 1 7 Build System, Metadata & Runtime OE-Core core 3.0 x86 Multiple RESOLVED FIXED AB-INT Medium+ normal 5.2 M4 1 richard.purdie unassigned hongxu.jia liezhi.yang mathieu.dubois-briand meta.mr.watcher meta.watcher paulg pavel randy.macleod ross.burton tgamblin yoann.congal --- --- No (bug/feature does not impact docs) oldest_to_newest 85697 0 richard.purdie 2019-12-04 13:06:51 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/60/builds/1299 NOTE: recipe core-image-sato-1.0-r0: task do_testsdk: Succeeded Started HTTPService on 192.168.7.7:45100 Started HTTPService on 192.168.7.3:41791 Traceback (most recent call last): File "/home/pokybuild/yocto-worker/qemumips/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemumips/build/meta/lib/oeqa/runtime/cases/dnf.py", line 95, in test_dnf_install_dependency self.dnf_with_repo('install -y run-postinsts-dev') File "/home/pokybuild/yocto-worker/qemumips/build/meta/lib/oeqa/runtime/cases/dnf.py", line 68, in dnf_with_repo output = self.dnf(" ".join(cmdlinerepoopts) + " --nogpgcheck " + command) File "/home/pokybuild/yocto-worker/qemumips/build/meta/lib/oeqa/runtime/cases/dnf.py", line 21, in dnf self.assertEqual(status, expected, message) AssertionError: 255 != 0 : dnf --repofrompath=oe-testimage-repo-qemumips,http://192.168.7.7:45100/qemumips --repofrompath=oe-testimage-repo-mips32r2,http://192.168.7.7:45100/mips32r2 --repofrompath=oe-testimage-repo-noarch,http://192.168.7.7:45100/noarch --nogpgcheck install -y run-postinsts-dev ssh: connect to host 192.168.7.8 port 22: No route to host Test case dnf.DnfRepoTest.test_dnf_install_from_disk depends on dnf.DnfRepoTest.test_dnf_install_dependency but it didn't pass/run. Test case dnf.DnfRepoTest.test_dnf_install_from_http depends on dnf.DnfRepoTest.test_dnf_install_from_disk but it didn't pass/run. Traceback (most recent call last): File "/home/pokybuild/yocto-worker/qemumips/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemumips/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemumips/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemumips/build/meta/lib/oeqa/runtime/cases/dnf.py", line 139, in test_dnf_installroot self.dnf_with_repo('install --installroot=%s -v -y --rpmverbosity=debug busybox run-postinsts' % rootpath) File "/home/pokybuild/yocto-worker/qemumips/build/meta/lib/oeqa/runtime/cases/dnf.py", line 68, in dnf_with_repo output = self.dnf(" ".join(cmdlinerepoopts) + " --nogpgcheck " + command) File "/home/pokybuild/yocto-worker/qemumips/build/meta/lib/oeqa/runtime/cases/dnf.py", line 21, in dnf self.assertEqual(status, expected, message) AssertionError: 1 != 0 : dnf --repofrompath=oe-testimage-repo-qemumips,http://192.168.7.7:45100/qemumips --repofrompath=oe-testimage-repo-mips32r2,http://192.168.7.7:45100/mips32r2 --repofrompath=oe-testimage-repo-noarch,http://192.168.7.7:45100/noarch --nogpgcheck install --installroot=/home/root/chroot/test -v -y --rpmverbosity=debug busybox run-postinsts Unable to detect release version (use '--releasever' to specify release version) DNF version: 4.2.2 cachedir: /home/root/chroot/test/var/cache/dnf Added oe-testimage-repo-qemumips repo from http://192.168.7.7:45100/qemumips Added oe-testimage-repo-mips32r2 repo from http://192.168.7.7:45100/mips32r2 Added oe-testimage-repo-noarch repo from http://192.168.7.7:45100/noarch repo: downloading from remote: oe-testimage-repo-qemumips oe-testimage-repo-qemumips 1.6 kB/s | 257 B 00:00 not found other for: not found modules for: not found deltainfo for: not found updateinfo for: oe-testimage-repo-qemumips: using metadata from Tue Dec 3 02:54:27 2019. repo: downloading from remote: oe-testimage-repo-mips32r2 oe-testimage-repo-mips32r2 460 kB/s | 43 kB 00:00 not found other for: not found modules for: not found deltainfo for: not found updateinfo for: oe-testimage-repo-mips32r2: using metadata from Tue Dec 3 02:54:27 2019. repo: downloading from remote: oe-testimage-repo-noarch oe-testimage-repo-noarch 5.8 kB/s | 1.4 kB 00:00 not found other for: not found modules for: not found deltainfo for: not found updateinfo for: oe-testimage-repo-noarch: using metadata from Tue Dec 3 02:54:24 2019. No module defaults found --> Starting dependency resolution --> Finished dependency resolution Error: Problem 1: conflicting requests - package busybox-1.31.1-r0.mips32r2 does not have a compatible architecture - nothing provides ld.so.1 needed by busybox-1.31.1-r0.mips32r2 - nothing provides ld.so.1(GLIBC_2.4) needed by busybox-1.31.1-r0.mips32r2 - nothing provides update-alternatives-opkg needed by busybox-1.31.1-r0.mips32r2 - nothing provides libc.so.6(GLIBC_2.28) needed by busybox-1.31.1-r0.mips32r2 - nothing provides libm.so.6 needed by busybox-1.31.1-r0.mips32r2 - nothing provides libm.so.6(GLIBC_2.0) needed by busybox-1.31.1-r0.mips32r2 - nothing provides libm.so.6(GLIBC_2.29) needed by busybox-1.31.1-r0.mips32r2 - nothing provides libc6 >= 2.30 needed by busybox-1.31.1-r0.mips32r2 Problem 2: conflicting requests - nothing provides /bin/sh needed by run-postinsts-1.0-r10.noarch (try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages) Test run when enable usrmege Traceback (most recent call last): File "/home/pokybuild/yocto-worker/qemumips/build/meta/lib/oeqa/core/decorator/__init__.py", line 36, in wrapped_f return func(*args, **kwargs) File "/home/pokybuild/yocto-worker/qemumips/build/meta/lib/oeqa/runtime/cases/dnf.py", line 116, in test_dnf_reinstall self.dnf_with_repo('reinstall -y run-postinsts-dev') File "/home/pokybuild/yocto-worker/qemumips/build/meta/lib/oeqa/runtime/cases/dnf.py", line 68, in dnf_with_repo output = self.dnf(" ".join(cmdlinerepoopts) + " --nogpgcheck " + command) File "/home/pokybuild/yocto-worker/qemumips/build/meta/lib/oeqa/runtime/cases/dnf.py", line 21, in dnf self.assertEqual(status, expected, message) AssertionError: 1 != 0 : dnf --repofrompath=oe-testimage-repo-qemumips,http://192.168.7.7:45100/qemumips --repofrompath=oe-testimage-repo-mips32r2,http://192.168.7.7:45100/mips32r2 --repofrompath=oe-testimage-repo-noarch,http://192.168.7.7:45100/noarch --nogpgcheck reinstall -y run-postinsts-dev Added oe-testimage-repo-qemumips repo from http://192.168.7.7:45100/qemumips Added oe-testimage-repo-mips32r2 repo from http://192.168.7.7:45100/mips32r2 Added oe-testimage-repo-noarch repo from http://192.168.7.7:45100/noarch Last metadata expiration check: 0:06:05 ago on Tue Dec 3 02:59:16 2019. Package run-postinsts-dev available, but not installed. No match for argument: run-postinsts-dev 85731 1 randy.macleod 2019-12-05 16:09:48 +0000 Occurs relatively rarely (every month or so) but it is important that the autobuilder tests be fixed. 86424 2 randy.macleod 2020-02-13 16:07:18 +0000 Not seen on autobuilder but it has been encountering other errors so let's check next week. 90915 3 randy.macleod 2021-07-13 13:18:50 +0000 Unassigned assuming that this is only mips, I don't think WR has an interest in this bug. 96051 4 randy.macleod 2023-07-26 21:21:57 +0000 Bulk move of 64 bugs to 4.3 M3 after a quick review. If a bug is actually fixed, please add a commit link and resolve it. -- Randy for YP bug team. 96459 5 tgamblin 2023-09-14 16:35:35 +0000 Moved to M4. 97054 6 randy.macleod 2023-10-30 15:29:08 +0000 Bulk move to 5.0 M1. -- Randy 97691 7 randy.macleod 2023-12-21 23:28:32 +0000 Bulk move of 52 5.0-M1 bugs to M2. 97980 8 randy.macleod 2024-01-25 14:28:42 +0000 Bulk move of bugs owned by "unassigned" from 5.0-M2 to 5.0-M3. 98292 9 randy.macleod 2024-02-22 19:53:21 +0000 Bulk move of bugs from 5.0-M3 to 5.99. The bugs in this list are all unassigned and are ones that I consider to be nice to do some time but not essential track from milestone to milestone in 5.0 or 5.1. We agreed on this bug handling approach in the YP bug review on Feb 22, 2024. If you disagree, please move back to a 5.[012...]-M* milestone. 100460 10 mathieu.dubois-briand 2024-12-19 15:18:19 +0000 qemumips64 debian11-vk-2 master completed at 2024-12-15T11:36:09Z https://valkyrie.yoctoproject.org/#/builders/32/builds/635/steps/14/logs/stdio qemumips64 debian12-vk-6 mathieu/master-next completed at 2024-12-18T10:18:05Z https://valkyrie.yoctoproject.org/#/builders/32/builds/662/steps/13/logs/stdio 100523 11 mathieu.dubois-briand 2025-01-03 12:09:13 +0000 Also on ARM: meta-arm ubuntu2004-vk-2 mathieu/master-next completed at 2024-12-23T10:01:34Z https://valkyrie.yoctoproject.org/#/builders/75/builds/635/steps/17/logs/stdio qemumips64 ubuntu2004-vk-3 master completed at 2024-12-26T01:23:12Z https://valkyrie.yoctoproject.org/#/builders/32/builds/690/steps/14/logs/stdio And now this one is on qemuarmv5, so maybe the bug seen during the last few days is not exactly the same as previously. qemuarmv5 alma9-vk-1 master completed at 2024-12-28T04:21:37Z https://valkyrie.yoctoproject.org/#/builders/80/builds/641/steps/14/logs/stdio qemuppc alma9-vk-1 master completed at 2024-12-30T04:02:21Z https://valkyrie.yoctoproject.org/#/builders/31/builds/707/steps/14/logs/stdio qemumips64 fedora39-vk-1 master completed at 2025-01-01T01:24:43Z https://valkyrie.yoctoproject.org/#/builders/32/builds/710/steps/14/logs/stdio qemumips64 alma9-vk-2 master-next completed at 2025-01-01T12:14:40Z https://valkyrie.yoctoproject.org/#/builders/32/builds/711/steps/14/logs/stdio qemumips64 opensuse155-vk-1 mathieu/master-next completed at 2025-01-02T09:44:32Z https://valkyrie.yoctoproject.org/#/builders/32/builds/714/steps/14/logs/stdio qemumips64 debian12-vk-8 mathieu/master-next completed at 2025-01-02T17:34:04Z https://valkyrie.yoctoproject.org/#/builders/32/builds/719/steps/13/logs/stdio 100648 12 mathieu.dubois-briand 2025-01-15 13:08:55 +0000 qemumips64 alma9-vk-1 abelloni/master-next completed at 2025-01-05T19:57:32Z https://valkyrie.yoctoproject.org/#/builders/32/builds/736/steps/14/logs/stdio qemumips64 rocky9-vk-1 master completed at 2025-01-06T01:32:04Z https://valkyrie.yoctoproject.org/#/builders/32/builds/737/steps/14/logs/stdio qemumips64 debian11-vk-1 abelloni/master-next completed at 2025-01-06T11:25:01Z https://valkyrie.yoctoproject.org/#/builders/32/builds/738/steps/14/logs/stdio qemumips64 alma9-vk-1 master-next completed at 2025-01-06T15:08:13Z https://valkyrie.yoctoproject.org/#/builders/32/builds/739/steps/14/logs/stdio qemumips64 stream9-vk-1 master completed at 2025-01-07T01:33:36Z https://valkyrie.yoctoproject.org/#/builders/32/builds/744/steps/14/logs/stdio qemumips64 ubuntu2204-vk-4 master-next completed at 2025-01-07T22:57:15Z https://valkyrie.yoctoproject.org/#/builders/32/builds/754/steps/14/logs/stdio meta-arm alma8-vk-2 abelloni/master-next completed at 2025-01-12T19:45:43Z https://valkyrie.yoctoproject.org/#/builders/75/builds/731/steps/18/logs/stdio 100695 13 mathieu.dubois-briand 2025-01-20 08:39:05 +0000 meta-arm debian11-vk-1 master-next completed at 2025-01-17T11:41:58Z https://autobuilder.yoctoproject.org/valkyrie/#/builders/75/builds/762/steps/17/logs/stdio qemumips64 rocky9-vk-2 master-next completed at 2025-01-16T12:11:38Z https://autobuilder.yoctoproject.org/valkyrie/#/builders/32/builds/808/steps/14/logs/stdio 100697 14 liezhi.yang 2025-01-20 08:40:23 +0000 it's strange, I tried it many times locally, but can't reproduce the error 100715 15 mathieu.dubois-briand 2025-01-22 14:14:22 +0000 qemumips64 ubuntu2404-vk-2 mathieu/master-next completed at 2025-01-22T09:55:12Z https://autobuilder.yoctoproject.org/valkyrie/#/builders/32/builds/848/steps/13/logs/stdio qemumips64 ubuntu2004-vk-1 master-next completed at 2025-01-22T12:21:01Z https://autobuilder.yoctoproject.org/valkyrie/#/builders/32/builds/850/steps/14/logs/stdio 100740 16 richard.purdie 2025-01-24 14:52:51 +0000 I did a deep dive into one of these. https://autobuilder.yocto.io/pub/failed-builds-data/qemumips64-dnf/log.do_testimage.293138 - is the testimage log of the test_dnf_installroot test failure. Inside https://autobuilder.yocto.io/pub/failed-builds-data/qemumips64-dnf/tests_artifacts.tar.gz is a dnf.rpm.log There is weirdness is this set of lines: 2025-01-22T19:42:32+0000 INFO --- logging initialized --- 2025-01-22T19:43:22+0000 SUBDEBUG Reinstall: dnf-test-main-4.22.0-r0.mips64r2 2025-01-22T19:43:22+0000 SUBDEBUG Reinstalled: dnf-test-main-4.22.0-r0.mips64r2 2025-01-22T19:43:44+0000 INFO --- logging initialized --- 2025-01-22T19:46:00+0000 INFO D: exiting chroot /home/root/chroot/test/ which implies the reinstall test is running when the installroot test is still active too. Since the installroot is separate, that is possible. The ssh connection for the test seems to fail, exit code 255. I don't believe the command fails, I believe the connection fails. In the "messages" log in the above archive, there is also this: Jan 22 19:40:34 qemumips64 authpriv.info dropbear[693]: Child connection from 192.168.7.1:41410 Jan 22 19:40:34 qemumips64 authpriv.notice dropbear[693]: Auth succeeded with blank password for 'root' from 192.168.7.1:41410 Jan 22 19:42:05 qemumips64 authpriv.info dropbear[2213]: Child connection from 192.168.7.1:44380 Jan 22 19:42:07 qemumips64 authpriv.notice dropbear[2213]: Auth succeeded with blank password for 'root' from 192.168.7.1:44380 Jan 22 19:43:22 qemumips64 authpriv.info dropbear[693]: Exit (root) from <192.168.7.1:41410>: Error writing: Broken pipe Jan 22 19:43:27 qemumips64 authpriv.info dropbear[2213]: Exit (root) from <192.168.7.1:44380>: Disconnect received Jan 22 19:43:27 qemumips64 authpriv.info dropbear[3541]: Child connection from 192.168.7.1:40790 i.e. it failed to write to the remote. I was wondering if the ssh server had OOM'd however this looks like like the networking to 192.168.7.1 glitched. 100753 17 mathieu.dubois-briand 2025-01-27 13:39:06 +0000 qemumips64 rocky9-vk-1 master-next completed at 2025-01-22T19:52:41Z https://autobuilder.yoctoproject.org/valkyrie/#/builders/32/builds/857/steps/14/logs/stdio 100769 18 mathieu.dubois-briand 2025-01-30 08:46:14 +0000 qemumips64 debian12-vk-9 mathieu/master-next completed at 2025-01-27T08:38:55Z https://autobuilder.yoctoproject.org/valkyrie/#/builders/32/builds/882/steps/13/logs/stdio 100778 19 ross.burton 2025-01-30 15:49:49 +0000 Potentially maybe fixed with oe-core 9d91a5674c515a43ae76d8615f72e5e2dc16c961? 100785 20 randy.macleod 2025-01-30 16:05:05 +0000 Bulk move. 100856 21 mathieu.dubois-briand 2025-02-06 14:58:36 +0000 qemumips64 debian12-vk-4 mathieu/master-next completed at 2025-02-06T12:08:28Z https://autobuilder.yoctoproject.org/valkyrie/#/builders/32/builds/937/steps/13/logs/stdio 100893 22 mathieu.dubois-briand 2025-02-10 13:02:01 +0000 qemumips64 opensuse155-vk-1 master-next completed at 2025-02-07T12:08:35Z Lot of dnf failures this week because of bad patches. I believe this one is because of the AB-INT, but I might be wrong. https://autobuilder.yoctoproject.org/valkyrie/#/builders/32/builds/948/steps/15/logs/stdio 100904 23 mathieu.dubois-briand 2025-02-12 14:39:24 +0000 qemumips64 ubuntu2004-vk-2 master completed at 2025-02-12T02:11:54Z https://autobuilder.yoctoproject.org/valkyrie/#/builders/32/builds/980/steps/14/logs/stdio 100937 24 randy.macleod 2025-02-13 15:44:05 +0000 Robert, this is a frequent issue for the YP AB. Can we get someone working on this ASAP? 101286 25 mathieu.dubois-briand 2025-03-05 08:37:09 +0000 qemumips64 opensuse156-vk-1 master-next completed at 2025-03-03T13:44:13Z https://autobuilder.yoctoproject.org/valkyrie/#/builders/32/builds/1091/steps/15/logs/stdio 101328 26 mathieu.dubois-briand 2025-03-10 09:43:02 +0000 qemumips64 debian12-vk-1 mathieu/master-next completed at 2025-03-07T07:23:32Z https://autobuilder.yoctoproject.org/valkyrie/#/builders/32/builds/1119/steps/13/logs/stdio qemumips64 ubuntu2004-vk-1 master completed at 2025-03-07T12:07:33Z https://autobuilder.yoctoproject.org/valkyrie/#/builders/32/builds/1122/steps/14/logs/stdio 101361 27 mathieu.dubois-briand 2025-03-13 09:29:52 +0000 qemumips64 alma8-vk-2 master-next completed at 2025-03-12T17:36:12Z https://autobuilder.yoctoproject.org/valkyrie/#/builders/32/builds/1159/steps/15/logs/stdio 101371 28 randy.macleod 2025-03-13 14:52:37 +0000 This is a difficult bug to reproduce, it's likely load related. Can we add some logs to help diagnose the issue. 101375 29 randy.macleod 2025-03-13 18:42:31 +0000 Something along the lines of the 'run_cmds' function in : https://git.yoctoproject.org/poky/tree/scripts/oe-time-dd-test.sh#n31 This is rather verbose so we have an offline script to generate a summary: https://git.yoctoproject.org/yocto-autobuilder-helper/tree/scripts/summarize_top_output.py It's not clear to me what the conditions merit such data collection. I'm off work tomorrow (Friday) so I'll try to make time to work on that next week if no one has gotten there first. 101382 30 richard.purdie 2025-03-16 22:51:31 +0000 Locally I couldn't reproduce this. I tried to reproduce using a failed autobuilder build directory and initially it didn't. When there were builds running in parallel (including an oe-selftest), it did reproduce. Locally, I did manage test failures with ssh hanging if I ran "cpulimit -l 50" on the qemu pid as it started from a "bitbake core-image-sato -c testimage" but not the same test failure. The kernel looks showed rcu-stalls and weird timestamps. 101383 31 richard.purdie 2025-03-17 00:17:07 +0000 I noticed there is a suspicious looking CONFIG_QEMUMIPS64 kernel config we're nt setting which changes clock/interrupt config. sched_clock: 64 bits at 100 Hz, resolution 10000000ns, wraps every 36028797015000000ns becomes: clocksource: MIPS: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 11945313065 ns sched_clock: 32 bits at 160MHz, resolution 6ns, wraps every 13421700092ns clocksource: Switched to clocksource MIPS with that config set. We should probably try setting that and seeing if these issues change in frequency. 101384 32 paulg 2025-03-17 00:34:36 +0000 I do 100% agree. Using a 100Hz jiffy clock is so y2k that it is not funny. It made sense back in the day as a valid trade-off in order to optimize toward bulk processing with a larger time slice on these old processors, before scribbling the tiny cache with a timer interrupt. But time moves on, and even 15(?) years ago the default was 1kHz because processors got faster and people started to care about resolution. Using a 6.x kernel on an ancient CPU model with a 100Hz clock definitely puts us in a spot where very few (if nobody else?) is testing. 101388 33 richard.purdie 2025-03-17 12:57:18 +0000 CONFIG_QEMUMIPS64 comes from one of our kernel patches. This disables a couple of different clock sources. Looking at upstream, there is https://web.git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/arch/mips?id=078b831638e1aa06dd7ffa9f244c8ac6b2995561 which also changes one of the two clock sources the above changes. Testing locally, I don't see the rcu-stalls with this backported so I think the best thing we can do is backport the above revert from mainline. 101391 34 mathieu.dubois-briand 2025-03-17 15:33:02 +0000 qemumips64 alma9-vk-2 master completed at 2025-03-14T01:24:20Z https://autobuilder.yoctoproject.org/valkyrie/#/builders/32/builds/1170/steps/14/logs/stdio 101402 35 mathieu.dubois-briand 2025-03-18 09:53:07 +0000 qemumips64 ubuntu2204-vk-2 master-next completed at 2025-03-17T22:52:25Z https://autobuilder.yoctoproject.org/valkyrie/#/builders/32/builds/1186/steps/14/logs/stdio 101430 36 mathieu.dubois-briand 2025-03-20 14:11:31 +0000 qemumips64 alma9-vk-2 master-next completed at 2025-03-19T15:45:45Z https://autobuilder.yoctoproject.org/valkyrie/#/builders/32/builds/1202/steps/14/logs/stdio 101468 37 richard.purdie 2025-03-24 09:41:51 +0000 The fix was included in the kernel upgrade: https://git.yoctoproject.org/poky/commit/?id=808fb969bc8c3dafda87ddfd813fae832eb6917c so we'll work on the basis that fixes this. We should start a new bug for any issuea after this is merged.