14937 2022-10-14 15:42:59 +0000 AB-INT: debuginfod.Debuginfod.test_debuginfod failure 2024-05-16 15:15:43 +0000 1 1 10 QA/Testing Functional (self) Testing oe-selftest unspecified x86 Multiple RESOLVED FIXED AB-INT High normal 5.1 M1 1 alexandre.belloni ross.burton alexandre.belloni joe.slater randy.macleod ross.burton steve --- --- No (bug/feature does not impact docs) oldest_to_newest 94177 0 alexandre.belloni 2022-10-14 15:42:59 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/4209/steps/14/logs/stdio oe-selftest-ubuntu ubuntu1804-ty-3 2022-09-29 22:04:54,677 - oe-selftest - INFO - debuginfod.Debuginfod.test_debuginfod (subunit.RemotedTestCase) 2022-09-29 22:04:54,678 - oe-selftest - INFO - ... FAIL 2022-09-29 22:04:54,678 - oe-selftest - INFO - 14: 1/18 43/503 (263.98s) (debuginfod.Debuginfod.test_debuginfod) 2022-09-29 22:04:54,678 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/selftest/cases/debuginfod.py", line 53, in test_debuginfod self.assertIn("/.cache/debuginfod_client/", output) File "/usr/lib/python3.6/unittest/case.py", line 1089, in assertIn self.fail(self._formatMessage(msg, standardMsg)) File "/usr/lib/python3.6/unittest/case.py", line 670, in fail raise self.failureException(msg) AssertionError: '/.cache/debuginfod_client/' not found in 'Server query failed: No such file or directory' 94204 1 ross.burton 2022-10-20 10:31:16 +0000 I was really hoping this was a release branch, but it happened in master. Maybe debuginfod is taking too long to startup? 94205 2 ross.burton 2022-10-20 10:32:48 +0000 The debuginfod server: [Thu 29 Sep 2022 10:04:43 PM GMT] (19923/19929): fts traversed source paths in 0.333763s, scanned=6033, regex-skipped=0 [Thu 29 Sep 2022 10:04:53 PM GMT] (19923/23460): not found [Thu 29 Sep 2022 10:04:53 PM GMT] (19923/23460): 192.168.7.2:40036 UA:elfutils/0.187,Linux/x86_64,/ XFF: GET /buildid/989e57318bc0367c5347af0cfc116b28329cedd1/debuginfo 404 9 0+276ms So, no. The debuginfod had scanned in time before the request arrived. 94300 3 alexandre.belloni 2022-10-27 19:58:22 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/4203/steps/14/logs/stdio oe-selftest-fedora fedora36-ty-2 94702 4 ross.burton 2023-01-12 15:47:49 +0000 Fixed in d65729748253eaa640333198ca8aec05946cb9e8. 98051 5 alexandre.belloni 2024-01-26 17:03:56 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6348/steps/14/logs/stdio oe-selftest-centos alma9-ty-1 98107 6 alexandre.belloni 2024-02-01 18:25:26 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6348/steps/14/logs/stdio oe-selftest-centos alma9-ty-1 98627 7 alexandre.belloni 2024-03-27 19:10:28 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6567/steps/14/logs/stdio oe-selftest-centos alma9-ty-1 98632 8 alexandre.belloni 2024-03-27 20:43:53 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6564/steps/14/logs/stdio oe-selftest-centos alma9-ty-1 98633 9 alexandre.belloni 2024-03-27 20:44:47 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/6576/steps/14/logs/stdio oe-selftest-ubuntu ubuntu2304-ty-1 98642 10 alexandre.belloni 2024-03-27 21:50:35 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6524/steps/14/logs/stdio oe-selftest-centos rocky9-ty-1 98648 11 alexandre.belloni 2024-03-27 22:27:19 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5868/steps/15/logs/stdio oe-selftest-centos alma8-ty-1 98703 12 alexandre.belloni 2024-04-04 21:25:32 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6581/steps/14/logs/stdio oe-selftest-centos alma9-ty-2 98704 13 alexandre.belloni 2024-04-04 21:28:35 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/3115/steps/14/logs/stdio oe-selftest-armhost ubuntu2004-arm-1 98705 14 alexandre.belloni 2024-04-04 21:29:16 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6587/steps/14/logs/stdio oe-selftest-centos alma9-ty-2 98708 15 alexandre.belloni 2024-04-04 21:48:14 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/6510/steps/14/logs/stdio oe-selftest-ubuntu ubuntu2304-ty-1 98717 16 alexandre.belloni 2024-04-05 17:13:54 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5913/steps/14/logs/stdio oe-selftest-centos alma9-ty-2 98718 17 alexandre.belloni 2024-04-05 17:15:12 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5917/steps/14/logs/stdio oe-selftest-centos alma9-ty-1 98720 18 alexandre.belloni 2024-04-05 21:35:06 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6424/steps/15/logs/stdio oe-selftest-centos alma8-ty-1 98760 19 randy.macleod 2024-04-18 14:43:29 +0000 Ross, Joe has been looking at debuginfod recently so I've CCed him on this defect. 98976 20 steve 2024-05-07 21:47:24 +0000 scarthgap: https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/3290 oe-selftest-armhost ubuntu2004-arm-1 98981 21 alexandre.belloni 2024-05-08 20:50:06 +0000 https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/3284/steps/14/logs/stdio oe-selftest-armhost ubuntu2004-arm-1 98982 22 randy.macleod 2024-05-09 14:38:19 +0000 Happening frequently so raise to High Importance. 98983 23 ross.burton 2024-05-09 14:47:12 +0000 In the latest run: [Sun May 5 01:58:32 2024] (3833224/3833234): fts traversed source paths in 2.73113s, scanned=33924, regex-skipped=0 It shouldn't have found 34K packages to scan... 98995 24 ross.burton 2024-05-10 15:52:40 +0000 So the log in a recent failure said this: [Sun May 5 01:58:32 2024] (3833224/3833234): fts traversed source paths in 2.73113s, scanned=33924, regex-skipped=0 That means that it did a walk of the feed directory and found 33924 packages to scan. It then timed out waiting for the scan to finish because that is a lot of packages! This is happening occasionally because it depends on both the scan happening on a slow/loaded machine _and_ the build tree splitting/sharing to have meant that something massive was previously built, and thus populated the feed with lots of files. The solution would be to create a subset of the package feed for the test case to use, similar to how a subset of the whole feed is used when creating an image. I have a patch that needs cleaning up. 98996 25 ross.burton 2024-05-10 15:57:50 +0000 My proof: by adding gtk+3 to the test build we can make the feed explode in size: [Fri May 10 15:55:29 2024] (1451180/1451190): fts traversed source paths in 0.0454599s, scanned=3537, regex-skipped=0 However for the debuginfo we only care about xz and its dependents, so we can hardlink a new feed: [Fri May 10 15:56:43 2024] (1490852/1490862): fts traversed source paths in 0.00358932s, scanned=72, regex-skipped=0 99007 26 ross.burton 2024-05-14 10:54:43 +0000 Patches on the list (oeqa/selftest/debuginfod: use localpkgfeed to speed server startup). 99022 27 ross.burton 2024-05-16 15:15:43 +0000 Hopefully fixed in master with 8a04f529f2603acaf4ae72fbeccbd2e4ef862be1.