https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/4209/steps/14/logs/stdio oe-selftest-ubuntu ubuntu1804-ty-3 2022-09-29 22:04:54,677 - oe-selftest - INFO - debuginfod.Debuginfod.test_debuginfod (subunit.RemotedTestCase) 2022-09-29 22:04:54,678 - oe-selftest - INFO - ... FAIL 2022-09-29 22:04:54,678 - oe-selftest - INFO - 14: 1/18 43/503 (263.98s) (debuginfod.Debuginfod.test_debuginfod) 2022-09-29 22:04:54,678 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last): File "/home/pokybuild/yocto-worker/oe-selftest-ubuntu/build/meta/lib/oeqa/selftest/cases/debuginfod.py", line 53, in test_debuginfod self.assertIn("/.cache/debuginfod_client/", output) File "/usr/lib/python3.6/unittest/case.py", line 1089, in assertIn self.fail(self._formatMessage(msg, standardMsg)) File "/usr/lib/python3.6/unittest/case.py", line 670, in fail raise self.failureException(msg) AssertionError: '/.cache/debuginfod_client/' not found in 'Server query failed: No such file or directory'
I was really hoping this was a release branch, but it happened in master. Maybe debuginfod is taking too long to startup?
The debuginfod server: [Thu 29 Sep 2022 10:04:43 PM GMT] (19923/19929): fts traversed source paths in 0.333763s, scanned=6033, regex-skipped=0 [Thu 29 Sep 2022 10:04:53 PM GMT] (19923/23460): not found [Thu 29 Sep 2022 10:04:53 PM GMT] (19923/23460): 192.168.7.2:40036 UA:elfutils/0.187,Linux/x86_64,/ XFF: GET /buildid/989e57318bc0367c5347af0cfc116b28329cedd1/debuginfo 404 9 0+276ms So, no. The debuginfod had scanned in time before the request arrived.
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/4203/steps/14/logs/stdio oe-selftest-fedora fedora36-ty-2
Fixed in d65729748253eaa640333198ca8aec05946cb9e8.
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6348/steps/14/logs/stdio oe-selftest-centos alma9-ty-1
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6567/steps/14/logs/stdio oe-selftest-centos alma9-ty-1
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6564/steps/14/logs/stdio oe-selftest-centos alma9-ty-1
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/6576/steps/14/logs/stdio oe-selftest-ubuntu ubuntu2304-ty-1
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6524/steps/14/logs/stdio oe-selftest-centos rocky9-ty-1
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5868/steps/15/logs/stdio oe-selftest-centos alma8-ty-1
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6581/steps/14/logs/stdio oe-selftest-centos alma9-ty-2
https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/3115/steps/14/logs/stdio oe-selftest-armhost ubuntu2004-arm-1
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6587/steps/14/logs/stdio oe-selftest-centos alma9-ty-2
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/6510/steps/14/logs/stdio oe-selftest-ubuntu ubuntu2304-ty-1
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5913/steps/14/logs/stdio oe-selftest-centos alma9-ty-2
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5917/steps/14/logs/stdio oe-selftest-centos alma9-ty-1
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6424/steps/15/logs/stdio oe-selftest-centos alma8-ty-1
Ross, Joe has been looking at debuginfod recently so I've CCed him on this defect.
scarthgap: https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/3290 oe-selftest-armhost ubuntu2004-arm-1
https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/3284/steps/14/logs/stdio oe-selftest-armhost ubuntu2004-arm-1
Happening frequently so raise to High Importance.
In the latest run: [Sun May 5 01:58:32 2024] (3833224/3833234): fts traversed source paths in 2.73113s, scanned=33924, regex-skipped=0 It shouldn't have found 34K packages to scan...
So the log in a recent failure said this: [Sun May 5 01:58:32 2024] (3833224/3833234): fts traversed source paths in 2.73113s, scanned=33924, regex-skipped=0 That means that it did a walk of the feed directory and found 33924 packages to scan. It then timed out waiting for the scan to finish because that is a lot of packages! This is happening occasionally because it depends on both the scan happening on a slow/loaded machine _and_ the build tree splitting/sharing to have meant that something massive was previously built, and thus populated the feed with lots of files. The solution would be to create a subset of the package feed for the test case to use, similar to how a subset of the whole feed is used when creating an image. I have a patch that needs cleaning up.
My proof: by adding gtk+3 to the test build we can make the feed explode in size: [Fri May 10 15:55:29 2024] (1451180/1451190): fts traversed source paths in 0.0454599s, scanned=3537, regex-skipped=0 However for the debuginfo we only care about xz and its dependents, so we can hardlink a new feed: [Fri May 10 15:56:43 2024] (1490852/1490862): fts traversed source paths in 0.00358932s, scanned=72, regex-skipped=0
Patches on the list (oeqa/selftest/debuginfod: use localpkgfeed to speed server startup).
Hopefully fixed in master with 8a04f529f2603acaf4ae72fbeccbd2e4ef862be1.