| Summary: | AB-INT: debuginfod.Debuginfod.test_debuginfod failure | ||
|---|---|---|---|
| Product: | [QA/Testing] Functional (self) Testing | Reporter: | Alexandre Belloni <alexandre.belloni> |
| Component: | oe-selftest | Assignee: | Ross Burton <ross.burton> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | normal | ||
| Priority: | High | CC: | alexandre.belloni, joe.slater, randy.macleod, ross.burton, steve |
| Version: | unspecified | ||
| Target Milestone: | 5.1 M1 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | AB-INT | ||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |
|
Description
Alexandre Belloni
2022-10-14 15:42:59 UTC
I was really hoping this was a release branch, but it happened in master. Maybe debuginfod is taking too long to startup? The debuginfod server: [Thu 29 Sep 2022 10:04:43 PM GMT] (19923/19929): fts traversed source paths in 0.333763s, scanned=6033, regex-skipped=0 [Thu 29 Sep 2022 10:04:53 PM GMT] (19923/23460): not found [Thu 29 Sep 2022 10:04:53 PM GMT] (19923/23460): 192.168.7.2:40036 UA:elfutils/0.187,Linux/x86_64,/ XFF: GET /buildid/989e57318bc0367c5347af0cfc116b28329cedd1/debuginfo 404 9 0+276ms So, no. The debuginfod had scanned in time before the request arrived. https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/4203/steps/14/logs/stdio oe-selftest-fedora fedora36-ty-2 Fixed in d65729748253eaa640333198ca8aec05946cb9e8. https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6348/steps/14/logs/stdio oe-selftest-centos alma9-ty-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6348/steps/14/logs/stdio oe-selftest-centos alma9-ty-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6567/steps/14/logs/stdio oe-selftest-centos alma9-ty-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6564/steps/14/logs/stdio oe-selftest-centos alma9-ty-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/6576/steps/14/logs/stdio oe-selftest-ubuntu ubuntu2304-ty-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6524/steps/14/logs/stdio oe-selftest-centos rocky9-ty-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5868/steps/15/logs/stdio oe-selftest-centos alma8-ty-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6581/steps/14/logs/stdio oe-selftest-centos alma9-ty-2 https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/3115/steps/14/logs/stdio oe-selftest-armhost ubuntu2004-arm-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6587/steps/14/logs/stdio oe-selftest-centos alma9-ty-2 https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/6510/steps/14/logs/stdio oe-selftest-ubuntu ubuntu2304-ty-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5913/steps/14/logs/stdio oe-selftest-centos alma9-ty-2 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5917/steps/14/logs/stdio oe-selftest-centos alma9-ty-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6424/steps/15/logs/stdio oe-selftest-centos alma8-ty-1 Ross, Joe has been looking at debuginfod recently so I've CCed him on this defect. scarthgap: https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/3290 oe-selftest-armhost ubuntu2004-arm-1 https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/3284/steps/14/logs/stdio oe-selftest-armhost ubuntu2004-arm-1 Happening frequently so raise to High Importance. In the latest run: [Sun May 5 01:58:32 2024] (3833224/3833234): fts traversed source paths in 2.73113s, scanned=33924, regex-skipped=0 It shouldn't have found 34K packages to scan... So the log in a recent failure said this: [Sun May 5 01:58:32 2024] (3833224/3833234): fts traversed source paths in 2.73113s, scanned=33924, regex-skipped=0 That means that it did a walk of the feed directory and found 33924 packages to scan. It then timed out waiting for the scan to finish because that is a lot of packages! This is happening occasionally because it depends on both the scan happening on a slow/loaded machine _and_ the build tree splitting/sharing to have meant that something massive was previously built, and thus populated the feed with lots of files. The solution would be to create a subset of the package feed for the test case to use, similar to how a subset of the whole feed is used when creating an image. I have a patch that needs cleaning up. My proof: by adding gtk+3 to the test build we can make the feed explode in size: [Fri May 10 15:55:29 2024] (1451180/1451190): fts traversed source paths in 0.0454599s, scanned=3537, regex-skipped=0 However for the debuginfo we only care about xz and its dependents, so we can hardlink a new feed: [Fri May 10 15:56:43 2024] (1490852/1490862): fts traversed source paths in 0.00358932s, scanned=72, regex-skipped=0 Patches on the list (oeqa/selftest/debuginfod: use localpkgfeed to speed server startup). Hopefully fixed in master with 8a04f529f2603acaf4ae72fbeccbd2e4ef862be1. |