| Summary: | Get timeout connecting to local hashserve under very specific conditions | ||
|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] BitBake | Reporter: | Bill Mills <wm.a.mills> |
| Component: | bitbake | Assignee: | Joshua Watt <JPEWhacker> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | minor | ||
| Priority: | Undecided | CC: | bruce.ashfield, JPEWhacker, poky.bs.watcher, poky.watcher, ross.burton, wm.a.mills |
| Version: | 5.0 | ||
| Target Milestone: | 0.0.0 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |
|
Description
Bill Mills
2023-12-02 15:56:51 UTC
Copying Bruce who is also seeing this and Joshua who's patches may be at fault :/ Bill: Would you be willing/able to revert the changes from: https://git.yoctoproject.org/poky/commit/bitbake?id=2378e4bdfa0045643b7d18390c454a458656de2d through to: https://git.yoctoproject.org/poky/commit/bitbake?id=95481c8668238b291fd693dd8b8589d49850dfd9 and see if that "resolves" the issue? That would at least narrow it down to the recent hashserv series. You may need to revert https://git.yoctoproject.org/poky/commit/bitbake?id=6b6374c336016ee56ff7418c45d1eba2eac95b21 as well but hopefully they're standalone apart from that. I an also confirm that I was seeing this on an up to date poky clone, building core-image-minimal for x86/arm.
I was using an old/existing poky clone that has been around for years, so it was unclear if that was partially to blame during my testing. I have done a new clone of poky, with a similar configuration and have started builds. No work on whether the timeouts are still present.
I was also seeing the very long build times when the timeouts were present.
For my debug, I did try increasing the debug level in:
diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index c5bfef55d6..856aa8fea4 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -330,7 +330,7 @@ class BBCooker:
sync=False,
upstream=upstream,
)
- self.hashserv.serve_as_process(log_level=logging.WARNING)
+ self.hashserv.serve_as_process(log_level=logging.DEBUG)
for mc in self.databuilder.mcdata:
self.databuilder.mcorigdata[mc].setVar("BB_HASHSERVE", self.hashservaddr)
self.databuilder.mcdata[mc].setVar("BB_HASHSERVE", self.hashservaddr)
But didn't get anything more useful in the logs.
I also configured my build to run a manually started hashserver, and
didn't get any more useful debug information as to why the timeouts
are happening.
Both unix and tcp were tried for the server, no difference based on
the protocol.
I was seeing a lot of connects/disconnects, and a few other minor
things in the cooker log, but none of them looked to be directly
related to the timeouts.
FWIW, the increased build times are a direct result of the timeouts. The timeouts are the direct problem, and shouldn't be happening, so solving those should also fix the increased built times. Clean build from JPEW's commit ID works with 0 warnings. However the build time is 8869 Sec which is better but not great. This machine is pretty isolated and not doing anything other than the build. This of course may be a different issue or just building more things than the old commit. For completeness sake I did a clean build from the parent commit of JPEW's commit to verify the warnings came back. They did. I got 341 warnings and a build time of 11,888 Sec. Richard reports the fix has been merged to master. Merged in bitbake c5b8c91d325ed1ca8abe5fe28d989693555c0622 |