Under very specific conditions, I get multiple (476 last build) timeout warnings like the below: WARNING: eudev-3.2.12-r0 do_package_write_rpm: Error contacting Hash Equivalence Server unix:///media/big_scrap/bill/yp-generic-arm64/master/qemuarm64/build/hashserve.sock: Timed out waiting for data I am using only Poky and building core-image-base for qemuarm64 The local.conf is defaults except I inherit build-stats. I use kas to set all this up. I can publish my tiny repo if you wish to look at it. poky commit is from 2023-11-27 d069eb3af4f8a20ca4bad84b5e529e92d152ccef This error goes away if I use a poky commit from 2023-10-05 2b9044361f2855866eed831f8bdb770f2c7d42dc I have done multiple builds of each and can reproduce the warnings (~ same number) when using the new commit and have never seen the warning on the old commit. I suspect something is wrong with the HD on that build machine but it is strange that I can do new builds with the old commit and get zero warnings. In addition to the warnings the build time goes up drastically. old commit: 6954 Sec (still too long for this machine) new commit: 14,421 Sec So the new commits are less tolerant of load issues somehow. If I move the build to the (smaller) SSD on the same machine I get 0 timeouts and the build time goes to 3756 Sec. If I leave the build on the HD but cap BB_THREADS to 8 and -j 8, the warnings reduce (32 on the test I did) but don't go away. Build time of 11,950 If I do a build on my main desktop machine which is similar (i7-10700 @2.9GHz 8 core/16 threads 32 GB ram) and still on a spinning metal HD I get zero warnings and 4069 Sec. I have also done builds on AWS EC2 machine c7a.4xlarge which is also 16 vcpus and 32GB ram. It is an AMD based x86x machine but the closest I could come. It of course has nvme storage. I get no warnings for the default setup and also if I oversubscribe the BB_THREADS to 32 (leaving -j at 16). (2417 Sec for default and 2401 Sec for oversubscribed.) Build machine: 8 core, 16 threads i7-11700K @ 3.6GHz (800 to 5000 min/maz MHz) 32 GB Ram 1 TB SSD for OS ( and 500GB for /fast volume) 6 TB HD for /media/big_scrap which is the normal position of builds I would not report the issue at all is others in IRC had not reported that they thought they had seen it AND that I can make the warnings go away by using an old commit. Yes, I have purchased a 4TB SSD to replace the HD on the build machine but I would still like to understand what is going on here.
Copying Bruce who is also seeing this and Joshua who's patches may be at fault :/ Bill: Would you be willing/able to revert the changes from: https://git.yoctoproject.org/poky/commit/bitbake?id=2378e4bdfa0045643b7d18390c454a458656de2d through to: https://git.yoctoproject.org/poky/commit/bitbake?id=95481c8668238b291fd693dd8b8589d49850dfd9 and see if that "resolves" the issue? That would at least narrow it down to the recent hashserv series.
You may need to revert https://git.yoctoproject.org/poky/commit/bitbake?id=6b6374c336016ee56ff7418c45d1eba2eac95b21 as well but hopefully they're standalone apart from that.
I an also confirm that I was seeing this on an up to date poky clone, building core-image-minimal for x86/arm. I was using an old/existing poky clone that has been around for years, so it was unclear if that was partially to blame during my testing. I have done a new clone of poky, with a similar configuration and have started builds. No work on whether the timeouts are still present. I was also seeing the very long build times when the timeouts were present. For my debug, I did try increasing the debug level in: diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py index c5bfef55d6..856aa8fea4 100644 --- a/bitbake/lib/bb/cooker.py +++ b/bitbake/lib/bb/cooker.py @@ -330,7 +330,7 @@ class BBCooker: sync=False, upstream=upstream, ) - self.hashserv.serve_as_process(log_level=logging.WARNING) + self.hashserv.serve_as_process(log_level=logging.DEBUG) for mc in self.databuilder.mcdata: self.databuilder.mcorigdata[mc].setVar("BB_HASHSERVE", self.hashservaddr) self.databuilder.mcdata[mc].setVar("BB_HASHSERVE", self.hashservaddr) But didn't get anything more useful in the logs. I also configured my build to run a manually started hashserver, and didn't get any more useful debug information as to why the timeouts are happening. Both unix and tcp were tried for the server, no difference based on the protocol. I was seeing a lot of connects/disconnects, and a few other minor things in the cooker log, but none of them looked to be directly related to the timeouts.
FWIW, the increased build times are a direct result of the timeouts. The timeouts are the direct problem, and shouldn't be happening, so solving those should also fix the increased built times.
https://patchwork.yoctoproject.org/project/bitbake/patch/20231204160343.1373871-1-JPEWhacker@gmail.com/
Clean build from JPEW's commit ID works with 0 warnings. However the build time is 8869 Sec which is better but not great. This machine is pretty isolated and not doing anything other than the build. This of course may be a different issue or just building more things than the old commit.
For completeness sake I did a clean build from the parent commit of JPEW's commit to verify the warnings came back. They did. I got 341 warnings and a build time of 11,888 Sec. Richard reports the fix has been merged to master.
Merged in bitbake c5b8c91d325ed1ca8abe5fe28d989693555c0622