| Summary: | add documentation on stabilizing host build machine performance for kernel>=4.3 and systemd>=228 | ||
|---|---|---|---|
| Product: | [Documentation] General Docs | Reporter: | Trevor Woerner <twoerner> |
| Component: | docs-general | Assignee: | Scott Rifenbark <srifenbark> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | normal | ||
| Priority: | Medium | CC: | bluelightning, joshuagloe, richard.purdie, srifenbark |
| Version: | unspecified | ||
| Target Milestone: | 2.3 M3 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | 10 Jan 2017: RESOLVED | ||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | Done (doc changes complete) | |
Apparently, this has been seen on Debian Testing as well. Hi Trevor, I am copying Paul Eggleton on this issue. Reading through this seems like it is all about the host build machine configuration. Maybe I am wrong on that. Are there any Yocto-specific things going on here? This is where I need help. To me, this seems like a build-debugging or release note situation? Not sure. Scott Hi Scott, (In reply to comment #2) > Reading through this seems like > it is all about the host build machine configuration. Maybe I am wrong on > that. You are exactly correct, this is a host build machine issue. > Are there any Yocto-specific things going on here? This is where I > need help. > > To me, this seems like a build-debugging or release note situation? Not > sure. If someone has an openSUSE machine running the latest desktop release (Leap 42.2), followed the instructions in, say, the Getting Started Guide, and started a build, there's a chance that build might fail for completely mysterious reasons with an "out of resources"-type message. Ideally, this will prompt them to go looking for a fix to find out why their build failed and to ask on the mailing list/stackoverflow/IRC. But sometimes when someone's first build (or one of their first builds) goes bad and causes them to have to dig deeply to find a solution (to something that is known but not mentioned anywhere) they might not be so happy. Although I gave openSUSE as an example, I want to make absolutely clear this is not an openSUSE-specific issue. The problem is a result of the host running a recent systemd and a recent kernel (not anything openSUSE does). openSUSE very recently had their release, I upgraded, and stumbled on this issue. In time every systemd-based distribution will upgrade their kernels and systemds and they will all be affected. One of the annoying things about this issue is that it doesn't happen on every build, it happens sporadically, making it all the more annoying because it will cause overnight Jenkins builds (for example) to fail when they shouldn't (for example). I hope the people who manage the Yocto Project's build machines are aware of this issue so they can tweak the build machines' configurations (if they haven't already). I think a note about this configuration tweak should be added *everywhere* because, in time, any build without this tweak could fail mysteriously. I mentioned the Getting Started Guide; ideally any document, blog post, or article that guides a user through their first Yocto/OE build should have an asterisk somewhere referring the reader to this issue to avoid the "bad first impression" syndrome. After guiding a user through their first build, a document could *optionally* give tips on how to make a build faster. But I don't think providing them with information to help keep their builds from failing for no apparent reason is something that I would consider to be optional. Trevor, Perfect detail.. This helps a lot thanks. Tevor, I have crafted a note and put it in a couple steps of the YP Quick Start. I will also put it in the "Build Overview" section of the YP Reference Manual once we are happy with the verbiage of the note. These are the two areas in the YP general documentation where a new user is taken through the process of a first build. See step 5 of http://www.yoctoproject.org/docs/2.3/yocto-project-qs/yocto-project-qs.html#building-an-image-for-emulation The note is replicated here as well - Step 3 of http://www.yoctoproject.org/docs/2.3/yocto-project-qs/yocto-project-qs.html#building-an-image-for-hardware Let me know if the note is okay. Once we are good with it, I will get it into the reference manual as well. Regarding other non-mainstream documented areas that might benefit from this note, you might set up a separate bug for those? I am not sure as I don't track that part of the documentation. Scott (In reply to comment #5) > See step 5 of > http://www.yoctoproject.org/docs/2.3/yocto-project-qs/yocto-project-qs. > html#building-an-image-for-emulation > > The note is replicated here as well - Step 3 of > http://www.yoctoproject.org/docs/2.3/yocto-project-qs/yocto-project-qs. > html#building-an-image-for-hardware > > Let me know if the note is okay. Once we are good with it, I will get it > into the reference manual as well. It's great to see this information being added! A reboot isn't needed, simply restarting the build will often cause it to mysteriously succeed when this is the culprit. I also don't think it's necessary to single out openSUSE 42.2 specifically; simply saying "This parameter is found in the system.conf file which is located in /etc/systemd on most systems" would suffice. Thanks Trevor for the further feedback on the note. See the following links and let me know if we are solid for the information. Once solid, I am going to add similar wording into the appropriate section in the ref-manual: The note in Step 5 of http://www.yoctoproject.org/docs/2.3/yocto-project-qs/yocto-project-qs.html#building-an-image-for-emulation The note in Step 3 of http://www.yoctoproject.org/docs/2.3/yocto-project-qs/yocto-project-qs.html#building-an-image-for-hardware Scott Hi Scott, I like this new wording; looks great, thanks! Thanks Trevor again... So I put the note here in the ref-manual (exact same wording) - http://www.yoctoproject.org/docs/2.3/ref-manual/ref-manual.html#build-overview. That covers it for the YP mainstream documentation. As I mentioned earlier, you might want to open a different bug against documentation that is not in my realm (YP mainstream manuals) if you want someone putting something similar in other areas you think need it. I will leave this at IN PROGRESS REVIEW just until you give me the okay on the placement of the note in the ref-manual. thanks, Scott (In reply to comment #9) > So I put the note here in the ref-manual (exact same wording) - > http://www.yoctoproject.org/docs/2.3/ref-manual/ref-manual.html#build- > overview. > > I will leave this at IN PROGRESS REVIEW just until you give me the okay on > the placement of the note in the ref-manual. Looks great! Thanks Trevor, Setting to RESOLVED and putting the doc flag to "done." Scott |
This issue might be related to 5594, but 5594 deals with host build machine tweaks for *speed*, whereas this one focuses on *stability*. I've noticed on a busy build machine sporadic issues where a build will fail with the error message: fork: Resource temporarily unavailable and nothing else wrong. Simply restarting the build causes it to succeed without incident. I had been seeing this, maybe, twice a week. Investigating led me to: http://unix.stackexchange.com/questions/253903/creating-threads-fails-with-resource-temporarily-unavailable-with-4-3-kernel which suggests the issue is a combination of - linux >= 4.3 - systemd >= 228 both of which are true when upgrading openSUSE from 42.1 (linux-4.1.36-41/210) to 42.2 (linux-4.4.36-8/228). The fix appears to be to modify the "DefaultTasksMax" systemd parameter (uncommenting it and setting it to "infinity"). On openSUSE 42.2 this is found in /etc/systemd/system.conf, then # systemctl daemon-reexec or reboot. I just made this adjustment but given it only happened once or twice a week on a reasonably busy build machine it might be hard to confirm. I'm also not sure if setting it to "infinity" is sensible ;-) I'm guessing other host distros probably also fall into this category. It would probably be a good idea to update various project documentation (e.g. the quick start guide?) to include a NOTE to this effect.