| Summary: | eSDK: performance issues when assembling sstate for full SDK | ||
|---|---|---|---|
| Product: | [Yocto Project Subprojects] eSDK | Reporter: | Paul Eggleton <bluelightning> |
| Component: | eSDK | Assignee: | Unassigned <unassigned> |
| Status: | RESOLVED FIXED | QA Contact: | Francisco Pedraza <francisco.j.pedraza.gonzalez> |
| Severity: | major | ||
| Priority: | Medium+ | CC: | alex.kanavin, randy.macleod, richard.purdie, ross.burton |
| Version: | unspecified | ||
| Target Milestone: | 3.1 M1 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |
|
Description
Paul Eggleton
2019-09-19 11:29:25 UTC
As the sstate cache on the autobuilder grows this is costing significant time on non-minimal esdk builds. The sstate cache is over NFS and directory listings are extremely slow (see the qa-extras step4b). Somewhat bad example on the AB last night: https://autobuilder.yoctoproject.org/typhoon/#/builders/69/builds/1066/steps/8/logs/stdio Step 5/7: Running bitbake core-image-sato core-image-sato:do_populate_sdk_ext (1568993016.3: 45086.1) That's twelve and a half hours. I am trying to reproduce the problem locally but it will take time. Why not disable sstate for these builds as a workaround? Looks like building from scratch will be faster and will not block the release. We build the esdk in many different places on the autobuilder, e.g. for each qemu machine, during selftest and so on. It looks like this performance issue affects minimal eSDK builds as well as non-minimal ones to some extent, the non-minimal ones are just much more badly affected. Bottom line is that we can't easily disable it just for those pieces. It would also leave our release artefacts incomplete. What we may have to do is empty the sstate cache to speed things back up again... I have sent a patch that replaces the glob with a hash to filename lookup. Unfortunately I was not able to reproduce the performance issues locally, but I was able to verify that file list produced was identical. I also added a basic time counter and logging so it would be easier to capture performance data in the future. Please review. To update this bug, we did update the script to perform better (Thanks Konrad!). Unfortunately this wasn't enough with long timeouts on the autobuilder. We therefore also cleaned out the sstate cache which has improved the timing. There is another way we could improve performance which is not to involve listing the directories at all but calculating the filenames directly. To quote me from elsewhere: """ Just to elaborate on what teaching the code would entail, as well as using the 'hashfn' data as part of the hash validate function in from runqueue.py: sq_data['hashfn'][tid] = self.rqdata.dataCaches[mc].hashfn[taskfn] we could inject that data into BB_TASKDEPDATA. If we did that, the oecore code could probably create the full sstate tasknames for the tasks it needs using logic extracted out of sstate_checkhashes() from sstate.bbclass. This would mean the script wouldn't have to guess at filenames, it could access them directly. """ I suggest we retarget this bug for 3.1 as an enhancement for doing this. As a final piece, we split up the contexts of the sstate directories tom mitigate the performance issues: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=4cb1b4b409c4067c669fedef4a02be27689e3d26 |