13539 2019-09-19 11:29:25 +0000 eSDK: performance issues when assembling sstate for full SDK 2020-01-16 16:26:35 +0000 1 1 6 Yocto Project Subprojects eSDK eSDK unspecified x86 Multiple RESOLVED FIXED Medium+ major 3.1 M1 1 bluelightning unassigned alex.kanavin randy.macleod richard.purdie ross.burton francisco.j.pedraza.gonzalez --- --- No (bug/feature does not impact docs) oldest_to_newest 85008 0 bluelightning 2019-09-19 11:29:25 +0000 When generating the sstate-cache for the eSDK (which is implemented in scripts/gen-lockedsig-cache) we are using glob to find the files matching each signature since we don't have the context to generate the full filename for each sstate archive; this is extremely expensive and starting to be problematic on the autobuilder, especially with the size of the local sstate-cache dirs there. We could at least try rewriting this to generate an in-memory mapping of hash to filename and that should be a big improvement. 85030 1 richard.purdie 2019-09-19 20:35:45 +0000 As the sstate cache on the autobuilder grows this is costing significant time on non-minimal esdk builds. The sstate cache is over NFS and directory listings are extremely slow (see the qa-extras step4b). 85035 2 ross.burton 2019-09-20 16:26:36 +0000 Somewhat bad example on the AB last night: https://autobuilder.yoctoproject.org/typhoon/#/builders/69/builds/1066/steps/8/logs/stdio Step 5/7: Running bitbake core-image-sato core-image-sato:do_populate_sdk_ext (1568993016.3: 45086.1) That's twelve and a half hours. 85106 3 konrad.scherer 2019-09-26 15:49:16 +0000 I am trying to reproduce the problem locally but it will take time. Why not disable sstate for these builds as a workaround? Looks like building from scratch will be faster and will not block the release. 85122 4 richard.purdie 2019-09-27 12:58:00 +0000 We build the esdk in many different places on the autobuilder, e.g. for each qemu machine, during selftest and so on. It looks like this performance issue affects minimal eSDK builds as well as non-minimal ones to some extent, the non-minimal ones are just much more badly affected. Bottom line is that we can't easily disable it just for those pieces. It would also leave our release artefacts incomplete. What we may have to do is empty the sstate cache to speed things back up again... 85124 5 konrad.scherer 2019-09-27 18:59:02 +0000 I have sent a patch that replaces the glob with a hash to filename lookup. Unfortunately I was not able to reproduce the performance issues locally, but I was able to verify that file list produced was identical. I also added a basic time counter and logging so it would be easier to capture performance data in the future. Please review. 85291 6 richard.purdie 2019-10-10 14:32:40 +0000 To update this bug, we did update the script to perform better (Thanks Konrad!). Unfortunately this wasn't enough with long timeouts on the autobuilder. We therefore also cleaned out the sstate cache which has improved the timing. There is another way we could improve performance which is not to involve listing the directories at all but calculating the filenames directly. To quote me from elsewhere: """ Just to elaborate on what teaching the code would entail, as well as using the 'hashfn' data as part of the hash validate function in from runqueue.py: sq_data['hashfn'][tid] = self.rqdata.dataCaches[mc].hashfn[taskfn] we could inject that data into BB_TASKDEPDATA. If we did that, the oecore code could probably create the full sstate tasknames for the tasks it needs using logic extracted out of sstate_checkhashes() from sstate.bbclass. This would mean the script wouldn't have to guess at filenames, it could access them directly. """ I suggest we retarget this bug for 3.1 as an enhancement for doing this. 86054 7 richard.purdie 2020-01-16 16:26:35 +0000 As a final piece, we split up the contexts of the sstate directories tom mitigate the performance issues: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=4cb1b4b409c4067c669fedef4a02be27689e3d26