Created attachment 4599 [details] Example bblayers.conf file I am seeing extreme recipe parsing times after I remove the tmp directory if I have the resident bitbake server enabled (BB_SERVER_TIMEOUT = "600"). See the attached bblayers.conf for the layers I used for the example (Poky + some layers from OpenEmbedded). To set up I devtool a recipe (bash) to cause a reparse each time bitbake is run, then I kill any existing server and remove the tmp directory to start from scratch: $ devtool modify -w bash $ bitbake -m $ rm -rf tmp $ time bitbake -p Parsing recipes: 100% |##########################################| Time: 0:00:18 Parsing of 2376 .bb files complete (0 cached, 2376 parsed). 3614 targets, 58 skipped, 0 masked, 0 errors. bitbake -p 0.20s user 0.02s system 1% cpu 20.050 total So, 20 seconds to parse 2376 recipes. So far so good. Then I remove the tmp directory (while the bitbake server is still running) and run bitbake -p again: $ rm -rf tmp $ time bitbake -p NOTE: Reconnecting to bitbake server... Parsing recipes: 100% |##########################################| Time: 0:00:54 Parsing of 2376 .bb files complete (0 cached, 2376 parsed). 3614 targets, 58 skipped, 0 masked, 0 errors. bitbake -p 0.20s user 0.02s system 0% cpu 55.959 total Ok, almost three times as long to parse the recipes now. (For the record, with our actual setup with 4000+ recipes, the recipe parsing time goes from 42 seconds to three and a half minutes under the same conditions...) Trying another bitbake -p, this time without removing tmp so this should only parse the bash recipe: $ time bitbake -p NOTE: Reconnecting to bitbake server... Loading cache: 100% |############################################| Time: 0:00:00 Loaded 3613 entries from dependency cache. Parsing recipes: 100% |##########################################| Time: 0:00:43 Parsing of 2376 .bb files complete (2375 cached, 1 parsed). 3614 targets, 58 skipped, 0 masked, 0 errors. bitbake -p 0.45s user 0.04s system 0% cpu 50.267 total It take almost as long, and doing it one more time just maintains the parsing time. $ time bitbake -p NOTE: Reconnecting to bitbake server... Loading cache: 100% |############################################| Time: 0:00:00 Loaded 3613 entries from dependency cache. Parsing recipes: 100% |##########################################| Time: 0:00:43 Parsing of 2376 .bb files complete (2375 cached, 1 parsed). 3614 targets, 58 skipped, 0 masked, 0 errors. bitbake -p 0.43s user 0.05s system 0% cpu 50.329 total Ok, now I remove the devtooled bash recipe: $ devtool reset -nr bash $ time bitbake -p NOTE: Reconnecting to bitbake server... Loading cache: 100% |############################################| Time: 0:00:01 Loaded 3613 entries from dependency cache. Parsing recipes: 100% |##########################################| Time: 0:00:43 Parsing of 2376 .bb files complete (2375 cached, 1 parsed). 3614 targets, 58 skipped, 0 masked, 0 errors. bitbake -p 0.42s user 0.05s system 0% cpu 50.802 total The parsing time is still the same (which was expected since the bash recipe needed to be parsed one last time). But what about the following two? There should be no parsing at all, but it still takes as long... $ time bitbake -p NOTE: Reconnecting to bitbake server... Loading cache: 100% |############################################| Time: 0:00:01 Loaded 3614 entries from dependency cache. bitbake -p 0.47s user 0.06s system 1% cpu 50.277 total $ time bitbake -p NOTE: Reconnecting to bitbake server... Loading cache: 100% |############################################| Time: 0:00:01 Loaded 3614 entries from dependency cache. bitbake -p 0.44s user 0.06s system 0% cpu 50.599 total Finally, by stopping the bitbake server and starting over, the times are back to normal: $ bitbake -m $ rm -rf tmp $ time bitbake -p Parsing recipes: 100% |##########################################| Time: 0:00:18 Parsing of 2376 .bb files complete (0 cached, 2376 parsed). 3614 targets, 58 skipped, 0 masked, 0 errors. bitbake -p 0.20s user 0.01s system 1% cpu 19.908 total And now a run without having to parse any recipes takes almost no time, as expected: $ time bitbake -p NOTE: Reconnecting to bitbake server... bitbake -p 0.17s user 0.02s system 88% cpu 0.209 total I have no real idea as to what is going on unfortunately. My only vague suspicion is that it is related to the use of inotify.
Hi Peter, Sorry about the delay in getting to it. I tried these steps, admittedly just with oe-core but I always got consistent parsing timings. Is this still an issue you're seeing? Are you still able to reproduce? There have been changes in the lasy couple of years which could have fixed this... Cheers, Richard
No, I do not seem to be able to reproduce it any longer. So I guess it was just a matter of waiting long enough for the problem to solve itself. :)
Thanks for confirming. I suspect we did fix some bug and this was one of the side effects of it. Would be nice to know where/when but I'm just happy it is working now!