Python has a make target, profile-opt, which uses profile-guided-optimization (with either a default workload, or one the user provides) to benchmark python and rebuild it with optimisations for the benchmarked workload. According to Python upstream "Not using a profile-opt builds in production is costing you ~10-40% in CPU performance."[1] We'd like to support this with OE-Core, however there are extra issues to workaround due to the nature of the cross-compilation environment. Most likely we need to support: 1) building a python which can generate a profile 2) building python with a supplied profile to We may even want a target image which will just boot into a profiling application (defaulting to pybench as upstream Python does) to help the user generate the profile on the target. 1. http://bugs.python.org/issue24318#msg244327
So Alejandro spoke about doing exactly this at ELC: http://events.linuxfoundation.org/sites/events/files/slides/ELC_2016.pdf
My plan is to implement this so that the usage would be as simple as possible. I'm thinking of having a separate build target, "build_pgo" or something, that would automagically: 1. build python with profile guided optimization enabled 2. build a specialized image with this special python package installed 3. boot the image in the target machine and run the profile target 4. copy profile data from the target machine to the build host 5. build python, utilizing the profile data from #4
Yeah that's pretty much exactly what Alejandro did as a proof of concept, so no doubt that work can be taken, updated, and reviewed.
(In reply to comment #2) > My plan is to implement this so that the usage would be as simple as > possible. I'm thinking of having a separate build target, "build_pgo" or > something, that would automagically: > 1. build python with profile guided optimization enabled > 2. build a specialized image with this special python package installed > 3. boot the image in the target machine and run the profile target > 4. copy profile data from the target machine to the build host > 5. build python, utilizing the profile data from #4 Like Ross just said, I implemented this using the autofdo class, and I automated the process of running the application (profile) and getting back the profile data from the image by adding a testsuite and running bitbake <image> -c testimage. The process is better explained on the previously mentioned slides, and I'm sure the ELC published a video somewhere. Here's the link to the code: https://github.com/aehs29/meta-autofdo This layer contains the required extra recipes, the class, the oeqa test, and some sample applications (I just need to check the latest version is upstream). Now, why hasn't this been merged/plublished? Last I checked, running perf (to get the profile data) on master was broken, and I haven't had time to figure out what was going on, but for all I know it could be fixed by now. Also there's still some work to do here, to get the profile data, we need to run perf to catch certain type of events (br_inst_retired.near_taken), now this event has a code associated with it: cpu/event=0xc4,umask=0x20,name=br_inst_retired_near_taken (on x86), and this event code is architecture dependent, ocperf is supposed to take care of this, but it doesnt, or at least it didnt, this is also something that needs to be checked, more info about this on Andi Kleen's github (https://github.com/andikleen/pmu-tools).
OK, please check that the latest code is upstream. Last time I checked it there were only two commits and not much automation in the process. IIRC it required manually copying the profile data, manually tweaking config between builds etc.
(In reply to comment #5) > OK, please check that the latest code is upstream. Last time I checked it > there were only two commits and not much automation in the process. IIRC it > required manually copying the profile data, manually tweaking config between > builds etc. Yeah, I was just checking, the latest version isn't upstream yet, I will upload it in a bit, you can check the slides or the video for now, they both cotain the updated instructions.
I just noticed the slides were moved, here's an updated link: http://events.linuxfoundation.org/sites/events/files/slides/ELC_2016_0.pdf Also, the code on github is pretty much updated now. Still missing a few things, I've got some other sample recipes for example, but I would like to check if theyre working before uploading those, everything else should be ok.
I utilized some concepts from the autofdo work but took a bit different approach and implemented optimization via the "profile-opt make target" as the bug subject suggests. A set of RFC patches are here: http://git.openembedded.org/openembedded-core-contrib/log/?h=marquiz/fixes-9338 The patchset basically has three parts. 1. Add new recipe for building python with profile instrumentation enabled 2. Add new minimalistic image for running the profile task 3. Add support for building optimized python, utilizing the profile data from 2. The upsides of the solution are that using it doesn't require any back-and-forth editing of local.conf or any manual tweaking of recipes. Also, it should be relatively easy to use in a automated build environment (provided that target hw is available). At least one downside is that it's Python-only.
Great!, it would be nice to see some performance results, comparing the two approaches, another setback of this approach is that it will only be available for python (I know this bug is for python only), whereas the other one could be used for programs coded on other languages as well.
I've run some benchmarks With simple pystone benchmark (python -m test.pystone 100000) I got about 24% improvement on qemux86 on my i7-3770K desktop: 3750 pystones/second -> 4655 pystones/second With pybench (python /usr/share/doc/python/Tools/pybench/pybench.py) I got also a significant improvement but I accidentally deleted the data. I'll re-run the benchmark and provide numbers shortly.
(In reply to comment #9) > another setback of this approach is that it will only be > available for python (I know this bug is for python only), whereas the other > one could be used for programs coded on other languages as well. Yes, the downside is that my solution only optimizes Python. It would probably make sense to create a separate bug about the autofdo feature. I'd be delighted to see that improved and integrated and would probably spend some time on it. What do you think?
Created attachment 3513 [details] pybench report on qemux86 Pybench result with non-optimized Python on qemux86
Created attachment 3514 [details] Pybench report on qemux86 - optimized Pybench result with optimized Python on qemux86
Pybench shows about 4.6% improvement. Interestingly, some of the tests are even slower with the optimized Python. I'll try with some other profile target to see if there's difference.
Created attachment 3517 [details] Pybench report on qemux86 - optimized with pybench target Pybench results improved dramatically when I used pybench itself as the profile target. Now the improvement is about 13% (up from 4.6%)
Moving this to 2.3 M3 for now. I have sent RFC patches for review: https://patchwork.openembedded.org/series/4960/ https://patchwork.openembedded.org/series/5027/
Patchset has been under review for few days now: https://patchwork.openembedded.org/series/5405/#
v3 of the patchset under review: https://patchwork.openembedded.org/series/5404/
I just sent v4: http://lists.openembedded.org/pipermail/openembedded-core/2017-August/140536.html
Rebased the patchset, with some changes: https://patchwork.openembedded.org/series/9511/
v6 is out
v9 under review
Just sent v10
Merged in oe-core 05a2a53f9cc7e75b4a3838ab9368cadf0f15ba1b.