Bug 9338 - Support building optimised Python (with profile-opt make target)
Summary: Support building optimised Python (with profile-opt make target)
Status: RESOLVED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: devtools / tool chain (show other bugs)
Version: 2.2
Hardware: x86 Multiple
: Medium+ enhancement
Target Milestone: 4.99
Assignee: Ross Burton
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2016-03-24 11:08 UTC by Joshua Lock
Modified: 2018-09-06 23:20 UTC (History)
8 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Yes (doc changes required)


Attachments
pybench report on qemux86 (5.67 KB, text/plain)
2016-10-28 18:39 UTC, Markus Lehtonen
no flags Details
Pybench report on qemux86 - optimized (5.67 KB, text/x-log)
2016-10-28 18:40 UTC, Markus Lehtonen
no flags Details
Pybench report on qemux86 - optimized with pybench target (5.67 KB, text/plain)
2016-11-01 14:41 UTC, Markus Lehtonen
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Joshua Lock 2016-03-24 11:08:08 UTC
Python has a make target, profile-opt, which uses profile-guided-optimization (with either a default workload, or one the user provides) to benchmark python and rebuild it with optimisations for the benchmarked workload.

According to Python upstream "Not using a profile-opt builds in production is costing you ~10-40% in CPU performance."[1]

We'd like to support this with OE-Core, however there are extra issues to workaround due to the nature of the cross-compilation environment.

Most likely we need to support:
1) building a python which can generate a profile
2) building python with a supplied profile to 

We may even want a target image which will just boot into a profiling application (defaulting to pybench as upstream Python does) to help the user generate the profile on the target.

1. http://bugs.python.org/issue24318#msg244327
Comment 1 Ross Burton 2016-04-11 08:39:38 UTC
So Alejandro spoke about doing exactly this at ELC:

http://events.linuxfoundation.org/sites/events/files/slides/ELC_2016.pdf
Comment 2 Markus Lehtonen 2016-09-15 07:28:29 UTC
My plan is to implement this so that the usage would be as simple as possible. I'm thinking of having a separate build target, "build_pgo" or something, that would automagically:
1. build python with profile guided optimization enabled
2. build a specialized image with this special python package installed
3. boot the image in the target machine and run the profile target
4. copy profile data from the target machine to the build host
5. build python, utilizing the profile data from #4
Comment 3 Ross Burton 2016-09-15 11:43:39 UTC
Yeah that's pretty much exactly what Alejandro did as a proof of concept, so no doubt that work can be taken, updated, and reviewed.
Comment 4 Alejandro Hernandez 2016-09-15 16:23:46 UTC
(In reply to comment #2)
> My plan is to implement this so that the usage would be as simple as
> possible. I'm thinking of having a separate build target, "build_pgo" or
> something, that would automagically:
> 1. build python with profile guided optimization enabled
> 2. build a specialized image with this special python package installed
> 3. boot the image in the target machine and run the profile target
> 4. copy profile data from the target machine to the build host
> 5. build python, utilizing the profile data from #4

Like Ross just said, I implemented this using the autofdo class, and I automated the process of running the application (profile) and getting back the profile data from the image by adding a testsuite and running bitbake <image> -c testimage.

The process is better explained on the previously mentioned slides, and I'm sure the ELC published a video somewhere.

Here's the link to the code:

https://github.com/aehs29/meta-autofdo

This layer contains the required extra recipes, the class, the oeqa test, and some sample applications (I just need to check the latest version is upstream).


Now, why hasn't this been merged/plublished?

Last I checked, running perf (to get the profile data) on master was broken, and I haven't had time to figure out what was going on, but for all I know it could be fixed by now.

Also there's still some work to do here, to get the profile data, we need to run perf to catch certain type of events (br_inst_retired.near_taken), now this event has a code associated with it: cpu/event=0xc4,umask=0x20,name=br_inst_retired_near_taken (on x86), and this event code is architecture dependent, ocperf is supposed to take care of this, but it doesnt, or at least it didnt, this is also something that needs to be checked, more info about this on Andi Kleen's github (https://github.com/andikleen/pmu-tools).
Comment 5 Markus Lehtonen 2016-09-15 17:21:08 UTC
OK, please check that the latest code is upstream. Last time I checked it there were only two commits and not much automation in the process. IIRC it required manually copying the profile data, manually tweaking config between builds etc.
Comment 6 Alejandro Hernandez 2016-09-15 17:27:17 UTC
(In reply to comment #5)
> OK, please check that the latest code is upstream. Last time I checked it
> there were only two commits and not much automation in the process. IIRC it
> required manually copying the profile data, manually tweaking config between
> builds etc.

Yeah, I was just checking, the latest version isn't upstream yet, I will upload it in a bit, you can check the slides or the video for now, they both cotain the updated instructions.
Comment 7 Alejandro Hernandez 2016-09-15 19:45:45 UTC
I just noticed the slides were moved, here's an updated link:

http://events.linuxfoundation.org/sites/events/files/slides/ELC_2016_0.pdf

Also, the code on github is pretty much updated now.

Still missing a few things, I've got some other sample recipes for example, but I would like to check if theyre working before uploading those, everything else should be ok.
Comment 8 Markus Lehtonen 2016-10-27 11:15:57 UTC
I utilized some concepts from the autofdo work but took a bit different approach and implemented optimization via the "profile-opt make target" as the bug subject suggests.

A set of RFC patches are here:
http://git.openembedded.org/openembedded-core-contrib/log/?h=marquiz/fixes-9338

The patchset basically has three parts.
1. Add new recipe for building python with profile instrumentation enabled
2. Add new minimalistic image for running the profile task
3. Add support for building optimized python, utilizing the profile data from 2.

The upsides of the solution are that using it doesn't require any back-and-forth editing of local.conf or any manual tweaking of recipes. Also, it should be relatively easy to use in a automated build environment (provided that target hw is available). At least one downside is that it's Python-only.
Comment 9 Alejandro Hernandez 2016-10-27 20:47:32 UTC
Great!, it would be nice to see some performance results, comparing the two approaches, another setback of this approach is that it will only be available for python (I know this bug is for python only), whereas the other one could be used for programs coded on other languages as well.
Comment 10 Markus Lehtonen 2016-10-28 08:53:36 UTC
I've run some benchmarks

With simple pystone benchmark (python -m test.pystone 100000) I got about 24% improvement on qemux86 on my i7-3770K desktop:
3750 pystones/second -> 4655 pystones/second

With pybench (python /usr/share/doc/python/Tools/pybench/pybench.py) I got also a significant improvement but I accidentally deleted the data. I'll re-run the benchmark and provide numbers shortly.
Comment 11 Markus Lehtonen 2016-10-28 08:57:34 UTC
(In reply to comment #9)
> another setback of this approach is that it will only be
> available for python (I know this bug is for python only), whereas the other
> one could be used for programs coded on other languages as well.

Yes, the downside is that my solution only optimizes Python.

It would probably make sense to create a separate bug about the autofdo feature. I'd be delighted to see that improved and integrated and would probably spend some time on it. What do you think?
Comment 12 Markus Lehtonen 2016-10-28 18:39:12 UTC
Created attachment 3513 [details]
pybench report on qemux86

Pybench result with non-optimized Python on qemux86
Comment 13 Markus Lehtonen 2016-10-28 18:40:19 UTC
Created attachment 3514 [details]
Pybench report on qemux86 - optimized

Pybench result with optimized Python on qemux86
Comment 14 Markus Lehtonen 2016-10-28 18:44:06 UTC
Pybench shows about 4.6% improvement.

Interestingly, some of the tests are even slower with the optimized Python. I'll try with some other profile target to see if there's difference.
Comment 15 Markus Lehtonen 2016-11-01 14:41:43 UTC
Created attachment 3517 [details]
Pybench report on qemux86 - optimized with pybench target

Pybench results improved dramatically when I used pybench itself as the profile target. Now the improvement is about 13% (up from 4.6%)
Comment 16 Markus Lehtonen 2017-01-31 07:34:39 UTC
Moving this to 2.3 M3 for now.

I have sent RFC patches for review:
https://patchwork.openembedded.org/series/4960/
https://patchwork.openembedded.org/series/5027/
Comment 17 Markus Lehtonen 2017-02-23 08:22:21 UTC
Patchset has been under review for few days now:
https://patchwork.openembedded.org/series/5405/#
Comment 18 Markus Lehtonen 2017-05-18 11:25:46 UTC
v3 of the patchset under review:
https://patchwork.openembedded.org/series/5404/
Comment 20 Markus Lehtonen 2017-10-27 10:28:34 UTC
Rebased the patchset, with some changes:
https://patchwork.openembedded.org/series/9511/
Comment 21 Markus Lehtonen 2017-11-20 12:35:14 UTC
v6 is out
Comment 22 Markus Lehtonen 2018-01-11 11:47:52 UTC
v9 under review
Comment 23 Markus Lehtonen 2018-01-30 13:44:24 UTC
Just sent v10
Comment 24 Ross Burton 2018-09-06 23:20:59 UTC
Merged in oe-core 05a2a53f9cc7e75b4a3838ab9368cadf0f15ba1b.