Bug 14466 - python: Should we add this optimization: -fno-semantic-interposition for 1.3x speed improvment?
Summary: python: Should we add this optimization: -fno-semantic-interposition for 1.3x...
Status: RESOLVED WONTFIX
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: core (show other bugs)
Version: 3.4
Hardware: x86 Multiple
: Medium+ enhancement
Target Milestone: 4.3 M2
Assignee: Trevor Gamblin
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2021-07-01 16:01 UTC by Randy MacLeod
Modified: 2023-10-23 20:45 UTC (History)
6 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Don't know


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Randy MacLeod 2021-07-01 16:01:21 UTC
The Fedora packaging has been modified to compile libpython with -fno-semantic-interposition flag: it makes Python up to 1.3x faster without having to touch any line of the C code! Should we do the same?
   https://bugs.python.org/issue38980

From a recent build, it seems that flag is not used:

$ rgrep no-semantic-interposition tmp-glibc/work/core2-64-wrs-linux/python3/3.9.2-r0
$ echo "No hits"

Trevor has expressed interest in this but I didn't want it to get lost so I've created this enhancement.
Comment 1 Richard Purdie 2021-07-01 21:40:24 UTC
You may want to read https://stackoverflow.com/questions/35745543/new-option-in-gcc-5-3-fno-semantic-interposition first and then see whether visibility settings might be safer?
Comment 2 Richard Purdie 2021-07-02 06:28:58 UTC
It is probably ok to enable this for an individual recipe, we just need to comment why and explain tit means library intercept/replacement won't work.
Comment 3 Randy MacLeod 2021-07-08 14:38:33 UTC
It will break things if we do it on python-native so only sdk and target.
Comment 4 Trevor Gamblin 2023-06-20 11:57:01 UTC
python3 ptest durations without the flag added to the recipe:

root@qemux86-64:~# grep -r "Total duration"
python3-ptest-time-1.log:Total duration: 22 min 37 sec
python3-ptest-time-2.log:Total duration: 24 min
python3-ptest-time-4.log:Total duration: 23 min 17 sec
python3-ptest-time-5.log:Total duration: 22 min 33 sec
python3-ptest-time-3.log:Total duration: 30 min 3 sec
root@qemux86-64:~# 

python3 ptest durations with -fno-semantic-interposition added to CPPFLAGS:

root@qemux86-64:~# grep -r "Total duration"
python3-ptest-with-opt-2.log:Total duration: 22 min 5 sec
python3-ptest-with-opt-4.log:Total duration: 22 min 30 sec
python3-ptest-with-opt-1.log:Total duration: 22 min 11 sec
python3-ptest-with-opt-3.log:Total duration: 22 min 36 sec
python3-ptest-with-opt-5.log:Total duration: 22 min 14 sec
root@qemux86-64:~# 

This doesn't seem like a worthwhile change, but I'll keep looking to see if it manifests more meaningfully in other areas.
Comment 5 Khem Raj 2023-06-20 18:19:33 UTC
I agree, gains seems minimal

I think python 3.13 is doing some useful work in this regard

https://github.com/faster-cpython/ideas/blob/main/3.13/README.md

but it wont be available until 2024
Comment 6 Trevor Gamblin 2023-06-21 13:48:26 UTC
Closing this for now as it doesn't seem to provide a meaningful gain to target builds.
Comment 7 Randy MacLeod 2023-06-21 14:32:13 UTC
Thanks for checking on this again Trevor.

I don't expect to see much of a difference in ptest results.
It would be interesting to see if there's a difference in https://github.com/python/pyperformance as mentioned in:
https://fedoraproject.org/wiki/Changes/PythonNoSemanticInterpositionSpeedup

but you likely have better things to do...
Comment 8 Randy MacLeod 2023-06-21 15:57:51 UTC
In case you or someone else decides to test pyperformance, note that it is part of the  phoronix-test-suite but I don't know if it's build by default.
There's current isn't a pyperformance recipe.

tmp-glibc/work/core2-64-wrs-linux/phoronix-test-suite/10.8.4-r0/phoronix-test-suite/ob-cache/test-profiles/pts/pyperformance-1.0.1
Comment 9 Trevor Gamblin 2023-06-21 18:55:06 UTC
It doesn't appear to be installed with phoronix by default. I wrote a standalone recipe for pyperformance, but after getting all of the dependencies sorted, along with providing enough extra storage and a nameserver in /etc/resolv.conf (part of the benchmark setup involves creating a venv and installing packages with pip, even if you have the dependencies in the image), the benchmark ran for a while and then stalled. Not sure what to make of it.
Comment 10 Khem Raj 2023-10-23 20:45:07 UTC
gcc has enabled this option now as part of -Ofast with

https://gcc.gnu.org/git/?p=gcc.git;a=commit;h=458d2c689963d8461d84670a3d8988cd6ecbfd81

Therefore it would be interesting to see if we change SELECTED_OPTIMIZATIONS from -O2 to -Ofast for python and cpython recipes, we might get some interesting results.