Bug 12627 - python3 distutils INCLUDEPY native wrong path
Summary: python3 distutils INCLUDEPY native wrong path
Status: RESOLVED INVALID
Alias: None
Product: Runtime
Classification: Build System, Metadata & Runtime
Component: build-appliance (show other bugs)
Version: 2.5
Hardware: x86 Multiple
: Undecided normal
Target Milestone: ---
Assignee: Juro Bystricky
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2018-03-26 05:02 UTC by luzik
Modified: 2018-03-26 15:59 UTC (History)
0 users

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description luzik 2018-03-26 05:02:38 UTC
STAGING_INCDIR=/opt/oe-core4/build/tmp/work/armv7vehf-neon-poky-linux-gnueabi/python-pyside2/5.9-r0/recipe-sysroot/usr/include STAGING_LIBDIR=/opt/oe-core4/build/tmp/work/armv7vehf-neon-poky-linux-gnueabi/python-pyside2/5.9-r0/recipe-sysroot/usr/lib /opt/oe-core4/build/tmp/work/armv7vehf-neon-poky-linux-gnueabi/python-pyside2/5.9-r0/recipe-sysroot-native/usr/bin/python3-native/python3
Python 3.5.3 (default, Mar 21 2018, 17:25:54) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import distutils.sysconfig
>>> distutils.sysconfig.get_config_var('INCLUDEPY')
'/python3.5m'
>>> distutils.sysconfig.get_config_var('LIBDIR')
'/opt/oe-core4/build/tmp/work/armv7vehf-neon-poky-linux-gnueabi/python-pyside2/5.9-r0/recipe-sysroot/usr/lib'
Comment 1 Juro Bystricky 2018-03-26 08:17:38 UTC
Can you please document exact steps to reproduce the problem?
The more details, the better, not entirely sure what to make of this bug report.
(I am a bit puzzled with the GCC being of version 5.4.)
Comment 2 luzik 2018-03-26 09:57:34 UTC
You can try build wayland or any other package contains 'recipe-sysroot/usr/lib/python3.5/config-3.5m/Makefile'. Problem is that this Makefile sets INCLUDEPY=  $(INCLUDEDIR)/python$(LDVERSION)
And if there is no INCLUDEDIR ENV defined then python distutils return wrong value. In my special case pyside2 using this to setup build env.
Maybe poky/meta/classes/distutils3.bbclass should contains INCLUDEDIR next to STAGING_INCDIR and STAGING_LIBDIR ?

Was it done to manipulate INCLUDEDIR between recipe-sysroot/ and recipe-sysroot-native/  ? 

gcc5.4 is my system compiler. Should that python3-native also be recompiled using internal gcc ?

Sorry for a mess in my messages. Yocto is quite new for me again and I am still trying to get into it.
Comment 3 Juro Bystricky 2018-03-26 10:39:04 UTC
(In reply to comment #2)
> You can try build wayland or any other package contains
> 'recipe-sysroot/usr/lib/python3.5/config-3.5m/Makefile'. Problem is that
> this Makefile sets INCLUDEPY=  $(INCLUDEDIR)/python$(LDVERSION)
> And if there is no INCLUDEDIR ENV defined then python distutils return wrong
> value. In my special case pyside2 using this to setup build env.
> Maybe poky/meta/classes/distutils3.bbclass should contains INCLUDEDIR next
> to STAGING_INCDIR and STAGING_LIBDIR ?
> 
> Was it done to manipulate INCLUDEDIR between recipe-sysroot/ and
> recipe-sysroot-native/  ? 
> 
> gcc5.4 is my system compiler. Should that python3-native also be recompiled
> using internal gcc ?
> 
> Sorry for a mess in my messages. Yocto is quite new for me again and I am
> still trying to get into it.

I'm sorry, I am still a bit confused about the nature of the problem.
Are you using the BuildAppliance for your builds? (The bug report is filed BuildAppliance)
Which Yocto distro are you using/ What is your Build Configuration?
What are you trying to build/achieve? (Please list all your exact steps, so I can reproduce the problem!).
Comment 4 luzik 2018-03-26 11:03:08 UTC
Oh, I missunderstud what BuildAppliance is. Could you please help me move this to correct one ?

I'm using Rocko.
I'm trying to cross compile Pyside2 on my Karo-TXUL (imx6ul) board using customized machine configuration, so it will be quite hard to fully reproduce, but I do believe that my issue is not related to any specific MACHINE so..

1. Create env with poky, meta-qt5, meta-clang.
2. Create python-pyside2_5.9.bb with:
LICENSE = "LGPLv2.1"
LIC_FILES_CHKSUM = "file://LICENSE.LGPLv21;md5=fb91571854638f10b2e5f36562661a5a"
SRC_URI = "git://code.qt.io/pyside/pyside-setup.git;tag=b09fde6260b255e8b93b0d20a337e701bc940a99;branch=${PV} \
        file://cmake_params.patch"

EXTRA_OECMAKE_prepend = " \
    -DOE_QMAKE_PATH_EXTERNAL_HOST_BINS=${OE_QMAKE_PATH_EXTERNAL_HOST_BINS} \
"
DISTUTILS_BUILD_ARGS = "--qmake=${OE_QMAKE_PATH_EXTERNAL_HOST_BINS}/qmake --ignore-git --cmake-params='${EXTRA_OECMAKE}'"
inherit distro_features_check qmake5_base cmake-native clang setuptools3
S = "${WORKDIR}/git"
B = "${S}"
TOOLCHAIN = "clang"
DEPENDS = "qtbase qtbase-native python3-setuptools-native libxslt-native python3-numpy-native qtxmlpatterns-native libxml2-native icu-native python3-native"
REQUIRED_DISTRO_FEATURES = "x11"

and cmake_params.patch:

diff --git a/setup.py b/setup.py
index ffb561c..734100a 100644
--- a/setup.py
+++ b/setup.py
@@ -257,6 +257,8 @@ OPTION_RELWITHDEBINFO = has_option('relwithdebinfo')
 OPTION_QMAKE = option_value("qmake")
 OPTION_QT_VERSION = option_value("qt")
 OPTION_CMAKE = option_value("cmake")
+OPTION_CMAKE_PARAMS = option_value("cmake-params")
+OPTION_INCLUDE_PY = option_value("include-py")
 OPTION_OPENSSL = option_value("openssl")
 OPTION_ONLYPACKAGE = has_option("only-package")
 OPTION_STANDALONE = has_option("standalone")
@@ -669,6 +671,8 @@ class pyside_build(_build):
                 py_libdir = os.path.join(py_prefix, "libs")
             else:
                 py_libdir = os.path.join(py_prefix, "lib")
+        if OPTION_INCLUDE_PY:
+            py_include_dir = OPTION_INCLUDE_PY
         if py_include_dir is None or not os.path.exists(py_include_dir):
             if sys.platform == "win32":
                 py_include_dir = os.path.join(py_prefix, "include")
@@ -1019,6 +1023,9 @@ class pyside_build(_build):
             "-DCMAKE_INSTALL_PREFIX=%s" % self.install_dir,
             module_src_dir
         ]
+        if OPTION_CMAKE_PARAMS:
+            for extra_cmake_param in OPTION_CMAKE_PARAMS.split():
+                cmake_cmd.append(extra_cmake_param.strip())
         cmake_cmd.append("-DPYTHON_EXECUTABLE=%s" % self.py_executable)
         cmake_cmd.append("-DPYTHON_INCLUDE_DIR=%s" % self.py_include_dir)
         cmake_cmd.append("-DPYTHON_LIBRARY=%s" % self.py_library)


and as You can see in this patch I have to pass extra include-py to properly configure python include directory.
Comment 5 Juro Bystricky 2018-03-26 15:59:29 UTC
(In reply to comment #4)
> Oh, I missunderstud what BuildAppliance is. Could you please help me move
> this to correct one ?
> 
> I'm using Rocko.
> I'm trying to cross compile Pyside2 on my Karo-TXUL (imx6ul) board using
> customized machine configuration, so it will be quite hard to fully
> reproduce, but I do believe that my issue is not related to any specific
> MACHINE so..
> 
> 1. Create env with poky, meta-qt5, meta-clang.
> 2. Create python-pyside2_5.9.bb with:
> LICENSE = "LGPLv2.1"
> LIC_FILES_CHKSUM =
> "file://LICENSE.LGPLv21;md5=fb91571854638f10b2e5f36562661a5a"
> SRC_URI =
> "git://code.qt.io/pyside/pyside-setup.git;
> tag=b09fde6260b255e8b93b0d20a337e701bc940a99;branch=${PV} \
>         file://cmake_params.patch"
> 
> EXTRA_OECMAKE_prepend = " \
>     -DOE_QMAKE_PATH_EXTERNAL_HOST_BINS=${OE_QMAKE_PATH_EXTERNAL_HOST_BINS} \
> "
> DISTUTILS_BUILD_ARGS = "--qmake=${OE_QMAKE_PATH_EXTERNAL_HOST_BINS}/qmake
> --ignore-git --cmake-params='${EXTRA_OECMAKE}'"
> inherit distro_features_check qmake5_base cmake-native clang setuptools3
> S = "${WORKDIR}/git"
> B = "${S}"
> TOOLCHAIN = "clang"
> DEPENDS = "qtbase qtbase-native python3-setuptools-native libxslt-native
> python3-numpy-native qtxmlpatterns-native libxml2-native icu-native
> python3-native"
> REQUIRED_DISTRO_FEATURES = "x11"
> 
> and cmake_params.patch:
> 
> diff --git a/setup.py b/setup.py
> index ffb561c..734100a 100644
> --- a/setup.py
> +++ b/setup.py
> @@ -257,6 +257,8 @@ OPTION_RELWITHDEBINFO = has_option('relwithdebinfo')
>  OPTION_QMAKE = option_value("qmake")
>  OPTION_QT_VERSION = option_value("qt")
>  OPTION_CMAKE = option_value("cmake")
> +OPTION_CMAKE_PARAMS = option_value("cmake-params")
> +OPTION_INCLUDE_PY = option_value("include-py")
>  OPTION_OPENSSL = option_value("openssl")
>  OPTION_ONLYPACKAGE = has_option("only-package")
>  OPTION_STANDALONE = has_option("standalone")
> @@ -669,6 +671,8 @@ class pyside_build(_build):
>                  py_libdir = os.path.join(py_prefix, "libs")
>              else:
>                  py_libdir = os.path.join(py_prefix, "lib")
> +        if OPTION_INCLUDE_PY:
> +            py_include_dir = OPTION_INCLUDE_PY
>          if py_include_dir is None or not os.path.exists(py_include_dir):
>              if sys.platform == "win32":
>                  py_include_dir = os.path.join(py_prefix, "include")
> @@ -1019,6 +1023,9 @@ class pyside_build(_build):
>              "-DCMAKE_INSTALL_PREFIX=%s" % self.install_dir,
>              module_src_dir
>          ]
> +        if OPTION_CMAKE_PARAMS:
> +            for extra_cmake_param in OPTION_CMAKE_PARAMS.split():
> +                cmake_cmd.append(extra_cmake_param.strip())
>          cmake_cmd.append("-DPYTHON_EXECUTABLE=%s" % self.py_executable)
>          cmake_cmd.append("-DPYTHON_INCLUDE_DIR=%s" % self.py_include_dir)
>          cmake_cmd.append("-DPYTHON_LIBRARY=%s" % self.py_library)
> 
> 
> and as You can see in this patch I have to pass extra include-py to properly
> configure python include directory.

Thanks, I see now. You created your own recipe and ran into the problem. So definitely a misfiled bug, nothing to do with BuildAppliance. The recipe uses meta-qt5, so I would contact meta-qt5 developers regarding building pyside-setup. Also, please note that there are various python recipes in meta/recipes-devtools/python you can consult. Generally, the heavy lifting is done by "inherit setuptools3", it should set up all needed dependencies and paths. 

I am going to close this bug report as RESOLVED/INVALID as Yocto does not maintain meta-qt5.

However, if you feel strongly this is truly an oe-core bug, you can re-open it under oe-core category. (but be prepared to demonstrate exact steps to reproduce the problem.)