Bug 7873

Summary: python3-ctypes fails to cross compile
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Peter Bergin <petan679>
Component: devtools / tool chainAssignee: Alejandro Hernandez <alejandro.hernandez>
Status: VERIFIED FIXED QA Contact: Cristina Agurida <cristina-danielax.agurida>
Severity: major    
Priority: Medium+ CC: benjamin.esquivel, cristina-danielax.agurida, jonas.goransson, meta.mr.watcher, meta.watcher, ross.burton
Version: 1.8   
Target Milestone: 1.9 M2   
Hardware: All   
OS: arm   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
patch to fix reconfigure issue in _ctypes none

Description Peter Bergin 2015-06-10 21:40:17 UTC 
Created attachment 2543 [details]
patch to fix reconfigure issue in _ctypes

There is a regression when building python3-ctypes. It worked fine in daisy but in fido and master it is broken. The symptom is that the library for _ctypes in missing in the rpm and on target.

When python3-ctypes is built with the recipes in fido do_configure fails with the printout in do_configure. The build goes on but the _ctypes module is not built.

DEBUG: Executing shell function do_configure
autoreconf: 'configure.ac' or 'configure.in' is required
NOTE: _ctypes failed to autoreconf


The cause of this error is an error with a path in the recipe for python3. With the attached patch that is backported from master the configure step works.

The next stop for the build is in the compile step. When trying to compile _ctypes the file setup.py does not take care of the configure parameter --with-system-ffi. This is because in setup.py the CONFIG_ARGS is taken from sysconfig lib for the host machine.

In some way when building for a x86 target on a x86_64 host the compilation works and the _ctypes library is present in the rpm. But when building for arm on a x86_64 the library for _ctype is not in the rpm.

To reproduce this issue in fido:

> Setup a build for beaglebone
> Apply the attached patch to poky
> bitbake python3-ctypes
> look in the do_compile log for failure building ctypes extension

I am currently using this with Freescla community BSP and my main problem is ctypes. But with this example building for Beaglebone no extension at all seems to be built.
Comment 1 Peter Bergin 2015-06-11 07:54:34 UTC 
More information collected about this and it is regarding the PYTHONPATH when building.

When buildng in Daisy (working) PYTHONPATH is set to:

'PYTHONPATH': '/work/yocto/qep/qep-media-cgt_build/tmp/work/cortexa9hf-vfp-neon-poky-linux-gnueabi/python3/3.3.3-r0.0/Python-3.3.3/build/lib.linux-arm-3.3:./Lib:./Lib/plat-linux'

When building in fido (not working) PYTHONPATH is empty:

'PYTHONPATH': ''

Importing sysconfigdata in fido case will take the host one and in daisy case will take the target one.

How should this work?
Comment 2 Peter Bergin 2015-06-11 08:46:02 UTC 
From tmp/work/cortexa9hf-vfp-neon-poky-linux-gnueabi/python3/3.3.3-r1.0/build/Makefile:

PYTHON_FOR_BUILD=_PYTHON_PROJECT_BASE=$(abs_builddir) _PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) PYTHONPATH=$(CROSSPYTHONPATH) python3.3

It seems that CROSSPYTHONPATH shall be passed in as PYTHONPATH but in my case this one is empty when executing in the Makefile.
Comment 3 Ross Burton 2015-06-11 14:33:37 UTC 
FWIW, this works in master for me.
Comment 4 Peter Bergin 2015-06-11 18:04:08 UTC 
(In reply to comment #3)
> FWIW, this works in master for me.

Which MACHINE are you building against?

With 'works', do you mean that you get your _ctype.so file in your RPM?
Comment 5 Jonas Göransson 2015-06-15 20:14:11 UTC 
The recipe python3_3.3.3.bb applies the patch python3-use-CROSSPYTHONPATH-for-PYTHON_FOR_BUILD.patch:

   -	PYTHON_FOR_BUILD='_PYTHON_PROJECT_BASE=$(abs_builddir) _PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) PYTHONPATH=$(shell test -f pybuilddir.txt && echo $(abs_builddir)/`cat pybuilddir.txt`:)$(srcdir)/Lib:$(srcdir)/Lib/$(PLATDIR) '$interp
   +	PYTHON_FOR_BUILD='_PYTHON_PROJECT_BASE=$(abs_builddir) _PYTHON_HOST_PLATFORM=$(_PYTHON_HOST_PLATFORM) PYTHONPATH=$(CROSSPYTHONPATH) '$interp

This results in the generated python Makefile being dependent on CROSSPYTHONPATH being defined in order to work properly during crosscompilation. Adding the following CROSSPYTHONPATH export to a .bbappend file makes ctypes build successfully without any warnings.

   export CROSSPYTHONPATH = "${B}/build/lib.linux-arm-${PYTHON_MAJMIN}:${S}/Lib:${S}/Lib/plat-linux"

The CROSSPYTHONPATH is the resolved version of the original "$(shell test -f pybuilddir.txt && echo $(abs_builddir)/`cat pybuilddir.txt`:)$(srcdir)/Lib:$(srcdir)/Lib/$(PLATDIR)".

Now setup.py loads the correct sysconfig target specific dicitionary _sysconfigdata.py found in "${B}/build/lib.linux-arm-${PYTHON_MAJMIN}". When CROSSPYTHONPATH is missing it uses the native (x86_64) python sysconfig instead and on my platform fails to build ctypes for arm.
Comment 6 Alejandro Hernandez 2015-06-16 21:58:49 UTC 
I can confirm ctypes fails to compile on arm, hence ctypes*.so is not on the final rpm or target, I will send a patch soon to fix this for all archs on fido and master
Comment 8 Cristina Agurida 2015-06-24 15:25:04 UTC 
Verified on master: 8ef99a00dc75c6eed87aa1bc1528614a2a27eddf