Bug 11513

Summary: Python3 isn't multilib compatible
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Ross Burton <ross.burton>
Component: devtools / tool chainAssignee: Ross Burton <ross.burton>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium+ CC: alex.kanavin, leonardo.sandoval.gonzalez, meta.mr.watcher, meta.watcher, philip, randy.macleod, sgw
Version: 2.3   
Target Milestone: 2.5 M2   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Bug Depends on: 11472    
Bug Blocks:    

Description Ross Burton 2017-05-16 15:12:42 UTC
If you build an image with python3-core and lib64-python3-core you get file conflicts and the rootfs won't build (whereas python-core and lib64-python-core work together).

This is because /usr/bin/python3-config contains lines such as:

CFLAGS=$(echo " -O2 -pipe -g -feliminate-unused-debug-types -fdebug-prefix-map=/data/poky-master/tmp/work/x86_64-pokymllib64-linux/lib6
4-python3/3.5.3-r1.0=/usr/src/debug/lib64-python3/3.5.3-r1.0 -fdebug-prefix-map=/data/poky-master/tmp/work/x86_64-pokymllib64-linux/lib
64-python3/3.5.3-r1.0/recipe-sysroot-native= -fdebug-prefix-map=/data/poky-master/tmp/work/x86_64-pokymllib64-linux/lib64-python3/3.5.3
-r1.0/lib64-recipe-sysroot=  -fstack-protector-strong -D_FORTIFY_SOURCE=2 -Wformat -Wformat-security -Werror=format-security" | sed "s#
$prefix_build#$prefix_real#")

So the files differ, and rpm won't overlay them.
Comment 1 Alejandro Hernandez 2017-05-16 17:01:10 UTC
Hey Ross,

Could you specify what your multilib configuration is, I am indeed getting an error on rootfs, but I also get it with python-core and lib64-python-core, so I'm assuming I screwed up when setting multilib.
Comment 2 Ross Burton 2017-05-16 17:28:40 UTC
Okay, if you're seeing python2 do the same then that's also a bug, I assumed it worked as the AB broke when we moved opkg-utils to py3 from py2.

master will have no py2 in shortly, so lets just sort out py3.
Comment 3 Alejandro Hernandez 2017-05-24 22:14:41 UTC
I still don't believe I have the correct configuration, since I am getting a different error.

This is the configuration I've got:
CORE_IMAGE_EXTRA_INSTALL += "lib64-python-core python-core"
require conf/multilib.conf
MULTILIBS = "multilib:lib64"
DEFAULTTUNE_virtclass-multilib-lib64 = "x86-64"


These are the errors I am getting (along with many others):

file /usr/lib64/python2.7/types.pyc conflicts between attempted installs of lib64-python-core-2.7.13-r1.x86_64 and python-core-2.7.13-r1.core2_64
  file /usr/lib64/python2.7/warnings.pyc conflicts between attempted installs of lib64-python-core-2.7.13-r1.x86_64 and python-core-2.7.13-r1.core2_64
  file /lib64/libtinfo.so.5.9 conflicts between attempted installs of libtinfo5-6.0+20161126-r0.core2_64 and lib64-libtinfo5-6.0+20161126-r0.x86_64
  file /usr/lib64/libhistory.so.7.0 conflicts between attempted installs of libreadline7-7.0-r0.core2_64 and lib64-libreadline7-7.0-r0.x86_64
  file /usr/lib64/libreadline.so.7.0 conflicts between attempted installs of libreadline7-7.0-r0.core2_64 and lib64-libreadline7-7.0-r0.x86_64
  file /usr/lib64/python2.7/lib-dynload/readline.so conflicts between attempted installs of lib64-python-readline-2.7.13-r1.x86_64 and python-readline-2.7.13-r1.core2_64
  file /usr/lib64/python2.7/rlcompleter.pyc conflicts between attempted installs of lib64-python-readline-2.7.13-r1.x86_64 and python-readline-2.7.13-r1.core2_64



I am not sure what the behavior should be here, but it looks to me that both lib64-python-core and python-core are the same package (because of my multilib configuration), but they are considered different packages by rpm.
Comment 4 Ross Burton 2017-06-19 22:20:42 UTC
*** Bug 11690 has been marked as a duplicate of this bug. ***
Comment 5 Leonardo Sandoval Gonzalez 2017-07-17 15:35:17 UTC
With the following configuration (which I believe is that Ross used)

IMAGE_INSTALL_append = "lib64-python3-core python3-core"
require conf/multilib.conf
MULTILIBS = "multilib:lib64"
DEFAULTTUNE_virtclass-multilib-lib64 = "x86-64"

This is what I am getting this error

ERROR: Could not invoke dnf. Command 'TOPDIR/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/recipe-sysroot-native/usr/bin/dnf -y -c TOPDIR/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/rootfs/etc/dnf/dnf.conf --setopt=reposdir=TOPDIR/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/rootfs/etc/yum.repos.d --repofrompath=oe-repo,TOPDIR/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo --installroot=TOPDIR/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/rootfs --setopt=logdir=TOPDIR/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/temp --nogpgcheck install lib64-python3-core packagegroup-core-boot python3-core run-postinsts' returned 1:
Unable to detect release version (use '--releasever' to specify release version)
Added oe-repo repo from TOPDIR/tmp/work/qemux86-poky-linux/core-image-minimal/1.0-r0/oe-rootfs-repo


Full log at: http://errors.yoctoproject.org/Errors/Details/147659/
Comment 6 Alexander Kanavin 2017-07-17 15:46:43 UTC
Would it be okay to move python3-config to the -dev package, as it's only for developers?
Comment 7 Leonardo Sandoval Gonzalez 2017-07-17 19:41:56 UTC
(In reply to comment #6)
> Would it be okay to move python3-config to the -dev package, as it's only
> for developers?

but in this case, the problem is just moved to another package, right?
Comment 8 Leonardo Sandoval Gonzalez 2017-07-17 20:33:57 UTC
(In reply to comment #0)
> If you build an image with python3-core and lib64-python3-core you get file
> conflicts and the rootfs won't build (whereas python-core and
> lib64-python-core work together).
> 
> This is because /usr/bin/python3-config contains lines such as:
> 
> CFLAGS=$(echo " -O2 -pipe -g -feliminate-unused-debug-types
> -fdebug-prefix-map=/data/poky-master/tmp/work/x86_64-pokymllib64-linux/lib6
> 4-python3/3.5.3-r1.0=/usr/src/debug/lib64-python3/3.5.3-r1.0
> -fdebug-prefix-map=/data/poky-master/tmp/work/x86_64-pokymllib64-linux/lib
> 64-python3/3.5.3-r1.0/recipe-sysroot-native=
> -fdebug-prefix-map=/data/poky-master/tmp/work/x86_64-pokymllib64-linux/lib64-
> python3/3.5.3
> -r1.0/lib64-recipe-sysroot=  -fstack-protector-strong -D_FORTIFY_SOURCE=2
> -Wformat -Wformat-security -Werror=format-security" | sed "s#
> $prefix_build#$prefix_real#")
> 
> So the files differ, and rpm won't overlay them.

Right Ross. The exact diff is the following

 diff -Naur i586-poky-linux/python3/3.5.3-r1.0/packages-split/python3-core/usr/bin/python3.5m-config x86_64-pokymllib64-linux/lib64-python3/3.5.3-r1.0/packages-split/lib64-python3-core/usr/bin/python3.5m-config
--- i586-poky-linux/python3/3.5.3-r1.0/packages-split/python3-core/usr/bin/python3.5m-config    2017-07-17 12:48:23.061720905 -0700
+++ x86_64-pokymllib64-linux/lib64-python3/3.5.3-r1.0/packages-split/lib64-python3-core/usr/bin/python3.5m-config       2017-07-17 12:49:24.691221100 -0700
@@ -33,8 +33,8 @@
 exec_prefix_build="/usr"
 exec_prefix=$(echo "$exec_prefix_build" | sed "s#$exec_prefix_build#$prefix_real#")
 includedir=$(echo "/usr/include" | sed "s#$prefix_build#$prefix_real#")
-libdir=$(echo "/usr/lib" | sed "s#$prefix_build#$prefix_real#")
-CFLAGS=$(echo " -O2 -pipe -g -feliminate-unused-debug-types -fdebug-prefix-map=/home/lsandov1/poky-testing/poky-python3-multilib/build-p3/tmp/work/i586-poky-linux/python3/3.5.3-r1.0=/usr/src/debug/python3/3.5.3-r1.0 -fdebug-prefix-map=/home/lsandov1/poky-testing/poky-python3-multilib/build-p3/tmp/work/i586-poky-linux/python3/3.5.3-r1.0/recipe-sysroot-native= -fdebug-prefix-map=/home/lsandov1/poky-testing/poky-python3-multilib/build-p3/tmp/work/i586-poky-linux/python3/3.5.3-r1.0/recipe-sysroot= " | sed "s#$prefix_build#$prefix_real#")
+libdir=$(echo "/usr/lib64" | sed "s#$prefix_build#$prefix_real#")
+CFLAGS=$(echo " -O2 -pipe -g -feliminate-unused-debug-types -fdebug-prefix-map=/home/lsandov1/poky-testing/poky-python3-multilib/build-p3/tmp/work/x86_64-pokymllib64-linux/lib64-python3/3.5.3-r1.0=/usr/src/debug/lib64-python3/3.5.3-r1.0 -fdebug-prefix-map=/home/lsandov1/poky-testing/poky-python3-multilib/build-p3/tmp/work/x86_64-pokymllib64-linux/lib64-python3/3.5.3-r1.0/recipe-sysroot-native= -fdebug-prefix-map=/home/lsandov1/poky-testing/poky-python3-multilib/build-p3/tmp/work/x86_64-pokymllib64-linux/lib64-python3/3.5.3-r1.0/lib64-recipe-sysroot= " | sed "s#$prefix_build#$prefix_real#")
 VERSION="3.5"
 LIBM="-lm"
 LIBC=""
@@ -49,7 +49,7 @@
 LDVERSION="${VERSION}${ABIFLAGS}"
 LIBDEST=${prefix}/lib/python${VERSION}
 LIBPL=$(echo "${LIBDIR}/python3.5/config-${VERSION}${ABIFLAGS}" | sed "s#$prefix_build#$prefix_real#")
-SO=".cpython-35m-i386-linux-gnu.so"
+SO=".cpython-35m-x86_64-linux-gnu.so"
 PYTHONFRAMEWORK=""
 INCDIR="-I$includedir/python${VERSION}${ABIFLAGS}"
 PLATINCDIR="-I$includedir/python${VERSION}${ABIFLAGS}"
Comment 9 Leonardo Sandoval Gonzalez 2017-07-24 16:24:46 UTC
(In reply to comment #6)
> Would it be okay to move python3-config to the -dev package, as it's only
> for developers?

Alex, after doing some review on this area, your suggestion is what I sent to the mailing list. Thanks for the tip.

http://lists.openembedded.org/pipermail/openembedded-core/2017-July/140066.html
Comment 10 Leonardo Sandoval Gonzalez 2017-08-02 21:12:42 UTC
I proposed a solution to fix this problem [1] but according to Alex, this wont fit the new model to be proposed soon. So, assigning this task to him so he can latter integrate the pieces I worked on.


[1] http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=lsandov1/python3-manifest
Comment 11 Alejandro Hernandez 2017-08-28 21:44:01 UTC
Integrated with the new manifest, but the changes were moved to 2.5 per Richard's request.
Comment 12 Ross Burton 2018-02-20 20:13:29 UTC
Apparently fixed with the new manifest code so merged in oe-core 54ac820b8a639950ccb534dcd9d6eaf8b2b736e0.