1937 2012-01-27 02:16:00 +0000 oe-core build of python 2.7.2 is broken 2012-06-04 07:20:44 +0000 1 1 7 Build System, Metadata & Runtime OE-Core devtools / tool chain unspecified x86 x86_64 RESOLVED FIXED Medium major 1.2.1 1 slapinid andrei andrei meta.layer.watcher meta.mr.watcher meta.watcher nitin.a.kamble obi scott.a.garman sgw --- --- --- oldest_to_newest 18393 0 slapinid 2012-01-27 02:16:00 +0000 Hi, oe-core build of python 2.7.2 is broken regarding the following modules: Python build finished, but the necessary bits to build these modules were not found: _bsddb _curses _curses_panel _tkinter bsddb185 dbm dl imageop linuxaudiodev ossaudiodev sunaudiodev To find the necessary bits, look in setup.py in detect_modules() for the module's name. This is 100% reproducible for more than 2 months with build from scratch on my build system at least. This results in almost NO C-based built in modules in python rendering python almost useless. resulting python-misc: dpkg-deb -c ~/slave/angstrom_build/build/build/tmp-angstrom_2010_x-eglibc/deploy/ipk/armv5te/python-misc_2.7.2-r0.5_armv5te.ipk drwxr-xr-x root/root 0 2012-01-26 19:23 ./ drwxr-xr-x root/root 0 2012-01-26 19:18 ./usr/ drwxr-xr-x root/root 0 2012-01-26 19:18 ./usr/lib/ drwxr-xr-x root/root 0 2012-01-26 19:18 ./usr/lib/python2.7/ -rw-r--r-- root/root 5467 2012-01-26 19:16 ./usr/lib/python2.7/macpath.pyo -rw-r--r-- root/root 11805 2012-01-26 19:14 ./usr/lib/python2.7/ast.py -rw-r--r-- root/root 1218 2012-01-26 19:16 ./usr/lib/python2.7/this.pyo drwxr-xr-x root/root 0 2012-01-26 19:18 ./usr/lib/python2.7/importlib/ -rw-r--r-- root/root 1142 2012-01-26 19:16 ./usr/lib/python2.7/importlib/__init__.pyo -rw-r--r-- root/root 1327 2012-01-26 19:16 ./usr/lib/python2.7/importlib/__init__.py -rw-r--r-- root/root 87802 2012-01-26 19:14 ./usr/lib/python2.7/argparse.py -rw-r--r-- root/root 17452 2012-01-26 19:16 ./usr/lib/python2.7/asyncore.pyo drwxr-xr-x root/root 0 2012-01-26 19:18 ./usr/lib/python2.7/site-packages/ -rw-r--r-- root/root 119 2012-01-26 19:14 ./usr/lib/python2.7/site-packages/README -rw-r--r-- root/root 24283 2012-01-26 19:14 ./usr/lib/python2.7/modulefinder.py -rw-r--r-- root/root 3658 2012-01-26 19:16 ./usr/lib/python2.7/Bastion.pyo -rw-r--r-- root/root 5968 2012-01-26 19:16 ./usr/lib/python2.7/timeit.pyo -rw-r--r-- root/root 10698 2012-01-26 19:14 ./usr/lib/python2.7/runpy.py -rw-r--r-- root/root 25764 2012-01-26 19:14 ./usr/lib/python2.7/imputil.py -rw-r--r-- root/root 4603 2012-01-26 19:14 ./usr/lib/python2.7/os2emxpath.py -rw-r--r-- root/root 18082 2012-01-26 19:14 ./usr/lib/python2.7/ntpath.py -rw-r--r-- root/root 3572 2012-01-26 19:16 ./usr/lib/python2.7/os2emxpath.pyo -rw-r--r-- root/root 2371 2012-01-26 19:14 ./usr/lib/python2.7/nturl2path.py -rw-r--r-- root/root 7216 2012-01-26 19:16 ./usr/lib/python2.7/filecmp.pyo -rw-r--r-- root/root 60 2012-01-26 19:14 ./usr/lib/python2.7/antigravity.py -rw-r--r-- root/root 4576 2012-01-26 19:14 ./usr/lib/python2.7/netrc.py -rw-r--r-- root/root 16392 2012-01-26 19:16 ./usr/lib/python2.7/plistlib.pyo -rw-r--r-- root/root 6840 2012-01-26 19:16 ./usr/lib/python2.7/runpy.pyo -rw-r--r-- root/root 8783 2012-01-26 19:16 ./usr/lib/python2.7/fileinput.pyo -rw-r--r-- root/root 20650 2012-01-26 19:14 ./usr/lib/python2.7/asyncore.py -rwxr-xr-x root/root 15021 2012-01-26 19:14 ./usr/lib/python2.7/mimify.py -rw-r--r-- root/root 6733 2012-01-26 19:16 ./usr/lib/python2.7/pyclbr.pyo -rw-r--r-- root/root 1477 2012-01-26 19:16 ./usr/lib/python2.7/nturl2path.pyo -rw-r--r-- root/root 26782 2012-01-26 19:16 ./usr/lib/python2.7/mhlib.pyo -rw-r--r-- root/root 6252 2012-01-26 19:14 ./usr/lib/python2.7/macpath.py -rw-r--r-- root/root 7427 2012-01-26 19:14 ./usr/lib/python2.7/mailcap.py -rw-r--r-- root/root 7518 2012-01-26 19:14 ./usr/lib/python2.7/symtable.py -rw-r--r-- root/root 12293 2012-01-26 19:16 ./usr/lib/python2.7/fractions.pyo -rw-r--r-- root/root 17500 2012-01-26 19:16 ./usr/lib/python2.7/ihooks.pyo -rw-r--r-- root/root 2138 2012-01-26 19:16 ./usr/lib/python2.7/sched.pyo -rw-r--r-- root/root 777 2012-01-26 19:16 ./usr/lib/python2.7/user.pyo -rw-r--r-- root/root 14143 2012-01-26 19:14 ./usr/lib/python2.7/fileinput.py -rw-r--r-- root/root 187 2012-01-26 19:14 ./usr/lib/python2.7/wsgiref.egg-info drwxr-xr-x root/root 0 2012-01-26 19:18 ./usr/lib/python2.7/lib-dynload/ -rw-r--r-- root/root 1524 2012-01-26 19:14 ./usr/lib/python2.7/lib-dynload/Python-2.7.2-py2.7.egg-info -rwxr-xr-x root/root 40428 2012-01-26 19:18 ./usr/lib/python2.7/lib-dynload/_testcapi.so -rwxr-xr-x root/root 4900 2012-01-26 19:18 ./usr/lib/python2.7/lib-dynload/future_builtins.so -rwxr-xr-x root/root 6960 2012-01-26 19:18 ./usr/lib/python2.7/lib-dynload/spwd.so -rw-r--r-- root/root 15176 2012-01-26 19:14 ./usr/lib/python2.7/plistlib.py drwxr-xr-x root/root 0 2012-01-26 19:18 ./usr/lib/python2.7/wsgiref/ -rw-r--r-- root/root 5224 2012-01-26 19:16 ./usr/lib/python2.7/wsgiref/simple_server.pyo -rw-r--r-- root/root 13120 2012-01-26 19:16 ./usr/lib/python2.7/wsgiref/validate.pyo -rw-r--r-- root/root 5879 2012-01-26 19:16 ./usr/lib/python2.7/wsgiref/headers.py -rw-r--r-- root/root 5576 2012-01-26 19:16 ./usr/lib/python2.7/wsgiref/util.py -rw-r--r-- root/root 127 2012-01-26 19:16 ./usr/lib/python2.7/wsgiref/__init__.pyo -rw-r--r-- root/root 14731 2012-01-26 19:16 ./usr/lib/python2.7/wsgiref/validate.py -rw-r--r-- root/root 15932 2012-01-26 19:16 ./usr/lib/python2.7/wsgiref/handlers.py -rw-r--r-- root/root 10370 2012-01-26 19:16 ./usr/lib/python2.7/wsgiref/handlers.pyo -rw-r--r-- root/root 4743 2012-01-26 19:16 ./usr/lib/python2.7/wsgiref/simple_server.py -rw-r--r-- root/root 586 2012-01-26 19:16 ./usr/lib/python2.7/wsgiref/__init__.py -rw-r--r-- root/root 4187 2012-01-26 19:16 ./usr/lib/python2.7/wsgiref/headers.pyo -rw-r--r-- root/root 4372 2012-01-26 19:16 ./usr/lib/python2.7/wsgiref/util.pyo -rw-r--r-- root/root 13282 2012-01-26 19:14 ./usr/lib/python2.7/pyclbr.py -rw-r--r-- root/root 3052 2012-01-26 19:16 ./usr/lib/python2.7/contextlib.pyo -rw-r--r-- root/root 33434 2012-01-26 19:14 ./usr/lib/python2.7/mhlib.py -rw-r--r-- root/root 7297 2012-01-26 19:16 ./usr/lib/python2.7/ast.pyo -rw-r--r-- root/root 16909 2012-01-26 19:16 ./usr/lib/python2.7/webbrowser.pyo -rw-r--r-- root/root 64 2012-01-26 19:14 ./usr/lib/python2.7/__phello__.foo.py -rw-r--r-- root/root 7264 2012-01-26 19:16 ./usr/lib/python2.7/tabnanny.pyo -rw-r--r-- root/root 4699 2012-01-26 19:14 ./usr/lib/python2.7/fpformat.py -rw-r--r-- root/root 14231 2012-01-26 19:16 ./usr/lib/python2.7/LICENSE.txt -rw-r--r-- root/root 9252 2012-01-26 19:16 ./usr/lib/python2.7/ntpath.pyo -rw-r--r-- root/root 21704 2012-01-26 19:14 ./usr/lib/python2.7/webbrowser.py -rw-r--r-- root/root 23948 2012-01-26 19:16 ./usr/lib/python2.7/aifc.pyo -rw-r--r-- root/root 22390 2012-01-26 19:14 ./usr/lib/python2.7/fractions.py -rw-r--r-- root/root 3275 2012-01-26 19:14 ./usr/lib/python2.7/macurl2path.py -rw-r--r-- root/root 5590 2012-01-26 19:16 ./usr/lib/python2.7/mailcap.pyo -rw-r--r-- root/root 520 2012-01-26 19:16 ./usr/lib/python2.7/statvfs.pyo -rw-r--r-- root/root 123 2012-01-26 19:16 ./usr/lib/python2.7/__phello__.foo.pyo -rw-r--r-- root/root 12059 2012-01-26 19:14 ./usr/lib/python2.7/timeit.py -rw-r--r-- root/root 9503 2012-01-26 19:14 ./usr/lib/python2.7/filecmp.py -rw-r--r-- root/root 5091 2012-01-26 19:14 ./usr/lib/python2.7/sched.py -rw-r--r-- root/root 4381 2012-01-26 19:16 ./usr/lib/python2.7/multifile.pyo -rw-r--r-- root/root 14739 2012-01-26 19:16 ./usr/lib/python2.7/binhex.pyo -rw-r--r-- root/root 10901 2012-01-26 19:16 ./usr/lib/python2.7/symtable.pyo -rw-r--r-- root/root 3677 2012-01-26 19:16 ./usr/lib/python2.7/fpformat.pyo -rw-r--r-- root/root 11402 2012-01-26 19:14 ./usr/lib/python2.7/asynchat.py -rw-r--r-- root/root 12050 2012-01-26 19:16 ./usr/lib/python2.7/imputil.pyo -rw-r--r-- root/root 12073 2012-01-26 19:14 ./usr/lib/python2.7/cgitb.py -rw-r--r-- root/root 54799 2012-01-26 19:16 ./usr/lib/python2.7/argparse.pyo -rw-r--r-- root/root 16353 2012-01-26 19:16 ./usr/lib/python2.7/rexec.pyo -rw-r--r-- root/root 20184 2012-01-26 19:16 ./usr/lib/python2.7/trace.pyo -rw-r--r-- root/root 2376 2012-01-26 19:16 ./usr/lib/python2.7/macurl2path.pyo -rw-r--r-- root/root 498 2012-01-26 19:14 ./usr/lib/python2.7/dbhash.py drwxr-xr-x root/root 0 2012-01-26 19:18 ./usr/lib/python2.7/plat-linux3/ -rw-r--r-- root/root 195 2012-01-26 19:16 ./usr/lib/python2.7/plat-linux3/regen -rwxr-xr-x root/root 11336 2012-01-26 19:14 ./usr/lib/python2.7/tabnanny.py -rw-r--r-- root/root 633 2012-01-26 19:16 ./usr/lib/python2.7/dbhash.pyo -rw-r--r-- root/root 20148 2012-01-26 19:14 ./usr/lib/python2.7/rexec.py -rw-r--r-- root/root 201 2012-01-26 19:16 ./usr/lib/python2.7/antigravity.pyo -rw-r--r-- root/root 33246 2012-01-26 19:14 ./usr/lib/python2.7/aifc.py -rw-r--r-- root/root 8003 2012-01-26 19:14 ./usr/lib/python2.7/posixfile.py -rw-r--r-- root/root 4424 2012-01-26 19:14 ./usr/lib/python2.7/contextlib.py -rw-r--r-- root/root 4820 2012-01-26 19:14 ./usr/lib/python2.7/multifile.py -rw-r--r-- root/root 898 2012-01-26 19:14 ./usr/lib/python2.7/statvfs.py -rw-r--r-- root/root 5549 2012-01-26 19:16 ./usr/lib/python2.7/posixfile.pyo -rw-r--r-- root/root 1002 2012-01-26 19:14 ./usr/lib/python2.7/this.py -rw-r--r-- root/root 17706 2012-01-26 19:16 ./usr/lib/python2.7/modulefinder.pyo -rw-r--r-- root/root 10475 2012-01-26 19:16 ./usr/lib/python2.7/cgitb.pyo -rw-r--r-- root/root 3649 2012-01-26 19:16 ./usr/lib/python2.7/netrc.pyo -rw-r--r-- root/root 30156 2012-01-26 19:14 ./usr/lib/python2.7/trace.py -rw-r--r-- root/root 18986 2012-01-26 19:14 ./usr/lib/python2.7/ihooks.py -rw-r--r-- root/root 7200 2012-01-26 19:16 ./usr/lib/python2.7/asynchat.pyo -rw-r--r-- root/root 1627 2012-01-26 19:14 ./usr/lib/python2.7/user.py -rw-r--r-- root/root 14476 2012-01-26 19:14 ./usr/lib/python2.7/binhex.py -rw-r--r-- root/root 10582 2012-01-26 19:16 ./usr/lib/python2.7/mimify.pyo -rw-r--r-- root/root 5744 2012-01-26 19:14 ./usr/lib/python2.7/Bastion.py so basic modules like IN, _curses, etc. are missing. Logs for build are available at http://ossfans.org/py/ but I don't see any errors there. Any help is appreciated, I can't see where I can go from here. All the best, S. 18397 1 slapinid 2012-01-27 04:19:33 +0000 python-native fails the same way. Compiling native python on this machine without OE produces no such errors. 18398 2 slapinid 2012-01-27 05:36:19 +0000 08:34 < slapin> if I symlink Lib/plat-linux2 to Lib/plat-linux3 most of stuff is installed properly. 08:34 < slapin> that is for native 08:34 < slapin> but I don't know if it is proper way 08:35 < slapin> probably another simlink is needed in package 08:35 < slapin> or renaming post-processing is needed 18399 3 slapinid 2012-01-27 06:13:22 +0000 This patch fixes this problem for me: diff --git a/meta/recipes-devtools/python/python-native_2.7.2.bb b/meta/recipes-devtools/python/python-native_2.7.2.bb index 8bf0af0..a882e9a 100644 --- a/meta/recipes-devtools/python/python-native_2.7.2.bb +++ b/meta/recipes-devtools/python/python-native_2.7.2.bb @@ -1,6 +1,6 @@ require python.inc DEPENDS = "openssl-native bzip2-full-native zlib-native readline-native sqlite3-native" -PR = "${INC_PR}.3" +PR = "${INC_PR}.4" SRC_URI += "file://04-default-is-optimized.patch \ file://05-enable-ctypes-cross-build.patch \ @@ -26,6 +26,7 @@ EXTRA_OEMAKE = '\ LIBC="" \ STAGING_LIBDIR=${STAGING_LIBDIR_NATIVE} \ STAGING_INCDIR=${STAGING_INCDIR_NATIVE} \ + MACHDEP=linux2 \ ' do_configure_prepend() { diff --git a/meta/recipes-devtools/python/python_2.7.2.bb b/meta/recipes-devtools/python/python_2.7.2.bb index 654c850..035d38b 100644 --- a/meta/recipes-devtools/python/python_2.7.2.bb +++ b/meta/recipes-devtools/python/python_2.7.2.bb @@ -1,7 +1,7 @@ require python.inc DEPENDS = "python-native db gdbm openssl readline sqlite3 zlib" DEPENDS_sharprom = "python-native db readline zlib gdbm openssl" -PR = "${INC_PR}.5" +PR = "${INC_PR}.6" DISTRO_SRC_URI ?= "file://sitecustomize.py" DISTRO_SRC_URI_linuxstdbase = "" @@ -23,6 +23,9 @@ SRC_URI += "\ " S = "${WORKDIR}/Python-${PV}" +EXTRA_OEMAKE += ' \ + MACHDEP=linux2 \ + ' inherit autotools 18400 4 nitin.a.kamble 2012-01-27 08:57:41 +0000 Hi Sergey, Good to see you also working on the fix. Is the issue fixed now ? Nitin 18402 5 slapinid 2012-01-27 09:16:57 +0000 Not yet. Some of modules are still not built properly. This patch solves only part of the problem. ncurses modules and others are still not built while there's no obvious reason for them not to. 18403 6 nitin.a.kamble 2012-01-27 09:43:56 +0000 Sergey, so what part is fixed by your patch ? Nitin 18404 7 slapinid 2012-01-27 09:57:41 +0000 'import IN' works now., modules in Lib/plat-linux2 directory are installed and function properly. 18407 8 nitin.a.kamble 2012-01-27 11:46:25 +0000 Sergey: You can send a commit pull request on the OE-core mailing list for your patch. Nitin 18952 9 obi 2012-03-10 02:07:37 +0000 I've sent a patch adressing part of this issue to the OE-Core mailing list. See http://patches.openembedded.org/patch/22993/ "dl" and "imageop" still fail on 64-bit build systems, because setup.py uses the build system's word size to perform some autodetection instead of the target word size. I haven't investigated the issue with the _curses* modules. Regards, Andreas 18954 10 scott.a.garman 2012-03-10 02:40:06 +0000 I'm working on a fix for the curses issue and should have a fix ready by Monday. See bug #2070. 19386 11 andrei 2012-03-23 09:59:34 +0000 I can reproduce but with fewer modules: Python build finished, but the necessary bits to build these modules were not found: _bsddb _tkinter bsddb185........ dbm sunaudiodev........................ To find the necessary bits, look in setup.py in detect_modules() for the module's name. 19419 12 andrei 2012-03-23 15:46:05 +0000 bsddb - in work _tkinter - in work bsddb185 - in work dbm - done sunaudiodev - not needed as it is SunOS specific module 19441 13 andrei 2012-03-23 23:02:23 +0000 bsddb - not needed as it is a darwin specific module _tkinter - tk needed and integrated bsddb185 - Is is a module specific for Berkeley db 1.85 and it's again platfrom specific (osf1 / darwin) dbm - done sunaudiodev - not needed as it is SunOS specific module Patches here: http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=ag/master For the darwin / sun specific modules i can push a patch to avoid these warnings but i don't know if this is wanted. 19442 14 nitin.a.kamble 2012-03-23 23:27:03 +0000 Andrei, Thanks for great work! And is there any update for this issue ? "dl" and "imageop" still fail on 64-bit build systems, because setup.py uses the build system's word size to perform some autodetection instead of the target word size. Nitin 19447 15 andrei 2012-03-24 10:35:02 +0000 Thank you. I don't have a 64b machine right now but i will setup one today. So i will fix those too. 19454 16 andrei 2012-03-25 00:44:55 +0000 bsddb - not needed as it is a darwin specific module _tkinter - tk needed and integrated bsddb185 - It is a module specific for Berkeley db 1.85 and it's again platfrom specific (osf1 / darwin) dbm - done sunaudiodev - not needed as it is SunOS specific module dl - done imageop - done These modules are now built on 32bit and 64bit host machines. Thanks to Nitin Kamble whose hint was the actual fix. Sergey Lapin, as you reported this issue, would you be so kind and test these fixes? Commits: http://lists.linuxtogo.org/pipermail/openembedded-core/2012-March/019978.html Thanks, @g 20019 17 sgw 2012-04-06 17:54:59 +0000 Partial Fixes: http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=0dacda90b98b2e66055427e9dead08b40188b8a9