Python test.regrtest is broken. First of all, the package dependencies are incorrect. After getting around that, some tests segfault: <pre> root@qemux86:~# for t in test_bytes test_str test_string test_tuple test_unicode test_userstring; do python -m test.regrtest $t; done [1/1] test_bytes [ 98.185825] python[376]: segfault at b719e000 ip 4269cad2 sp bfea34f4 error 7 in libc-2.24.so[42625000+175000] Segmentation fault [1/1] test_str [ 113.812423] python[381]: segfault at b7080000 ip 4269cac4 sp bfee0594 error 7 in libc-2.24.so[42625000+175000] Segmentation fault [1/1] test_string [ 127.986049] python[384]: segfault at b7178000 ip 4269caea sp bfeecf64 error 7 in libc-2.24.so[42625000+175000] Segmentation fault [1/1] test_tuple [ 129.136838] python[385]: segfault at b7732000 ip 428a8ebc sp bfdab840 error 6 in libpython2.7.so.1.0[42824000+1a3000] Segmentation fault [1/1] test_unicode [ 148.463642] python[386]: segfault at 82d6010 ip 4269cab9 sp bfc7b5b4 error 4 in libc-2.24.so[42625000+175000] Segmentation fault [1/1] test_userstring [ 179.787759] python[387]: segfault at b709b000 ip 4269cac4 sp bf9c2a54 error 7 in libc-2.24.so[42625000+175000] Segmentation fault </pre> Some other tests fail more gracefully: <pre> python -m test.regrtest test_StringIO test_builtin test_calendar test_cmath test_ctypes test_distutils test_exceptions test_getargs test_gzip test_json test_math test_shutil test_socket test_sqlite test_traceback test_warnings [ 1/16] test_StringIO test test_StringIO failed -- Traceback (most recent call last): File "/usr/lib/python2.7/test/test_StringIO.py", line 197, in test_unicode self.assertRaises(UnicodeEncodeError, self.MODULE.StringIO, u'\xf4') AssertionError: UnicodeEncodeError not raised [ 2/16/1] test_builtin test test_builtin failed -- multiple errors occurred; run in verbose mode for details [ 3/16/2] test_calendar test test_calendar failed -- Traceback (most recent call last): File "/usr/lib/python2.7/test/test_calendar.py", line 212, in test_output_htmlcalendar result_2004_html.strip() AssertionError: '<?xml version="1.0" encoding="utf8"?>\n<!DOCTYPE html PUBLIC "-// ... [ 4/16/3] test_cmath test test_cmath failed -- Traceback (most recent call last): File "/usr/lib/python2.7/test/test_cmath.py", line 352, in test_specific_values msg=error_message) File "/usr/lib/python2.7/test/test_cmath.py", line 112, in rAssertAlmostEqual '{!r} and {!r} are not sufficiently close'.format(a, b)) AssertionError: acos0034: acos(complex(-1.0000000000000002, 0.0)) Expected: complex(3.1415926535897931, -2.1073424255447014e-08) Received: complex(3.1415926535897931, -2.1073424338879928e-08) Received value insufficiently close to expected value. [ 5/16/4] test_ctypes test test_ctypes failed -- Traceback (most recent call last): File "/usr/lib/python2.7/ctypes/test/test_structures.py", line 258, in test_invalid_name self.assertRaises(TypeError, declare_with_name, u"x\xe9") AssertionError: TypeError not raised [ 6/16/5] test_distutils unable to execute 'i586-poky-linux-gcc': No such file or directory unable to execute 'i586-poky-linux-gcc': No such file or directory unable to execute 'i586-poky-linux-gcc': No such file or directory unable to execute 'i586-poky-linux-gcc': No such file or directory Warning -- os.environ was modified by test_distutils test test_distutils failed -- multiple errors occurred; run in verbose mode for details [ 7/16/6] test_exceptions test test_exceptions failed -- multiple errors occurred; run in verbose mode for details [ 8/16/7] test_getargs test test_getargs failed -- Traceback (most recent call last): File "/usr/lib/python2.7/test/test_getargs.py", line 25, in test_with_marshal self.assertRaises(UnicodeError, marshal.loads, arg) File "/usr/lib/python2.7/unittest/case.py", line 473, in assertRaises callableObj(*args, **kwargs) ValueError: bad marshal data (unknown type code) [ 9/16/8] test_gzip test test_gzip failed -- Traceback (most recent call last): File "/usr/lib/python2.7/test/test_gzip.py", line 84, in test_write_incompatible_type f.write(u'\xff') AssertionError: UnicodeEncodeError not raised [10/16/9] test_json t test_json failed -- multiple errors occurred; run in verbose mode for details [11/16/10] test_math test test_math failed -- multiple errors occurred; run in verbose mode for details [12/16/11] test_shutil test test_shutil failed -- Traceback (most recent call last): File "/usr/lib/python2.7/test/test_shutil.py", line 535, in test_unzip_zipfile self.fail(msg.format(exc, details)) AssertionError: Command '['unzip', '-t', '/tmp/tmpdGc2VR/archive.zip']' returned non-zero exit status 1 **Unzip Output** unzip: invalid option -- 't' BusyBox v1.24.1 (2016-10-28 14:03:27 EEST) multi-call binary. Usage: unzip [-lnopq] FILE[.zip] [FILE]... [-x FILE...] [-d DIR] [13/16/12] test_socket test test_socket failed -- Traceback (most recent call last): File "/usr/lib/python2.7/test/test_socket.py", line 309, in testSendtoErrors s.sendto(u'\u2620', sockname) AssertionError: UnicodeEncodeError not raised [14/16/13] test_sqlite test test_sqlite failed -- Traceback (most recent call last): File "/usr/lib/python2.7/sqlite3/test/regression.py", line 158, in CheckSetIsolationLevel "isolation_level", u"\xe9") AssertionError: UnicodeEncodeError not raised [15/16/14] test_traceback test test_traceback failed -- Traceback (most recent call last): File "/usr/lib/python2.7/test/test_traceback.py", line 179, in test_unicode self.assertEqual(lines, ['AssertionError: \\xe9\n']) AssertionError: Lists differ: ['AssertionError: \xc3\xa9\n'] != ['AssertionError: \\xe9\n'] First differing element 0: 'AssertionError: \xc3\xa9\n' 'AssertionError: \\xe9\n' - ['AssertionError: \xc3\xa9\n'] ? ^^^^^ + ['AssertionError: \\xe9\n'] ? + ^ [16/16/15] test_warnings /usr/lib/python2.7/unittest/case.py:503: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal if not first == second: test test_warnings failed -- multiple errors occurred; run in verbose mode for details 16 tests failed: test_StringIO test_builtin test_calendar test_cmath test_ctypes test_distutils test_exceptions test_getargs test_gzip test_json test_math test_shutil test_socket test_sqlite test_traceback test_warnings </pre> The above lists all the tests that are failing on qemux86 in my case. Other tests succeed.
I found the failures when implementing profile-guided-optimization for Python as test.regrtest is the default profile target being used there.
Fixed in oe-core a8dc4bc039d1c01a78fdc1f727cbfc6f2feffe93.
(In reply to comment #2) > Fixed in oe-core a8dc4bc039d1c01a78fdc1f727cbfc6f2feffe93. That was just a very small piece of a fix, fixing import errors but not the actual test case failures.
Sorry, thanks for noticing!
Just for reference, I quickly tried regrtest on python-native and the results are better: $ tmp-glibc/sysroots/x86_64-linux/usr/bin/nativepython -m test.regrtest -w ... 4 tests failed again: test_distutils test_gdb test_urllib test_urllib2_localnet I'm just not entirely confident that it loads/imports the correct sysroot libraries/modules instead of host system ones in all cases
I've been able to reproduce this issue, showing that the failures are caused by multiple causes. Preliminary findings are at the following comment; however, before continuing, I'd like to point out that python-test was dropped from python modules in commit [1] due to being for internal Python developer's use only [2]. Markus: What are your thoughts on the relevance/need for fixing these tests? So far, there are no signs that any of the failures is caused by our building/configuration but mostly due to a required test code refactoring/fixing. [1] c760d99466c270d18294f5a07ccd91185fa0ef83 (from OE-Core a5bb13a5d7d7a668ca61da6b17884e3b05b95355) [2] https://docs.python.org/2/library/test.html
Tests ran in qemu with qemux86 core-image-minimal with python-tests at IMAGE_FEATURES_append Following tests are showing a segmentation fault caused in all cases by the "test_replace_overflow" function: test_bytes test_string test_str test_unicode test_userstring Following tests are all showing AssertionError: UnicodeError not raised: test_StringIO test_builtin test_exceptions test_gzip test_socket test_sqlite Following tests are showing errors that may be also related to encoding: test_ctypes (AssertionError: TypeError not raised) test_traceback (AssertionError: Lists differ) test_getargs (ValueError: bad marshal data (unknown type code)) Following tests are showing OSError: [Error 12] Cannot allocate memory: test_parser test_platform test_poll test_posix test_pty test_readline test_select test_ssl Other errors: test_tuple (AssertionError: MemoryError not raised) test_cmath (Received value insufficiently close to expected value.) test_calendar (AssertionError)) test_shutil (unzip: invalid option -- 't') test_warnings (x18) AttributeError: 'CatchWarningTests' object has no attribute 'module' (x2) AssertionError test_distutils (unable to execute file) test_json test_pdb test_math test_multiprocessing test_os test_pep247 test_pipes test_popen test_popen2 (x2)(Exception AttributeError) test_pydoc test_quopri test_resource (Out of memory) test_source_encoding test_sysconfig (AssertionError: /usr/lib/python2.7/config/Makefile) self.assertTrue(os.path.isfile(makefile), makefile)
(In reply to comment #7) > Tests ran in qemu with qemux86 core-image-minimal with python-tests at > IMAGE_FEATURES_append > > Following tests are showing a segmentation fault caused in all cases by the > "test_replace_overflow" function: > test_bytes > test_string > test_str > test_unicode > test_userstring > > Following tests are all showing AssertionError: UnicodeError not raised: > test_StringIO > test_builtin > test_exceptions > test_gzip > test_socket > test_sqlite > > Following tests are showing errors that may be also related to encoding: > test_ctypes (AssertionError: TypeError not raised) > test_traceback (AssertionError: Lists differ) > test_getargs (ValueError: bad marshal data (unknown type code)) > > Following tests are showing OSError: [Error 12] Cannot allocate memory: > test_parser > test_platform > test_poll > test_posix > test_pty > test_readline > test_select > test_ssl > > Other errors: > test_tuple (AssertionError: MemoryError not raised) > test_cmath (Received value insufficiently close to expected value.) > test_calendar (AssertionError)) > test_shutil (unzip: invalid option -- 't') > test_warnings (x18) AttributeError: 'CatchWarningTests' object has no > attribute 'module' > (x2) AssertionError > test_distutils (unable to execute file) > test_json > test_pdb > test_math > test_multiprocessing > test_os > test_pep247 > test_pipes > test_popen > test_popen2 (x2)(Exception AttributeError) > test_pydoc > test_quopri > test_resource (Out of memory) > test_source_encoding > test_sysconfig (AssertionError: /usr/lib/python2.7/config/Makefile) > self.assertTrue(os.path.isfile(makefile), makefile) Jose, For the memomry issues, you may need to increase the memory used by the qemu. This is done through extra parameters when launching the image.
(In reply to comment #7) > Tests ran in qemu with qemux86 core-image-minimal with python-tests at > IMAGE_FEATURES_append > > Following tests are showing a segmentation fault caused in all cases by the > "test_replace_overflow" function: > test_bytes > test_string > test_str Great work narrowing this down Jose, couldyou upload the full test log as an attachment, so we can see the specific errors for each one. Thanks!
Created attachment 3972 [details] Logs from tests showing segmentation fault from test_replace_overflow function
I've attached the logs from tests showing segmentation faults from test_replace_overflow function (I'm afraid there's not much information). Leo: You are right, I ran qemu with 256M of memory and the "OSError: [Error 12] Cannot allocate memory" is gone. Thanks
Regarding the segmentation faults, those are caused by a known issue in Python's overflow checks (see [1]) for which a fix is available in Python 3 but unfortunately there are no plans to backport it to Python 2 (see [2]). This should add to the question on whether this regression tests are still relevant or not. [1] http://bugs.python.org/issue29145 [2] http://bugs.python.org/issue29145#msg285022
(In reply to comment #12) > Regarding the segmentation faults, those are caused by a known issue in > Python's overflow checks (see [1]) for which a fix is available in Python 3 > but unfortunately there are no plans to backport it to Python 2 (see [2]). > > This should add to the question on whether this regression tests are still > relevant or not. > > [1] http://bugs.python.org/issue29145 > > [2] http://bugs.python.org/issue29145#msg285022 So this is a know issue at upstream, so I suggest to resolved it as wontfix unless you think the patch is trivial to backport.
(In reply to comment #12) > Regarding the segmentation faults, those are caused by a known issue in > Python's overflow checks (see [1]) for which a fix is available in Python 3 > but unfortunately there are no plans to backport it to Python 2 (see [2]). > > This should add to the question on whether this regression tests are still > relevant or not. > > [1] http://bugs.python.org/issue29145 > > [2] http://bugs.python.org/issue29145#msg285022 I tried to reproduce it on my computer and it froze beyond repair, I had to manually turn it off, to me that pretty much settles it, this is an upstream issue, and while the ptest package may be useful to us in some ways, we can't base our work on those tests. We may still want to try and fix the Unicode errors.
I further investigated the cause for the encode-related failures to discard any error related to our build process: All of the regression tests are based on the assumption that the default target encoding is 'ascii', whereas Yocto-build images use 'utf-8' as default encoding for Python 2 (defined in sitecustomize.py). This causes the unexpected behavior. For example: the expected outcome in test_StringIO.TestcStringIO.test_unicode is a UnicodeEncodeError when the cStringIO module tries to convert Unicode strings to character strings in the default (ascii assumed) encoding. Evidently, in our case there is no encoding error since both the input string and the default encoding (used by Python's str() function) are 'utf-8'. So, these encoding errors would also need the regression tests to be re-written/refactored.
(In reply to comment #15) > I further investigated the cause for the encode-related failures to discard > any error related to our build process: > > All of the regression tests are based on the assumption that the default > target encoding is 'ascii', whereas Yocto-build images use 'utf-8' as > default encoding for Python 2 (defined in sitecustomize.py). This causes the > unexpected behavior. Jose, I believe using setdefaultencoding() is discouraged, this will be fixed on M1 with the new manifest patches, can you please check what happens if you remove the setting from sitecustomize.py?, do the errors go away? Thanks
Confirmed that all of the encoding-related errors are no longer present when the image is created WITHOUT the setdefaultencoding() setting in sitecustomize.py
Following are the summarized findings for the reported behavior: - python-tests module was dropped from python-modules [1], so it must be explicitly added to an image in order to be available. - Some of the failures are only present when executing the tests in a qemu emulated machine and are caused by its inherent limited resources. For example: executing the tests in qemu with 128 MB of memory causes at least 8 test failures due to memory that can not be allocated. - When executed on real hardware (Minnowboard Turbot 32 bits), failures can be divided into four main causes: 1 - Segmentation fault caused by a known issue in Python's replace_* function [2]. There are no plans to fix this for Python 2.7. 2 - Failures related to Unicode encoding. These are caused by the default encoding used in Yocto-build images ('utf-8') being different from the assumed 'ascii' by the tests. This will no longer be the case after 2.5 M1 milestone, when manifest patches that remove the mentioned default encoding are scheduled to be implemented. 3 - Other: both test_distutils and test_sysconfig failure due to dev-related resources not available in limited-size images (gcc compiler and makefiles, respectively). These failures are not present when executing tests in SDK images or when tools-sdk is added as an extra image feature. test_shutil failure due to invalid option in BusyBox's unzip. This is not the case when using an SDK image. test_tuple failure due to __mul__ operator not raising MemoryError when using a large multiplier. This may be the result of an optimized memory management implemented after the test was developed. After these findings, recommendation is to run the regression tests under the following configuration: - Use an SDK image (for example: core-image-sato-sdk) or add 'tools-sdk' to EXTRA_IMAGE_FEATURES - Ensure default encoding is 'ascii' - Exclude the tests that use the "replace_*" function and test_tuple test. (i.e.: use '--exclude' option followed by 'test_bytes test_string test_str test_tuple') For all of the above, and given that any of the failures is directly caused by our building system, I'm closing this bug as "won't fix". [1] http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=c760d99466c270d18294f5a07ccd91185fa0ef83 [2] https://bugs.python.org/issue29145
(In reply to comment #18) > Following are the summarized findings for the reported behavior: your report is pretty conclusive so not much to add. BTW, on all this exploration did you instruct bitbake to use the 'profile-guided-optimization' when compiling the interpreter? and did you run the ptests under python3? I believe this wont change the bugzilla status, but just wondering if output from the latter may be different.