Bug 10522

I found the failures when implementing profile-guided-optimization for Python as test.regrtest is the default profile target being used there.

Fixed in oe-core a8dc4bc039d1c01a78fdc1f727cbfc6f2feffe93.

(In reply to comment #2)
> Fixed in oe-core a8dc4bc039d1c01a78fdc1f727cbfc6f2feffe93.

That was just a very small piece of a fix, fixing import errors but not the actual test case failures.

Sorry, thanks for noticing!

Just for reference, I quickly tried regrtest on python-native and the results are better:

$ tmp-glibc/sysroots/x86_64-linux/usr/bin/nativepython -m test.regrtest -w
...
4 tests failed again:
    test_distutils test_gdb test_urllib test_urllib2_localnet

I'm just not entirely confident that it loads/imports the correct sysroot libraries/modules instead of host system ones in all cases

I've been able to reproduce this issue, showing that the failures are caused by multiple causes. 

Preliminary findings are at the following comment; however, before continuing, I'd like to point out that python-test was dropped from python modules in commit [1] due to being for internal Python developer's use only [2].

Markus: What are your thoughts on the relevance/need for fixing these tests? So far, there are no signs that any of the failures is caused by our building/configuration but mostly due to a required test code refactoring/fixing.


[1] c760d99466c270d18294f5a07ccd91185fa0ef83 (from OE-Core a5bb13a5d7d7a668ca61da6b17884e3b05b95355)

[2] https://docs.python.org/2/library/test.html

Tests ran in qemu with qemux86 core-image-minimal with python-tests at IMAGE_FEATURES_append

Following tests are showing a segmentation fault caused in all cases by the "test_replace_overflow" function:
    test_bytes
    test_string
    test_str
    test_unicode
    test_userstring

Following tests are all showing AssertionError: UnicodeError not raised:
    test_StringIO
    test_builtin
    test_exceptions
    test_gzip
    test_socket
    test_sqlite

Following tests are showing errors that may be also related to encoding:    
    test_ctypes     (AssertionError: TypeError not raised)
    test_traceback  (AssertionError: Lists differ)
    test_getargs    (ValueError: bad marshal data (unknown type code))

Following tests are showing OSError: [Error 12] Cannot allocate memory: 
    test_parser
    test_platform
    test_poll
    test_posix
    test_pty
    test_readline
    test_select
    test_ssl

Other errors:
    test_tuple      (AssertionError: MemoryError not raised)
    test_cmath      (Received value insufficiently close to expected value.)
    test_calendar   (AssertionError))
    test_shutil     (unzip: invalid option -- 't')
    test_warnings   (x18) AttributeError: 'CatchWarningTests' object has no attribute 'module'
                    (x2) AssertionError
    test_distutils  (unable to execute file)
    test_json
    test_pdb
    test_math
    test_multiprocessing
    test_os
    test_pep247
    test_pipes
    test_popen
    test_popen2     (x2)(Exception AttributeError)
    test_pydoc
    test_quopri
    test_resource   (Out of memory)
    test_source_encoding
    test_sysconfig  (AssertionError: /usr/lib/python2.7/config/Makefile) self.assertTrue(os.path.isfile(makefile), makefile)

(In reply to comment #7)
> Tests ran in qemu with qemux86 core-image-minimal with python-tests at
> IMAGE_FEATURES_append
> 
> Following tests are showing a segmentation fault caused in all cases by the
> "test_replace_overflow" function:
>     test_bytes
>     test_string
>     test_str
>     test_unicode
>     test_userstring
> 
> Following tests are all showing AssertionError: UnicodeError not raised:
>     test_StringIO
>     test_builtin
>     test_exceptions
>     test_gzip
>     test_socket
>     test_sqlite
> 
> Following tests are showing errors that may be also related to encoding:    
>     test_ctypes     (AssertionError: TypeError not raised)
>     test_traceback  (AssertionError: Lists differ)
>     test_getargs    (ValueError: bad marshal data (unknown type code))
> 
> Following tests are showing OSError: [Error 12] Cannot allocate memory: 
>     test_parser
>     test_platform
>     test_poll
>     test_posix
>     test_pty
>     test_readline
>     test_select
>     test_ssl
> 
> Other errors:
>     test_tuple      (AssertionError: MemoryError not raised)
>     test_cmath      (Received value insufficiently close to expected value.)
>     test_calendar   (AssertionError))
>     test_shutil     (unzip: invalid option -- 't')
>     test_warnings   (x18) AttributeError: 'CatchWarningTests' object has no
> attribute 'module'
>                     (x2) AssertionError
>     test_distutils  (unable to execute file)
>     test_json
>     test_pdb
>     test_math
>     test_multiprocessing
>     test_os
>     test_pep247
>     test_pipes
>     test_popen
>     test_popen2     (x2)(Exception AttributeError)
>     test_pydoc
>     test_quopri
>     test_resource   (Out of memory)
>     test_source_encoding
>     test_sysconfig  (AssertionError: /usr/lib/python2.7/config/Makefile)
> self.assertTrue(os.path.isfile(makefile), makefile)


Jose,

For the memomry issues, you may need to increase the memory used by the qemu. This is done through extra parameters when launching the image.

(In reply to comment #7)
> Tests ran in qemu with qemux86 core-image-minimal with python-tests at
> IMAGE_FEATURES_append
> 
> Following tests are showing a segmentation fault caused in all cases by the
> "test_replace_overflow" function:
>     test_bytes
>     test_string
>     test_str


Great work narrowing this down Jose, couldyou upload the full test log as an attachment, so we can see the specific errors for each one.

Thanks!

Created attachment 3972 [details]
Logs from tests showing segmentation fault from test_replace_overflow function

I've attached the logs from tests showing segmentation faults from test_replace_overflow function (I'm afraid there's not much information).

Leo: You are right, I ran qemu with 256M of memory and the "OSError: [Error 12] Cannot allocate memory" is gone. Thanks

Regarding the segmentation faults, those are caused by a known issue in Python's overflow checks (see [1]) for which a fix is available in Python 3 but unfortunately there are no plans to backport it to Python 2 (see [2]).

This should add to the question on whether this regression tests are still relevant or not. 

[1] http://bugs.python.org/issue29145

[2] http://bugs.python.org/issue29145#msg285022

(In reply to comment #12)
> Regarding the segmentation faults, those are caused by a known issue in
> Python's overflow checks (see [1]) for which a fix is available in Python 3
> but unfortunately there are no plans to backport it to Python 2 (see [2]).
> 
> This should add to the question on whether this regression tests are still
> relevant or not. 
> 
> [1] http://bugs.python.org/issue29145
> 
> [2] http://bugs.python.org/issue29145#msg285022

So this is a know issue at upstream, so I suggest to resolved it as wontfix unless you think the patch is trivial to backport.

(In reply to comment #12)
> Regarding the segmentation faults, those are caused by a known issue in
> Python's overflow checks (see [1]) for which a fix is available in Python 3
> but unfortunately there are no plans to backport it to Python 2 (see [2]).
> 
> This should add to the question on whether this regression tests are still
> relevant or not. 
> 
> [1] http://bugs.python.org/issue29145
> 
> [2] http://bugs.python.org/issue29145#msg285022

I tried to reproduce it on my computer and it froze beyond repair, I had to manually turn it off, to me that pretty much settles it, this is an upstream issue, and while the ptest package may be useful to us in some ways, we can't base our work on those tests.

We may still want to try and fix the Unicode errors.

I further investigated the cause for the encode-related failures to discard any error related to our build process:

All of the regression tests are based on the assumption that the default target encoding is 'ascii', whereas Yocto-build images use 'utf-8' as default encoding for Python 2 (defined in sitecustomize.py). This causes the unexpected behavior.

For example: the expected outcome in test_StringIO.TestcStringIO.test_unicode is a UnicodeEncodeError when the cStringIO module tries to convert Unicode strings to character strings in the default (ascii assumed) encoding. Evidently, in our case there is no encoding error since both the input string and the default encoding (used by Python's str() function) are 'utf-8'.

So, these encoding errors would also need the regression tests to be re-written/refactored.

(In reply to comment #15)
> I further investigated the cause for the encode-related failures to discard
> any error related to our build process:
> 
> All of the regression tests are based on the assumption that the default
> target encoding is 'ascii', whereas Yocto-build images use 'utf-8' as
> default encoding for Python 2 (defined in sitecustomize.py). This causes the
> unexpected behavior.


Jose, I believe using setdefaultencoding() is discouraged, this will be fixed on M1 with the new manifest patches, can you please check what happens if you remove the setting from sitecustomize.py?, do the errors go away?

Thanks

Confirmed that all of the encoding-related errors are no longer present when the image is created WITHOUT the setdefaultencoding() setting in sitecustomize.py

Following are the summarized findings for the reported behavior:

    - python-tests module was dropped from  python-modules [1], so it must
      be explicitly added to an image in order to be available.

    - Some of the failures are only present when executing the tests in a qemu
      emulated machine and are caused by its inherent limited resources.
      For example: executing the tests in qemu with 128 MB of memory causes at
      least 8 test failures due to memory that can not be allocated.

    - When executed on real hardware (Minnowboard Turbot 32 bits), failures
      can be divided into four main causes:

      1 - Segmentation fault caused by a known issue in Python's replace_*
          function [2]. There are no plans to fix this for Python 2.7.

      2 - Failures related to Unicode encoding. These are caused by the default
          encoding used in Yocto-build images ('utf-8') being different from
          the assumed 'ascii' by the tests. This will no longer be the case
          after 2.5 M1 milestone, when manifest patches that remove the
          mentioned default encoding are scheduled to be implemented.

      3 - Other:
          both test_distutils and test_sysconfig failure due to dev-related
          resources not available in limited-size images (gcc compiler and
          makefiles, respectively). These failures are not present when
          executing tests in SDK images or when tools-sdk is added as an extra
          image feature.

          test_shutil failure due to invalid option in BusyBox's unzip. This
          is not the case when using an SDK image.

          test_tuple failure due to __mul__ operator not raising MemoryError
          when using a large multiplier. This may be the result of an
          optimized memory management implemented after the test was developed.
          
After these findings, recommendation is to run the regression tests under the
following configuration:

      - Use an SDK image (for example: core-image-sato-sdk) or add 'tools-sdk'
        to EXTRA_IMAGE_FEATURES

      - Ensure default encoding is 'ascii'

      - Exclude the tests that use the "replace_*" function and test_tuple
        test.
        (i.e.: use '--exclude' option followed by
         'test_bytes test_string test_str test_tuple')

For all of the above, and given that any of the failures is directly caused by
our building system, I'm closing this bug as "won't fix".
      
[1] http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=c760d99466c270d18294f5a07ccd91185fa0ef83

[2] https://bugs.python.org/issue29145

(In reply to comment #18)
> Following are the summarized findings for the reported behavior:

your report is pretty conclusive so not much to add. BTW, on all this exploration did you instruct bitbake to use the 'profile-guided-optimization' when compiling the interpreter? and did you run the ptests under python3? I believe this wont change the bugzilla status, but just wondering  if output from the latter may be different.