Bug 12788

Summary:	[2.4.3.rc2] python ptest failed
Product:	[Build System, Metadata & Runtime] OE-Core	Reporter:	yeoh ee peng <ee.peng.yeoh>
Component:	core	Assignee:	Anuj Mittal <anuj.mittal>
Status:	RESOLVED FIXED	QA Contact:
Severity:	normal
Priority:	Medium+	CC:	manish.das, meta.mr.watcher, meta.watcher, randy.macleod, ross.burton
Version:	2.6
Target Milestone:	2.6 M4
Hardware:	x86
OS:	Multiple
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	Regression (Found by QA test)
Verified:		Documentation change:	No (bug/feature does not impact docs)

Description yeoh ee peng 2018-06-14 07:17:54 UTC

Perform diff and found that python tests (test_compile_files, test_magic_number, test_mtime, and more)  that passing in 2.4.2.rc2 are failing on 2.4.3.rc2

--- a/python (2.4.2.rc2)
+++ b/python (2.4.3.rc2)

-PASS: test_compile_files (test.test_compileall.CompileallTests)
-PASS: test_magic_number (test.test_compileall.CompileallTests)
-PASS: test_mtime (test.test_compileall.CompileallTests)
+FAIL: test_compile_files (test.test_compileall.CompileallTests)
+FAIL: test_magic_number (test.test_compileall.CompileallTests)
+FAIL: test_mtime (test.test_compileall.CompileallTests)
-PASS: test_http2time (test.test_cookielib.DateTimeTests)
-PASS: test_http2time_formats (test.test_cookielib.DateTimeTests)
+FAIL: test_http2time (test.test_cookielib.DateTimeTests)
+FAIL: test_http2time_formats (test.test_cookielib.DateTimeTests)

-PASS: test_check_hostname (test.test_ftplib.TestTLS_FTPClass)
-PASS: test_context (test.test_ftplib.TestTLS_FTPClass)
-PASS: test_control_connection (test.test_ftplib.TestTLS_FTPClass)
-PASS: test_data_connection (test.test_ftplib.TestTLS_FTPClass)
-PASS: test_login (test.test_ftplib.TestTLS_FTPClass)
+FAIL: test_check_hostname (test.test_ftplib.TestTLS_FTPClass)
+FAIL: test_context (test.test_ftplib.TestTLS_FTPClass)
+FAIL: test_control_connection (test.test_ftplib.TestTLS_FTPClass)
+FAIL: test_data_connection (test.test_ftplib.TestTLS_FTPClass)
+FAIL: test_login (test.test_ftplib.TestTLS_FTPClass)

-PASS: test_basics (test.test_import.PycRewritingTests)
-PASS: test_foreign_code (test.test_import.PycRewritingTests)
-PASS: test_incorrect_code_name (test.test_import.PycRewritingTests)
+FAIL: test_basics (test.test_import.PycRewritingTests)
+FAIL: test_foreign_code (test.test_import.PycRewritingTests)
+FAIL: test_incorrect_code_name (test.test_import.PycRewritingTests)

-PASS: test_read_write_after_close_raises_valuerror (test.test_ssl.ThreadedTests)
+FAIL: test_read_write_after_close_raises_valuerror (test.test_ssl.ThreadedTests)
 PASS: A brutal shutdown of an SSL server should raise an OSError
-PASS: test_server_accept (test.test_ssl.ThreadedTests)
+FAIL: test_server_accept (test.test_ssl.ThreadedTests)

-PASS: test_date (test.test_strptime.StrptimeTests)
+FAIL: test_date (test.test_strptime.StrptimeTests)


-PASS: test_year (test.test_strptime.StrptimeTests)
+FAIL: test_year (test.test_strptime.StrptimeTests)

-PASS: test_redirect_no_path (test.test_urllib2.HandlerTests)
+FAIL: test_redirect_no_path (test.test_urllib2.HandlerTests)

-PASS: test_basic_auth_success (test.test_urllib2_localnet.BasicAuthTests)
+FAIL: test_basic_auth_success (test.test_urllib2_localnet.BasicAuthTests)
-PASS: test_compile_files (test.test_compileall.CompileallTests)
-PASS: test_magic_number (test.test_compileall.CompileallTests)
-PASS: test_mtime (test.test_compileall.CompileallTests)
+FAIL: test_compile_files (test.test_compileall.CompileallTests)
+FAIL: test_magic_number (test.test_compileall.CompileallTests)
+FAIL: test_mtime (test.test_compileall.CompileallTests)


-PASS: test_http2time (test.test_cookielib.DateTimeTests)
-PASS: test_http2time_formats (test.test_cookielib.DateTimeTests)
+FAIL: test_http2time (test.test_cookielib.DateTimeTests)
+FAIL: test_http2time_formats (test.test_cookielib.DateTimeTests)

-PASS: test_check_hostname (test.test_ftplib.TestTLS_FTPClass)
-PASS: test_context (test.test_ftplib.TestTLS_FTPClass)
-PASS: test_control_connection (test.test_ftplib.TestTLS_FTPClass)
-PASS: test_data_connection (test.test_ftplib.TestTLS_FTPClass)
-PASS: test_login (test.test_ftplib.TestTLS_FTPClass)
+FAIL: test_check_hostname (test.test_ftplib.TestTLS_FTPClass)
+FAIL: test_context (test.test_ftplib.TestTLS_FTPClass)
+FAIL: test_control_connection (test.test_ftplib.TestTLS_FTPClass)
+FAIL: test_data_connection (test.test_ftplib.TestTLS_FTPClass)
+FAIL: test_login (test.test_ftplib.TestTLS_FTPClass)

-PASS: test_basics (test.test_import.PycRewritingTests)
-PASS: test_foreign_code (test.test_import.PycRewritingTests)
-PASS: test_incorrect_code_name (test.test_import.PycRewritingTests)
+FAIL: test_basics (test.test_import.PycRewritingTests)
+FAIL: test_foreign_code (test.test_import.PycRewritingTests)
+FAIL: test_incorrect_code_name (test.test_import.PycRewritingTests)

-PASS: test_read_write_after_close_raises_valuerror (test.test_ssl.ThreadedTests)
+FAIL: test_read_write_after_close_raises_valuerror (test.test_ssl.ThreadedTests)
 PASS: A brutal shutdown of an SSL server should raise an OSError
-PASS: test_server_accept (test.test_ssl.ThreadedTests)
+FAIL: test_server_accept (test.test_ssl.ThreadedTests)

Comment 1 Ross Burton 2018-06-14 14:58:02 UTC

My hunch is that python-ptest doesn't depend on *all* of the Python library packages.

Comment 2 Armin Kuster 2018-08-15 16:21:34 UTC

running test against latest.

Comment 3 Armin Kuster 2018-08-15 16:56:56 UTC

currently seeing these failures:

FAIL: test_getattr (test.test_builtin.BuiltinTest)
FAIL: test_hasattr (test.test_builtin.BuiltinTest)
FAIL: test_hasattr (test.test_builtin.BuiltinTest)
FAIL: Check same msg for exceptions with 0 args and overridden __str__
FAIL: Check same msg for Exceptions with 1 arg
FAIL: Check same msg for exceptions with overridden __str__ and 1 arg
FAIL: Check same msg for exceptions with overridden __str__ and many args
FAIL: test_0_args_with_overridden___str__ (test.test_exceptions.TestSameStrAndUnicodeMsg)
FAIL: test_1_arg (test.test_exceptions.TestSameStrAndUnicodeMsg)
FAIL: test_1_arg_with_overridden___str__ (test.test_exceptions.TestSameStrAndUnicodeMsg)
FAIL: test_many_args_with_overridden___str__ (test.test_exceptions.TestSameStrAndUnicodeMsg)
FAIL: test_unicode (test.test_StringIO.TestcStringIO)
FAIL: test_unicode (test.test_StringIO.TestcStringIO)
FAIL: test_output_htmlcalendar (test.test_calendar.OutputTestCase)
FAIL: test_output_htmlcalendar (test.test_calendar.OutputTestCase)
FAIL: test_basic_script (test.test_cmd_line_script.CmdLineTest) ... Output from test script '/var/volatile/tmp/tmpXzFI_5/script.py':
FAIL: test_directory (test.test_cmd_line_script.CmdLineTest) ... Output from test script '/var/volatile/tmp/tmp3bo47W':
FAIL: test_directory_compiled (test.test_cmd_line_script.CmdLineTest) ... Output from test script '/var/volatile/tmp/tmpCRBJAD':
FAIL: test_directory_error (test.test_cmd_line_script.CmdLineTest) ... Output from test script '/var/volatile/tmp/tmp3JEJ7X':
FAIL: test_module_in_package (test.test_cmd_line_script.CmdLineTest) ... Output from test script '/var/volatile/tmp/tmpVSBSXY/launch.py':
FAIL: test_module_in_package_in_zipfile (test.test_cmd_line_script.CmdLineTest) ... Output from test script '/var/volatile/tmp/tmpqx0kZy/launch.py':
FAIL: test_module_in_subpackage_in_zipfile (test.test_cmd_line_script.CmdLineTest) ... Output from test script '/var/volatile/tmp/tmpc2YAYT/launch.py':
FAIL: test_package (test.test_cmd_line_script.CmdLineTest) ... Output from test script '/var/volatile/tmp/tmpgVJEhI/launch.py':
FAIL: test_package_compiled (test.test_cmd_line_script.CmdLineTest) ... Output from test script '/var/volatile/tmp/tmpmUaQro/launch.py':
FAIL: test_package_error (test.test_cmd_line_script.CmdLineTest) ... Output from test script '/var/volatile/tmp/tmp3BA0Nh/launch.py':
FAIL: test_package_recursion (test.test_cmd_line_script.CmdLineTest) ... Output from test script '/var/volatile/tmp/tmpfrnWsV/launch.py':
FAIL: test_script_compiled (test.test_cmd_line_script.CmdLineTest) ... Output from test script '/var/volatile/tmp/tmp2sWkG8/script.pyc':
FAIL: test_zipfile (test.test_cmd_line_script.CmdLineTest) ... Output from test script '/var/volatile/tmp/tmp6NffD_/test_zip.zip':
FAIL: test_zipfile_compiled (test.test_cmd_line_script.CmdLineTest) ... Output from test script '/var/volatile/tmp/tmpoVHT1r/test_zip.zip':
FAIL: test_zipfile_error (test.test_cmd_line_script.CmdLineTest) ... Output from test script '/var/volatile/tmp/tmpIHoD_v/test_zip.zip':
FAIL: test_invalid_name (ctypes.test.test_structures.StructureTestCase)
FAIL: test_getattr (test.test_builtin.BuiltinTest)
FAIL: test_hasattr (test.test_builtin.BuiltinTest)
FAIL: test_hasattr (test.test_builtin.BuiltinTest)
FAIL: Check same msg for exceptions with 0 args and overridden __str__
FAIL: Check same msg for Exceptions with 1 arg
FAIL: Check same msg for exceptions with overridden __str__ and 1 arg
FAIL: Check same msg for exceptions with overridden __str__ and many args
FAIL: test_0_args_with_overridden___str__ (test.test_exceptions.TestSameStrAndUnicodeMsg)
FAIL: test_1_arg (test.test_exceptions.TestSameStrAndUnicodeMsg)
FAIL: test_1_arg_with_overridden___str__ (test.test_exceptions.TestSameStrAndUnicodeMsg)
FAIL: test_many_args_with_overridden___str__ (test.test_exceptions.TestSameStrAndUnicodeMsg)
FAIL: test_unicode (test.test_StringIO.TestcStringIO)
FAIL: test_unicode (test.test_StringIO.TestcStringIO)
FAIL: test_output_htmlcalendar (test.test_calendar.OutputTestCase)
FAIL: test_output_htmlcalendar (test.test_calendar.OutputTestCase)
FAIL: test_basic_script (test.test_cmd_line_script.CmdLineTest) ... Output from test script '/var/volatile/tmp/tmpTfONN6/script.py':
FAIL: test_directory (test.test_cmd_line_script.CmdLineTest) ... Output from test script '/var/volatile/tmp/tmpgtbLth':
FAIL: test_directory_compiled (test.test_cmd_line_script.CmdLineTest) ... Output from test script '/var/volatile/tmp/tmpemZyS6':
FAIL: test_directory_error (test.test_cmd_line_script.CmdLineTest) ... Output from test script '/var/volatile/tmp/tmpMUTrNN':
FAIL: test_module_in_package (test.test_cmd_line_script.CmdLineTest) ... Output from test script '/var/volatile/tmp/tmpSMinIW/launch.py':
FAIL: test_module_in_package_in_zipfile (test.test_cmd_line_script.CmdLineTest) ... Output from test script '/var/volatile/tmp/tmpZGGV90/launch.py':
FAIL: test_module_in_subpackage_in_zipfile (test.test_cmd_line_script.CmdLineTest) ... Output from test script '/var/volatile/tmp/tmpbiYMd3/launch.py':
FAIL: test_package (test.test_cmd_line_script.CmdLineTest) ... Output from test script '/var/volatile/tmp/tmpN3s55C/launch.py':
FAIL: test_package_compiled (test.test_cmd_line_script.CmdLineTest) ... Output from test script '/var/volatile/tmp/tmptB125_/launch.py':
FAIL: test_package_error (test.test_cmd_line_script.CmdLineTest) ... Output from test script '/var/volatile/tmp/tmpSfRzJf/launch.py':
FAIL: test_package_recursion (test.test_cmd_line_script.CmdLineTest) ... Output from test script '/var/volatile/tmp/tmpRXoCvq/launch.py':
FAIL: test_script_compiled (test.test_cmd_line_script.CmdLineTest) ... Output from test script '/var/volatile/tmp/tmpeYMwv4/script.pyc':
FAIL: test_zipfile (test.test_cmd_line_script.CmdLineTest) ... Output from test script '/var/volatile/tmp/tmpNohLD7/test_zip.zip':
FAIL: test_zipfile_compiled (test.test_cmd_line_script.CmdLineTest) ... Output from test script '/var/volatile/tmp/tmpLaqYMv/test_zip.zip':
FAIL: test_zipfile_error (test.test_cmd_line_script.CmdLineTest) ... Output from test script '/var/volatile/tmp/tmpnY8i8L/test_zip.zip':
FAIL: test_invalid_name (ctypes.test.test_structures.StructureTestCase)

Comment 4 yeoh ee peng 2018-08-16 01:22:33 UTC

The focus here was not so much on how many tests pass or fail in current release, but rather the tests that were passing in previous release (eg. 2.4.2) but now failing on current release (eg. 2.4.3) showing that some changes in current release may caused the package functionality to be broken.

Comment 5 Manish Das 2018-10-03 10:15:04 UTC

Performed diff and found that python tests that passed in 2.6 M2 rc1 are failing on 2.6 M3 rc1

------------------------------------------------------------------------------
Package Name:  python
------------------------------------------------------------------------------

-PASS: test_options (test.test_ssl.ContextTests)

+FAIL: test_options (test.test_ssl.ContextTests)

-PASS: test_options (test.test_ssl.ContextTests)

+FAIL: test_options (test.test_ssl.ContextTests)

------------------------------------------------------------------------------
Total number of test cases passing in Previous release but failing in Current release:  2
------------------------------------------------------------------------------

Comment 6 Ross Burton 2018-10-03 11:08:29 UTC

Looks like openssl 1.1 broke Python2 too.  Anuj, you worked wonders with Python3, can you take py2 too?

Comment 7 Anuj Mittal 2018-10-04 00:04:24 UTC

I cherry-picked some patches and looks like it's fixed now. Will send to list.

Comment 8 Anuj Mittal 2018-10-08 23:40:49 UTC

Merged for ssl tests:

https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=2b4dce4925627c81ee2f27a0eeb2422e15029c3e

If there are any other python tests failing as compared to last runs (like the ones in first comment), please re-open or file a new bug.