11818 – Allow oe-selftest to work with BB_SERVER_TIMEOUT

Bug 11818 - Allow oe-selftest to work with BB_SERVER_TIMEOUT

Summary: Allow oe-selftest to work with BB_SERVER_TIMEOUT

Status:	RESOLVED FIXED

Alias:	None

Product:	Build Testing
Classification:	QA/Testing
Component:	general (show other bugs)
Version:	2.4
Hardware:	x86 Multiple

Importance:	Medium+ normal
Target Milestone:	2.4 M4
Assignee:	Leonardo Sandoval Gonzalez
QA Contact:

URL:
Whiteboard:

Depends on:	11993
Blocks:
	Show dependency tree

Reported:	2017-07-21 08:12 UTC by Richard Purdie
Modified:	2017-09-15 16:46 UTC (History)
CC List:	2 users (show)

See Also:
OS type for building Yocto:	---
Type of Regression:	---
Verified:
Documentation change:	No (bug/feature does not impact docs)

Attachments
oe-selftest run bb_server_timeout=100 failure log (1.64 MB, text/plain) 2017-08-08 16:05 UTC, Aníbal Limón	no flags	Details
oe-selftest fail master rev 51cc0b07 (609.76 KB, text/plain) 2017-08-28 20:06 UTC, Aníbal Limón	no flags	Details
oe-selftest failures after reload config fix (534.41 KB, text/plain) 2017-08-30 21:52 UTC, Aníbal Limón	no flags	Details
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.

Description Richard Purdie 2017-07-21 08:12:15 UTC

Now that we have memres bitbake by default, we need to make oe-selftest complete successfully when BB_SERVER_TIMEOUT is set to a non-zero value.

Comment 1 Aníbal Limón 2017-08-08 16:05:38 UTC

Created attachment 3922 [details]
oe-selftest run bb_server_timeout=100 failure log

First run the tests that fails, see full log,

2017-08-07 20:21:14,503 - oe-selftest - INFO - RESULTS - archiver.Archiver.test_archiver_allows_to_filter_on_recipe_name - Testcase 1345: FAILED
2017-08-07 20:21:14,503 - oe-selftest - INFO - RESULTS - archiver.Archiver.test_archiver_filters_by_type - Testcase -1: FAILED
2017-08-07 20:21:14,503 - oe-selftest - INFO - RESULTS - archiver.Archiver.test_archiver_filters_by_type_and_name - Testcase -1: FAILED
2017-08-07 20:21:14,506 - oe-selftest - INFO - RESULTS - bbtests.BitbakeTests.test_bbappend_order - Testcase 1425: FAILED
2017-08-07 20:21:14,508 - oe-selftest - INFO - RESULTS - bbtests.BitbakeTests.test_event_handler - Testcase 806: FAILED
2017-08-07 20:21:14,509 - oe-selftest - INFO - RESULTS - bbtests.BitbakeTests.test_image_manifest - Testcase 899: FAILED
2017-08-07 20:21:14,509 - oe-selftest - INFO - RESULTS - bbtests.BitbakeTests.test_invalid_patch - Testcase 108: FAILED
2017-08-07 20:21:14,511 - oe-selftest - INFO - RESULTS - bbtests.BitbakeTests.test_prefile - Testcase 1032: FAILED
2017-08-07 20:21:14,512 - oe-selftest - INFO - RESULTS - bbtests.BitbakeTests.test_setscene_only - Testcase 1422: FAILED
2017-08-07 20:21:14,513 - oe-selftest - INFO - RESULTS - buildoptions.BuildhistoryTests.test_buildhistory_basic - Testcase 293: FAILED
2017-08-07 20:21:14,514 - oe-selftest - INFO - RESULTS - buildoptions.ImageOptionsTests.test_ccache_tool - Testcase 286: FAILED
2017-08-07 20:21:14,514 - oe-selftest - INFO - RESULTS - buildoptions.ImageOptionsTests.test_incremental_image_generation - Testcase 761: FAILED
2017-08-07 20:21:14,516 - oe-selftest - INFO - RESULTS - containerimage.ContainerImageTests.test_expected_files - Testcase 1619: FAILED
2017-08-07 20:21:14,528 - oe-selftest - INFO - RESULTS - distrodata.Distrodata.test_checkpkg - Testcase -1: FAILED
2017-08-07 20:21:14,531 - oe-selftest - INFO - RESULTS - imagefeatures.ImageFeatures.test_long_chain_conversion - Testcase -1: FAILED
2017-08-07 20:21:14,531 - oe-selftest - INFO - RESULTS - imagefeatures.ImageFeatures.test_non_root_user_can_connect_via_ssh_without_password - Testcase 1107: FAILED
2017-08-07 20:21:14,545 - oe-selftest - INFO - RESULTS - prservice.BitbakePrTests.test_import_export_override_db - Testcase 931: FAILED
2017-08-07 20:21:14,545 - oe-selftest - INFO - RESULTS - prservice.BitbakePrTests.test_import_export_replace_db - Testcase 930: FAILED
2017-08-07 20:21:14,546 - oe-selftest - INFO - RESULTS - prservice.BitbakePrTests.test_pr_service_deb_arch_dep - Testcase 934: FAILED
2017-08-07 20:21:14,546 - oe-selftest - INFO - RESULTS - prservice.BitbakePrTests.test_pr_service_deb_arch_indep - Testcase 937: FAILED
2017-08-07 20:21:14,546 - oe-selftest - INFO - RESULTS - prservice.BitbakePrTests.test_pr_service_ipk_arch_dep - Testcase 933: FAILED
2017-08-07 20:21:14,546 - oe-selftest - INFO - RESULTS - prservice.BitbakePrTests.test_pr_service_ipk_arch_indep - Testcase 936: FAILED
2017-08-07 20:21:14,546 - oe-selftest - INFO - RESULTS - prservice.BitbakePrTests.test_pr_service_rpm_arch_dep - Testcase 932: FAILED
2017-08-07 20:21:14,547 - oe-selftest - INFO - RESULTS - prservice.BitbakePrTests.test_pr_service_rpm_arch_indep - Testcase 935: FAILED
2017-08-07 20:21:14,558 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolTests.test_recipetool_create_github - Testcase 1638: FAILED
2017-08-07 20:21:14,570 - oe-selftest - INFO - RESULTS - runtime_test.TestImage.test_testimage_dnf - Testcase 1883: FAILED
2017-08-07 20:21:14,572 - oe-selftest - INFO - RESULTS - sstatetests.SStateTests.test_cleansstate_task_distro_nonspecific - Testcase 1376: FAILED
2017-08-07 20:21:14,573 - oe-selftest - INFO - RESULTS - sstatetests.SStateTests.test_rebuild_distro_specific_sstate_cross_native_targets - Testcase 175: FAILED
2017-08-07 20:21:14,573 - oe-selftest - INFO - RESULTS - sstatetests.SStateTests.test_rebuild_distro_specific_sstate_native_target - Testcase 1373: FAILED
2017-08-07 20:21:14,575 - oe-selftest - INFO - RESULTS - sstatetests.SStateTests.test_sstate_cache_management_script_using_machine - Testcase 974: FAILED
2017-08-07 20:21:14,575 - oe-selftest - INFO - RESULTS - sstatetests.SStateTests.test_sstate_cache_management_script_using_pr_1 - Testcase 973: FAILED
2017-08-07 20:21:14,575 - oe-selftest - INFO - RESULTS - sstatetests.SStateTests.test_sstate_cache_management_script_using_pr_2 - Testcase 978: FAILED
2017-08-07 20:21:14,576 - oe-selftest - INFO - RESULTS - sstatetests.SStateTests.test_sstate_creation_distro_nonspecific_pass - Testcase 976: FAILED
2017-08-07 20:21:14,576 - oe-selftest - INFO - RESULTS - sstatetests.SStateTests.test_sstate_creation_distro_specific_fail - Testcase 1374: FAILED
2017-08-07 20:21:14,577 - oe-selftest - INFO - RESULTS - sstatetests.SStateTests.test_sstate_creation_distro_specific_pass - Testcase 975: FAILED

Comment 2 Aníbal Limón 2017-08-08 16:14:02 UTC

(In reply to comment #1)
> Created attachment 3922 [details]
> oe-selftest run bb_server_timeout=100 failure log
> 
> First run the tests that fails, see full log,
> 
> 2017-08-07 20:21:14,503 - oe-selftest - INFO - RESULTS -
> archiver.Archiver.test_archiver_allows_to_filter_on_recipe_name - Testcase
> 1345: FAILED
> 2017-08-07 20:21:14,503 - oe-selftest - INFO - RESULTS -
> archiver.Archiver.test_archiver_filters_by_type - Testcase -1: FAILED
> 2017-08-07 20:21:14,503 - oe-selftest - INFO - RESULTS -
> archiver.Archiver.test_archiver_filters_by_type_and_name - Testcase -1:
> FAILED
> 2017-08-07 20:21:14,506 - oe-selftest - INFO - RESULTS -
> bbtests.BitbakeTests.test_bbappend_order - Testcase 1425: FAILED
> 2017-08-07 20:21:14,508 - oe-selftest - INFO - RESULTS -
> bbtests.BitbakeTests.test_event_handler - Testcase 806: FAILED
> 2017-08-07 20:21:14,509 - oe-selftest - INFO - RESULTS -
> bbtests.BitbakeTests.test_image_manifest - Testcase 899: FAILED
> 2017-08-07 20:21:14,509 - oe-selftest - INFO - RESULTS -
> bbtests.BitbakeTests.test_invalid_patch - Testcase 108: FAILED
> 2017-08-07 20:21:14,511 - oe-selftest - INFO - RESULTS -
> bbtests.BitbakeTests.test_prefile - Testcase 1032: FAILED
> 2017-08-07 20:21:14,512 - oe-selftest - INFO - RESULTS -
> bbtests.BitbakeTests.test_setscene_only - Testcase 1422: FAILED
> 2017-08-07 20:21:14,513 - oe-selftest - INFO - RESULTS -
> buildoptions.BuildhistoryTests.test_buildhistory_basic - Testcase 293: FAILED
> 2017-08-07 20:21:14,514 - oe-selftest - INFO - RESULTS -
> buildoptions.ImageOptionsTests.test_ccache_tool - Testcase 286: FAILED
> 2017-08-07 20:21:14,514 - oe-selftest - INFO - RESULTS -
> buildoptions.ImageOptionsTests.test_incremental_image_generation - Testcase
> 761: FAILED
> 2017-08-07 20:21:14,516 - oe-selftest - INFO - RESULTS -
> containerimage.ContainerImageTests.test_expected_files - Testcase 1619:
> FAILED
> 2017-08-07 20:21:14,528 - oe-selftest - INFO - RESULTS -
> distrodata.Distrodata.test_checkpkg - Testcase -1: FAILED
> 2017-08-07 20:21:14,531 - oe-selftest - INFO - RESULTS -
> imagefeatures.ImageFeatures.test_long_chain_conversion - Testcase -1: FAILED
> 2017-08-07 20:21:14,531 - oe-selftest - INFO - RESULTS -
> imagefeatures.ImageFeatures.
> test_non_root_user_can_connect_via_ssh_without_password - Testcase 1107:
> FAILED
> 2017-08-07 20:21:14,545 - oe-selftest - INFO - RESULTS -
> prservice.BitbakePrTests.test_import_export_override_db - Testcase 931:
> FAILED
> 2017-08-07 20:21:14,545 - oe-selftest - INFO - RESULTS -
> prservice.BitbakePrTests.test_import_export_replace_db - Testcase 930: FAILED
> 2017-08-07 20:21:14,546 - oe-selftest - INFO - RESULTS -
> prservice.BitbakePrTests.test_pr_service_deb_arch_dep - Testcase 934: FAILED
> 2017-08-07 20:21:14,546 - oe-selftest - INFO - RESULTS -
> prservice.BitbakePrTests.test_pr_service_deb_arch_indep - Testcase 937:
> FAILED
> 2017-08-07 20:21:14,546 - oe-selftest - INFO - RESULTS -
> prservice.BitbakePrTests.test_pr_service_ipk_arch_dep - Testcase 933: FAILED
> 2017-08-07 20:21:14,546 - oe-selftest - INFO - RESULTS -
> prservice.BitbakePrTests.test_pr_service_ipk_arch_indep - Testcase 936:
> FAILED
> 2017-08-07 20:21:14,546 - oe-selftest - INFO - RESULTS -
> prservice.BitbakePrTests.test_pr_service_rpm_arch_dep - Testcase 932: FAILED
> 2017-08-07 20:21:14,547 - oe-selftest - INFO - RESULTS -
> prservice.BitbakePrTests.test_pr_service_rpm_arch_indep - Testcase 935:
> FAILED
> 2017-08-07 20:21:14,558 - oe-selftest - INFO - RESULTS -
> recipetool.RecipetoolTests.test_recipetool_create_github - Testcase 1638:
> FAILED
> 2017-08-07 20:21:14,570 - oe-selftest - INFO - RESULTS -
> runtime_test.TestImage.test_testimage_dnf - Testcase 1883: FAILED
> 2017-08-07 20:21:14,572 - oe-selftest - INFO - RESULTS -
> sstatetests.SStateTests.test_cleansstate_task_distro_nonspecific - Testcase
> 1376: FAILED
> 2017-08-07 20:21:14,573 - oe-selftest - INFO - RESULTS -
> sstatetests.SStateTests.
> test_rebuild_distro_specific_sstate_cross_native_targets - Testcase 175:
> FAILED
> 2017-08-07 20:21:14,573 - oe-selftest - INFO - RESULTS -
> sstatetests.SStateTests.test_rebuild_distro_specific_sstate_native_target -
> Testcase 1373: FAILED
> 2017-08-07 20:21:14,575 - oe-selftest - INFO - RESULTS -
> sstatetests.SStateTests.test_sstate_cache_management_script_using_machine -
> Testcase 974: FAILED
> 2017-08-07 20:21:14,575 - oe-selftest - INFO - RESULTS -
> sstatetests.SStateTests.test_sstate_cache_management_script_using_pr_1 -
> Testcase 973: FAILED
> 2017-08-07 20:21:14,575 - oe-selftest - INFO - RESULTS -
> sstatetests.SStateTests.test_sstate_cache_management_script_using_pr_2 -
> Testcase 978: FAILED
> 2017-08-07 20:21:14,576 - oe-selftest - INFO - RESULTS -
> sstatetests.SStateTests.test_sstate_creation_distro_nonspecific_pass -
> Testcase 976: FAILED
> 2017-08-07 20:21:14,576 - oe-selftest - INFO - RESULTS -
> sstatetests.SStateTests.test_sstate_creation_distro_specific_fail - Testcase
> 1374: FAILED
> 2017-08-07 20:21:14,577 - oe-selftest - INFO - RESULTS -
> sstatetests.SStateTests.test_sstate_creation_distro_specific_pass - Testcase
> 975: FAILED

The errors are two different ones, 

1st - One related to not deterministic meta-data
2nd - Related to fetcher failures and it doesn't matter

Comment 3 Aníbal Limón 2017-08-28 20:06:12 UTC

Created attachment 3955 [details]
oe-selftest fail master rev 51cc0b07

After a run with the current poky master [1] other errors appears, i will load specific bugs.

[1] 51cc0b07a14277c71a3b6184575fe30ab489c717

Comment 4 Aníbal Limón 2017-08-30 21:52:39 UTC

Created attachment 3970 [details]
oe-selftest failures after reload config fix

Comment 5 Aníbal Limón 2017-08-30 21:54:11 UTC

(In reply to comment #4)
> Created attachment 3970 [details]
> oe-selftest failures after reload config fix

A run after apply, 

http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?h=master-next&id=a22779d00762923d8329eff4e84bfe8a71232c81

There are some tests that are still failing,

2017-08-30 12:24:36,110 - oe-selftest - INFO - RESULTS - bblayers.BitbakeLayers.test_bitbakelayers_flatten - Testcase 95: FAILED
2017-08-30 12:24:36,110 - oe-selftest - INFO - RESULTS - bblayers.BitbakeLayers.test_bitbakelayers_showappends - Testcase 93: FAILED
2017-08-30 12:24:36,111 - oe-selftest - INFO - RESULTS - bblayers.BitbakeLayers.test_bitbakelayers_showrecipes - Testcase 1384: FAILED
2017-08-30 12:24:36,119 - oe-selftest - INFO - RESULTS - bbtests.BitbakeTests.test_prefile - Testcase 1032: FAILED
2017-08-30 12:24:36,137 - oe-selftest - INFO - RESULTS - devtool.DevtoolTests.test_devtool_reset_all - Testcase 1168: FAILED
2017-08-30 12:24:36,139 - oe-selftest - INFO - RESULTS - devtool.DevtoolTests.test_devtool_update_recipe_local_files - Testcase 1370: FAILED
2017-08-30 12:24:36,140 - oe-selftest - INFO - RESULTS - devtool.DevtoolTests.test_devtool_update_recipe_local_files_subdir - Testcase 1628: FAILED
2017-08-30 12:24:36,140 - oe-selftest - INFO - RESULTS - devtool.DevtoolTests.test_devtool_upgrade - Testcase 1367: FAILED
2017-08-30 12:24:36,142 - oe-selftest - INFO - RESULTS - distrodata.Distrodata.test_checkpkg - Testcase -1: FAILED
2017-08-30 12:24:36,145 - oe-selftest - INFO - RESULTS - imagefeatures.ImageFeatures.test_long_chain_conversion - Testcase -1: FAILED
2017-08-30 12:24:36,148 - oe-selftest - INFO - RESULTS - manifest.VerifyManifest.test_SDK_manifest_entries - Testcase 1380: FAILED
2017-08-30 12:24:36,163 - oe-selftest - INFO - RESULTS - prservice.BitbakePrTests.test_import_export_override_db - Testcase 931: FAILED
2017-08-30 12:24:36,163 - oe-selftest - INFO - RESULTS - prservice.BitbakePrTests.test_import_export_replace_db - Testcase 930: FAILED
2017-08-30 12:24:36,163 - oe-selftest - INFO - RESULTS - prservice.BitbakePrTests.test_pr_service_deb_arch_dep - Testcase 934: FAILED
2017-08-30 12:24:36,164 - oe-selftest - INFO - RESULTS - prservice.BitbakePrTests.test_pr_service_deb_arch_indep - Testcase 937: FAILED
2017-08-30 12:24:36,164 - oe-selftest - INFO - RESULTS - prservice.BitbakePrTests.test_pr_service_ipk_arch_dep - Testcase 933: FAILED
2017-08-30 12:24:36,164 - oe-selftest - INFO - RESULTS - prservice.BitbakePrTests.test_pr_service_ipk_arch_indep - Testcase 936: FAILED
2017-08-30 12:24:36,164 - oe-selftest - INFO - RESULTS - prservice.BitbakePrTests.test_pr_service_rpm_arch_dep - Testcase 932: FAILED
2017-08-30 12:24:36,164 - oe-selftest - INFO - RESULTS - prservice.BitbakePrTests.test_pr_service_rpm_arch_indep - Testcase 935: FAILED
2017-08-30 12:24:36,168 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendsrcTests.test_recipetool_appendsrcfile_subdir_basic_dirdest - Testcase 1282: FAILED
2017-08-30 12:24:36,194 - oe-selftest - INFO - RESULTS - runtime_test.Postinst.test_verify_postinst - Testcase 1540: FAILED
2017-08-30 12:24:36,194 - oe-selftest - INFO - RESULTS - runtime_test.TestExport.test_testexport_basic - Testcase 1499: FAILED
2017-08-30 12:24:36,194 - oe-selftest - INFO - RESULTS - runtime_test.TestExport.test_testexport_sdk - Testcase 1641: FAILED
2017-08-30 12:24:36,195 - oe-selftest - INFO - RESULTS - runtime_test.TestImage.test_testimage_dnf - Testcase 1883: FAILED
2017-08-30 12:24:36,197 - oe-selftest - INFO - RESULTS - signing.Signing.test_signing_packages - Testcase 1362: FAILED

Comment 6 Leonardo Sandoval Gonzalez 2017-09-15 16:46:48 UTC

http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=92903ced3ac318c6a7e7e43bcdd1895d3c7b83d1