Bug 11818

Summary: Allow oe-selftest to work with BB_SERVER_TIMEOUT
Product: [QA/Testing] Build Testing Reporter: Richard Purdie <richard.purdie>
Component: generalAssignee: Leonardo Sandoval Gonzalez <leonardo.sandoval.gonzalez>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium+ CC: anibal.limon, juan.m.cruz.alcaraz
Version: 2.4   
Target Milestone: 2.4 M4   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Bug Depends on: 11993    
Bug Blocks:    
Attachments:
Description Flags
oe-selftest run bb_server_timeout=100 failure log
none
oe-selftest fail master rev 51cc0b07
none
oe-selftest failures after reload config fix none

Description Richard Purdie 2017-07-21 08:12:15 UTC 
Now that we have memres bitbake by default, we need to make oe-selftest complete successfully when BB_SERVER_TIMEOUT is set to a non-zero value.
Comment 1 Aníbal Limón 2017-08-08 16:05:38 UTC 
Created attachment 3922 [details]
oe-selftest run bb_server_timeout=100 failure log

First run the tests that fails, see full log,

2017-08-07 20:21:14,503 - oe-selftest - INFO - RESULTS - archiver.Archiver.test_archiver_allows_to_filter_on_recipe_name - Testcase 1345: FAILED
2017-08-07 20:21:14,503 - oe-selftest - INFO - RESULTS - archiver.Archiver.test_archiver_filters_by_type - Testcase -1: FAILED
2017-08-07 20:21:14,503 - oe-selftest - INFO - RESULTS - archiver.Archiver.test_archiver_filters_by_type_and_name - Testcase -1: FAILED
2017-08-07 20:21:14,506 - oe-selftest - INFO - RESULTS - bbtests.BitbakeTests.test_bbappend_order - Testcase 1425: FAILED
2017-08-07 20:21:14,508 - oe-selftest - INFO - RESULTS - bbtests.BitbakeTests.test_event_handler - Testcase 806: FAILED
2017-08-07 20:21:14,509 - oe-selftest - INFO - RESULTS - bbtests.BitbakeTests.test_image_manifest - Testcase 899: FAILED
2017-08-07 20:21:14,509 - oe-selftest - INFO - RESULTS - bbtests.BitbakeTests.test_invalid_patch - Testcase 108: FAILED
2017-08-07 20:21:14,511 - oe-selftest - INFO - RESULTS - bbtests.BitbakeTests.test_prefile - Testcase 1032: FAILED
2017-08-07 20:21:14,512 - oe-selftest - INFO - RESULTS - bbtests.BitbakeTests.test_setscene_only - Testcase 1422: FAILED
2017-08-07 20:21:14,513 - oe-selftest - INFO - RESULTS - buildoptions.BuildhistoryTests.test_buildhistory_basic - Testcase 293: FAILED
2017-08-07 20:21:14,514 - oe-selftest - INFO - RESULTS - buildoptions.ImageOptionsTests.test_ccache_tool - Testcase 286: FAILED
2017-08-07 20:21:14,514 - oe-selftest - INFO - RESULTS - buildoptions.ImageOptionsTests.test_incremental_image_generation - Testcase 761: FAILED
2017-08-07 20:21:14,516 - oe-selftest - INFO - RESULTS - containerimage.ContainerImageTests.test_expected_files - Testcase 1619: FAILED
2017-08-07 20:21:14,528 - oe-selftest - INFO - RESULTS - distrodata.Distrodata.test_checkpkg - Testcase -1: FAILED
2017-08-07 20:21:14,531 - oe-selftest - INFO - RESULTS - imagefeatures.ImageFeatures.test_long_chain_conversion - Testcase -1: FAILED
2017-08-07 20:21:14,531 - oe-selftest - INFO - RESULTS - imagefeatures.ImageFeatures.test_non_root_user_can_connect_via_ssh_without_password - Testcase 1107: FAILED
2017-08-07 20:21:14,545 - oe-selftest - INFO - RESULTS - prservice.BitbakePrTests.test_import_export_override_db - Testcase 931: FAILED
2017-08-07 20:21:14,545 - oe-selftest - INFO - RESULTS - prservice.BitbakePrTests.test_import_export_replace_db - Testcase 930: FAILED
2017-08-07 20:21:14,546 - oe-selftest - INFO - RESULTS - prservice.BitbakePrTests.test_pr_service_deb_arch_dep - Testcase 934: FAILED
2017-08-07 20:21:14,546 - oe-selftest - INFO - RESULTS - prservice.BitbakePrTests.test_pr_service_deb_arch_indep - Testcase 937: FAILED
2017-08-07 20:21:14,546 - oe-selftest - INFO - RESULTS - prservice.BitbakePrTests.test_pr_service_ipk_arch_dep - Testcase 933: FAILED
2017-08-07 20:21:14,546 - oe-selftest - INFO - RESULTS - prservice.BitbakePrTests.test_pr_service_ipk_arch_indep - Testcase 936: FAILED
2017-08-07 20:21:14,546 - oe-selftest - INFO - RESULTS - prservice.BitbakePrTests.test_pr_service_rpm_arch_dep - Testcase 932: FAILED
2017-08-07 20:21:14,547 - oe-selftest - INFO - RESULTS - prservice.BitbakePrTests.test_pr_service_rpm_arch_indep - Testcase 935: FAILED
2017-08-07 20:21:14,558 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolTests.test_recipetool_create_github - Testcase 1638: FAILED
2017-08-07 20:21:14,570 - oe-selftest - INFO - RESULTS - runtime_test.TestImage.test_testimage_dnf - Testcase 1883: FAILED
2017-08-07 20:21:14,572 - oe-selftest - INFO - RESULTS - sstatetests.SStateTests.test_cleansstate_task_distro_nonspecific - Testcase 1376: FAILED
2017-08-07 20:21:14,573 - oe-selftest - INFO - RESULTS - sstatetests.SStateTests.test_rebuild_distro_specific_sstate_cross_native_targets - Testcase 175: FAILED
2017-08-07 20:21:14,573 - oe-selftest - INFO - RESULTS - sstatetests.SStateTests.test_rebuild_distro_specific_sstate_native_target - Testcase 1373: FAILED
2017-08-07 20:21:14,575 - oe-selftest - INFO - RESULTS - sstatetests.SStateTests.test_sstate_cache_management_script_using_machine - Testcase 974: FAILED
2017-08-07 20:21:14,575 - oe-selftest - INFO - RESULTS - sstatetests.SStateTests.test_sstate_cache_management_script_using_pr_1 - Testcase 973: FAILED
2017-08-07 20:21:14,575 - oe-selftest - INFO - RESULTS - sstatetests.SStateTests.test_sstate_cache_management_script_using_pr_2 - Testcase 978: FAILED
2017-08-07 20:21:14,576 - oe-selftest - INFO - RESULTS - sstatetests.SStateTests.test_sstate_creation_distro_nonspecific_pass - Testcase 976: FAILED
2017-08-07 20:21:14,576 - oe-selftest - INFO - RESULTS - sstatetests.SStateTests.test_sstate_creation_distro_specific_fail - Testcase 1374: FAILED
2017-08-07 20:21:14,577 - oe-selftest - INFO - RESULTS - sstatetests.SStateTests.test_sstate_creation_distro_specific_pass - Testcase 975: FAILED
Comment 2 Aníbal Limón 2017-08-08 16:14:02 UTC 
(In reply to comment #1)
> Created attachment 3922 [details]
> oe-selftest run bb_server_timeout=100 failure log
> 
> First run the tests that fails, see full log,
> 
> 2017-08-07 20:21:14,503 - oe-selftest - INFO - RESULTS -
> archiver.Archiver.test_archiver_allows_to_filter_on_recipe_name - Testcase
> 1345: FAILED
> 2017-08-07 20:21:14,503 - oe-selftest - INFO - RESULTS -
> archiver.Archiver.test_archiver_filters_by_type - Testcase -1: FAILED
> 2017-08-07 20:21:14,503 - oe-selftest - INFO - RESULTS -
> archiver.Archiver.test_archiver_filters_by_type_and_name - Testcase -1:
> FAILED
> 2017-08-07 20:21:14,506 - oe-selftest - INFO - RESULTS -
> bbtests.BitbakeTests.test_bbappend_order - Testcase 1425: FAILED
> 2017-08-07 20:21:14,508 - oe-selftest - INFO - RESULTS -
> bbtests.BitbakeTests.test_event_handler - Testcase 806: FAILED
> 2017-08-07 20:21:14,509 - oe-selftest - INFO - RESULTS -
> bbtests.BitbakeTests.test_image_manifest - Testcase 899: FAILED
> 2017-08-07 20:21:14,509 - oe-selftest - INFO - RESULTS -
> bbtests.BitbakeTests.test_invalid_patch - Testcase 108: FAILED
> 2017-08-07 20:21:14,511 - oe-selftest - INFO - RESULTS -
> bbtests.BitbakeTests.test_prefile - Testcase 1032: FAILED
> 2017-08-07 20:21:14,512 - oe-selftest - INFO - RESULTS -
> bbtests.BitbakeTests.test_setscene_only - Testcase 1422: FAILED
> 2017-08-07 20:21:14,513 - oe-selftest - INFO - RESULTS -
> buildoptions.BuildhistoryTests.test_buildhistory_basic - Testcase 293: FAILED
> 2017-08-07 20:21:14,514 - oe-selftest - INFO - RESULTS -
> buildoptions.ImageOptionsTests.test_ccache_tool - Testcase 286: FAILED
> 2017-08-07 20:21:14,514 - oe-selftest - INFO - RESULTS -
> buildoptions.ImageOptionsTests.test_incremental_image_generation - Testcase
> 761: FAILED
> 2017-08-07 20:21:14,516 - oe-selftest - INFO - RESULTS -
> containerimage.ContainerImageTests.test_expected_files - Testcase 1619:
> FAILED
> 2017-08-07 20:21:14,528 - oe-selftest - INFO - RESULTS -
> distrodata.Distrodata.test_checkpkg - Testcase -1: FAILED
> 2017-08-07 20:21:14,531 - oe-selftest - INFO - RESULTS -
> imagefeatures.ImageFeatures.test_long_chain_conversion - Testcase -1: FAILED
> 2017-08-07 20:21:14,531 - oe-selftest - INFO - RESULTS -
> imagefeatures.ImageFeatures.
> test_non_root_user_can_connect_via_ssh_without_password - Testcase 1107:
> FAILED
> 2017-08-07 20:21:14,545 - oe-selftest - INFO - RESULTS -
> prservice.BitbakePrTests.test_import_export_override_db - Testcase 931:
> FAILED
> 2017-08-07 20:21:14,545 - oe-selftest - INFO - RESULTS -
> prservice.BitbakePrTests.test_import_export_replace_db - Testcase 930: FAILED
> 2017-08-07 20:21:14,546 - oe-selftest - INFO - RESULTS -
> prservice.BitbakePrTests.test_pr_service_deb_arch_dep - Testcase 934: FAILED
> 2017-08-07 20:21:14,546 - oe-selftest - INFO - RESULTS -
> prservice.BitbakePrTests.test_pr_service_deb_arch_indep - Testcase 937:
> FAILED
> 2017-08-07 20:21:14,546 - oe-selftest - INFO - RESULTS -
> prservice.BitbakePrTests.test_pr_service_ipk_arch_dep - Testcase 933: FAILED
> 2017-08-07 20:21:14,546 - oe-selftest - INFO - RESULTS -
> prservice.BitbakePrTests.test_pr_service_ipk_arch_indep - Testcase 936:
> FAILED
> 2017-08-07 20:21:14,546 - oe-selftest - INFO - RESULTS -
> prservice.BitbakePrTests.test_pr_service_rpm_arch_dep - Testcase 932: FAILED
> 2017-08-07 20:21:14,547 - oe-selftest - INFO - RESULTS -
> prservice.BitbakePrTests.test_pr_service_rpm_arch_indep - Testcase 935:
> FAILED
> 2017-08-07 20:21:14,558 - oe-selftest - INFO - RESULTS -
> recipetool.RecipetoolTests.test_recipetool_create_github - Testcase 1638:
> FAILED
> 2017-08-07 20:21:14,570 - oe-selftest - INFO - RESULTS -
> runtime_test.TestImage.test_testimage_dnf - Testcase 1883: FAILED
> 2017-08-07 20:21:14,572 - oe-selftest - INFO - RESULTS -
> sstatetests.SStateTests.test_cleansstate_task_distro_nonspecific - Testcase
> 1376: FAILED
> 2017-08-07 20:21:14,573 - oe-selftest - INFO - RESULTS -
> sstatetests.SStateTests.
> test_rebuild_distro_specific_sstate_cross_native_targets - Testcase 175:
> FAILED
> 2017-08-07 20:21:14,573 - oe-selftest - INFO - RESULTS -
> sstatetests.SStateTests.test_rebuild_distro_specific_sstate_native_target -
> Testcase 1373: FAILED
> 2017-08-07 20:21:14,575 - oe-selftest - INFO - RESULTS -
> sstatetests.SStateTests.test_sstate_cache_management_script_using_machine -
> Testcase 974: FAILED
> 2017-08-07 20:21:14,575 - oe-selftest - INFO - RESULTS -
> sstatetests.SStateTests.test_sstate_cache_management_script_using_pr_1 -
> Testcase 973: FAILED
> 2017-08-07 20:21:14,575 - oe-selftest - INFO - RESULTS -
> sstatetests.SStateTests.test_sstate_cache_management_script_using_pr_2 -
> Testcase 978: FAILED
> 2017-08-07 20:21:14,576 - oe-selftest - INFO - RESULTS -
> sstatetests.SStateTests.test_sstate_creation_distro_nonspecific_pass -
> Testcase 976: FAILED
> 2017-08-07 20:21:14,576 - oe-selftest - INFO - RESULTS -
> sstatetests.SStateTests.test_sstate_creation_distro_specific_fail - Testcase
> 1374: FAILED
> 2017-08-07 20:21:14,577 - oe-selftest - INFO - RESULTS -
> sstatetests.SStateTests.test_sstate_creation_distro_specific_pass - Testcase
> 975: FAILED

The errors are two different ones, 

1st - One related to not deterministic meta-data
2nd - Related to fetcher failures and it doesn't matter
Comment 3 Aníbal Limón 2017-08-28 20:06:12 UTC 
Created attachment 3955 [details]
oe-selftest fail master rev 51cc0b07

After a run with the current poky master [1] other errors appears, i will load specific bugs.

[1] 51cc0b07a14277c71a3b6184575fe30ab489c717
Comment 4 Aníbal Limón 2017-08-30 21:52:39 UTC 
Created attachment 3970 [details]
oe-selftest failures after reload config fix
Comment 5 Aníbal Limón 2017-08-30 21:54:11 UTC 
(In reply to comment #4)
> Created attachment 3970 [details]
> oe-selftest failures after reload config fix

A run after apply, 

http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?h=master-next&id=a22779d00762923d8329eff4e84bfe8a71232c81

There are some tests that are still failing,

2017-08-30 12:24:36,110 - oe-selftest - INFO - RESULTS - bblayers.BitbakeLayers.test_bitbakelayers_flatten - Testcase 95: FAILED
2017-08-30 12:24:36,110 - oe-selftest - INFO - RESULTS - bblayers.BitbakeLayers.test_bitbakelayers_showappends - Testcase 93: FAILED
2017-08-30 12:24:36,111 - oe-selftest - INFO - RESULTS - bblayers.BitbakeLayers.test_bitbakelayers_showrecipes - Testcase 1384: FAILED
2017-08-30 12:24:36,119 - oe-selftest - INFO - RESULTS - bbtests.BitbakeTests.test_prefile - Testcase 1032: FAILED
2017-08-30 12:24:36,137 - oe-selftest - INFO - RESULTS - devtool.DevtoolTests.test_devtool_reset_all - Testcase 1168: FAILED
2017-08-30 12:24:36,139 - oe-selftest - INFO - RESULTS - devtool.DevtoolTests.test_devtool_update_recipe_local_files - Testcase 1370: FAILED
2017-08-30 12:24:36,140 - oe-selftest - INFO - RESULTS - devtool.DevtoolTests.test_devtool_update_recipe_local_files_subdir - Testcase 1628: FAILED
2017-08-30 12:24:36,140 - oe-selftest - INFO - RESULTS - devtool.DevtoolTests.test_devtool_upgrade - Testcase 1367: FAILED
2017-08-30 12:24:36,142 - oe-selftest - INFO - RESULTS - distrodata.Distrodata.test_checkpkg - Testcase -1: FAILED
2017-08-30 12:24:36,145 - oe-selftest - INFO - RESULTS - imagefeatures.ImageFeatures.test_long_chain_conversion - Testcase -1: FAILED
2017-08-30 12:24:36,148 - oe-selftest - INFO - RESULTS - manifest.VerifyManifest.test_SDK_manifest_entries - Testcase 1380: FAILED
2017-08-30 12:24:36,163 - oe-selftest - INFO - RESULTS - prservice.BitbakePrTests.test_import_export_override_db - Testcase 931: FAILED
2017-08-30 12:24:36,163 - oe-selftest - INFO - RESULTS - prservice.BitbakePrTests.test_import_export_replace_db - Testcase 930: FAILED
2017-08-30 12:24:36,163 - oe-selftest - INFO - RESULTS - prservice.BitbakePrTests.test_pr_service_deb_arch_dep - Testcase 934: FAILED
2017-08-30 12:24:36,164 - oe-selftest - INFO - RESULTS - prservice.BitbakePrTests.test_pr_service_deb_arch_indep - Testcase 937: FAILED
2017-08-30 12:24:36,164 - oe-selftest - INFO - RESULTS - prservice.BitbakePrTests.test_pr_service_ipk_arch_dep - Testcase 933: FAILED
2017-08-30 12:24:36,164 - oe-selftest - INFO - RESULTS - prservice.BitbakePrTests.test_pr_service_ipk_arch_indep - Testcase 936: FAILED
2017-08-30 12:24:36,164 - oe-selftest - INFO - RESULTS - prservice.BitbakePrTests.test_pr_service_rpm_arch_dep - Testcase 932: FAILED
2017-08-30 12:24:36,164 - oe-selftest - INFO - RESULTS - prservice.BitbakePrTests.test_pr_service_rpm_arch_indep - Testcase 935: FAILED
2017-08-30 12:24:36,168 - oe-selftest - INFO - RESULTS - recipetool.RecipetoolAppendsrcTests.test_recipetool_appendsrcfile_subdir_basic_dirdest - Testcase 1282: FAILED
2017-08-30 12:24:36,194 - oe-selftest - INFO - RESULTS - runtime_test.Postinst.test_verify_postinst - Testcase 1540: FAILED
2017-08-30 12:24:36,194 - oe-selftest - INFO - RESULTS - runtime_test.TestExport.test_testexport_basic - Testcase 1499: FAILED
2017-08-30 12:24:36,194 - oe-selftest - INFO - RESULTS - runtime_test.TestExport.test_testexport_sdk - Testcase 1641: FAILED
2017-08-30 12:24:36,195 - oe-selftest - INFO - RESULTS - runtime_test.TestImage.test_testimage_dnf - Testcase 1883: FAILED
2017-08-30 12:24:36,197 - oe-selftest - INFO - RESULTS - signing.Signing.test_signing_packages - Testcase 1362: FAILED