8862 2015-12-29 00:09:15 +0000 oe-selftest: test_user reported as valid testcase but fails when run 2016-01-07 15:37:08 +0000 1 1 10 QA/Testing Build Testing general 2.0 All Multiple RESOLVED FIXED Undecided minor 2.1 M2 1 humberto.ibarra.lopez daniel.alexandrux.istrate alexandru.c.georgescu bluelightning bogdanx.a.voiculescu daniel.alexandrux.istrate richard.purdie ross.burton bogdanx.a.voiculescu --- --- No (bug/feature does not impact docs) oldest_to_newest 57482 0 humberto.ibarra.lopez 2015-12-29 00:09:15 +0000 imagefeatures.ImageFeatures.test_user reported as a valid testcase by "oe-selftest --list-classes" but oe-selftest doesn't seem to recognize it. testcase displayed as valid: $ oe-selftest --list-classes 2015-12-28 15:22:44,967 - selftest - INFO - Running bitbake -e to get BBPATH 2015-12-28 15:22:46,056 - selftest - INFO - Listing all available test modules: [..] imagefeatures -- ImageFeatures -- -- test_all_users_can_connect_via_ssh_without_password -- -- test_clutter_image_can_be_built -- -- test_non_root_user_can_connect_via_ssh_without_password -- -- test_rpm_version_4_support_on_image -- -- test_user -- -- test_wayland_support_in_image [..] testcase is detected as available, but oe-selftest can't execute it: $ oe-selftest --run-tests imagefeatures.ImageFeatures.test_user 2015-12-28 18:00:17,989 - selftest - INFO - Running bitbake -e to get BBPATH 2015-12-28 18:00:19,045 - selftest - INFO - Checking that everything is in order before running the tests 2015-12-28 18:00:20,093 - selftest - INFO - Running bitbake -p 2015-12-28 18:00:57,230 - selftest - INFO - Loading tests from: oeqa.selftest.imagefeatures.ImageFeatures.test_user Traceback (most recent call last): File "/home/humberto/poky/scripts/oe-selftest", line 518, in <module> ret = main() File "/home/humberto/poky/scripts/oe-selftest", line 469, in main suite.addTests(loader.loadTestsFromName(test)) File "/usr/lib/python2.7/unittest/loader.py", line 124, in loadTestsFromName raise TypeError("don't know how to make test from: %s" % obj) TypeError: don't know how to make test from: tester 57485 1 daniel.alexandrux.istrate 2015-12-29 11:15:57 +0000 Hi, I took a quick look at this, here are my findings: --list-classes does a weak validation when determining the test cases names (if method.startswith("test_"):), that is why it finds the class variable 'test_user' as a valid test case name. This should be replaced with: if method.startswith("test_") and hasattr(vars(t)[method], '__call__'): like it is done in get_tests_from_module(tmod) from oe-selftest also. While applying this fix I noticed something else: Traceback (most recent call last): File "/home/daniel/poky/scripts/oe-selftest", line 524, in <module> ret = main() File "/home/daniel/poky/scripts/oe-selftest", line 413, in main if method.startswith("test_") and hasattr(vars(t)[method], '__call__'): KeyError: 'test_buildhistory_does_not_change_signatures' This is because 'test_buildhistory_does_not_change_signatures' is in buildhistory.py/BuildhistoryBase which is inherited in other modules like: layerappend, oescripts, buildoptions. Take a look at the next output and you will see 'test_buildhistory_does_not_change_signatures' appear many times. 12:26:16 [daniel@fedora-ws poky-build]$ oe-selftest --list-classes 2015-12-29 12:33:44,439 - selftest - INFO - Running bitbake -e to get BBPATH 2015-12-29 12:33:47,591 - selftest - INFO - Listing all available test modules: _sstatetests_noauto (hidden) -- SStateBase -- RebuildFromSState -- -- test_sstate_rebuild -- -- test_sstate_relocation _toaster (hidden) No module named django.db.backends.signals archiver -- Archiver -- -- test_archiver_allows_to_filter_on_recipe_name bblayers -- BitbakeLayers -- -- test_bitbakelayers_add_remove -- -- test_bitbakelayers_flatten -- -- test_bitbakelayers_showappends -- -- test_bitbakelayers_showcrossdepends -- -- test_bitbakelayers_showlayers -- -- test_bitbakelayers_showoverlayed -- -- test_bitbakelayers_showrecipes bbtests -- BitbakeTests -- -- test_bitbake_g -- -- test_bitbake_invalid_recipe -- -- test_bitbake_invalid_target -- -- test_checkuri -- -- test_continue -- -- test_dry_run -- -- test_environment -- -- test_event_handler -- -- test_force_task_1 -- -- test_force_task_2 -- -- test_image_manifest -- -- test_invalid_patch -- -- test_invalid_recipe_src_uri -- -- test_just_parse -- -- test_local_sstate -- -- test_non_gplv3 -- -- test_postfile -- -- test_prefile -- -- test_rename_downloaded_file -- -- test_run_bitbake_from_dir_1 -- -- test_run_bitbake_from_dir_2 -- -- test_version -- -- test_warnings_errors buildhistory -- BuildhistoryBase -- -- test_buildhistory_does_not_change_signatures buildoptions -- DiskMonTest -- -- test_stoptask_behavior -- SanityOptionsTest -- -- test_options_warnqa_errorqa_switch -- -- test_sanity_userspace_dependency -- ImageOptionsTests -- -- test_ccache_tool -- -- test_incremental_image_generation -- -- test_rm_old_image -- BuildImagesTest -- -- test_directfb -- ArchiverTest -- -- test_arch_work_dir_and_export_source -- BuildhistoryTests -- -- test_buildhistory_basic -- -- test_buildhistory_buildtime_pr_backwards -- -- test_buildhistory_does_not_change_signatures -- BuildhistoryBase -- -- test_buildhistory_does_not_change_signatures devtool -- DevtoolBase -- DevtoolTests -- -- test_create_workspace -- -- test_devtool_add -- -- test_devtool_add_fetch -- -- test_devtool_add_fetch_git -- -- test_devtool_add_fetch_simple -- -- test_devtool_add_library -- -- test_devtool_build_image -- -- test_devtool_deploy_target -- -- test_devtool_extract -- -- test_devtool_extract_virtual -- -- test_devtool_layer_plugins -- -- test_devtool_modify -- -- test_devtool_modify_git -- -- test_devtool_modify_invalid -- -- test_devtool_modify_localfiles -- -- test_devtool_modify_native -- -- test_devtool_modify_virtual -- -- test_devtool_reset_all -- -- test_devtool_update_recipe -- -- test_devtool_update_recipe_append -- -- test_devtool_update_recipe_append_git -- -- test_devtool_update_recipe_git -- -- test_devtool_update_recipe_local_files -- -- test_devtool_update_recipe_local_files_2 -- -- test_devtool_upgrade imagefeatures -- ImageFeatures -- -- test_all_users_can_connect_via_ssh_without_password -- -- test_clutter_image_can_be_built -- -- test_non_root_user_can_connect_via_ssh_without_password -- -- test_rpm_version_4_support_on_image -- -- test_user -- -- test_wayland_support_in_image layerappend -- LayerAppendTests -- -- test_layer_appends -- BuildhistoryBase -- -- test_buildhistory_does_not_change_signatures lic-checksum -- LicenseTests -- -- test_nonmatching_checksum manifest -- VerifyManifest -- -- test_SDK_manifest_entries -- -- test_image_manifest_entries oescripts -- TestScripts -- -- test_cleanup_workdir -- BuildhistoryDiffTests -- -- test_buildhistory_diff -- -- test_buildhistory_does_not_change_signatures -- BuildhistoryBase -- -- test_buildhistory_does_not_change_signatures pkgdata -- OePkgdataUtilTests -- -- test_find_path -- -- test_glob -- -- test_list_pkg_files -- -- test_list_pkgs -- -- test_lookup_pkg -- -- test_lookup_recipe -- -- test_read_value -- -- test_specify_pkgdatadir prservice -- BitbakePrTests -- -- test_import_export_override_db -- -- test_import_export_replace_db -- -- test_pr_service_deb_arch_dep -- -- test_pr_service_deb_arch_indep -- -- test_pr_service_ipk_arch_dep -- -- test_pr_service_ipk_arch_indep -- -- test_pr_service_rpm_arch_dep -- -- test_pr_service_rpm_arch_indep recipetool -- RecipetoolTests -- -- test_recipetool_appendfile_add -- -- test_recipetool_appendfile_add_bindir -- -- test_recipetool_appendfile_add_machine -- -- test_recipetool_appendfile_alternatives -- -- test_recipetool_appendfile_basic -- -- test_recipetool_appendfile_binary -- -- test_recipetool_appendfile_extlayer -- -- test_recipetool_appendfile_inst_func -- -- test_recipetool_appendfile_inst_glob -- -- test_recipetool_appendfile_inst_todir_glob -- -- test_recipetool_appendfile_invalid -- -- test_recipetool_appendfile_orig -- -- test_recipetool_appendfile_patch -- -- test_recipetool_appendfile_postinstall -- -- test_recipetool_appendfile_renamed -- -- test_recipetool_appendfile_script -- -- test_recipetool_appendfile_src_glob -- -- test_recipetool_appendfile_subdir -- -- test_recipetool_appendfile_todir -- -- test_recipetool_appendfile_wildcard -- -- test_recipetool_create -- -- test_recipetool_create_git -- -- test_recipetool_create_simple -- RecipetoolAppendsrcTests -- -- test_recipetool_appendsrcfile_basic -- -- test_recipetool_appendsrcfile_basic_wildcard -- -- test_recipetool_appendsrcfile_existing_in_src_uri -- -- test_recipetool_appendsrcfile_existing_in_src_uri_diff_params -- -- test_recipetool_appendsrcfile_replace_file_srcdir -- -- test_recipetool_appendsrcfile_srcdir_basic -- -- test_recipetool_appendsrcfile_subdir_basic -- -- test_recipetool_appendsrcfile_subdir_basic_dirdest -- -- test_recipetool_appendsrcfiles_basic -- -- test_recipetool_appendsrcfiles_basic_subdir -- RecipetoolBase -- RecipetoolAppendsrcBase signing -- Signing -- -- test_signing_packages -- -- test_signing_sstate_archive sstate -- SStateBase sstatetests -- SStateBase -- SStateTests -- -- test_cleansstate_task_distro_nonspecific -- -- test_cleansstate_task_distro_specific -- -- test_cleansstate_task_distro_specific_nonspecific -- -- test_rebuild_distro_specific_sstate_cross_native_targets -- -- test_rebuild_distro_specific_sstate_cross_target -- -- test_rebuild_distro_specific_sstate_native_target -- -- test_sstate_32_64_same_hash -- -- test_sstate_allarch_samesigs -- -- test_sstate_cache_management_script_using_machine -- -- test_sstate_cache_management_script_using_pr_1 -- -- test_sstate_cache_management_script_using_pr_2 -- -- test_sstate_cache_management_script_using_pr_3 -- -- test_sstate_creation_distro_nonspecific_fail -- -- test_sstate_creation_distro_nonspecific_pass -- -- test_sstate_creation_distro_specific_fail -- -- test_sstate_creation_distro_specific_pass -- -- test_sstate_nativelsbstring_same_hash -- -- test_sstate_sametune_samesigs wic -- Wic -- -- test_build_artifacts -- -- test_build_image_name -- -- test_compress_bzip2 -- -- test_compress_gzip -- -- test_compress_xz -- -- test_createhelp -- -- test_directdisk_bootloader_config -- -- test_gpt_image -- -- test_help -- -- test_help_kickstart -- -- test_help_overview -- -- test_help_plugins -- -- test_image_env -- -- test_iso_image -- -- test_listhelp -- -- test_mkefidisk -- -- test_mkgummidisk -- -- test_no_command -- -- test_qemux86_directdisk -- -- test_rootfs_artifacts -- -- test_rootfs_indirect_recipes -- -- test_unsupported_subcommand -- -- test_wic_image_type -- -- test_wrong_compressor gummiboot -- Gummiboot -- -- test_efi_gummiboot_images_can_be_built -- -- test_wic_command_can_create_efi_gummiboot_installation_images 12:33:47 [daniel@fedora-ws poky-build]$ Some solutions that I see: 1. move the 'test_buildhistory_does_not_change_signatures' somewhere else, like BuildhistoryTests 2. make --list-classes obsolete and implement a new option for --list-tests-by all which will nicely lay out all available test cases. 3. replace 'test_user' with 'testuser' :) Thanks, -- Daniel 57505 2 ross.burton 2016-01-04 12:11:18 +0000 (1) Definitely move the test somewhere else (a dedicated buildhistory test class in buildhistory.py?), and leave a comment that no tests should be added to the base class. (2) Either fixing the logic in --list-tests, or making --list-tests simply an alias for (and implementing) --list-tests-by=all, is good as far as I'm concerned. (3) Fixing (2) will fix this for us so no need to rename it really. 57507 3 daniel.alexandrux.istrate 2016-01-04 12:25:37 +0000 Patch sent to ML: http://lists.openembedded.org/pipermail/openembedded-core/2016-January/114746.html 57509 4 daniel.alexandrux.istrate 2016-01-04 12:49:01 +0000 Opened bug for buildhistory test issue (1. from Ross's comment) Bug 8867 57511 5 daniel.alexandrux.istrate 2016-01-04 13:44:06 +0000 Sent Patch V2 to ML: http://lists.openembedded.org/pipermail/openembedded-core/2016-January/114752.html 57516 6 daniel.alexandrux.istrate 2016-01-04 15:40:46 +0000 Opened bug for issue no 2. from Ross's comment: Bug 8868 57626 7 ross.burton 2016-01-07 15:37:08 +0000 Fixed in oe-core 175810503d5596370cf7d840539ebdf35cf30278.