Bug 8862

Hi,

I took a quick look at this, here are my findings:

--list-classes does a weak validation when determining the test cases names (if method.startswith("test_"):), that is why it finds the class variable 'test_user' as a valid test case name. This should be replaced with:
if method.startswith("test_") and hasattr(vars(t)[method], '__call__'): like it is done in get_tests_from_module(tmod) from oe-selftest also.

While applying this fix I noticed something else:
Traceback (most recent call last):
  File "/home/daniel/poky/scripts/oe-selftest", line 524, in <module>
    ret = main()
  File "/home/daniel/poky/scripts/oe-selftest", line 413, in main
    if method.startswith("test_") and hasattr(vars(t)[method], '__call__'):
KeyError: 'test_buildhistory_does_not_change_signatures'

This is because 'test_buildhistory_does_not_change_signatures' is in buildhistory.py/BuildhistoryBase which is inherited in other modules like: layerappend, oescripts, buildoptions. Take a look at the next output and you will see 'test_buildhistory_does_not_change_signatures' appear many times.

12:26:16 [daniel@fedora-ws poky-build]$ oe-selftest --list-classes
2015-12-29 12:33:44,439 - selftest - INFO - Running bitbake -e to get BBPATH
2015-12-29 12:33:47,591 - selftest - INFO - Listing all available test modules:
_sstatetests_noauto (hidden)
 -- SStateBase
 -- RebuildFromSState
 --  -- test_sstate_rebuild
 --  -- test_sstate_relocation
_toaster (hidden)
No module named django.db.backends.signals
archiver
 -- Archiver
 --  -- test_archiver_allows_to_filter_on_recipe_name
bblayers
 -- BitbakeLayers
 --  -- test_bitbakelayers_add_remove
 --  -- test_bitbakelayers_flatten
 --  -- test_bitbakelayers_showappends
 --  -- test_bitbakelayers_showcrossdepends
 --  -- test_bitbakelayers_showlayers
 --  -- test_bitbakelayers_showoverlayed
 --  -- test_bitbakelayers_showrecipes
bbtests
 -- BitbakeTests
 --  -- test_bitbake_g
 --  -- test_bitbake_invalid_recipe
 --  -- test_bitbake_invalid_target
 --  -- test_checkuri
 --  -- test_continue
 --  -- test_dry_run
 --  -- test_environment
 --  -- test_event_handler
 --  -- test_force_task_1
 --  -- test_force_task_2
 --  -- test_image_manifest
 --  -- test_invalid_patch
 --  -- test_invalid_recipe_src_uri
 --  -- test_just_parse
 --  -- test_local_sstate
 --  -- test_non_gplv3
 --  -- test_postfile
 --  -- test_prefile
 --  -- test_rename_downloaded_file
 --  -- test_run_bitbake_from_dir_1
 --  -- test_run_bitbake_from_dir_2
 --  -- test_version
 --  -- test_warnings_errors
buildhistory
 -- BuildhistoryBase
 --  -- test_buildhistory_does_not_change_signatures
buildoptions
 -- DiskMonTest
 --  -- test_stoptask_behavior
 -- SanityOptionsTest
 --  -- test_options_warnqa_errorqa_switch
 --  -- test_sanity_userspace_dependency
 -- ImageOptionsTests
 --  -- test_ccache_tool
 --  -- test_incremental_image_generation
 --  -- test_rm_old_image
 -- BuildImagesTest
 --  -- test_directfb
 -- ArchiverTest
 --  -- test_arch_work_dir_and_export_source
 -- BuildhistoryTests
 --  -- test_buildhistory_basic
 --  -- test_buildhistory_buildtime_pr_backwards
 --  -- test_buildhistory_does_not_change_signatures
 -- BuildhistoryBase
 --  -- test_buildhistory_does_not_change_signatures
devtool
 -- DevtoolBase
 -- DevtoolTests
 --  -- test_create_workspace
 --  -- test_devtool_add
 --  -- test_devtool_add_fetch
 --  -- test_devtool_add_fetch_git
 --  -- test_devtool_add_fetch_simple
 --  -- test_devtool_add_library
 --  -- test_devtool_build_image
 --  -- test_devtool_deploy_target
 --  -- test_devtool_extract
 --  -- test_devtool_extract_virtual
 --  -- test_devtool_layer_plugins
 --  -- test_devtool_modify
 --  -- test_devtool_modify_git
 --  -- test_devtool_modify_invalid
 --  -- test_devtool_modify_localfiles
 --  -- test_devtool_modify_native
 --  -- test_devtool_modify_virtual
 --  -- test_devtool_reset_all
 --  -- test_devtool_update_recipe
 --  -- test_devtool_update_recipe_append
 --  -- test_devtool_update_recipe_append_git
 --  -- test_devtool_update_recipe_git
 --  -- test_devtool_update_recipe_local_files
 --  -- test_devtool_update_recipe_local_files_2
 --  -- test_devtool_upgrade
imagefeatures
 -- ImageFeatures
 --  -- test_all_users_can_connect_via_ssh_without_password
 --  -- test_clutter_image_can_be_built
 --  -- test_non_root_user_can_connect_via_ssh_without_password
 --  -- test_rpm_version_4_support_on_image
 --  -- test_user
 --  -- test_wayland_support_in_image
layerappend
 -- LayerAppendTests
 --  -- test_layer_appends
 -- BuildhistoryBase
 --  -- test_buildhistory_does_not_change_signatures
lic-checksum
 -- LicenseTests
 --  -- test_nonmatching_checksum
manifest
 -- VerifyManifest
 --  -- test_SDK_manifest_entries
 --  -- test_image_manifest_entries
oescripts
 -- TestScripts
 --  -- test_cleanup_workdir
 -- BuildhistoryDiffTests
 --  -- test_buildhistory_diff
 --  -- test_buildhistory_does_not_change_signatures
 -- BuildhistoryBase
 --  -- test_buildhistory_does_not_change_signatures
pkgdata
 -- OePkgdataUtilTests
 --  -- test_find_path
 --  -- test_glob
 --  -- test_list_pkg_files
 --  -- test_list_pkgs
 --  -- test_lookup_pkg
 --  -- test_lookup_recipe
 --  -- test_read_value
 --  -- test_specify_pkgdatadir
prservice
 -- BitbakePrTests
 --  -- test_import_export_override_db
 --  -- test_import_export_replace_db
 --  -- test_pr_service_deb_arch_dep
 --  -- test_pr_service_deb_arch_indep
 --  -- test_pr_service_ipk_arch_dep
 --  -- test_pr_service_ipk_arch_indep
 --  -- test_pr_service_rpm_arch_dep
 --  -- test_pr_service_rpm_arch_indep
recipetool
 -- RecipetoolTests
 --  -- test_recipetool_appendfile_add
 --  -- test_recipetool_appendfile_add_bindir
 --  -- test_recipetool_appendfile_add_machine
 --  -- test_recipetool_appendfile_alternatives
 --  -- test_recipetool_appendfile_basic
 --  -- test_recipetool_appendfile_binary
 --  -- test_recipetool_appendfile_extlayer
 --  -- test_recipetool_appendfile_inst_func
 --  -- test_recipetool_appendfile_inst_glob
 --  -- test_recipetool_appendfile_inst_todir_glob
 --  -- test_recipetool_appendfile_invalid
 --  -- test_recipetool_appendfile_orig
 --  -- test_recipetool_appendfile_patch
 --  -- test_recipetool_appendfile_postinstall
 --  -- test_recipetool_appendfile_renamed
 --  -- test_recipetool_appendfile_script
 --  -- test_recipetool_appendfile_src_glob
 --  -- test_recipetool_appendfile_subdir
 --  -- test_recipetool_appendfile_todir
 --  -- test_recipetool_appendfile_wildcard
 --  -- test_recipetool_create
 --  -- test_recipetool_create_git
 --  -- test_recipetool_create_simple
 -- RecipetoolAppendsrcTests
 --  -- test_recipetool_appendsrcfile_basic
 --  -- test_recipetool_appendsrcfile_basic_wildcard
 --  -- test_recipetool_appendsrcfile_existing_in_src_uri
 --  -- test_recipetool_appendsrcfile_existing_in_src_uri_diff_params
 --  -- test_recipetool_appendsrcfile_replace_file_srcdir
 --  -- test_recipetool_appendsrcfile_srcdir_basic
 --  -- test_recipetool_appendsrcfile_subdir_basic
 --  -- test_recipetool_appendsrcfile_subdir_basic_dirdest
 --  -- test_recipetool_appendsrcfiles_basic
 --  -- test_recipetool_appendsrcfiles_basic_subdir
 -- RecipetoolBase
 -- RecipetoolAppendsrcBase
signing
 -- Signing
 --  -- test_signing_packages
 --  -- test_signing_sstate_archive
sstate
 -- SStateBase
sstatetests
 -- SStateBase
 -- SStateTests
 --  -- test_cleansstate_task_distro_nonspecific
 --  -- test_cleansstate_task_distro_specific
 --  -- test_cleansstate_task_distro_specific_nonspecific
 --  -- test_rebuild_distro_specific_sstate_cross_native_targets
 --  -- test_rebuild_distro_specific_sstate_cross_target
 --  -- test_rebuild_distro_specific_sstate_native_target
 --  -- test_sstate_32_64_same_hash
 --  -- test_sstate_allarch_samesigs
 --  -- test_sstate_cache_management_script_using_machine
 --  -- test_sstate_cache_management_script_using_pr_1
 --  -- test_sstate_cache_management_script_using_pr_2
 --  -- test_sstate_cache_management_script_using_pr_3
 --  -- test_sstate_creation_distro_nonspecific_fail
 --  -- test_sstate_creation_distro_nonspecific_pass
 --  -- test_sstate_creation_distro_specific_fail
 --  -- test_sstate_creation_distro_specific_pass
 --  -- test_sstate_nativelsbstring_same_hash
 --  -- test_sstate_sametune_samesigs
wic
 -- Wic
 --  -- test_build_artifacts
 --  -- test_build_image_name
 --  -- test_compress_bzip2
 --  -- test_compress_gzip
 --  -- test_compress_xz
 --  -- test_createhelp
 --  -- test_directdisk_bootloader_config
 --  -- test_gpt_image
 --  -- test_help
 --  -- test_help_kickstart
 --  -- test_help_overview
 --  -- test_help_plugins
 --  -- test_image_env
 --  -- test_iso_image
 --  -- test_listhelp
 --  -- test_mkefidisk
 --  -- test_mkgummidisk
 --  -- test_no_command
 --  -- test_qemux86_directdisk
 --  -- test_rootfs_artifacts
 --  -- test_rootfs_indirect_recipes
 --  -- test_unsupported_subcommand
 --  -- test_wic_image_type
 --  -- test_wrong_compressor
gummiboot
 -- Gummiboot
 --  -- test_efi_gummiboot_images_can_be_built
 --  -- test_wic_command_can_create_efi_gummiboot_installation_images
12:33:47 [daniel@fedora-ws poky-build]$


Some solutions that I see:
1. move the 'test_buildhistory_does_not_change_signatures' somewhere else, like BuildhistoryTests
2. make --list-classes obsolete and implement a new option for --list-tests-by all which will nicely lay out all available test cases.
3. replace 'test_user' with 'testuser' :)

Thanks,
-- Daniel

(1) Definitely move the test somewhere else (a dedicated buildhistory test class in buildhistory.py?), and leave a comment that no tests should be added to the base class.

(2) Either fixing the logic in --list-tests, or making --list-tests simply an alias for (and implementing) --list-tests-by=all, is good as far as I'm concerned.

(3) Fixing (2) will fix this for us so no need to rename it really.

Patch sent to ML: http://lists.openembedded.org/pipermail/openembedded-core/2016-January/114746.html

Opened bug for buildhistory test issue (1. from Ross's comment) Bug 8867

Sent Patch V2 to ML: http://lists.openembedded.org/pipermail/openembedded-core/2016-January/114752.html

Opened bug for issue no 2. from Ross's comment: Bug 8868

Fixed in oe-core 175810503d5596370cf7d840539ebdf35cf30278.