Bug 13930

Current work-in-progress on getting the CROPS toaster-container fully functional and passing all tests (also updating to more modern Selenium for web GUI tests):
https://github.com/moto-timo/toaster-container/tree/timo/wip-build-fixes

The CROPS toaster-container uses a selenium docker container to perform tests.
https://hub.docker.com/r/selenium/standalone-firefox

The tests/runtests.sh script installs the selenium python module and then launches the toaster-container and the selenium container and then calls tests/smoketests.py with appropriate parameters.

For Buildbot, we probably don't want to require running selenium in a docker container. The following blog gives some hints:
https://hristos.lol/selenium-testing-django-application-locally-and-ci/

We may also need to modify the tests/smoketests.py script for the Autobuilder/Buildbot environment.

Perhaps the tests could be integrated into oe-selftest/oeqa, but if so the system requirements for functional Selenium (including chromedriver and/or geckodriver) would need to be tested for presence and if missing other test steps would be skipped.

Also interesting is the Selenium IDE:
"If you want to create quick bug reproduction scripts, create scripts to aid in automation-aided exploratory testing, then you want to use Selenium IDE; a Chrome and Firefox add-on that will do simple record-and-playback of interactions with the browser."

Which sounds like we could "easily" create new test cases to test more aspects of Toaster. Or suggest to users so they can give us reproducers for bugs.

https://www.selenium.dev/

For the CROPS toaster-container, the tests are run with a selenium container. This simplifies setting up Selenium on the host. The tests are very basic and only run on Firefox currently.
https://github.com/crops/toaster-container/tree/master/tests

These tests do catch regressions and they are run weekly on GitHub Actions.
https://github.com/crops/toaster-container/actions

The tests in bitbake itself are out of date and probably do not work anymore.
https://git.openembedded.org/bitbake/tree/lib/toaster/tests/browser/README references ancient versions of Selenium and steps that are likely broken.

Not likely to get to this anytime soon. Moving to "Unassigned" and "NEW".

During the YP Technical Meeting, we discussed installing Selenium on e.g. only the Ubuntu worker(s) (rather than using Docker and a container).

The instructions to install Selenium can be found in the getting started guide [1].

The steps are:
1. Install the Selenium Library (e.g. pip install selenium) [2]
2. Install the web browser drivers (currently our tests are for Firefox) [3][4]

The version of selenium installed (pip install) and the version of the web drivers would need to be kept in sync with the test cases/code. This is somewhat easier with the Docker container [5], but it is important to tear down the selenium container after running the tests or if the tests fail.

[1] https://www.selenium.dev/documentation/webdriver/getting_started/
[2] https://www.selenium.dev/documentation/webdriver/getting_started/install_library/
[3] https://www.selenium.dev/documentation/webdriver/getting_started/install_drivers/
[4] https://github.com/mozilla/geckodriver/releases
[5] https://hub.docker.com/r/selenium/standalone-firefox/tags

Build move to 5.0 -- ../Randy

Bulk move of 5.0 medium importance issues to 5.99.
Move to 5.1 if you want to actively work on an issue.

Move to 5.99 since no one is actively working on this issue.
Move to 5.x if you start on a task, please.

We added toaster testing to the autobuilder as part of the STF work.