Bug 9484

(In reply to comment #0)
> Port the Selenium tests owned by the QA team to the Django+Selenium style
> tests we have in Toaster under bitbake/lib/toaster/tests/browser.
> 
> These test could either be converted to unit tests and fitted into the
> existing test suite; or a new test suite for end-to-end tests, where whole
> workflows are being tested, could be added. 
> 
> Any UI tests which don't require bitbake to be run can be converted to this
> style of test. (The main criterion is that these tests can be run in a CI
> environment which does not require command-line bitbake to be run.)
> 
> For tests which need a lot of data, suggest that QA create a database
> containing the required records in the db, then dump it to a fixture file
> and load it into the tests where it is required.
> 
> Tests which depend on bitbake runs (i.e. they are actually testing bitbake
> and how Toaster gathers its data) may need a separate QA test suite.

Hi Elliot,

The current tests of QA team for toaster correspond to Functional testing, i mean the original idea was to be able to test user case scenarios, that's the reason why the current test suite loads a toaster instance, then build images and run the test cases.

In other hand your test cases correspond to Unit testing because you are generating scenarios into the database and then validate the well behavior into the Web browser UI. 

We agree that we need to maintain a common software libraries like the selenium helper and to have a common folder for all the test cases (this will be toaster/tests/).

But to be able to load a toaster instance and then the Functional tests against it we need a runner (it exist already and needs clean-up) [1][2] or there is a way to load a full toaster instance inside django testing env?

I hope i was clear and i'll expect your feedback.

Cheers,
   alimon

[1] http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/tree/bitbake/lib/toaster/tests/helpers.py?h=alimon/toaster_tests_squash
[2] http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/tree/bitbake/bin/toaster-test?h=alimon/toaster_tests_squash

Hi Alimon.

You're quite right, our tests are unit tests.

I would definitely like to retain the functional tests, no question there.

In the unit tests, you effectively *are* loading a full Django instance: the Selenium client fetches web pages from a running Toaster web server in a browser. The only difference is in the database, where we load fixtures rather than generate the data by running actual builds.

I was hoping that we could make use of the extensive work you've done writing UI tests, which could still make sense even if the data were stub data (i.e. loading from a fixture).

You may also be interested in the toaster-eventreplay script, which now works again thanks to Ed. This can load a bitbake event JSON file into toasterui, which effectively allows you to mimic a bitbake build in a fraction of the time.

So, in an ideal world, we should be able to run the QA tests in the following ways:

1. By doing the actual builds (scripted by Selenium, open the browser, press the build button, wait for the results etc.) to populate the database (indirectly via toasterui) then testing the outputs in the UI (with Selenium).

2. By running a bitbake JSON event file through toaster-eventreplay to populate the database, then testing the outputs in the UI (with Selenium).

3. By loading fake data into the database, then testing the outputs in the UI (with Selenium).

The "testing the outputs in the UI" part is the piece I would like to see translated into Selenium tests in the style of our Toaster unit tests. The same set of tests could work against the UI regardless of how the data gets into the database, whether by running the actual builds, running toaster-eventreplay with a bitbake JSON event file, or loading from fixtures.

So I'm happy for you to continue with writing scripts etc. to populate the database any way you want. But I'd like to be able to easily run those scripts to create data dumps (either JSON event files or database fixtures), then be able to run your Selenium UI tests in our unit testing environment against that dumped data.

Hi Elliot,

(In reply to comment #2)
> Hi Alimon.
> 
> You're quite right, our tests are unit tests.
> 
> I would definitely like to retain the functional tests, no question there.

Great!.

> 
> In the unit tests, you effectively *are* loading a full Django instance: the
> Selenium client fetches web pages from a running Toaster web server in a
> browser. The only difference is in the database, where we load fixtures
> rather than generate the data by running actual builds.
> 
> I was hoping that we could make use of the extensive work you've done
> writing UI tests, which could still make sense even if the data were stub
> data (i.e. loading from a fixture).
> 
> You may also be interested in the toaster-eventreplay script, which now
> works again thanks to Ed. This can load a bitbake event JSON file into
> toasterui, which effectively allows you to mimic a bitbake build in a
> fraction of the time.
> 
> So, in an ideal world, we should be able to run the QA tests in the
> following ways:
> 
> 1. By doing the actual builds (scripted by Selenium, open the browser, press
> the build button, wait for the results etc.) to populate the database
> (indirectly via toasterui) then testing the outputs in the UI (with
> Selenium).

This way are our functional tests.

> 
> 2. By running a bitbake JSON event file through toaster-eventreplay to
> populate the database, then testing the outputs in the UI (with Selenium).
> 
> 3. By loading fake data into the database, then testing the outputs in the
> UI (with Selenium).

I like the idea, we need to review our current tests and if one/many of them match to this case, we migrate into UI test cases.

> 
> The "testing the outputs in the UI" part is the piece I would like to see
> translated into Selenium tests in the style of our Toaster unit tests. The
> same set of tests could work against the UI regardless of how the data gets
> into the database, whether by running the actual builds, running
> toaster-eventreplay with a bitbake JSON event file, or loading from fixtures.
> 
> So I'm happy for you to continue with writing scripts etc. to populate the
> database any way you want. But I'd like to be able to easily run those
> scripts to create data dumps (either JSON event files or database fixtures),
> then be able to run your Selenium UI tests in our unit testing environment
> against that dumped data.

For clarify, Libertad and I will start to work on review the current Functional tests and discover if some of them match the criteria of point 2 and point 3 and the migrate into browser unit test cases.

The remaining tests will be the Functional testing (point 1) and we need create a separated folder into toaster/tests called fuctional also for this test is a plan to reuse the Selenium helper that you wrote.

Sounds like a good plan? what you think?

Cheers,
    alimon

Yes, sounds good to me, please go ahead.

If you are using a separate runner for your tests, we just need to be careful if you create a directory under tests called functional: this is because if you call manage.py test tests, test*.py files in all of the directories under tests/ are added to the test suite. There may be a way to avoid this behaviour. 

Alternatively, we could have a folder called tests-functional inside bitbake/lib/toaster which houses the functional tests and isn't scanned by the Django test runner.

Brian, do you have any thoughts on this?

Yes, sounds good to me, please go ahead.

If you are using a separate runner for your tests, we just need to be careful if you create a directory under tests called functional: this is because if you call manage.py test tests, test*.py files in all of the directories under tests/ are added to the test suite. There may be a way to avoid this behaviour. 

Alternatively, we could have a folder called tests-functional inside bitbake/lib/toaster which houses the functional tests and isn't scanned by the Django test runner.

Brian, do you have any thoughts on this?

(In reply to comment #5)
> Yes, sounds good to me, please go ahead.
> 
> If you are using a separate runner for your tests, we just need to be
> careful if you create a directory under tests called functional: this is
> because if you call manage.py test tests, test*.py files in all of the
> directories under tests/ are added to the test suite. There may be a way to
> avoid this behaviour. 
> 
> Alternatively, we could have a folder called tests-functional inside
> bitbake/lib/toaster which houses the functional tests and isn't scanned by
> the Django test runner.
> 
> Brian, do you have any thoughts on this?

Hi Elliot,

In order to reuse your Selenium helper, i created a base class that don't depend on django here are the patches rebased to current master.

https://github.com/alimon/bitbake/commits/toaster_tests

Also, i'll send it to the ML.

Cheers,
    alimon

The first part for decouple the SeleniumHelper for use without Django was integrated at,

http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=1bf1e2c208ad137b9b91ca0687d6fb9486045dfd

This bug makes reference to the test cases of the toaster_automation_tests.py.

These test cases were erased from master due to the following problem stated by Michael Wood (Comments below). Action plan for this is to recheck every test case and determine if we reintegrate it, fixed or if we erase it from our test plan.

> There are a bunch of old tests in a file called 
> toaster_automation_tests.py in morty (removed in master) which 
> reference some of the test cases which correspond to test cases on the 
> bugzilla plugin here 
> https://bugzilla.yoctoproject.org/tr_list_cases.cgi?product_id=61&limi
> t=25&current_tab=case
> (needs login to bugzilla)
> 
> http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/lib/toaste
> r/contrib/tts/toasteruitest/toaster_automation_test.py?h=morty
> 
> 
> These tests were unfortunately written in a very fragile way which is 
> why they currently don't work and were removed from the tree in master.
> 
> 
> Problems with the old tests:
> -------------------------------------------
> 
> * Accessing the database directly
> 
> This is an absolute no no, Django has an abstraction on top of the 
> selected database back-end (of which there are many) and everything 
> underneath is internal to Django framework; field names, tables and 
> relationships are internal implementation details which can change at 
> any point and are not necessarily in the 'control' of the Toaster 
> developers, therefore there are no guarantees that running raw sql 
> queries will ever work.
> 
> Example in toaster_automation_test.py:
> "
>  con=sqlite.connect('toaster.sqlite')
>  cursor = con.cursor()
>  query = "select value from orm_toastersetting where name like 'DEFCONF%';"
> 
> "
> 
> The correct way to do this is:
> ToasterSettings.objects.filter(name__startswith="DEFCONF")
> 
> 
> * Use of xpath to find html elements/properties
> 
> Using the xml xpath of the elements in a html page is very fragile. If 
> one of the elements is moved in the document then the whole path 
> becomes invalid. E.g. if a button is inside a nav menu and the xpath 
> is /div/nav/button[@id="start"]/ if the button were to be moved then 
> the xpath is no longer valid. The xpaths are also quite hard to read.
> 
> Example in toaster_automation_test.py
> "
>    # Check the default table is sorted by Recipe
>    tasks_column_count =
> len(self.driver.find_elements_by_xpath("/html/body/div[2]/div/div[2]/d
> iv[2]/table/tbody/tr/td[1]"))
> 
>    print(tasks_column_count)
> ...
>  "
> 
> The correct way to do this is make sure that there is class or id in 
> the html document which can be referenced.
> 
> self.driver.find_element_by_id('start')
> 
> http://selenium-python.readthedocs.io/locating-elements.html
> 
> We've been running tests as a patchwork <->CI using Travis here:
> https://travis-ci.org/toastertester/toaster-next
>
>

starting work on this

testecases were erased obsolete