Bug 8647

Notes after discussion with Paul:

This script is intended to be automatically run when someone sends a patch to the mailing list - see https://bugzilla.yoctoproject.org/show_bug.cgi?id=8648

It should be outputting some sort of directives that the calling script can interpret - it shouldn't be outputting a list of shell commands. It's intended for consumption by another script, not a human, so brief and easy to parse.

It needs to be able to handle change to any file in the tree... now, it can be that it doesn't know how to test everything and that's fine - this is meant to be for a quick "smoke test" of the patch, not exhaustive testing, but at the base level for example if it changes something in scripts/lib/devtool/ it needs to be telling me to run the devtool tests (or a subset of them)

It'll end up being a big map of files to tests; it may be possible to have decorators on the tests to define some of it.


This script would take as input a commit and suggest what tests to run.

$ analyze_patch 94decbce394f74f11f79f6779598426e423e1745
--> selftest bblayers.BitbakeLayers.test_bitbakelayers_showrecipes

Patch sent to ML for review:
http://lists.openembedded.org/pipermail/openembedded-core/2015-November/113117.html

The files can also be obtained from here:
https://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=distrate/bug_8647_script_analyse_patch_changes

To give it a try:

1. Get first 100 commits from poky
[daniel@fedora-ws poky]$ git log --oneline -n 100 | cut -d ' ' -f 1 > /tmp/first_100_commits.txt
2. Run the script against all of them
[daniel@fedora-ws poky-build]$ for i in `cat /tmp/first_100_commits.txt` ; do analyze_patch -r $i; done

This script works from the sourced env also.

Attached mail sent to Paul the previous week. I am still open to suggestions.

Hi Paul,

I managed to determine how the files (.py) influence one another, by following the import chain.  I attached a graph illustrating this for whoever is curious.
This is useful if you want to see what influence a commit change has.

Now I need a method to determine for the given affected files what tests to suggest.
How should I map the runtine/oe-selftest/bitbake (or other) tests?

We did a brainstorming here at QA + Cristian, in short this is what we came up with:
-	This is a good candidate for a neural net implementation :)
-	We need a mechanism that computes  some sort of dictionary based on the existing test cases that we have. So for each test case/ category of test cases to have some keywords associated. If this process is not fast enough we should consider caching it, can we use sstate for that? Also this process should be automated. Now, when a file is changed, what criteria should we consider for testing? Its name, its location, its content, its type?
-	What are all the tests that could be suggested? Based on what?
-	If there is a .bb, .bbappend file it means it’s a recipe and suggest to rebuild the recipe.

Richard suggested to do a mapping for bitbake tests (bitbake/lib/bb/tests) to bitbake/lib/bb/, as they have the same filename. So if a file from bitbake/lib/bb/ changed to look for a corresponding test in bitbake/lib/bb/tests/.


This is how the graph was generated:
daniel@ubuntu-ws:~/poky$ sfood . | grep -v /usr/lib/python | grep -v /usr/share/pyshared/ | grep -v None | grep ".py')," | grep ".py'))" | sfood-graph | dot -Tsvg -o deps.svg


Thanks,
--Daniel

Created attachment 2865 [details]
poky modules import dependencies graph

Some questions for Paul

(In reply to comment #1)
> Notes after discussion with Paul:
> 
> This script is intended to be automatically run when someone sends a patch
> to the mailing list - see
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=8648
> 
> It should be outputting some sort of directives that the calling script can
> interpret - it shouldn't be outputting a list of shell commands. It's
> intended for consumption by another script, not a human, so brief and easy
> to parse.

> 
> It needs to be able to handle change to any file in the tree... now, it can
> be that it doesn't know how to test everything and that's fine - this is
> meant to be for a quick "smoke test" of the patch, not exhaustive testing,
> but at the base level for example if it changes something in
> scripts/lib/devtool/ it needs to be telling me to run the devtool tests (or
> a subset of them)
> 

I was thinking of using oe-selftest to figure out the mapping between unit tests and the files being exercise. This way, there is no need to have some extra mapping (on a separate file, which at the end it will need maintenance).

> It'll end up being a big map of files to tests; it may be possible to have
> decorators on the tests to define some of it.
> 

Can you elaborate more on the decorators?

> 
> This script would take as input a commit and suggest what tests to run.
> 
> $ analyze_patch 94decbce394f74f11f79f6779598426e423e1745
> --> selftest bblayers.BitbakeLayers.test_bitbakelayers_showrecipes

Perhaps for M1, we should just throw the full command, and latter abstract a bit more as you mentioned before (<script to run the test> <parameters>)

Bug 8750 is sent for review.

Hi,

    To get a mapping of test case to files they 'touch' I was thinking to build upon bug 8679 (https://bugzilla.yoctoproject.org/show_bug.cgi?id=8679#c4). With a little parsing I was able to get a dictionary tests -> list of files they touch (will attach to bug).

    Doing analysis on this dictionary I noticed that there are lot of files 'touched' by tests that doesn't necessarily has something to do with the test (will attach to bug). Most of them are 'touched' because of the bitbake process that most tests spawn, which loads all it's modules even though some of the bitbake features are not specifically tested in that test.
    Take for instance 'bitbake/lib/bb/fetch2/clearcase.py'  which is 'touched' by all test cases (according to coverage), but as RP confirmed that there is no test atm that tests that.

    So this approach doesn't gives us an accurate mapping of 'random file' to test cases. Even though this will make it dependent of the output of the coverage that has to be cached, so it will always be a little out of sync.

    The other idea that I explored was to have feature tag decorators for test cases (bug 8750) that will describe what features a test case tests. This will allow to run tests based on that tag.

    So the problem is split into two parts: 'random file' -> feature tag -> test cases. Bug 8750 addresses the second part which maps feature tag to test cases.

    This still leaves the question open: How do you map a random file (from a commit message) to a feature tag/suite of tests to run?

Thanks,
--Daniel

Created attachment 2890 [details]
A dictionary of test cases mapped to list of files they touch (from coverage)

Created attachment 2891 [details]
by how many tests those files are touched

Assigning to Paul for more info.
Any suggestions on how to implement this, would be much appreciated.

Thanks,
--Daniel

M2 missed.

patchtest [1], the new tool to test patches from patchwork is almost done. A patch analysis should be included on the repository [2] and derived tests based on this.

[1] https://github.com/lsandoval/patchtest
[2] https://github.com/aphran/pt-suites

It appears the NEEDINFO has been answered.

Per our talks with Paul, this will move to 2.2 based on load.

Reviewing this, I think we don't have enough of an idea of how to practically implement it without signing ourselves up for a huge maintenance burden. It's also not something that should block the release. Therefore let's move it to Medium/Future until such time as we have a better idea of how to do it.

I don't think this is going to fly in the current state of the YP.

This is a good idea but it's not possible with current YP community participation levels.