bitbake-selftest is not fun when using a slow network connection, as it will clone several large repositories each time it is run. While one can skip the network tests, there is no strict reason to fetch large repositories to cover the same functional tests. Thus, this request to enhance these tests to improve the speed of these tests. For example, bitbake-selftest took ~35 minutes (2068s) to run over a 20Mbit DSL connection. Skipping those tests (with BB_SKIP_NETTESTS), bitbake-selftest took ~3 minutes (178s) on the same machine. Even with a faster connection, it's not particularly nice to be pulling that much data across the network for each test run. A couple of possible solutions: 1) Allow the current set of repositories to be pre-fetched to a fast local storage location. bitbake-selftest could then fetch from those locations. One downside would be the requirement to serve the local copies over the same transports, which would complicate matters entirely 2) Curate a custom set of minimal test repositories that manifest the same characteristics being tested in the "live" repositories. This has the advantage of making these tests more self-contained and resilient, while also ideally distilling each test case down to its most minimal reproducible configuration.
Running them locally took about 11 minutes and the autobuilder is running them in 7 mins. The difference is neither I nor the autobuilder have nodejs available. I suspect is could be the nodejs tests which are taking a lot of the time. We have designed the tests to be pretty efficient/minimal where we can... Can you confirm you had nodejs available? Ideally a test run that or with the npm tests disabled would be interesting too, just to confirm my guess about where the time is being spent is correct.
I did not have nodejs available on the host machine. After installing npm (and its plethora of dependencies), the selftest took 15 minutes. Twice as fast as my previous results, but still not as fast as your results. I then went ahead and purged npm and re-ran the test; it took the same 15 minutes. I am not sure why my test runs the other day were taking so long, but I suspect there was a transient confounding factor with my network connection.
Zack or someone can make a proposal about how to deal with the potentially slow network parts of the tests.
I suggest this one be closed. Running tests today $ bitbake-selftest Ran 472 tests in 509.452s OK $ BB_SKIP_NETTESTS=yes bitbake-selftest Ran 472 tests in 279.613s OK (skipped=75) Running each of the tests which use networking (marked by the presence of skipIfNoNetwork()) does not show any outliers. The longest is $ bitbake-selftest bb.tests.fetch.FetcherNetworkTest Ran 25 tests in 101.075s OK And NPM is negligeable $ bitbake-selftest bb.tests.fetch.NPMTest Ran 25 tests in 28.721s OK So yes networking tests add time to the tests but nothing I would call unexpected. We could look at doing more shallow or single branch clones or the likes to cut down some traffic without removing the items being tested. What are your thoughts Richard, Zach?
No further comments so closing as I'm not sure what we'd do to change things.