Bug 14817

Summary:	improve bitbake-selftest network tests
Product:	[Build System, Metadata & Runtime] BitBake	Reporter:	Zach Welch <zach>
Component:	bitbake	Assignee:	Richard Purdie <richard.purdie>
Status:	RESOLVED WORKSFORME	QA Contact:
Severity:	enhancement
Priority:	Low	CC:	mark.asselstine, poky.bs.watcher, poky.watcher, randy.macleod
Version:	5.99
Target Milestone:	Future
Hardware:	x86
OS:	Multiple
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	No (bug/feature does not impact docs)

Description Zach Welch 2022-05-26 18:06:43 UTC

bitbake-selftest is not fun when using a slow network connection, as it will clone several large repositories each time it is run.  While one can skip the network tests, there is no strict reason to fetch large repositories to cover the same functional tests.  Thus, this request to enhance these tests to improve the speed of these tests.

For example, bitbake-selftest took ~35 minutes (2068s) to run over a 20Mbit DSL connection.  Skipping those tests (with BB_SKIP_NETTESTS), bitbake-selftest took ~3 minutes (178s) on the same machine.  Even with a faster connection, it's not particularly nice to be pulling that much data across the network for each test run.

A couple of possible solutions:

1) Allow the current set of repositories to be pre-fetched to a fast local storage location.  bitbake-selftest could then fetch from those locations.  One downside would be the requirement to serve the local copies over the same transports, which would complicate matters entirely

2) Curate a custom set of minimal test repositories that manifest the same characteristics being tested in the "live" repositories.  This has the advantage of making these tests more self-contained and resilient, while also ideally distilling each test case down to its most minimal reproducible configuration.

Comment 1 Richard Purdie 2022-05-29 16:22:54 UTC

Running them locally took about 11 minutes and the autobuilder is running them in 7 mins. The difference is neither I nor the autobuilder have nodejs available. I suspect is could be the nodejs tests which are taking a lot of the time. We have designed the tests to be pretty efficient/minimal where we can...

Can you confirm you had nodejs available? Ideally a test run that or with the npm tests disabled would be interesting too, just to confirm my guess about where the time is being spent is correct.

Comment 2 Zach Welch 2022-06-02 14:28:38 UTC

I did not have nodejs available on the host machine.  After installing npm (and its plethora of dependencies), the selftest took 15 minutes.  Twice as fast as my previous results, but still not as fast as your results.

I then went ahead and purged npm and re-ran the test; it took the same 15 minutes.  I am not sure why my test runs the other day were taking so long, but I suspect there was a transient confounding factor with my network connection.

Comment 3 Randy MacLeod 2022-06-02 14:42:03 UTC

Zack or someone can make a proposal about how to deal with the potentially slow network parts of the tests.

Comment 4 Mark 2022-11-01 02:01:00 UTC

I suggest this one be closed. Running tests today

$ bitbake-selftest
Ran 472 tests in 509.452s
OK

$ BB_SKIP_NETTESTS=yes bitbake-selftest
Ran 472 tests in 279.613s
OK (skipped=75)

Running each of the tests which use networking (marked by the presence of skipIfNoNetwork()) does not show any outliers.

The longest is
$ bitbake-selftest bb.tests.fetch.FetcherNetworkTest
Ran 25 tests in 101.075s
OK

And NPM is negligeable
$ bitbake-selftest bb.tests.fetch.NPMTest
Ran 25 tests in 28.721s
OK

So yes networking tests add time to the tests but nothing I would call unexpected. We could look at doing more shallow or single branch clones or the likes to cut down some traffic without removing the items being tested.

What are your thoughts Richard, Zach?

Comment 5 Richard Purdie 2024-08-07 21:15:20 UTC

No further comments so closing as I'm not sure what we'd do to change things.