Try to clean-up this code for merging http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=1e8f440ac5c2f0232286a6533a8e5052863e9fdf
There are two stages for this. One is parallelising the "present in sstate" checks. The patches for this have been separated out, cleaned up and merged. There are also improvements to the wget fetcher, particularly so it can reuse conections rather than making a connection attempt per url. There is improved code for this in the branch which now uses TLS to hold the connection cache. To work correctly, the code had to be switched from wget to urllib2. We have to hack the class to make it reuse connections though (and also to support HEAD requests). We need to use the class to ensure proxy information is handled correctly. My rpurdie/t222 branch has the improved code but more cleanup needs to be done, along with better tests before it can be submitted.
In review at, http://lists.openembedded.org/pipermail/openembedded-core/2015-June/106430.html http://lists.openembedded.org/pipermail/openembedded-core/2015-June/106439.html http://lists.openembedded.org/pipermail/bitbake-devel/2015-June/005912.html Integration branch at, http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=alimon/fetch2-wget-checkstatus
Merged at, http://cgit.openembedded.org/bitbake/commit/?id=9fa6407e6cefe66c77467419a8040d6957a6bb01 http://cgit.openembedded.org/openembedded-core/commit/?id=e6f66370c6ce15aca18ef64491bab3dc92b80c57
Verified on master: 505a82673ac2487df5ea343a6422c2fc47018831 The tests from bitbake/lib/bb/tests/fetch.py --> FetchCheckStatusTest validate this feature. [daniel@fedora-ws poky-build]$ bitbake-selftest -v bb.tests.fetch.FetchCheckStatusTest test_wget_checkstatus (bb.tests.fetch.FetchCheckStatusTest) ... ok test_wget_checkstatus_connection_cache (bb.tests.fetch.FetchCheckStatusTest) ... ok ---------------------------------------------------------------------- Ran 2 tests in 23.763s OK [daniel@fedora-ws poky-build]$