Bug 7796 - SState: wget improvements
Summary: SState: wget improvements
Status: VERIFIED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: core (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium+ enhancement
Target Milestone: 1.9 M2
Assignee: Aníbal Limón
QA Contact: Daniel Istrate
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2015-05-20 16:34 UTC by Benjamin Esquivel
Modified: 2015-10-26 15:48 UTC (History)
6 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Don't know


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Comment 1 Richard Purdie 2015-06-04 14:25:34 UTC
There are two stages for this. One is parallelising the "present in sstate" checks. The patches for this have been separated out, cleaned up and merged.

There are also improvements to the wget fetcher, particularly so it can reuse conections rather than making a connection attempt per url. There is improved code for this in the branch which now uses TLS to hold the connection cache. 

To work correctly, the code had to be switched from wget to urllib2. We have to hack the class to make it reuse connections though (and also to support HEAD requests). We need to use the class to ensure proxy information is handled correctly.

My rpurdie/t222 branch has the improved code but more cleanup needs to be done, along with better tests before it can be submitted.
Comment 4 Daniel Istrate 2015-10-26 15:48:01 UTC
Verified on master: 505a82673ac2487df5ea343a6422c2fc47018831

The tests from bitbake/lib/bb/tests/fetch.py --> FetchCheckStatusTest validate this feature.

[daniel@fedora-ws poky-build]$ bitbake-selftest -v bb.tests.fetch.FetchCheckStatusTest
test_wget_checkstatus (bb.tests.fetch.FetchCheckStatusTest) ... ok
test_wget_checkstatus_connection_cache (bb.tests.fetch.FetchCheckStatusTest) ... ok

----------------------------------------------------------------------
Ran 2 tests in 23.763s

OK
[daniel@fedora-ws poky-build]$