| Summary: | It costs a lot of time when wget missing sstate file from http/ftp server | ||
|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] BitBake | Reporter: | Edwin Zhai <edwin.zhai> |
| Component: | bitbake | Assignee: | Edwin Zhai <edwin.zhai> |
| Status: | RESOLVED WONTFIX | QA Contact: | |
| Severity: | minor | ||
| Priority: | Low | CC: | kergoth, kevin.tian, poky.bs.watcher, poky.watcher, richard.purdie, sgw |
| Version: | unspecified | ||
| Target Milestone: | 1.0 M4 | ||
| Hardware: | x86 | ||
| OS: | Multiple | ||
| Whiteboard: | |||
| OS type for building Yocto: | --- | Type of Regression: | --- |
| Verified: | Documentation change: | --- | |
|
Description
Edwin Zhai
2011-02-16 23:05:23 UTC
I agree we should change the wget command to avoid the retires/timeouts. It may be worth using urllib here just to determine if creating the connection errors, to fairly quickly (I hope) determine if the file is even there, before calling wget to get it. This could also give us an easy way to tell, ahead of time if necessary, which things will be restored from sstates and which won't. (In reply to comment #2) > It may be worth using urllib here just to determine if creating the connection > errors, to fairly quickly (I hope) determine if the file is even there, before > calling wget to get it. This could also give us an easy way to tell, ahead of > time if necessary, which things will be restored from sstates and which won't. Of course, that only works for wget URIs, but we can't do everything :) (In reply to comment #2) > It may be worth using urllib here just to determine if creating the connection > errors, to fairly quickly (I hope) determine if the file is even there, before > calling wget to get it. This could also give us an easy way to tell, ahead of > time if necessary, which things will be restored from sstates and which won't. urllib has some limitation: google = urllib.urlopen('http://www.google.com') google.getcode() getcode() detect missing file behind http:// by error code 404, but only get none for ftp:// regardless of file existence. So we still need custom wget command line. On latest master, I can't reproduce this issue. Default retry parameter in poky, "wget -t 5" doesn't start a new retry upon missing file for both http and ftp. This retry mechanism only works when download fail.
Previous observation should be caused by abnormal network environment or wrong wgetrc. I'd like to close this bug and reopen it if we can reproduce it in future.
To change the wget command line for sstate, we need change meta/classes/sstate.bbclass: pstaging_fetch() by adding following before fetcher.download():
bb.data.setVar("FETCHCOMMAND_wget", "/usr/bin/env wget WITH RIGHT PARAMETER", localdata)
Default config is @ meta/conf/bitbake.conf
This change only affect local data, so global environ val is safe.
|