Bug 732 - It costs a lot of time when wget missing sstate file from http/ftp server
Summary: It costs a lot of time when wget missing sstate file from http/ftp server
Status: RESOLVED WONTFIX
Alias: None
Product: BitBake
Classification: Build System, Metadata & Runtime
Component: bitbake (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Low minor
Target Milestone: 1.0 M4
Assignee: Edwin Zhai
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2011-02-16 23:05 UTC by Edwin Zhai
Modified: 2011-04-29 01:20 UTC (History)
6 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Edwin Zhai 2011-02-16 23:05:23 UTC
After enabling sstate cache behind http/ftp server, bitbake try to get required sstate file from the server, and work hard even the file is missing on the server.

Wget, the default fetching command, will have several retries, and several seconds  timeout each for every missing sstate file. This cause the build process very very slow.

This behavior is okay for fetching source tar ball, as they always exist there. But sstate file does not.

One work around is use NFS server, but we can't export the sstate cache externally.

Possible solution is changing the fetching behavior for sstate file, or using extra parameters for sstate fetching to make wget failed early upon missing file.
Comment 1 Richard Purdie 2011-02-27 02:22:32 UTC
I agree we should change the wget command to avoid the retires/timeouts.
Comment 2 Christopher Larson 2011-04-06 08:52:34 UTC
It may be worth using urllib here just to determine if creating the connection errors, to fairly quickly (I hope) determine if the file is even there, before calling wget to get it.  This could also give us an easy way to tell, ahead of time if necessary, which things will be restored from sstates and which won't.
Comment 3 Christopher Larson 2011-04-06 08:53:15 UTC
(In reply to comment #2)
> It may be worth using urllib here just to determine if creating the connection
> errors, to fairly quickly (I hope) determine if the file is even there, before
> calling wget to get it.  This could also give us an easy way to tell, ahead of
> time if necessary, which things will be restored from sstates and which won't.

Of course, that only works for wget URIs, but we can't do everything :)
Comment 4 Edwin Zhai 2011-04-29 01:02:19 UTC
(In reply to comment #2)
> It may be worth using urllib here just to determine if creating the connection
> errors, to fairly quickly (I hope) determine if the file is even there, before
> calling wget to get it.  This could also give us an easy way to tell, ahead of
> time if necessary, which things will be restored from sstates and which won't.

urllib has some limitation:
google = urllib.urlopen('http://www.google.com')
google.getcode()
getcode() detect missing file behind http:// by error code 404, but only get none for ftp:// regardless  of file existence.

So we still need custom wget command line.
Comment 5 Edwin Zhai 2011-04-29 01:20:02 UTC
On latest master, I can't reproduce this issue.  Default retry parameter in poky, "wget -t 5" doesn't start a new retry upon missing file for both http and ftp. This retry mechanism only works when download fail.

Previous observation should be caused by abnormal network environment or wrong wgetrc. I'd like to close this bug and reopen it if we can reproduce it in future.

To change the wget command line for sstate, we need change meta/classes/sstate.bbclass: pstaging_fetch() by adding following before fetcher.download():

bb.data.setVar("FETCHCOMMAND_wget", "/usr/bin/env wget WITH RIGHT PARAMETER", localdata)

Default config is @ meta/conf/bitbake.conf

This change only affect local data, so global environ val is safe.