Bug 7268

Summary: Oracle Berkley DB doesn't download
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Landgraf <vincent.landgraf>
Component: coreAssignee: Saul Wold <sgw>
Status: RESOLVED WONTFIX QA Contact:
Severity: critical    
Priority: Low CC: bluelightning, meta.mr.watcher, meta.watcher, poky.bs.watcher, poky.watcher, srifenbark
Version: 1.7.1   
Target Milestone: 1.8   
Hardware: All   
OS: x86_64   
Whiteboard: 02 March 2015: Suggested YP manual change
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Yes (doc changes required)

Description Landgraf 2015-01-30 08:39:58 UTC
I manually tried to download the package. It asks me to enter OTN credentials. Maybe that is the problem and only the latest version 6.1.19 can be downloaded without login.

-------- Output -------

$ bitbake core-image-minimal
WARNING: Host distribution "Ubuntu-14.10" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution.
Parsing recipes: 100% |#################################################################################################| Time: 00:00:54
Parsing of 893 .bb files complete (0 cached, 893 parsed). 1289 targets, 101 skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies

Build Configuration:
BB_VERSION        = "1.24.0"
BUILD_SYS         = "x86_64-linux"
NATIVELSBSTRING   = "Ubuntu-14.10"
TARGET_SYS        = "i586-poky-linux"
MACHINE           = "qemux86"
DISTRO            = "poky-tiny"
DISTRO_VERSION    = "1.7.1"
TUNE_FEATURES     = "m32 i586"
TARGET_FPU        = ""
meta
meta-yocto
meta-yocto-bsp    = "foo:6dd21a9f152a93e2df1178d7a5bd903d7edcf4be"

NOTE: Preparing runqueue
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
WARNING: Failed to fetch URL ftp://ftp.gmplib.org/pub/gmp-6.0.0/gmp-6.0.0a.tar.bz2, attempting MIRRORS if available
WARNING: Failed to fetch URL http://download.oracle.com/berkeley-db/db-6.0.30.tar.gz, attempting MIRRORS if available
ERROR: Fetcher failure: Fetch command failed with exit code 4, output:
Read error (Connection timed out) in headers.
Read error (Connection timed out) in headers.

ERROR: Function failed: Fetcher failure for URL: 'http://download.oracle.com/berkeley-db/db-6.0.30.tar.gz'. Unable to fetch URL from any source.
ERROR: Logfile of failure stored in: /home/vlandgraf/poky/build/tmp/work/x86_64-linux/db-native/6.0.30-r0/temp/log.do_fetch.20714
ERROR: Task 646 (virtual:native:/home/vlandgraf/poky/meta/recipes-support/db/db_6.0.30.bb, do_fetch) failed with exit code '1'
NOTE: Tasks Summary: Attempted 259 tasks of which 0 didn't need to be rerun and 1 failed.
Waiting for 0 running tasks to finish:

Summary: 1 task failed:
  virtual:native:/home/vlandgraf/poky/meta/recipes-support/db/db_6.0.30.bb, do_fetch
Summary: There were 3 WARNING messages shown.
Summary: There were 2 ERROR messages shown, returning a non-zero exit code.
Comment 1 Paul Eggleton 2015-01-30 17:35:31 UTC
The original URL seems to be working for me. Additionally, this file is part of our source mirror which is enabled by default, so even if the upstream URL is unavailable it should have been able to download it from there:

http://downloads.yoctoproject.org/mirror/sources/db-6.0.30.tar.gz

Was this just an intermittent error or are you still receiving it if you retry now?
Comment 2 Landgraf 2015-02-02 12:04:40 UTC
It seems I have a workaround for the problem. Background: We have a transparent firewall, that inspects downloads (archives) and delivers the content after inspection. Therefore the default timeout of 30 sec may gets exceeded. I first increased it to 5 min. but that was problematic, when downloading the linux kernel. So i increased it to 3000 seconds.

Here my patch: 

$ git diff -p
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 74813a2..55d8e96 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -572,7 +572,7 @@ CPAN_MIRROR = "http://search.cpan.org/CPAN"

 FETCHCMD_svn = "/usr/bin/env svn --non-interactive --trust-server-cert"
 FETCHCMD_cvs = "/usr/bin/env cvs"
-FETCHCMD_wget = "/usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate"
+FETCHCMD_wget = "/usr/bin/env wget -t 2 -T 3000 -nv --passive-ftp --no-check-certificate"
 FETCHCMD_bzr = "/usr/bin/env bzr"
 FETCHCMD_hg = "/usr/bin/env hg"


I'm not sure if you can do something about this problem in the default configuration. If not, it might be worth documenting this in the FAQ, since i needed some time to find that config...
Comment 3 Saul Wold 2015-02-05 15:38:39 UTC
This is not something we will not fix, this is specific to your firewall situation and as you already have a workaround for your situation. We will not change the default nor does it make sense to add an additional variable.

You can override the FETCHCMD_wget in your local.conf file.

We will add a section to the wiki/FAQ or Troubleshooting guide noting that if one has issues with downloading to extend the timeout and give this as an example
Comment 4 Scott Rifenbark 2015-02-12 17:52:54 UTC
Hi, 

So we currently do not document the FETCHCMD variable in the ref-manual glossary.  Should we be documenting this variable?  And, if we do, this is where I could note this behavior for this example firewall situation.  

Scott
Comment 5 Paul Eggleton 2015-02-12 18:17:10 UTC
FETCHCMD is documented in the bitbake manual. It is one of quite a number that is waiting for us to go through the exercise of bringing over bitbake variable definitions to the YP ref manual (and figure out a means of ensuring both are kept in sync).
Comment 6 Scott Rifenbark 2015-02-12 21:52:09 UTC
Hi, 

So here is a proposed change to the FETCHCMD variable documentation for the BB manual that would use this particular issue in an example:

-----------

FETCHCMD

Defines the command the BitBake fetcher module executes when running fetch operations. You need to use an override suffix when you use the variable (e.g. FETCHCMD_git or FETCHCMD_svn).

Typically, you do not need to hand-manipulate this variable.  However, if necessary (e.g. to increase the default timeout interval used when downloading files), you would set this variable in the bitbake.conf file.  Here is an example that increases the default wait time from 30 seconds to 3000 seconds.  Increasing the time could help in an environment that uses a transparent firewall where the system needs extra time to examine files downloading across the Internet:

     FETCHCMD_wget = "/usr/bin/env wget -t 2 -T 3000 -nv --passive-ftp --no-check-certificate"

----------

Let me know if this would work to help with the documentation part of this bug.  Also, would be good to know if I could then take the FETCHCMD variable description and simply drop it into the ref-manual. 

Thanks, 
Scott
Comment 7 Landgraf 2015-02-13 07:12:45 UTC
Hi,

this looks very helpful! Thx
Comment 8 Paul Eggleton 2015-02-13 09:15:18 UTC
Scott, I don't think extending the FETCHCMD entry in the variable reference is really what we need. New users encountering this problem aren't going to know to look here in order to fix this - this is more something we should mention in a "troubleshooting issues" section of the documentation.
Comment 9 Scott Rifenbark 2015-03-02 20:30:28 UTC
Paul, 

I agree that this proposed example in the FETCHCMD description does not directly address the trouble-shooting nature of issue.  However, I don't think it takes away from or hurts the existing docs.  It is an "oh-by-the-way" type of addition that enhances the description of FETCHCMD.  

We currently don't document FETCHCMD in the ref-manual.  I would like to add it as suggested here.  I think separately we could think of some way to address trouble-shooting issues that occasionally pop up within the team the community.  Maybe that could be added as new bug to the YP docs... "Create a general trouble-shooting area that would help users".  

Scott