| Summary: | bitbake can use alternate download methods than git | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] BitBake | Reporter: | Libertad <libertad.gonzalez.de.la.cruz> | ||||||||
| Component: | bitbake | Assignee: | Libertad <libertad.gonzalez.de.la.cruz> | ||||||||
| Status: | RESOLVED NOTABUG | QA Contact: | Libertad <libertad.gonzalez.de.la.cruz> | ||||||||
| Severity: | normal | ||||||||||
| Priority: | Medium+ | CC: | jose.perez.carranza, poky.bs.watcher, poky.watcher, richard.purdie, sgw | ||||||||
| Version: | 2.2 | ||||||||||
| Target Milestone: | 2.2 | ||||||||||
| Hardware: | x86 | ||||||||||
| OS: | Multiple | ||||||||||
| Whiteboard: | |||||||||||
| OS type for building Yocto: | --- | Type of Regression: | --- | ||||||||
| Verified: | Documentation change: | Don't know | |||||||||
| Attachments: |
|
||||||||||
I tested this by adding: 192.168.1.123 github.com to my /etc/hosts which basically means it cannot contact github.com, then running "bitbake file -c cleanall", then "bitbake file -c fetch". In the log for the fetch task I see: $ cat tmp/work/core2-64-poky-linux/file/5.28-r0/temp/log.do_fetch DEBUG: Executing python function do_fetch DEBUG: Executing python function base_do_fetch DEBUG: Trying PREMIRRORS DEBUG: For url ['git', 'github.com', '/file/file.git', '', '', OrderedDict()] comparing ['bzr', '.*', '/.*', '', '', OrderedDict()] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', OrderedDict()] DEBUG: For url ['git', 'github.com', '/file/file.git', '', '', OrderedDict()] comparing ['cvs', '.*', '/.*', '', '', OrderedDict()] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', OrderedDict()] DEBUG: For url ['git', 'github.com', '/file/file.git', '', '', OrderedDict()] comparing ['git', '.*', '/.*', '', '', OrderedDict()] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', OrderedDict()] DEBUG: For url git://github.com/file/file.git returning http://downloads.yoctoproject.org/mirror/sources/git2_github.com.file.file.git.tar.gz DEBUG: For url ['http', 'downloads.yoctoproject.org', '/mirror/sources/git2_github.com.file.file.git.tar.gz', '', '', OrderedDict()] comparing ['bzr', '.*', '/.*', '', '', OrderedDict()] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', OrderedDict()] DEBUG: For url ['http', 'downloads.yoctoproject.org', '/mirror/sources/git2_github.com.file.file.git.tar.gz', '', '', OrderedDict()] comparing ['cvs', '.*', '/.*', '', '', OrderedDict()] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', OrderedDict()] DEBUG: For url ['http', 'downloads.yoctoproject.org', '/mirror/sources/git2_github.com.file.file.git.tar.gz', '', '', OrderedDict()] comparing ['gitsm', '.*', '/.*', '', '', OrderedDict()] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', OrderedDict()] DEBUG: For url ['http', 'downloads.yoctoproject.org', '/mirror/sources/git2_github.com.file.file.git.tar.gz', '', '', OrderedDict()] comparing ['hg', '.*', '/.*', '', '', OrderedDict()] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', OrderedDict()] DEBUG: For url ['http', 'downloads.yoctoproject.org', '/mirror/sources/git2_github.com.file.file.git.tar.gz', '', '', OrderedDict()] comparing ['osc', '.*', '/.*', '', '', OrderedDict()] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', OrderedDict()] DEBUG: For url ['http', 'downloads.yoctoproject.org', '/mirror/sources/git2_github.com.file.file.git.tar.gz', '', '', OrderedDict()] comparing ['p4', '.*', '/.*', '', '', OrderedDict()] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', OrderedDict()] DEBUG: For url ['http', 'downloads.yoctoproject.org', '/mirror/sources/git2_github.com.file.file.git.tar.gz', '', '', OrderedDict()] comparing ['svn', '.*', '/.*', '', '', OrderedDict()] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', OrderedDict()] DEBUG: For url ['git', 'github.com', '/file/file.git', '', '', OrderedDict()] comparing ['gitsm', '.*', '/.*', '', '', OrderedDict()] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', OrderedDict()] DEBUG: For url ['git', 'github.com', '/file/file.git', '', '', OrderedDict()] comparing ['hg', '.*', '/.*', '', '', OrderedDict()] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', OrderedDict()] DEBUG: For url ['git', 'github.com', '/file/file.git', '', '', OrderedDict()] comparing ['osc', '.*', '/.*', '', '', OrderedDict()] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', OrderedDict()] DEBUG: For url ['git', 'github.com', '/file/file.git', '', '', OrderedDict()] comparing ['p4', '.*', '/.*', '', '', OrderedDict()] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', OrderedDict()] DEBUG: For url ['git', 'github.com', '/file/file.git', '', '', OrderedDict()] comparing ['svn', '.*', '/.*', '', '', OrderedDict()] to ['http', 'downloads.yoctoproject.org', '/mirror/sources/', '', '', OrderedDict()] DEBUG: Fetching http://downloads.yoctoproject.org/mirror/sources/git2_github.com.file.file.git.tar.gz using command '/usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /media/build3/sources/ 'http://downloads.yoctoproject.org/mirror/sources/git2_github.com.file.file.git.tar.gz'' DEBUG: Fetcher accessed the network with the command /usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /media/build3/sources/ 'http://downloads.yoctoproject.org/mirror/sources/git2_github.com.file.file.git.tar.gz' DEBUG: Running export PATH="/media/build1/poky/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/media/build1/poky/scripts:/media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/bin/x86_64-poky-linux:/media/build1/poky/build/tmp/sysroots/qemux86-64/usr/bin/crossscripts:/media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/sbin:/media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/bin:/media/build1/poky/build/tmp/sysroots/x86_64-linux/sbin:/media/build1/poky/build/tmp/sysroots/x86_64-linux/bin:/media/build1/poky/scripts:/media/build1/poky/bitbake/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"; export HOME="/home/richard"; /usr/bin/env wget -t 2 -T 30 -nv --passive-ftp --no-check-certificate -P /media/build3/sources/ 'http://downloads.yoctoproject.org/mirror/sources/git2_github.com.file.file.git.tar.gz' --progress=dot -v --2016-07-28 16:49:27-- http://downloads.yoctoproject.org/mirror/sources/git2_github.com.file.file.git.tar.gz Resolving downloads.yoctoproject.org (downloads.yoctoproject.org)... 198.145.20.127 Connecting to downloads.yoctoproject.org (downloads.yoctoproject.org)|198.145.20.127|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 9900919 (9.4M) [application/octet-stream] Saving to: ‘/media/build3/sources/git2_github.com.file.file.git.tar.gz’ 2016-07-28 16:50:16 (197 KB/s) - ‘/media/build3/sources/git2_github.com.file.file.git.tar.gz’ saved [9900919/9900919] DEBUG: Running export PATH="/media/build1/poky/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/media/build1/poky/scripts:/media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/bin/x86_64-poky-linux:/media/build1/poky/build/tmp/sysroots/qemux86-64/usr/bin/crossscripts:/media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/sbin:/media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/bin:/media/build1/poky/build/tmp/sysroots/x86_64-linux/sbin:/media/build1/poky/build/tmp/sysroots/x86_64-linux/bin:/media/build1/poky/scripts:/media/build1/poky/bitbake/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"; export HOME="/home/richard"; tar -xzf /media/build3/sources/git2_github.com.file.file.git.tar.gz DEBUG: Running export PATH="/media/build1/poky/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/media/build1/poky/scripts:/media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/bin/x86_64-poky-linux:/media/build1/poky/build/tmp/sysroots/qemux86-64/usr/bin/crossscripts:/media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/sbin:/media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/bin:/media/build1/poky/build/tmp/sysroots/x86_64-linux/sbin:/media/build1/poky/build/tmp/sysroots/x86_64-linux/bin:/media/build1/poky/scripts:/media/build1/poky/bitbake/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"; export HOME="/home/richard"; git -c core.fsyncobjectfiles=0 branch --contains 3c521817322a6bf5160cfeb09b9145ccde587b2a --list master 2> /dev/null | wc -l DEBUG: Running export PATH="/media/build1/poky/build/tmp/sysroots-uninative/x86_64-linux/usr/bin:/media/build1/poky/scripts:/media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/bin/x86_64-poky-linux:/media/build1/poky/build/tmp/sysroots/qemux86-64/usr/bin/crossscripts:/media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/sbin:/media/build1/poky/build/tmp/sysroots/x86_64-linux/usr/bin:/media/build1/poky/build/tmp/sysroots/x86_64-linux/sbin:/media/build1/poky/build/tmp/sysroots/x86_64-linux/bin:/media/build1/poky/scripts:/media/build1/poky/bitbake/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin"; export HOME="/home/richard"; git -c core.fsyncobjectfiles=0 branch --contains 3c521817322a6bf5160cfeb09b9145ccde587b2a --list master 2> /dev/null | wc -l DEBUG: Searching for debian-742262.patch in paths: /media/build1/poky/meta/recipes-devtools/file/file-5.28/poky /media/build1/poky/meta/recipes-devtools/file/file/poky /media/build1/poky/meta/recipes-devtools/file/files/poky /media/build1/poky/meta/recipes-devtools/file/file-5.28/qemux86-64 /media/build1/poky/meta/recipes-devtools/file/file/qemux86-64 /media/build1/poky/meta/recipes-devtools/file/files/qemux86-64 /media/build1/poky/meta/recipes-devtools/file/file-5.28/qemuall /media/build1/poky/meta/recipes-devtools/file/file/qemuall /media/build1/poky/meta/recipes-devtools/file/files/qemuall /media/build1/poky/meta/recipes-devtools/file/file-5.28/x86-64 /media/build1/poky/meta/recipes-devtools/file/file/x86-64 /media/build1/poky/meta/recipes-devtools/file/files/x86-64 /media/build1/poky/meta/recipes-devtools/file/file-5.28/ /media/build1/poky/meta/recipes-devtools/file/file/ /media/build1/poky/meta/recipes-devtools/file/files/ DEBUG: Searching for 0001-Add-P-prompt-into-Usage-info.patch in paths: /media/build1/poky/meta/recipes-devtools/file/file-5.28/poky /media/build1/poky/meta/recipes-devtools/file/file/poky /media/build1/poky/meta/recipes-devtools/file/files/poky /media/build1/poky/meta/recipes-devtools/file/file-5.28/qemux86-64 /media/build1/poky/meta/recipes-devtools/file/file/qemux86-64 /media/build1/poky/meta/recipes-devtools/file/files/qemux86-64 /media/build1/poky/meta/recipes-devtools/file/file-5.28/qemuall /media/build1/poky/meta/recipes-devtools/file/file/qemuall /media/build1/poky/meta/recipes-devtools/file/files/qemuall /media/build1/poky/meta/recipes-devtools/file/file-5.28/x86-64 /media/build1/poky/meta/recipes-devtools/file/file/x86-64 /media/build1/poky/meta/recipes-devtools/file/files/x86-64 /media/build1/poky/meta/recipes-devtools/file/file-5.28/ /media/build1/poky/meta/recipes-devtools/file/file/ /media/build1/poky/meta/recipes-devtools/file/files/ DEBUG: Python function base_do_fetch finished DEBUG: Python function do_fetch finished DEBUG: Executing python function write_srcrev DEBUG: Python function write_srcrev finished which shows it successfully downloading a mirror tarball from PREMIRRORS and never touching git. Is this a test that has worked before and is failing or is there some other setup error? Is the git proxy you're using also being used for http? It does seem the system is working as expected... What does the full failing do_fetch log look like in your case? Created attachment 3331 [details]
log.do_fetch
(In reply to comment #2) > What does the full failing do_fetch log look like in your case? The log.do_fetch file is attached in attachment 3331 [details]. This testcase has been ran successfully in my environment before. And there has been no changes to my environment or proxies. The following information is from my proxy configuration. /TC1058/poky/build$ cat /usr/local/bin/git-proxy #!/bin/bash PROXY=proxy-socks.fm.intel.com PORT=1080 METHOD="-X 5 -x ${PROXY}:${PORT}" /bin/nc ${METHOD} $* _________________________________________________- /TC1058/poky/build$ env|grep -i proxy http_proxy=http://proxy.fm.intel.com:911 ftp_proxy=http://proxy.fm.intel.com:911 GIT_PROXY_COMMAND=/usr/local/bin/git-proxy https_proxy=http://proxy.fm.intel.com:911 no_proxy=localhost,eam.intel.com,.devtools.intel.com,intelpedia.intel.com,zpn.intel.com,.jf.intel.com,.fm.intel.com,.intel.com,10.0.0.0/8 BB_ENV_EXTRAWHITE=ALL_PROXY BBPATH_EXTRA BB_NO_NETWORK BB_NUMBER_THREADS BB_SETSCENE_ENFORCE BB_SRCREV_POLICY DISTRO FTPS_PROXY FTP_PROXY GIT_PROXY_COMMAND HTTPS_PROXY HTTP_PROXY MACHINE NO_PROXY PARALLEL_MAKE SCREENDIR SDKMACHINE SOCKS5_PASSWD SOCKS5_USER SSH_AGENT_PID SSH_AUTH_SOCK STAMPS_DIR TCLIBC TCMODE all_proxy ftp_proxy ftps_proxy http_proxy https_proxy no_proxy Could you retry this test please? I'm rather puzzled as to why its failing. If you read your log, it correctly downloads: http://downloads.yoctoproject.org/mirror/sources/git2_github.com.file.file.git.tar.gz If then runs: git -c core.fsyncobjectfiles=0 branch --contains 3c521817322a6bf5160cfeb09b9145ccde587b2a --list master 2> /dev/null | wc -l which presumably fails, then it tries to fetch from origin over git. That tarball *should* contain this reference. I guess before doing this, could you run "md5sum /home/libby/TC1058/poky/build/downloads/git2_github.com.file.file.git.tar.gz" and see what the checksum is? I see 3f7412e6ac188ef0587f25e90222e8c2 git2_github.com.file.file.git.tar.gz here. You could also try directly downloading the file and see what checksum you get. Created attachment 3332 [details]
Console output & log.do_fetch.30025
This attachment has the console output the TC rerun and also at the end it has the complete log.do_fetch output
hmmm, in trying to run the md5sum I see that it says no such file or directory.
So, I erased my poky folder and started all over again.
1. cloned poky
2. git checkout to 36feb38045b7a2af86ece147fec54b0db3bf491f
3. chmod -x /usr/local/bin/git-proxy
4. bitbake -c fetchall core-image-sato
It still gives me the errors and the file git2_github.com.file.file.git.tar.gz does not exist, only git2_github.com.file.file.git.tar.gz.done.
see attachment 3332 [details]
I suspect the build cleans the file out when it fails to find the reference it needs. If you run "wget http://downloads.yoctoproject.org/mirror/sources/git2_github.com.file.file.git.tar.gz" and then run md5sum on the file it fetches, does that checksum match? You may need to copy and paste the wget command out the logs so you have the right proxy variables. I'm worrying the proxy is providing an incorrect old, stale copy of this file... (In reply to comment #8) > I suspect the build cleans the file out when it fails to find the reference > it needs. If you run "wget > http://downloads.yoctoproject.org/mirror/sources/git2_github.com.file.file. > git.tar.gz" and then run md5sum on the file it fetches, does that checksum > match? You may need to copy and paste the wget command out the logs so you > have the right proxy variables. > > I'm worrying the proxy is providing an incorrect old, stale copy of this > file... Downloading with wget in a different folder the first md5sum does not match all other downloads after the first I get the same checksum as you. libby@liberta1-MOBL1:~$ mkdir ZipFile libby@liberta1-MOBL1:~$ cd ZipFile/ libby@liberta1-MOBL1:~/ZipFile$ wget http://downloads.yoctoproject.org/mirror/sources/git2_github.com.file.file.git.tar.gz --2016-07-28 16:49:55-- http://downloads.yoctoproject.org/mirror/sources/git2_github.com.file.file.git.tar.gz Resolving proxy.fm.intel.com (proxy.fm.intel.com)... 10.1.192.48 Connecting to proxy.fm.intel.com (proxy.fm.intel.com)|10.1.192.48|:911... connected. Proxy request sent, awaiting response... 200 OK Length: 9086339 (8.7M) [application/octet-stream] Saving to: ‘git2_github.com.file.file.git.tar.gz’ git2_github.com.fil 100%[=====================>] 8.67M 1.30MB/s in 8.0s 2016-07-28 16:50:04 (1.09 MB/s) - ‘git2_github.com.file.file.git.tar.gz’ saved [9086339/9086339] libby@liberta1-MOBL1:~/ZipFile$ md5sum git2_github.com.file.file.git.tar.gz ae0c695c4e4da26d0cf3a5c10846e095 git2_github.com.file.file.git.tar.gz libby@liberta1-MOBL1:~/ZipFile$ libby@liberta1-MOBL1:~/ZipFile$ wget http://downloads.yoctoproject.org/mirror/sources/git2_github.com.file.file.git.tar.gz --2016-07-28 16:56:29-- http://downloads.yoctoproject.org/mirror/sources/git2_github.com.file.file.git.tar.gz Resolving proxy.fm.intel.com (proxy.fm.intel.com)... 10.1.192.48 Connecting to proxy.fm.intel.com (proxy.fm.intel.com)|10.1.192.48|:911... connected. Proxy request sent, awaiting response... 200 OK Length: 9900919 (9.4M) [application/octet-stream] Saving to: ‘git2_github.com.file.file.git.tar.gz.1’ git2_github.com.file.file.git.tar.gz.1 100%[=======================================================================================================>] 9.44M 1.76MB/s in 6.2s 2016-07-28 16:56:35 (1.52 MB/s) - ‘git2_github.com.file.file.git.tar.gz.1’ saved [9900919/9900919] libby@liberta1-MOBL1:~/ZipFile$ md5sum git2_github.com.file.file.git.tar.gz.1 3f7412e6ac188ef0587f25e90222e8c2 git2_github.com.file.file.git.tar.gz.1 This is not a bitbake issue, it looks like there is an issue with your proxy servers since it should not return two different files for the same url. You need to contact your IT department and ask them to make the proxies work properly... |
Created attachment 3329 [details] output of console log DESCRIPTION: Bitbake cannot fetch when the git-proxy script has no execution permissions BUILD: 2.2_M2 : 36feb38045b7a2af86ece147fec54b0db3bf491f ENVIRONMENT: Debian 4.3.5-1~bpo8+1 STEPS TO REPRODUCE: 1. Start by creating a fresh build directory: source oe-initbuild-env newbuild 2. break the proxy settings for git by issuing the following command: sudo chmod -x /usr/local/bin/git-proxy 3. fetch all required packages for a core-image-sato image by issuing bitbake -c fetchall core-image-sato 4. once bitbake finishes repair the git proxy settings with: sudo chmod +x /usr/local/bin/git-proxy EXPECTED OUTCOME: bitbake command issued on step 3 should complete successfully