2558 2012-06-07 22:10:35 +0000 Fetcher code incorrectly forces fetch of files which are in the download directory or mirror 2012-06-20 09:39:28 +0000 1 1 7 Build System, Metadata & Runtime Meta-yocto meta-yocto 1.2 x86 Multiple VERIFIED FIXED Issue debugged and patch under discussion/testing High critical 1.3 M1 1 gary richard.purdie laurentiu.serban poky.bs.watcher poky.watcher sgw shane.wang --- --- --- oldest_to_newest 21921 0 gary 2012-06-07 22:10:35 +0000 Something in the fetcher code has changed such that the file gtk+-2.24.8.tar.bz2 is _always_ fetched from the network, even if it is already in a local mirror or download directory. Tested with poky master de4cdfd6bc1280ac7ac0559b87734d26294ef773 Note: this is easy to test, just set BB_NO_NETWORK="1" in local.conf and try to run 'bitbake gtk+ -c fetch' Since my customers insist on zero network use during builds, this is a critical bug/regression. 22057 1 richard.purdie 2012-06-12 14:00:35 +0000 This isn't as simple as you're describing it since I tried setting BB_NO_NETWORK = "1" in my local.conf (confirmed set with -e) and my fetch for gtk+ suceeded. I then ran a -c cleanall, and the fetch then failed afterwards. So is there something more specific about your environment needed to reproduce this? 22058 2 richard.purdie 2012-06-12 14:01:19 +0000 I'm wondering what values of PREMIRRORS and MIRRORS you're using? 22059 3 gary 2012-06-12 14:10:25 +0000 (In reply to comment #2) > I'm wondering what values of PREMIRRORS and MIRRORS you're using? I use "own-mirrors" set to a local directory, e.g. SOURCE_MIRROR_URL ?= "file:///work/misc/Poky/sources/" INHERIT += "own-mirrors" My comment about BB_NO_NETWORK was just to show when the file is being downloaded, not that setting it causes the problem. Sorry for any confusion. The real issue is that the source tarball is being fetched from the network even if it exists in the mirror. 22061 4 572 gary 2012-06-12 14:25:20 +0000 Created attachment 572 Log showing fetch behaviour 22136 5 richard.purdie 2012-06-13 15:27:45 +0000 A quick question: Are you only seeing this for gtk+ ? I think this patch might fix things: --- a/bitbake/lib/bb/fetch2/local.py +++ b/bitbake/lib/bb/fetch2/local.py @@ -26,6 +26,7 @@ BitBake build tools. # Based on functions from the base bb module, Copyright 2003 Holger Schurig import os +import urllib import bb import bb.utils from bb import data @@ -40,7 +41,7 @@ class Local(FetchMethod): def urldata_init(self, ud, d): # We don't set localfile as for this fetcher the file is already local! - ud.basename = os.path.basename(ud.url.split("://")[1].split(";")[0]) + ud.basename = os.path.basename(urllib.unquote(ud.url.split("://")[1].split(";")[0])) return def localpath(self, url, urldata, d): @@ -49,6 +50,7 @@ class Local(FetchMethod): """ path = url.split("://")[1] path = path.split(";")[0] + path = urllib.unquote(path) newpath = path if path[0] != "/": filespath = data.getVar('FILESPATH', d, True) 22141 6 richard.purdie 2012-06-13 15:44:33 +0000 *** Bug 2546 has been marked as a duplicate of this bug. *** 22143 7 gary 2012-06-13 15:53:14 +0000 (In reply to comment #5) > A quick question: Are you only seeing this for gtk+ ? Correct. It appears to be related to the '+' in the URL > > I think this patch might fix things: Tested this patch and it does work - the gtk+ tarball is no longer fetched from the network. > > --- a/bitbake/lib/bb/fetch2/local.py > +++ b/bitbake/lib/bb/fetch2/local.py > @@ -26,6 +26,7 @@ BitBake build tools. > # Based on functions from the base bb module, Copyright 2003 Holger Schurig > > import os > +import urllib > import bb > import bb.utils > from bb import data > @@ -40,7 +41,7 @@ class Local(FetchMethod): > > def urldata_init(self, ud, d): > # We don't set localfile as for this fetcher the file is already > local! > - ud.basename = os.path.basename(ud.url.split("://")[1].split(";")[0]) > + ud.basename = > os.path.basename(urllib.unquote(ud.url.split("://")[1].split(";")[0])) > return > > def localpath(self, url, urldata, d): > @@ -49,6 +50,7 @@ class Local(FetchMethod): > """ > path = url.split("://")[1] > path = path.split(";")[0] > + path = urllib.unquote(path) > newpath = path > if path[0] != "/": > filespath = data.getVar('FILESPATH', d, True) 22144 8 gary 2012-06-13 15:58:48 +0000 With the patch above, there is now a slight change in behaviour. As mentioned, I use "own-mirrors" set up like this: BB_GENERATE_MIRROR_TARBALLS ?= "1" SOURCE_MIRROR_URL ?= "file:///work/misc/Poky/sources/" INHERIT += "own-mirrors" Before, all files which exist on the mirror create a soft link in the local "downloads" directory, e.g. $ ls -l downloads/*v4l* lrwxrwxrwx 1 gthomas gthomas 63 Mar 27 10:57 downloads/git2_linuxtv.org.v4l-utils.git.tar.gz -> /local/poky-multi/sources/git2_linuxtv.org.v4l-utils.git.tar.gz After this change, they are not fetched from the network but seem to be copied from the mirror. I just ran this sequence: $ bitbake v4l-utils -c cleansstate $ rm downloads/*v4l* $ bitbake v4l-utils -c fetch which now shows $ ls -l downloads/*v4l* -rw-rw-r-- 1 gthomas gthomas 1439674 Jun 13 09:53 downloads/git2_linuxtv.org.v4l-utils.git.tar.gz I'll try to verify that the patch caused this change. 22145 9 gary 2012-06-13 16:01:28 +0000 (In reply to comment #8) > With the patch above, there is now a slight change in behaviour. As > mentioned, > I use "own-mirrors" set up like this: > BB_GENERATE_MIRROR_TARBALLS ?= "1" > SOURCE_MIRROR_URL ?= "file:///work/misc/Poky/sources/" > INHERIT += "own-mirrors" > > Before, all files which exist on the mirror create a soft link in the local > "downloads" directory, e.g. > $ ls -l downloads/*v4l* > lrwxrwxrwx 1 gthomas gthomas 63 Mar 27 10:57 > downloads/git2_linuxtv.org.v4l-utils.git.tar.gz -> > /local/poky-multi/sources/git2_linuxtv.org.v4l-utils.git.tar.gz > > After this change, they are not fetched from the network but seem to be > copied from the mirror. I just ran this sequence: > $ bitbake v4l-utils -c cleansstate > $ rm downloads/*v4l* > $ bitbake v4l-utils -c fetch > which now shows > $ ls -l downloads/*v4l* > -rw-rw-r-- 1 gthomas gthomas 1439674 Jun 13 09:53 > downloads/git2_linuxtv.org.v4l-utils.git.tar.gz > > I'll try to verify that the patch caused this change. Indeed, the patch is not the cause of this difference. Since it seems to be benign (still no network activity which is my main concern), I'm going to just note it but not worry about it. 22178 10 richard.purdie 2012-06-14 13:06:41 +0000 Fixed in master: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=1faf02599af12a62bbed0bce5fda6a72df63ec30 22406 11 laurentiu.serban 2012-06-20 09:39:28 +0000 Verified 572 2012-06-12 14:25:20 +0000 2012-06-12 14:25:20 +0000 Log showing fetch behaviour gtk.log text/x-log 4561 gary RG93bmxvYWQgc3RhdGUgZnJvbSBhIGJ1aWxkIG1hZGUgb24gSnVuZSAxIChiZWZvcmUgdGhlIGNo YW5nZSB0aGF0IGJyb2tlIHRoZSBmZXRjaGVyKToKICAgIFtndGhvbWFzQGV1cm9wYSBwNjBfcG9r eV0kIGxzIGRvd25sb2Fkcy9ndGsqIC1sCiAgICBscnd4cnd4cnd4IDEgZ3Rob21hcyBndGhvbWFz IDQ1IE1hciAyNyAxMDo0OSBkb3dubG9hZHMvZ3RrKy0yLjI0LjgudGFyLmJ6MiAtPiAvbG9jYWwv cG9reS1tdWx0aS9zb3VyY2VzL2d0aystMi4yNC44LnRhci5iejIKICAgIC1ydy1ydy1yLS0gMSBn dGhvbWFzIGd0aG9tYXMgIDAgSnVuICAxIDA3OjEwIGRvd25sb2Fkcy9ndGsrLTIuMjQuOC50YXIu YnoyLmRvbmUKICAgIFtndGhvbWFzQGV1cm9wYSBwNjBfcG9reV0kIGxzIC1sIC9sb2NhbC9wb2t5 LW11bHRpL3NvdXJjZXMvZ3RrKy0yLjI0LjgudGFyLmJ6MgogICAgLXJ3LXJ3LXItLSAxIGd0aG9t YXMgZ3Rob21hcyAxNjYwNzk4MCBOb3YgMTAgIDIwMTEgL2xvY2FsL3Bva3ktbXVsdGkvc291cmNl cy9ndGsrLTIuMjQuOC50YXIuYnoyCgpBcyB5b3UgY2FuIHNlZSwgdGhlIGZpbGUgd2FzIGZvdW5k IGluIG15IG1pcnJvci4KClJ1biBvbiBKdW5lIDEyLCBzaG93aW5nIHRoZSBmZXRjaCBmcm9tIHRo ZSBuZXR3b3JrOgogICAgW2d0aG9tYXNAZXVyb3BhIHA2MF9wb2t5XSQgcm0gZG93bmxvYWRzL2d0 ayoKICAgIFtndGhvbWFzQGV1cm9wYSBwNjBfcG9reV0kIGJpdGJha2UgZ3RrKyAtYyBjbGVhbnNz dGF0ZQogICAgTG9hZGluZyBjYWNoZTogMTAwJSB8IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj fCBFVEE6ICAwMDowMDowMAogICAgTG9hZGVkIDE0NTggZW50cmllcyBmcm9tIGRlcGVuZGVuY3kg Y2FjaGUuCiAgICAKICAgIEJ1aWxkIENvbmZpZ3VyYXRpb246CiAgICBCQl9WRVJTSU9OICAgICAg ICA9ICIxLjE1LjIiCiAgICBUQVJHRVRfQVJDSCAgICAgICA9ICJhcm0iCiAgICBUQVJHRVRfT1Mg ICAgICAgICA9ICJsaW51eC1nbnVlYWJpIgogICAgTUFDSElORSAgICAgICAgICAgPSAiY29icmEz NTMwcDYwIgogICAgRElTVFJPICAgICAgICAgICAgPSAiYW1sdGQiCiAgICBESVNUUk9fVkVSU0lP TiAgICA9ICIxLjIrc25hcHNob3QtMjAxMjA2MTIiCiAgICBUVU5FX0ZFQVRVUkVTICAgICA9ICJh cm12N2EgdmZwIG5lb24gY29ydGV4YTgiCiAgICBUQVJHRVRfRlBVICAgICAgICA9ICJ2ZnAtbmVv biIKICAgIG1ldGEgICAgICAgICAgICAgID0gIm15X21hc3RlcjplMWM3OWY3ZGUwNzY4YzUzM2Jj ZjA3ZDFiZTgxOTEwOTUyMGE1Njc0IgogICAgbWV0YS1vZSAgICAgICAgICAgPSAibWFzdGVyOmIy YzRmMjU4YjkzY2E1MDZmYzhmMjJkM2VjZTNkOWQ2NjM2M2FmZjYiCiAgICBtZXRhLWFtbHRkICAg ICAgICA9ICJtYXN0ZXI6YTkzYzNkY2VkODA0ODYxNTZhMjNkOGU0ZDI1YzRmMjM0MWQwN2E1YyIK ICAgIG1ldGEtYnJvd3NlciAgICAgID0gIm1hc3Rlcjo3NTYzMTU3MmM4NmVmMjI5OThlNGQzMThl YTk2NTY0YThlMmUxZDNlIgogICAgbWV0YS1jb2JyYTM1MzBwNjAgPSAibWFzdGVyOjg4NDMzNjIx MWFmYmMwZjY5NDY5MDBjNGY0NTk2OGNmY2ZmMGQ3M2MiCiAgICBtZXRhLXRpICAgICAgICAgICA9 ICJtYXN0ZXI6ZDMzMDU4MjI1ZGYxMTU2MTc0ZGIxMTMxZDk2NGRlMWVmMGFiM2I0OCIKICAgIG1l dGEtcGF0Y2hlcyAgICAgID0gIm1hc3RlcjphZWJkMTI3Y2U2NTg1NjNhM2RmZmViNDA0OTY1YjMw NWYzYzlkNjRlIgogICAgCiAgICBOT1RFOiBSZXNvbHZpbmcgYW55IG1pc3NpbmcgdGFzayBxdWV1 ZSBkZXBlbmRlbmNpZXMKICAgIE5PVEU6IFByZXBhcmluZyBydW5xdWV1ZQogICAgTk9URTogRXhl Y3V0aW5nIFJ1blF1ZXVlIFRhc2tzCiAgICBOT1RFOiBSdW5uaW5nIHRhc2sgMSBvZiAyIChJRDog MCwgL2xvY2FsL3Bva3ktbXVsdGkvbWV0YS9yZWNpcGVzLWdub21lL2d0aysvZ3RrK18yLjI0Ljgu YmIsIGRvX2NsZWFuKQogICAgTk9URTogcGFja2FnZSBndGsrLTIuMjQuOC1yNDogdGFzayBkb19j bGVhbjogU3RhcnRlZAogICAgTk9URTogcGFja2FnZSBndGsrLTIuMjQuOC1yNDogdGFzayBkb19j bGVhbjogU3VjY2VlZGVkCiAgICBOT1RFOiBSdW5uaW5nIHRhc2sgMiBvZiAyIChJRDogMSwgL2xv Y2FsL3Bva3ktbXVsdGkvbWV0YS9yZWNpcGVzLWdub21lL2d0aysvZ3RrK18yLjI0LjguYmIsIGRv X2NsZWFuc3N0YXRlKQogICAgTk9URTogcGFja2FnZSBndGsrLTIuMjQuOC1yNDogdGFzayBkb19j bGVhbnNzdGF0ZTogU3RhcnRlZAogICAgTk9URTogcGFja2FnZSBndGsrLTIuMjQuOC1yNDogdGFz ayBkb19jbGVhbnNzdGF0ZTogU3VjY2VlZGVkCiAgICBOT1RFOiBUYXNrcyBTdW1tYXJ5OiBBdHRl bXB0ZWQgMiB0YXNrcyBvZiB3aGljaCAwIGRpZG4ndCBuZWVkIHRvIGJlIHJlcnVuIGFuZCBhbGwg c3VjY2VlZGVkLgoKQXQgdGhpcyBwb2ludCwgdGhlIGRvd25sb2FkcyBbY2FjaGVdIGRvZXMgbm90 IGhhdmUgdGhlIHRhcmJhbGwuCiAgICBbZ3Rob21hc0BldXJvcGEgcDYwX3Bva3ldJCBiaXRiYWtl IGd0aysgLWMgZmV0Y2gKICAgIExvYWRpbmcgY2FjaGU6IDEwMCUgfCMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj IyMjIyMjIyMjI3wgRVRBOiAgMDA6MDA6MDAKICAgIExvYWRlZCAxNDU4IGVudHJpZXMgZnJvbSBk ZXBlbmRlbmN5IGNhY2hlLgogICAgCiAgICBCdWlsZCBDb25maWd1cmF0aW9uOgogICAgQkJfVkVS U0lPTiAgICAgICAgPSAiMS4xNS4yIgogICAgVEFSR0VUX0FSQ0ggICAgICAgPSAiYXJtIgogICAg VEFSR0VUX09TICAgICAgICAgPSAibGludXgtZ251ZWFiaSIKICAgIE1BQ0hJTkUgICAgICAgICAg ID0gImNvYnJhMzUzMHA2MCIKICAgIERJU1RSTyAgICAgICAgICAgID0gImFtbHRkIgogICAgRElT VFJPX1ZFUlNJT04gICAgPSAiMS4yK3NuYXBzaG90LTIwMTIwNjEyIgogICAgVFVORV9GRUFUVVJF UyAgICAgPSAiYXJtdjdhIHZmcCBuZW9uIGNvcnRleGE4IgogICAgVEFSR0VUX0ZQVSAgICAgICAg PSAidmZwLW5lb24iCiAgICBtZXRhICAgICAgICAgICAgICA9ICJteV9tYXN0ZXI6ZTFjNzlmN2Rl MDc2OGM1MzNiY2YwN2QxYmU4MTkxMDk1MjBhNTY3NCIKICAgIG1ldGEtb2UgICAgICAgICAgID0g Im1hc3RlcjpiMmM0ZjI1OGI5M2NhNTA2ZmM4ZjIyZDNlY2UzZDlkNjYzNjNhZmY2IgogICAgbWV0 YS1hbWx0ZCAgICAgICAgPSAibWFzdGVyOmE5M2MzZGNlZDgwNDg2MTU2YTIzZDhlNGQyNWM0ZjIz NDFkMDdhNWMiCiAgICBtZXRhLWJyb3dzZXIgICAgICA9ICJtYXN0ZXI6NzU2MzE1NzJjODZlZjIy OTk4ZTRkMzE4ZWE5NjU2NGE4ZTJlMWQzZSIKICAgIG1ldGEtY29icmEzNTMwcDYwID0gIm1hc3Rl cjo4ODQzMzYyMTFhZmJjMGY2OTQ2OTAwYzRmNDU5NjhjZmNmZjBkNzNjIgogICAgbWV0YS10aSAg ICAgICAgICAgPSAibWFzdGVyOmQzMzA1ODIyNWRmMTE1NjE3NGRiMTEzMWQ5NjRkZTFlZjBhYjNi NDgiCiAgICBtZXRhLXBhdGNoZXMgICAgICA9ICJtYXN0ZXI6YWViZDEyN2NlNjU4NTYzYTNkZmZl YjQwNDk2NWIzMDVmM2M5ZDY0ZSIKICAgIAogICAgTk9URTogUmVzb2x2aW5nIGFueSBtaXNzaW5n IHRhc2sgcXVldWUgZGVwZW5kZW5jaWVzCiAgICBOT1RFOiBQcmVwYXJpbmcgcnVucXVldWUKICAg IE5PVEU6IEV4ZWN1dGluZyBSdW5RdWV1ZSBUYXNrcwogICAgTk9URTogUnVubmluZyB0YXNrIDEg b2YgMSAoSUQ6IDAsIC9sb2NhbC9wb2t5LW11bHRpL21ldGEvcmVjaXBlcy1nbm9tZS9ndGsrL2d0 aytfMi4yNC44LmJiLCBkb19mZXRjaCkKICAgIE5PVEU6IHBhY2thZ2UgZ3RrKy0yLjI0LjgtcjQ6 IHRhc2sgZG9fZmV0Y2g6IFN0YXJ0ZWQKICAgIE5PVEU6IHBhY2thZ2UgZ3RrKy0yLjI0LjgtcjQ6 IHRhc2sgZG9fZmV0Y2g6IFN1Y2NlZWRlZAogICAgTk9URTogVGFza3MgU3VtbWFyeTogQXR0ZW1w dGVkIDEgdGFza3Mgb2Ygd2hpY2ggMCBkaWRuJ3QgbmVlZCB0byBiZSByZXJ1biBhbmQgYWxsIHN1 Y2NlZWRlZC4KICAgIFtndGhvbWFzQGV1cm9wYSBwNjBfcG9reV0kIGxzIC1sIGRvd25sb2Fkcy9n dGsqCiAgICAtcnctcnctci0tIDEgZ3Rob21hcyBndGhvbWFzIDE2NjA3OTgwIE5vdiAxMCAgMjAx MSBkb3dubG9hZHMvZ3RrKy0yLjI0LjgudGFyLmJ6MgogICAgLXJ3LXJ3LXItLSAxIGd0aG9tYXMg Z3Rob21hcyAgICAgICAgMCBKdW4gMTIgMDg6MTkgZG93bmxvYWRzL2d0aystMi4yNC44LnRhci5i ejIuZG9uZQoKVGhpcyBzaG93cyB0aGF0IHRoZSBmaWxlIHdhcyByZWZldGNoZWQgZnJvbSB0aGUg bmV0d29yaywgbm90IHVzaW5nIHRoZSBtaXJyb3IuCgpUaGUgbWlycm9yIHdvcmtzIHByb3Blcmx5 IGZvciBhbGwgb3RoZXIgZmlsZXMsIGp1c3Qgbm90IHRoaXMgb25lIHRoYXQgaGFwcGVucwp0byBo YXZlIGEgJysnIGluIHRoZSBuYW1lLiAgTm90ZTogdGhlcmUgd2FzIGEgdGhyZWFkIGFib3V0IHRo aXMgb24gdGhlIFlvY3RvCm1haWxpbmcgbGlzdCwgc3RhcnRpbmcgb24gMjAxMi0wNi0wMy4KICAg IA==