Bug 8267 - Traceback with recipe folder containing a space
Summary: Traceback with recipe folder containing a space
Status: VERIFIED FIXED
Alias: None
Product: BitBake
Classification: Build System, Metadata & Runtime
Component: bitbake (show other bugs)
Version: 1.8
Hardware: x86 Multiple
: Medium+ minor
Target Milestone: 2.0
Assignee: Richard Purdie
QA Contact: Costin
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2015-09-08 09:27 UTC by Paul Eggleton
Modified: 2015-11-16 13:06 UTC (History)
5 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: New (Never tested)
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Paul Eggleton 2015-09-08 09:27:24 UTC
If a directory containing a recipe has a space in its name and the recipe has a local file in SRC_URI, you will receive an error during parse:

----------- snip -----------
ERROR: Command execution failed: Traceback (most recent call last):
  File "/home/paul/poky/poky/bitbake/lib/bb/command.py", line 101, in runAsyncCommand
    self.cooker.updateCache()
  File "/home/paul/poky/poky/bitbake/lib/bb/cooker.py", line 1514, in updateCache
    self.parser = CookerParser(self, filelist, masked)
  File "/home/paul/poky/poky/bitbake/lib/bb/cooker.py", line 1963, in __init__
    if not self.bb_cache.cacheValid(filename, appends):
  File "/home/paul/poky/poky/bitbake/lib/bb/cache.py", line 471, in cacheValid
    self.cacheValidUpdate(fn, appends)
  File "/home/paul/poky/poky/bitbake/lib/bb/cache.py", line 534, in cacheValidUpdate
    f, exist = f.split(":")
ValueError: need more than 1 value to unpack
----------- snip -----------

I'm not sure how but at one point I was actually able to get past parsing and start a build but then it blew up in the runqueue:

----------- snip -----------
NOTE: Preparing RunQueue
ERROR: An uncaught exception occured in runqueue, please see the failure below:
ERROR: Running idle function
Traceback (most recent call last):
  File "/home/paul/poky/poky/bitbake/lib/bb/runqueue.py", line 1031, in RunQueue._execute_runqueue():
                 self.rqexe = RunQueueExecuteDummy(self)
    >            if self.rqdata.prepare() == 0:
                     self.state = runQueueComplete
  File "/home/paul/poky/poky/bitbake/lib/bb/runqueue.py", line 811, in RunQueueData.prepare():
                             procdep.append(self.taskData.fn_index[self.runq_fnid[dep]] + "." + self.runq_task[dep])
    >                    self.runq_hash[task] = bb.parse.siggen.get_taskhash(self.taskData.fn_index[self.runq_fnid[task]], self.runq_task[task], procdep, self.dataCache)

  File "/home/paul/poky/poky/meta/lib/oe/sstatesig.py", line 113, in SignatureGeneratorOEBasicHash.get_taskhash(fn='/home/paul/poky/poky/meta/recipes-core/fb-uc1701-mod (wip)/woot.bb', task='do_fetch', deps=[], dataCache=<bb.cache.CacheData object at 0x7fe7117d4e50>):
         def get_taskhash(self, fn, task, deps, dataCache):
    >        h = super(bb.siggen.SignatureGeneratorBasicHash, self).get_taskhash(fn, task, deps, dataCache)

  File "/home/paul/poky/poky/bitbake/lib/bb/siggen.py", line 192, in SignatureGeneratorOEBasicHash.get_taskhash(fn='/home/paul/poky/poky/meta/recipes-core/fb-uc1701-mod (wip)/woot.bb', task='do_fetch', deps=[], dataCache=<bb.cache.CacheData object at 0x7fe7117d4e50>):
             if task in dataCache.file_checksums[fn]:
    >            checksums = bb.fetch2.get_file_checksums(dataCache.file_checksums[fn][task], recipename)
                 for (f,cs) in checksums:
  File "/home/paul/poky/poky/bitbake/lib/bb/fetch2/__init__.py", line 1139, in get_file_checksums(filelist='/home/paul/poky/poky/meta/recipes-core/fb-uc1701-mod (wip)/woot-1.0/poky/testfile:False /home/paul/poky/poky/meta/recipes-core/fb-uc1701-mod (wip)/woot/poky/testfile:False /home/paul/poky/poky/meta/recipes-core/fb-uc1701-mod (wip)/files/poky/testfile:False /home/paul/poky/poky/meta/recipes-core/fb-uc1701-mod (wip)/woot-1.0/qemux86-64/testfile:False /home/paul/poky/poky/meta/recipes-core/fb-uc1701-mod (wip)/woot/qemux86-64/testfile:False /home/paul/poky/poky/meta/recipes-core/fb-uc1701-mod (wip)/files/qemux86-64/testfile:False /home/paul/poky/poky/meta/recipes-core/fb-uc1701-mod (wip)/woot-1.0/qemuall/testfile:False /home/paul/poky/poky/meta/recipes-core/fb-uc1701-mod (wip)/woot/qemuall/testfile:False /home/paul/poky/poky/meta/recipes-core/fb-uc1701-mod (wip)/files/qemuall/testfile:False /home/paul/poky/poky/meta/recipes-core/fb-uc1701-mod (wip)/woot-1.0/x86-64/testfile:False /home/paul/poky/poky/meta/recipes-core/fb-uc1701-mod (wip)/woot/x86-64/testfile:False /home/paul/poky/poky/meta/recipes-core/fb-uc1701-mod (wip)/files/x86-64/testfile:False /home/paul/poky/poky/meta/recipes-core/fb-uc1701-mod (wip)/woot-1.0/testfile:False /home/paul/poky/poky/meta/recipes-core/fb-uc1701-mod (wip)/woot/testfile:True  ', pn='woot'):
         for pth in filelist.split():
    >        exist = pth.split(":")[1]
             if exist == "False":
IndexError: list index out of range
----------- snip -----------

The reason for this is most likely that the "file-checksums" varflag that base.bbclass sets (which is read by bitbake) is space-separated. The filename and checksum in each item are separated by colons as well so a colon in the path would probably break the code too.

(Mostly this is a case of "don't do that!" but we should be displaying a proper error instead of these nasty tracebacks).
Comment 1 Richard Purdie 2015-09-15 21:04:07 UTC
From 21de5ef838d7f4aeeeb91752572cb19d877c1520 Mon Sep 17 00:00:00 2001
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Date: Tue, 15 Sep 2015 20:43:34 +0000
Subject: cache: Handle spaces and colons in directory names for file-checksums

If there is a space in a directory name containing a file in file-checksums
(e.g. from a file:// url), you currently get tracebacks from bitbake. This
improves the code to handle colons and spaces in the file-checksums names
since it possible to figure out the correct names.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

diff --git a/bitbake/lib/bb/cache.py b/bitbake/lib/bb/cache.py
index ef4d660..ab09b08 100644
--- a/bitbake/lib/bb/cache.py
+++ b/bitbake/lib/bb/cache.py
@@ -528,7 +528,20 @@ class Cache(object):
 
         if hasattr(info_array[0], 'file_checksums'):
             for _, fl in info_array[0].file_checksums.items():
-                for f in fl.split():
+                fl = fl.strip()
+                while fl:
+                    # A .split() would be simpler but means spaces or colons in filenames would break
+                    a = fl.find(":True")
+                    b = fl.find(":False")
+                    if ((a < 0) and b) or ((b > 0) and (b < a)):
+                       f = fl[:b+6]
+                       fl = fl[b+7:]
+                    elif ((b < 0) and a) or ((a > 0) and (a < b)):
+                       f = fl[:a+5]
+                       fl = fl[a+6:]
+                    else:
+                       break
+                    fl = fl.strip()
                     if "*" in f:
                         continue
                     f, exist = f.split(":")
Comment 3 Costin 2015-11-04 16:10:06 UTC
Tested on last master commit fc45deac89ef63ca1c44e763c38ced7dfd72cbe1 on an Ubuntu 15.04 OS.

Steps i performed:
1) ~/costin-poky/meta/recipes-graphics$ pwd
/home/autobuilder/costin-poky/meta/recipes-graphics
~/costin-poky/meta/recipes-graphics$ mv xcursor-transparent-theme/ xcursor-transparent-\ theme/

2)~/costin-poky/meta/recipes-graphics$ bitbake xcursor-transparent-theme

and i get the same error messages as on comment#1.

I observed after that the patch is on contrib. not on master

reopening.
Comment 4 Costin 2015-11-04 16:10:52 UTC
i guess the best status would be WaitingForUpstrem
Comment 5 Ross Burton 2015-11-05 16:22:21 UTC
This was merged into bitbake as 87282b283921a58426f24fb21151db457c5bca66 and poky as 07aef86c3a5fde8c335c6697ebbb751f79bbde1d, which made it into 2.0.
Comment 6 Costin 2015-11-06 15:17:05 UTC
Marking as verified