This bug doesn't seem to be as simple as states here as bitbake -b base-files -c unpack works fine for me :(

It is likely therefore some kind of complication in the layering but its making it harder to reproduce.

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=9d4f95587424c9e2d3ef7891c8ac7007f8db1f00 is a valid fix and seems to resolve the problem for Martin even if I can't reproduce it.

Part of original fix was reverted in
http://git.yoctoproject.org/cgit.cgi/poky/commit/bitbake/lib/bb/parse/parse_py/BBHandler.py?id=9d4f95587424c9e2d3ef7891c8ac7007f8db1f00

so it's broken again

Ah :/ not this commit but something else.. but still it's broken (and only broken when relative path is used in -b)

I think the working directories are not getting consistently set for some reason.

Martin, can you please provide a test case to reproduce this? I was hunting for a different problem today and came back to looking at this for various reasons but I can't replicate the bug :(.

I suspect it depends on some particular directory layout, some particular environment script or combination of things which I don't have...

I wasn't able to reproduce it today with oe-core only (but IIRC it was failing with oe-core only too), so I've tried to narrow it with default SHR setup


NOTE: Preparing runqueue
NOTE: Executing RunQueue Tasks
NOTE: Running task 1 of 3 (ID: 2, openembedded-core/meta/recipes-core/base-files/base-files_3.0.14.bb, do_rm_old_works)
NOTE: Running task 2 of 3 (ID: 1, openembedded-core/meta/recipes-core/base-files/base-files_3.0.14.bb, do_fetch)
NOTE: package base-files-3.0.14-r77: task do_rm_old_works: Started
NOTE: package base-files-3.0.14-r77: task do_rm_old_works: Succeeded
NOTE: package base-files-3.0.14-r77: task do_fetch: Started
ERROR: Function failed: Fetcher failure for URL: 'file://rotation'. Unable to fetch URL from any source.
ERROR: Logfile of failure stored in: /OE/shr-core/tmp-eglibc/work/om_gta02-oe-linux-gnueabi/base-files/base-files-3.0.14-r77/temp/log.do_fetch.5922
Log data follows:
| DEBUG: Trying PREMIRRORS
| DEBUG: For url ['file', '', 'rotation', '', '', {}] comparing ['cvs', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
| DEBUG: For url ['file', '', 'rotation', '', '', {}] comparing ['svn', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
| DEBUG: For url ['file', '', 'rotation', '', '', {}] comparing ['git', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
| DEBUG: For url ['file', '', 'rotation', '', '', {}] comparing ['hg', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
| DEBUG: For url ['file', '', 'rotation', '', '', {}] comparing ['bzr', '.*', '/.*', '', '', {}] to ['http', 'build.shr-project.org', '/sources/', '', '', {}]
| DEBUG: Trying Upstream
| ERROR: Function failed: Fetcher failure for URL: 'file://rotation'. Unable to fetch URL from any source.
NOTE: package base-files-3.0.14-r77: task do_fetch: Failed
ERROR: Task 1 (openembedded-core/meta/recipes-core/base-files/base-files_3.0.14.bb, do_fetch) failed with exit code '1'
NOTE: Tasks Summary: Attempted 2 tasks of which 0 didn't need to be rerun and 1 failed.

Tried to disable distro PREMIRRORS:

NOTE: Preparing runqueue
NOTE: Executing RunQueue Tasks
NOTE: Running task 2 of 3 (ID: 1, openembedded-core/meta/recipes-core/base-files/base-files_3.0.14.bb, do_fetch)
NOTE: package base-files-3.0.14-r77: task do_fetch: Started
ERROR: Function failed: Fetcher failure for URL: 'file://rotation'. Unable to fetch URL from any source.
ERROR: Logfile of failure stored in: /OE/shr-core/tmp-eglibc/work/om_gta02-oe-linux-gnueabi/base-files/base-files-3.0.14-r77/temp/log.do_fetch.6078
Log data follows:
| DEBUG: Trying PREMIRRORS
| DEBUG: Trying Upstream
| ERROR: Function failed: Fetcher failure for URL: 'file://rotation'. Unable to fetch URL from any source.
NOTE: package base-files-3.0.14-r77: task do_fetch: Failed
ERROR: Task 1 (openembedded-core/meta/recipes-core/base-files/base-files_3.0.14.bb, do_fetch) failed with exit code '1'
NOTE: Tasks Summary: Attempted 2 tasks of which 1 didn't need to be rerun and 1 failed.

Removed all .bbappends for base-files:
OE om-gta02@shr ~/shr-core $ rm -f meta-smartphone/meta-nokia/recipes-core/base-files/base-files_3.0.14.bbappend meta-smartphone/meta-shr/recipes-core/base-files/base-files_3.0.14.bbappend meta-smartphone/meta-openmoko/recipes-core/base-files/base-files_3.0.14.bbappend meta-smartphone/meta-samsung/recipes-core/base-files/base-files_3.0.14.bbappend

NOTE: Preparing runqueue
NOTE: Executing RunQueue Tasks
NOTE: Running task 1 of 3 (ID: 2, openembedded-core/meta/recipes-core/base-files/base-files_3.0.14.bb, do_rm_old_works)
NOTE: Running task 2 of 3 (ID: 1, openembedded-core/meta/recipes-core/base-files/base-files_3.0.14.bb, do_fetch)
NOTE: package base-files-3.0.14-r71: task do_rm_old_works: Started
NOTE: package base-files-3.0.14-r71: task do_rm_old_works: Succeeded
NOTE: package base-files-3.0.14-r71: task do_fetch: Started
ERROR: Function failed: Fetcher failure for URL: 'file://rotation'. Unable to fetch URL from any source.
ERROR: Logfile of failure stored in: /OE/shr-core/tmp-eglibc/work/om_gta02-oe-linux-gnueabi/base-files/base-files-3.0.14-r71/temp/log.do_fetch.6241
Log data follows:
| DEBUG: Trying PREMIRRORS
| DEBUG: Trying Upstream
| ERROR: Function failed: Fetcher failure for URL: 'file://rotation'. Unable to fetch URL from any source.
NOTE: package base-files-3.0.14-r71: task do_fetch: Failed
ERROR: Task 1 (openembedded-core/meta/recipes-core/base-files/base-files_3.0.14.bb, do_fetch) failed with exit code '1'
NOTE: Tasks Summary: Attempted 2 tasks of which 0 didn't need to be rerun and 1 failed.

Still the same, tried without -b to reparse (interestingly nothing was used from cache)

OE om-gta02@shr ~/shr-core $ bitbake -c unpack -f base-files
WARNING: /OE/shr-core/meta-openembedded/meta-oe/recipes-connectivity/networkmanager/networkmanager_0.9.2.0.bb: it is recommended to set SYSTEMD_PACKAGES as <existing-package>-systemd                                        | ETA:  0:00:16
NOTE: shr DOES NOT support libiconv because libiconv is provided by LIBC########################################################################################################                                              | ETA:  0:00:09
NOTE: shr DOES NOT support libiconv because libiconv is provided by LIBC##########################################################################################################################                            | ETA:  0:00:05
WARNING: /OE/shr-core/openembedded-core/meta/recipes-extended/lighttpd/lighttpd_1.4.30.bb: it is recommended to set SYSTEMD_PACKAGES as <existing-package>-systemd
NOTE: shr DOES NOT support initscripts because use service file for systemd now
NOTE: shr DOES NOT support external-csl-toolchain because use internal toolchain
NOTE: shr DOES NOT support sysvinit because use systemd now###############################################################################################################################################                    | ETA:  0:00:04
Parsing recipes: 100% |#######################################################################################################################################################################################################| Time: 0:00:42
Parsing of 1718 .bb files complete (0 cached, 1718 parsed). 2157 targets, 70 skipped, 0 masked, 0 errors.
WARNING: No bb files matched BBFILE_PATTERN_systemd-layer '^/OE/shr-core/meta-openembedded/meta-systemd/'

Build Configuration:
BB_VERSION        = "1.15.2"
TARGET_ARCH       = "arm"
TARGET_OS         = "linux-gnueabi"
MACHINE           = "om-gta02"
DISTRO            = "shr"
DISTRO_VERSION    = "20120522"
TUNE_FEATURES     = "armv4 thumb arm920t"
TARGET_FPU        = "soft"
meta-jama         = "master:aa61d65a0a21be6eab0619c1ad3a5bef6c912f92"
meta-shr
meta-aurora
meta-fso          = "jansa/test:a4b5b9c2e987dc87cf0a4f890428443a9cea3733"
meta-oe
toolchain-layer
meta-efl
meta-gnome
meta-gpe
meta-initramfs
meta-systemd      = "jansa/test:7b64162e153c3538ddc98e804cc0208d5f50212a"
meta-osmocombb
meta-nokia
meta-htc
meta-palm
meta-openmoko
meta-samsung      = "jansa/test:a4b5b9c2e987dc87cf0a4f890428443a9cea3733"
meta-handheld     = "jansa/spitz:f2169618f96b14491911a2361bf04f1b8c5f65a2"
meta-browser      = "master:c47f59df2e723495679c751cbdf6a8c6adec4b6a"
meta              = "jansa/test:89f4092b93452f77d0d30b394629a928341a01d0"

NOTE: Resolving any missing task queue dependencies
NOTE: Preparing runqueue
NOTE: Executing RunQueue Tasks
NOTE: Running task 2 of 3 (ID: 1, /OE/shr-core/openembedded-core/meta/recipes-core/base-files/base-files_3.0.14.bb, do_fetch)
NOTE: package base-files-3.0.14-r71: task do_fetch: Started
NOTE: package base-files-3.0.14-r71: task do_fetch: Succeeded
NOTE: Running task 3 of 3 (ID: 0, /OE/shr-core/openembedded-core/meta/recipes-core/base-files/base-files_3.0.14.bb, do_unpack)
NOTE: package base-files-3.0.14-r71: task do_unpack: Started
NOTE: package base-files-3.0.14-r71: task do_unpack: Succeeded
NOTE: Tasks Summary: Attempted 3 tasks of which 1 didn't need to be rerun and all succeeded.

Summary: There were 3 WARNING messages shown.

And with -b just after it:
OE om-gta02@shr ~/shr-core $ bitbake -f -c unpack -b openembedded-core/meta/recipes-core/base-files/base-files_3.0.14.bb
WARNING: Buildfile specified, dependencies will not be handled. If this is not what you want, do not use -b / --buildfile.

Build Configuration:
BB_VERSION        = "1.15.2"
TARGET_ARCH       = "arm"
TARGET_OS         = "linux-gnueabi"
MACHINE           = "om-gta02"
DISTRO            = "shr"
DISTRO_VERSION    = "20120522"
TUNE_FEATURES     = "armv4 thumb arm920t"
TARGET_FPU        = "soft"
meta-jama         = "master:aa61d65a0a21be6eab0619c1ad3a5bef6c912f92"
meta-shr
meta-aurora
meta-fso          = "jansa/test:a4b5b9c2e987dc87cf0a4f890428443a9cea3733"
meta-oe
toolchain-layer
meta-efl
meta-gnome
meta-gpe
meta-initramfs
meta-systemd      = "jansa/test:7b64162e153c3538ddc98e804cc0208d5f50212a"
meta-osmocombb
meta-nokia
meta-htc
meta-palm
meta-openmoko
meta-samsung      = "jansa/test:a4b5b9c2e987dc87cf0a4f890428443a9cea3733"
meta-handheld     = "jansa/spitz:f2169618f96b14491911a2361bf04f1b8c5f65a2"
meta-browser      = "master:c47f59df2e723495679c751cbdf6a8c6adec4b6a"
meta              = "jansa/test:89f4092b93452f77d0d30b394629a928341a01d0"

NOTE: Preparing runqueue
NOTE: Executing RunQueue Tasks
NOTE: Running task 3 of 3 (ID: 0, openembedded-core/meta/recipes-core/base-files/base-files_3.0.14.bb, do_unpack)
NOTE: package base-files-3.0.14-r71: task do_unpack: Started
ERROR: Error executing a python function in openembedded-core/meta/recipes-core/base-files/base-files_3.0.14.bb:
OSError: [Errno 2] No such file or directory: '/OE/shr-core/downloads/rotation'

ERROR: The stack trace of python calls that resulted in this exception/failure was:
ERROR:   File "base_do_unpack", line 19, in <module>
ERROR:
ERROR:   File "base_do_unpack", line 14, in base_do_unpack
ERROR:
ERROR:   File "/usr/lib64/python2.7/site-packages/bb/fetch2/__init__.py", line 1075, in unpack
ERROR:     ud.method.unpack(ud, root, self.d)
ERROR:
ERROR:   File "/usr/lib64/python2.7/site-packages/bb/fetch2/__init__.py", line 775, in unpack
ERROR:     if (file != dest) and not (os.path.exists(dest) and os.path.samefile(file, dest)):
ERROR:
ERROR:   File "/usr/lib64/python2.7/posixpath.py", line 153, in samefile
ERROR:     s1 = os.stat(f1)
ERROR:
ERROR: The code that was being executed was:
ERROR:      0015:        except bb.fetch2.BBFetchException, e:
ERROR:      0016:            raise bb.build.FuncFailed(e)
ERROR:      0017:
ERROR:      0018:
ERROR:  *** 0019:base_do_unpack(d)
ERROR:      0020:
ERROR: (file: 'base_do_unpack', lineno: 19, function: <module>)
ERROR:      0010:       rootdir = localdata.getVar('WORKDIR', True)
ERROR:      0011:
ERROR:      0012:        try:
ERROR:      0013:            fetcher = bb.fetch2.Fetch(src_uri, localdata)
ERROR:  *** 0014:            fetcher.unpack(rootdir)
ERROR:      0015:        except bb.fetch2.BBFetchException, e:
ERROR:      0016:            raise bb.build.FuncFailed(e)
ERROR:      0017:
ERROR:      0018:
ERROR: (file: 'base_do_unpack', lineno: 14, function: base_do_unpack)
ERROR: Function failed: base_do_unpack
ERROR: Logfile of failure stored in: /OE/shr-core/tmp-eglibc/work/om_gta02-oe-linux-gnueabi/base-files/base-files-3.0.14-r71/temp/log.do_unpack.10111
NOTE: package base-files-3.0.14-r71: task do_unpack: Failed
ERROR: Task 0 (openembedded-core/meta/recipes-core/base-files/base-files_3.0.14.bb, do_unpack) failed with exit code '1'
NOTE: Tasks Summary: Attempted 3 tasks of which 2 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
  openembedded-core/meta/recipes-core/base-files/base-files_3.0.14.bb, do_unpack
Summary: There was 1 WARNING message shown.
Summary: There were 34 ERROR messages shown, returning a non-zero exit code.


OE om-gta02@shr ~/shr-core $ ls /OE/shr-core/downloads/rotation*
/OE/shr-core/downloads/rotation.done  /OE/shr-core/downloads/rotation.lock

OE om-gta02@shr ~/shr-core $ rm -f /OE/shr-core/downloads/rotation*
OE om-gta02@shr ~/shr-core $ bitbake -f -c unpack -b openembedded-core/meta/recipes-core/base-files/base-files_3.0.14.bb

fails the same, but this time only .lock is created
OE om-gta02@shr ~/shr-core $ ls /OE/shr-core/downloads/rotation*
/OE/shr-core/downloads/rotation.lock

And now I'm sort of stuck here, -f -c unpack with -b always fails like this (even after another reparse and cleansstate, without -b still works) and if I do -c cleansstate and then -c unpack _without_ -f then I'm back to first error.

Layout is like this:
OE om-gta02@shr ~/shr-core $ which bitbake
/OE/shr-core/openembedded-core/scripts/bitbake

OE om-gta02@shr ~/shr-core $ ls -ld meta* open*
lrwxrwxrwx 1 bitbake bitbake 16 Nov 29 12:41 meta-angstrom -> ../meta-angstrom
lrwxrwxrwx 1 bitbake bitbake 16 May 17 13:15 meta-browser -> ../meta-browser/
lrwxrwxrwx 1 bitbake bitbake 17 Apr 20 10:50 meta-chromium -> ../meta-chromium/
lrwxrwxrwx 1 bitbake bitbake 16 Nov 29 12:41 meta-handheld -> ../meta-handheld
lrwxrwxrwx 1 bitbake bitbake 12 Nov 29 12:41 meta-jama -> ../meta-jama
lrwxrwxrwx 1 bitbake bitbake 16 Jan 16 16:15 meta-mozilla -> ../meta-mozilla/
lrwxrwxrwx 1 bitbake bitbake 20 Nov 29 12:41 meta-openembedded -> ../meta-openembedded
lrwxrwxrwx 1 bitbake bitbake 18 Nov 29 12:41 meta-smartphone -> ../meta-smartphone
lrwxrwxrwx 1 bitbake bitbake 21 Nov 29 12:41 openembedded-core -> ../openembedded-core/

OE om-gta02@shr ~/shr-core $ pwd
/OE/shr-core

OE om-gta02@shr ~/shr-core $ env | grep BB
BBFETCH2=True
BB_ENV_EXTRAWHITE=MACHINE DISTRO http_proxy ftp_proxy https_proxy all_proxy ALL_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS GIT_PROXY_COMMAND PSEUDO_DISABLED PSEUDO_BUILD

OE om-gta02@shr ~/shr-core $ cat conf/bblayers.conf
# LAYER_CONF_VERSION is increased each time build/conf/bblayers.conf
# changes incompatibly
LCONF_VERSION = "4"

BBFILES ?= ""

# Add your overlay location to BBLAYERS
# Make sure to have a conf/layers.conf in there
BBLAYERS = " \
  ${TOPDIR}/meta-jama \
  ${TOPDIR}/meta-smartphone/meta-shr \
  ${TOPDIR}/meta-smartphone/meta-aurora \
  ${TOPDIR}/meta-smartphone/meta-fso \
  ${TOPDIR}/meta-openembedded/meta-oe \
  ${TOPDIR}/meta-openembedded/toolchain-layer \
  ${TOPDIR}/meta-openembedded/meta-efl \
  ${TOPDIR}/meta-openembedded/meta-gnome \
  ${TOPDIR}/meta-openembedded/meta-gpe \
  ${TOPDIR}/meta-openembedded/meta-initramfs \
  ${TOPDIR}/meta-openembedded/meta-systemd \
  ${TOPDIR}/meta-smartphone/meta-osmocombb \
  ${TOPDIR}/meta-smartphone/meta-nokia \
  ${TOPDIR}/meta-smartphone/meta-htc \
  ${TOPDIR}/meta-smartphone/meta-palm \
  ${TOPDIR}/meta-smartphone/meta-openmoko \
  ${TOPDIR}/meta-smartphone/meta-samsung \
  ${TOPDIR}/meta-handheld \
  ${TOPDIR}/meta-browser \
  ${TOPDIR}/openembedded-core/meta \
"

OE om-gta02@shr ~/shr-core $ cat setup-env
# setup up the environment for a bitbake build.  This is used
# by the makefile and may also be directly sourced from an
# interactive shell.  The makefile uses 'env -i' to ensure
# no variables are inherited apart from CCACHE_DISABLE and
# CCACHE_DIR
#
# topdir must exist in the directory and define TOPDIR to
# the full path name of the working directory
. ./conf/topdir.conf
test -e ./setup-local || echo "setup-local does not exist, please update your conf/auto.conf and copy setup-local from common directory"
. ./setup-local
test -n "$TOPDIR" -a -d "$TOPDIR" || {
        echo "environment: TOPDIR not defined" >&2
        return
}

#
# the following must match the definitions in common/conf/site.conf
test ! -d "${TOPDIR}/bitbake/bin" || export PATH="${TOPDIR}/bitbake/bin:${PATH}"
test ! -d "${TOPDIR}/openembedded-core/scripts" || export PATH="${TOPDIR}/openembedded-core/scripts:${PATH}"

# used in bitbake wrapper for pseudodone location
export BUILDDIR="${TOPDIR}"

#export BBPATH="${TOPDIR}:${TOPDIR}/openembedded-core/meta"
export BB_ENV_EXTRAWHITE="MACHINE DISTRO http_proxy ftp_proxy https_proxy all_proxy ALL_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS GIT_PROXY_COMMAND PSEUDO_DISABLED PSEUDO_BUILD"
#export BB_ENV_EXTRAWHITE="MACHINE DISTRO http_proxy ftp_proxy https_proxy all_proxy ALL_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS GIT_PROXY_COMMAND PSEUDO_DISABLED PSEUDO_BUILD LD_LIBRARY_PATH"
#export LD_LIBRARY_PATH="/usr/lib/gcc/x86_64-pc-linux-gnu/4.6.2/"
export LANG=C
export MACHINE
export DISTRO
#NOTE: if you add export definitions here add them below too!
#
# unset the following (unnecessary for the makefile, but safe)
unset LC_CTYPE
unset CC
unset CXX
unset MFLAGS
unset MAKEFLAGS
unset MAKE_TARGET
#
# make bb into a 'safe' bitbake
alias bb-safe="env -i \
        HOME='${HOME}' \
        PYTHONPATH='${PYTHONPATH}' \
        BBPATH='${BBPATH}' \
        PATH='${PATH}' \
        LD_LIBRARY_PATH='${LD_LIBRARY_PATH}' \
        LANG='${LANG}' \
        '${TOPDIR}/bitbake/bin/bitbake'"
alias bb="bitbake"
#
# remove TOPDIR - not required
unset TOPDIR

if [ ${TERM:-dumb} = dumb ]; then
   export PS1='OE $MACHINE@$DISTRO \w \$ '
else
   export PS1='\[\033[01;32m\]OE $MACHINE@$DISTRO\[\033[01;34m\] \w \$\[\033[00m\] '
fi
echo "Altered environment for ${MACHINE}@${DISTRO} development"


What else could be usefull?

I think I have a handle on this now. Could you run:

bitbake -e | grep BBPATH and share the value of BBPATH that is being used?

I have a suspicion that something in your layer config is injecting a "::" or other empty entry in there, before the meta layer is included.

For example, if locally I have:

BBPATH="/media/data1/build1/poky/build/meta-yocto::/media/data1/build1/poky/meta-qt3:/media/data1/build1/poky/build/meta"

I can reproduce this error. I effecitvely stumbled upon this by accident trying to recreate your error. If you run "bitbake base-files -e" and look at the value of FILE, it will also likely not have an asbolute path which will corrupt FILE_DIRNAME and FILESPATH in turn.

Simplistically, we can do something like:

diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
index 7a73419..fc389a3 100644
--- a/bitbake/lib/bb/utils.py
+++ b/bitbake/lib/bb/utils.py
@@ -721,6 +721,8 @@ def which(path, item, direction = 0):
     for p in paths:
         next = os.path.join(p, item)
         if os.path.exists(next):
+            if not os.path.isabs(next):
+                next = os.path.abspath(next)
             return next
 
     return ""

to work around this. I'm wondering if we should error about empty directories in that variable though.

If you could see if the above guesses are correct for your environment I'd appreciate it. It looks like it comes down to the exact layer config you're using.

I don't have :: in BBPATH but there is non-existent dir "/OE/shr-core/openembedded"

from initial BBPATH defined setup-env
export BBPATH="${TOPDIR}:${TOPDIR}/openembedded"


BBPATH="/OE/shr-core:/OE/shr-core/openembedded:/OE/shr-core/meta-smartphone/meta-shr:/OE/shr-core/meta-smartphone/meta-aurora:/OE/shr-core/meta-smartphone/meta-fso:/OE/shr-core/meta-openembedded/meta-oe:/OE/shr-core/meta-openembedded/toolchain-layer:/OE/shr-core/meta-openembedded/meta-efl:/OE/shr-core/meta-openembedded/meta-gnome:/OE/shr-core/meta-openembedded/meta-gpe:/OE/shr-core/meta-openembedded/meta-systemd:/OE/shr-core/meta-smartphone/meta-osmocombb:/OE/shr-core/meta-smartphone/meta-nokia:/OE/shr-core/meta-smartphone/meta-htc:/OE/shr-core/meta-smartphone/meta-palm:/OE/shr-core/meta-smartphone/meta-openmoko:/OE/shr-core/meta-smartphone/meta-samsung:/OE/shr-core/meta-browser:/OE/shr-core/openembedded-core/meta"

Later today or tomorrow I'll try to remove it and then your patch.

Do you have the value of the FILE variable handy in the failing cases (bitbake -e vase-files | grep ^FILE). I'm pretty sure the value of FILE is wrong, the question is how/why...

OE om-gta02@shr ~/shr-core $ bitbake -e -b openembedded-core/meta/recipes-core/base-files/base-files_3.0.14.bb | grep ^FILE=
FILE="openembedded-core/meta/recipes-core/base-files/base-files_3.0.14.bb"
OE om-gta02@shr ~/shr-core $ bitbake -e base-files | grep ^FILE=
FILE="/OE/shr-core/openembedded-core/meta/recipes-core/base-files/base-files_3.0.14.bb"

Now I've found what could cause '::' in BBPATH

meta-openembedded/meta-initramfs/conf/layer.conf
and meta-handheld/conf/layer.conf are using
BBPATH := "${LAYERDIR}:${BBPATH}"

while all other layers are I'm using have
BBPATH .= ":${LAYERDIR}"

And with meta-initramfs/meta-handheld layers included I get '::'

base-files.b.env:BBPATH="/OE/shr-core/meta-handheld:/OE/shr-core/meta-openembedded/meta-initramfs::/OE/shr-core/meta-jama:/OE/shr-core/meta-smartphone/meta-shr:/OE/shr-core/meta-smartphone/meta-aurora:/OE/shr-core/meta-smartphone/meta-fso:/OE/shr-core/meta-openembedded/meta-oe:/OE/shr-core/meta-openembedded/toolchain-layer:/OE/shr-core/meta-openembedded/meta-efl:/OE/shr-core/meta-openembedded/meta-gnome:/OE/shr-core/meta-openembedded/meta-gpe:/OE/shr-core/meta-openembedded/meta-systemd:/OE/shr-core/meta-smartphone/meta-osmocombb:/OE/shr-core/meta-smartphone/meta-nokia:/OE/shr-core/meta-smartphone/meta-htc:/OE/shr-core/meta-smartphone/meta-palm:/OE/shr-core/meta-smartphone/meta-openmoko:/OE/shr-core/meta-smartphone/meta-samsung:/OE/shr-core/meta-browser:/OE/shr-core/openembedded-core/meta"

I can also reproduce it with BBPATH starting with ':'.

Thanks Martin, that helps a lot.

So to summarise, we have several issues:

a) bb.utils.which() can sometimes not return absolute paths, My patch fixes that.
b) :: can make it into BBPATH accidentally for whatever reason. We should detect and warn about that as it doesn't make sense.
c) buildFile() in cooker should always be resolving paths into absolute paths anyway. The code in matchFiles therefore also needs fixing so non-absolute paths don't get passed any further within bitbake.

b)
Can we somehow not only warn about leading/trailing ':' and '::', but also ignore /remove them in runtime?

I'm thinking about scenario when someone is using layers maintained by someone else or from some older release and cannot update/fix their layer.conf immediately.

Well if this "-c unpack -b" is only case when ':' matter then it's not important, but if there are some other hidden issues with such BBPATH..

I've sent patches for layer.conf in
meta-handheld:
http://patchwork.openembedded.org/patch/28485/
meta-initramfs:
http://patchwork.openembedded.org/patch/28487/

but there are at least 2 more in oe-core repository with the same BBPATH append:
meta-hob/conf/layer.conf:BBPATH := "${BBPATH}:${LAYERDIR}"
meta-skeleton/conf/layer.conf:BBPATH := "${BBPATH}:${LAYERDIR}"

(In reply to comment #15)
> I've sent patches for layer.conf in
> meta-handheld:
> http://patchwork.openembedded.org/patch/28485/
> meta-initramfs:
> http://patchwork.openembedded.org/patch/28487/
> 
> but there are at least 2 more in oe-core repository with the same BBPATH
> append:
> meta-hob/conf/layer.conf:BBPATH := "${BBPATH}:${LAYERDIR}"
> meta-skeleton/conf/layer.conf:BBPATH := "${BBPATH}:${LAYERDIR}"


These aren't wrong as such, they just assume they're being included after the oe-core layer...

Well, yes, they work, but order of layers in BBLAYERS together with BBFILE_PRIORITY in each layer.conf is confusing enough already for many people.. 

so IMHO it's better when layers respect BBLAYERS order not introduce another "variable" to priority mix.

Patches for issues a) and c) have merged:
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=0b14db452413df414fbadf14d6dae287bc5ae9ca
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=2d3ff5e6e17c3c8e0ba000c696da54e51167f15d

I'm going to leave b) to Laurentiu. I don't think fixing this up dynamically makes sense, we need to tell the user there is a problem and let them fix it. We need to detect any empty path variable, so "." and "" at any point of BBPATH are an issue. There is already a PATH check in sanity.bbclass which could easily be used as a reference for this.

Patch for b) has been also merged. For a while, we will only show a warning message and ask the user to fix the BBPATH issue, but the build process will continue. After all BBPATH issues in the tree will be fixed(let's give it a month), the warning will become a fatal error and the build will not continue unless the problem is fixed.

http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=d8d95d10bd6945047b4bab9ff9a4e948c73db205