Bug 1139

Created attachment 171 [details]
linux-omap_2.6.39.bbappend

There is insufficient information here to work out what the problem is. At the very least we need the list of files in the meta-ti layer, the list of files in the layer the .bbappend is in, what the contents of the .bb file is and some details about what you're expecting to happen.

I also note you're using angstrom as your DISTRO and having a test case we can reproduce against OE-Core would be extremely helpful in trying to resolve this.

I thought this is more or less a known bug, since Koen Kooi also noticed this and asked me to file a bug here.

(In reply to comment #2)
> There is insufficient information here to work out what the problem is. At the
> very least we need the list of files in the meta-ti layer, the list of files in
> the layer the .bbappend is in, what the contents of the .bb file is and some
> details about what you're expecting to happen.
> 

bitbake: c9bb37e588ee7ee95eca798b0eae57bad68e8caf
oe-core: 2a52f806f3789f717219651b97dc64fec3881f7f
meta-ti: cc4141a6869346ac89c881ea9dd1633cc3a9817b
meta-oe: accc0d62ae67e36acfed8a469f0deea716d9de70
meta-angstrom: c2e604c898124c7cce50cc175d59bc83ee5a73a8

Basically these are all HEADS from yesterday.

OE Build Configuration:
BB_VERSION        = "1.13.0"
METADATA_BRANCH   = "master"
METADATA_REVISION = "2a52f806f3789f717219651b97dc64fec3881f7f"
TARGET_ARCH       = "arm"
TARGET_OS         = "linux-gnueabi"
MACHINE           = "beagleboard"
DISTRO            = "angstrom"
DISTRO_VERSION    = "v2011.06-core"
TARGET_FPU        = "hard"

Files from the meta-ti layer that should matter:
recipes-kernel/linux/linux-omap_2.6.39.bb
recipes-kernel/linux/multi-kernel.inc (!)

The .bbappend is already attached.
Files from my local layer:
recipes-kernel/linux/linux-omap_2.6.39.bbappend
recipes-kernel/linux/linux-omap/defconfig
recipes-kernel/linux/files/configs/.empty (not really used)

Current state:
* linux-omap_2.6.39.bb requires multi-kernel.inc 
* multi-kernel.inc issues:

    SRC += "\
      file://configs/* \
    "


What I would've expected is that bitbake tries to load configs/* from the meta-ti overlay, OR my local overlay IFF the directory or files in there exist.

Bibtake seems to ignore the files/configs/.empty file from the meta-ti overlay and tries to fetch it from my local overlay.

Creating a linux-omap/configs/.empty file in my local overlay doesn't help either.

> I also note you're using angstrom as your DISTRO and having a test case we can
> reproduce against OE-Core would be extremely helpful in trying to resolve this.

I don't think DISTRO matters here, but it is Angstrom. 

The problem might be the handling of FILESEXTRAPATHS.

BBLAYERS = " \ 
  /home/mike/oe/openembedded-core/meta \ 
  /home/mike/oe/openembedded-core/meta-openembedded/meta-oe \
  /home/mike/oe/openembedded-core/meta-openembedded/meta-efl \
  /home/mike/oe/openembedded-core/meta-angstrom \
  /home/mike/oe/openembedded-core/meta-texasinstruments \
  /home/mike/oe/openembedded-core/meta-local \  
  "

Command: bitbake virtual/kernel

Created attachment 172 [details]
linux-omap_2.6.39.bb

Created attachment 173 [details]
multi-kernel.inc

Created attachment 174 [details]
log.do_unpack.2202

Since I needed some kind of fix right now, here's some more information:

As soon as FILESEXTRAPATHS is present only (!) the subdirectory "configs/*" in FILESEXTRAPATHS is used to search for these kind of files.

I could create ${PN}/configs/empty (without a dot, since cp path/* does not match dot-files) and get a working build. I don't think this is the intended behaviour though, since I only wanted an additional search path using FILESEXTRAPATHS and not an exclusive one.

(Patches specified directly in the linux-omap recipe get picked up at the right location.)

There may be some odd behaviour due to the wildcard (which I'm not sure is supported in OE-core). What if you change multi-kernel.inc to remove it:

SRC_URI += " \
           file://configs "

I've been able to reproduce this here, and the wildcard does not make a difference to the behaviour. We just do not handle this situation at all - the first item found in the search path (in this case, the configs *directory* not specifically its contents, from the append) is the one that gets copied.

It's worth noting that meta-ti has since dropped multi-kernel.inc; however it may be that we still need to address this.

The following recipes in oe-core have the same problem:
$ grep -r 'file:\/\/.*\*' *
meta/recipes-extended/man/man_1.6g.bb:           file://man*"
meta/recipes-extended/pam/libpam_1.1.6.bb:           file://pam.d/* \
meta/recipes-core/systemd/systemd-compat-units.bb:SRC_URI = "file://*.service"

if we have bbappend files for them in some other layers, e.g. a local layer:

BBLAYERS ?= " \
  /data/wr-taf/jhuang0/git/poky/meta \
  /data/wr-taf/jhuang0/git/poky/meta-yocto \
  /data/wr-taf/jhuang0/git/poky/meta-yocto-bsp \
  /data/wr-taf/jhuang0/local/ \
  "

and FILESEXTRAPATHS is set as:
FILESEXTRAPATHS_prepend := "${THISDIR}:"

Take libpam as example:

in meta/recipes-extended/pam/libpam_1.1.6.bb, we have "file://pam.d/*" in SRC_URI,
and we added /data/wr-taf/jhuang0/local/recipes-extended/pam/libpam_1.1.6.bbappend,

bitbake's fetch2 unpack() will try to find "pam.d/*" in FILESPATH and FILESDIR, but nothing will match, then if "*" is used it will return the first directory in FILESPATH that exists:

codes in bitbake/lib/bb/fetch2/local.py:
----------------------------------------
    def localpath(self, url, urldata, d):
        """
        Return the local filename of a given url assuming a successful fetch.
        """
        path = urldata.decodedurl
        newpath = path
        if path[0] != "/":
            filespath = data.getVar('FILESPATH', d, True)
            if filespath:
                logger.debug(2, "Searching for %s in paths:    \n%s" % (path, "\n    ".join(filespath.split(":"))))
                newpath = bb.utils.which(filespath, path)
            if not newpath:
                filesdir = data.getVar('FILESDIR', d, True)
                if filesdir:
                    logger.debug(2, "Searching for %s in path: %s" % (path, filesdir))
                    newpath = os.path.join(filesdir, path)
            if (not newpath or not os.path.exists(newpath)) and path.find("*") != -1:
                # For expressions using '*', best we can do is take the first directory in FILESPATH that exists
                newpath = bb.utils.which(filespath, ".")
                logger.debug(2, "Searching for %s in path: %s" % (path, newpath))
                return newpath
            if not os.path.exists(newpath):
                dldirfile = os.path.join(d.getVar("DL_DIR", True), path)
                logger.debug(2, "Defaulting to %s for %s" % (dldirfile, path))
                bb.utils.mkdirhier(os.path.dirname(dldirfile))
                return dldirfile
        return newpath
---------------------------------------

and since the direcotory set in FILESEXTRAPATHS will be the first one in FILESPATH, eventually only the /data/wr-taf/jhuang0/local/recipes-extended/pam/pam.d/* will be copied, which is actually not exist and is obviously not what we want.

BTW, I found that use the following in bbappend instead would work around this issue:
FILESPATH_append := ":${@base_set_filespath(['${THISDIR}'], d)}"


So will it be implemented to handle such kind of situation? But anyway, I think we should avoid using wildcard in SRC_URI currently, so we should fix the above recipe by replacing the wildcard with proper way.

I would suggest this bug be rejected / or 'fixed' by looking for wildcards in a sanity check and avoided.

As far as I can tell wild cards should not be used.

I'll add warning when wildcards are used in SRC_URI and test it in my world builds to see how many recipes are actually using it.

I don't think it'd be terribly difficult to implement wildcards a bit better than we do today, assuming it has explicit semantics. Instead of assuming it's /*, we could support arbitrary wildcards as long as we only use the first of the entries in FILESPATH for which the wildcard matches anything at all. That's the semantics I used in the meta-sourcery external sysroot file search / extraction code, which is basically reimplemented FILESPATH+PREMIRRORS.

We have now removed wildcard support from SRC_URI, partly due to issues like this so this issue is obsolete.