Bug 4790 - get_file_checksums(filelist, pn) is searching in different directories then FILESPATH used in do_fetch task
Summary: get_file_checksums(filelist, pn) is searching in different directories then F...
Status: RESOLVED FIXED
Alias: None
Product: BitBake
Classification: Build System, Metadata & Runtime
Component: bitbake (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium normal
Target Milestone: 1.6 M5
Assignee: Paul Eggleton
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2013-06-29 11:09 UTC by Martin Jansa
Modified: 2014-08-24 20:13 UTC (History)
4 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Martin Jansa 2013-06-29 11:09:40 UTC 
I have 2 mesa_git recipes and one .bbappend:
./meta-webos/recipes-graphics/mesa/mesa_git.bbappend
./meta-webos-backports/recipes-graphics/mesa/mesa_git.bb
./oe-core/meta/recipes-graphics/mesa/mesa_9.0.2.bb
./oe-core/meta/recipes-graphics/mesa/mesa_git.bb

0001-configure-Avoid-use-of-AC_CHECK_FILE-for-cross-compi.patch is only in meta-webos-backports layer
oe-core is dylan release which doesn't have it yet (that's why we have backports layer)
./meta-webos-backports/recipes-graphics/mesa/mesa/0001-configure-Avoid-use-of-AC_CHECK_FILE-for-cross-compi.patch

BBPATH=
/OE/
/OE/BUILD
/OE/meta-webos
/OE/meta-webos-backports
/OE/oe-core/meta

FILESEXTRAPATHS=
/OE/meta-webos/recipes-graphics/mesa/mesa

do_fetch finds it correctly:
bitbake -e mesa | grep ^FILESPATH= | sed "s/\"/\n/g; s/:/\n/g"
FILESPATH=
/OE/meta-webos/recipes-graphics/mesa/mesa/i586
/OE/meta-webos/recipes-graphics/mesa/mesa/x86
/OE/meta-webos/recipes-graphics/mesa/mesa/qemuall
/OE/meta-webos/recipes-graphics/mesa/mesa/qemux86
/OE/meta-webos/recipes-graphics/mesa/mesa/webos
/OE/meta-webos/recipes-graphics/mesa/mesa/
/OE/meta-webos-backports/recipes-graphics/mesa/mesa-9.1.3+gitAUTOINC+b3f1f665b0fef178ae193e6b111f14c9a5ad3b25/i586
/OE/meta-webos-backports/recipes-graphics/mesa/mesa-9.1.3+gitAUTOINC+b3f1f665b0fef178ae193e6b111f14c9a5ad3b25/x86
/OE/meta-webos-backports/recipes-graphics/mesa/mesa-9.1.3+gitAUTOINC+b3f1f665b0fef178ae193e6b111f14c9a5ad3b25/qemuall
/OE/meta-webos-backports/recipes-graphics/mesa/mesa-9.1.3+gitAUTOINC+b3f1f665b0fef178ae193e6b111f14c9a5ad3b25/qemux86
/OE/meta-webos-backports/recipes-graphics/mesa/mesa-9.1.3+gitAUTOINC+b3f1f665b0fef178ae193e6b111f14c9a5ad3b25/webos
/OE/meta-webos-backports/recipes-graphics/mesa/mesa-9.1.3+gitAUTOINC+b3f1f665b0fef178ae193e6b111f14c9a5ad3b25/
/OE/meta-webos-backports/recipes-graphics/mesa/mesa/i586
/OE/meta-webos-backports/recipes-graphics/mesa/mesa/x86
/OE/meta-webos-backports/recipes-graphics/mesa/mesa/qemuall
/OE/meta-webos-backports/recipes-graphics/mesa/mesa/qemux86
/OE/meta-webos-backports/recipes-graphics/mesa/mesa/webos
/OE/meta-webos-backports/recipes-graphics/mesa/mesa/
/OE/meta-webos-backports/recipes-graphics/mesa/files/i586
/OE/meta-webos-backports/recipes-graphics/mesa/files/x86
/OE/meta-webos-backports/recipes-graphics/mesa/files/qemuall
/OE/meta-webos-backports/recipes-graphics/mesa/files/qemux86
/OE/meta-webos-backports/recipes-graphics/mesa/files/webos
/OE/meta-webos-backports/recipes-graphics/mesa/files/

bitbake -DDD mesa:
DEBUG: Searching for 0001-configure-Avoid-use-of-AC_CHECK_FILE-for-cross-compi.patch in paths:
    /OE/meta-webos/recipes-graphics/mesa/mesa/i586
    /OE/meta-webos/recipes-graphics/mesa/mesa/x86
    /OE/meta-webos/recipes-graphics/mesa/mesa/qemuall
    /OE/meta-webos/recipes-graphics/mesa/mesa/qemux86
    /OE/meta-webos/recipes-graphics/mesa/mesa/webos
    /OE/meta-webos/recipes-graphics/mesa/mesa/
    /OE/oe-core/meta/recipes-graphics/mesa/mesa-9.1~gitAUTOINC+b3f1f665b0fef178ae193e6b111f14c9a5ad3b25/i586
    /OE/oe-core/meta/recipes-graphics/mesa/mesa-9.1~gitAUTOINC+b3f1f665b0fef178ae193e6b111f14c9a5ad3b25/x86
    /OE/oe-core/meta/recipes-graphics/mesa/mesa-9.1~gitAUTOINC+b3f1f665b0fef178ae193e6b111f14c9a5ad3b25/qemuall
    /OE/oe-core/meta/recipes-graphics/mesa/mesa-9.1~gitAUTOINC+b3f1f665b0fef178ae193e6b111f14c9a5ad3b25/qemux86
    /OE/oe-core/meta/recipes-graphics/mesa/mesa-9.1~gitAUTOINC+b3f1f665b0fef178ae193e6b111f14c9a5ad3b25/webos
    /OE/oe-core/meta/recipes-graphics/mesa/mesa-9.1~gitAUTOINC+b3f1f665b0fef178ae193e6b111f14c9a5ad3b25/
    /OE/oe-core/meta/recipes-graphics/mesa/mesa/i586
    /OE/oe-core/meta/recipes-graphics/mesa/mesa/x86
    /OE/oe-core/meta/recipes-graphics/mesa/mesa/qemuall
    /OE/oe-core/meta/recipes-graphics/mesa/mesa/qemux86
    /OE/oe-core/meta/recipes-graphics/mesa/mesa/webos
    /OE/oe-core/meta/recipes-graphics/mesa/mesa/
    /OE/oe-core/meta/recipes-graphics/mesa/files/i586
    /OE/oe-core/meta/recipes-graphics/mesa/files/x86
    /OE/oe-core/meta/recipes-graphics/mesa/files/qemuall
    /OE/oe-core/meta/recipes-graphics/mesa/files/qemux86
    /OE/oe-core/meta/recipes-graphics/mesa/files/webos
    /OE/oe-core/meta/recipes-graphics/mesa/files/
DEBUG: Defaulting to /OE/downloads/0001-configure-Avoid-use-of-AC_CHECK_FILE-for-cross-compi.patch for 0001-configure-Avoid-use-of-AC_CHECK_FILE-for-cross-compi.patch
WARNING: Unable to get checksum for mesa SRC_URI entry 0001-configure-Avoid-use-of-AC_CHECK_FILE-for-cross-compi.patch: file could not be found
Comment 1 Martin Jansa 2014-01-18 14:15:06 UTC 
I wonder if this is somehow related to stalled entries in checksums cache, I've noticed couple of "WARNING: Unable to get checksum for foo SRC_URI entry" when moving meta-qt5 patches to different directories (e.g. from qtbase-5.2.0 to qtbase).

After such move, first bitbake build was showing this WARNING and then fetching the files correctly, 2nd and other builds were again without WARNING.
Comment 2 Paul Eggleton 2014-02-25 10:35:02 UTC 
So I talked to Martin on IRC, but just to keep this bug updated - I'm not really sure how to reproduce this or what do to about it. 

If it is just about genuinely missing files, it's certainly true that if a file is missing it won't be included in the checksum, and thus adding it after the fact and not modifying the recipe won't change the cached list of files and thus the file could be missed (will still actually be fetched when do_fetch executes, but changes to the file won't trigger any task re-execution). We could possibly address this by not caching the recipe if it has missing files, or marking the cache entry in some way such that it will get reparsed the next time.
Comment 3 Paul Eggleton 2014-04-07 14:02:12 UTC 
So it turns out the fix to ensure the file is picked up on the second run of bitbake is trivial - I've sent it to the bitbake list; hopefully that should solve this issue finally.
Comment 4 Paul Eggleton 2014-04-09 14:36:58 UTC 
Fix merged to master:

http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=1094be01e3a5a89474a56d97d7c750e2e312a990

This ensures if a missing file is subsequently added, the recipe will be reparsed and the new file will be picked up and incorporated into the checksum (without the user having to touch the recipe or manually clear anything.) Hopefully this should take care of the reported problem.
Comment 5 Martin Jansa 2014-07-30 12:22:07 UTC 
It looks like there is still problem when some local file is removed from SRC_URI.

I've seen it with latest bitbake and oe-core (as of today), with cache generated before this meta-oe patch:

http://patchwork.openembedded.org/patch/76837/

then first build with this patch included shows the warning from stalled checksums entry for automake-once-fix.patch file:

bitbake -k monkey | tee log.monkey
NOTE: Started PRServer with DBfile: /OE/build/oe-core/cache/prserv.sqlite3, IP: 127.0.0.1, PORT: 33128, PID: 2191
Parsing recipes...WARNING: Unable to get checksum for jq SRC_URI entry automake-once-fix.patch: file could not be found
done.
Parsing of 1778 .bb files complete (0 cached, 1778 parsed). 2236 targets, 90 skipped, 0 masked, 0 errors.
Comment 6 Martin Jansa 2014-07-30 12:29:04 UTC 
Please ignore my previous comment, as Paul noticed, the warning is legitimate in this case and it was real issue in the jq change.
Comment 7 Jacob Kroon 2014-08-24 20:13:08 UTC 
I'd like to request we reopen this bug.

I'm using the kernel from meta-fsl-arm, which ships a defconfig.
I also have my own layer, in which I've added a .bbappend that does

FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"

and also put my own defconfig there.

Parsing the kernel recipe, I see that it is picking up my own defconfig correctly.

If I remove my defconfig, the recipe is re-parsed, and the defconfig in meta-fsl-arm is registered in bitbakes cache.

If I now add back my defconfig, although it is used during unpack (I think), the checksum for meta-fsl-arm's defconfig is still in bitbakes cache, and used, so changes I make to my defconfig will not trigger rebuilds.