Bug 10981 - devtool: upgrade fails, recipe file not found
Summary: devtool: upgrade fails, recipe file not found
Status: RESOLVED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: Scripts and Tools (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 2.4 M4
Assignee: Paul Eggleton
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2017-01-27 08:28 UTC by Patrick Ohly
Modified: 2017-09-01 00:12 UTC (History)
5 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Patrick Ohly 2017-01-27 08:28:09 UTC
Example (must have meta-oe in the bblayers.conf):

$ devtool upgrade --srcrev 922421cf795d53d339862bb2d99f1a85c96ad9a3 --version 0.6.4 multipath-tools
Loading cache: 100% |#################################################################################| Time: 0:00:00
Loaded 2934 entries from dependency cache.
NOTE: Executing do_fetch...
NOTE: Executing do_unpack...
NOTE: Patching...
NOTE: Executing do_patch...
NOTE: Adding local source files to srctree...
WARNING: Command 'git rebase 922421cf795d53d339862bb2d99f1a85c96ad9a3' failed:
First, rewinding head to replay your work on top of it...
Applying: multipath-tools: modify Makefile.inc for cross-compilation
error: Failed to merge in the changes.
Using index info to reconstruct a base tree...
M	Makefile.inc
Falling back to patching base and 3-way merge...
Auto-merging Makefile.inc
CONFLICT (content): Merge conflict in Makefile.inc
Patch failed at 0001 multipath-tools: modify Makefile.inc for cross-compilation
The copy of the patch that failed is found in: .git/rebase-apply/patch

When you have resolved this problem, run "git rebase --continue".
If you prefer to skip this patch, run "git rebase --skip" instead.
To check out the original branch and stop rebasing, run "git rebase --abort".

You will need to resolve conflicts in order to complete the upgrade.
Traceback (most recent call last):
  File "/work/openembedded-core/scripts/devtool", line 351, in <module>
    ret = main()
  File "/work/openembedded-core/scripts/devtool", line 338, in main
    ret = args.func(args, config, basepath, workspace)
  File "/work/openembedded-core/scripts/lib/devtool/upgrade.py", line 370, in upgrade
    rf, copied = _create_new_recipe(args.version, md5, sha256, args.srcrev, args.srcbranch, config.workspace_path, tinfoil, rd)
  File "/work/openembedded-core/scripts/lib/devtool/upgrade.py", line 323, in _create_new_recipe
    rd = tinfoil.parse_recipe_file(fullpath, False)
  File "/work/openembedded-core/bitbake/lib/bb/tinfoil.py", line 360, in parse_recipe_file
    dscon = self.run_command('parseRecipeFile', fn, appends, appendlist)
  File "/work/openembedded-core/bitbake/lib/bb/tinfoil.py", line 268, in run_command
    raise TinfoilCommandFailed(result[1])
bb.tinfoil.TinfoilCommandFailed: Traceback (most recent call last):
  File "/work/openembedded-core/bitbake/lib/bb/command.py", line 81, in runCommand
    result = command_method(self, commandline)
  File "/work/openembedded-core/bitbake/lib/bb/command.py", line 535, in parseRecipeFile
    envdata = parser.loadDataFull(fn, appendfiles)
  File "/work/openembedded-core/bitbake/lib/bb/cache.py", line 339, in loadDataFull
    bb_data = self.load_bbfile(virtualfn, appends, virtonly=True)
  File "/work/openembedded-core/bitbake/lib/bb/cache.py", line 352, in load_bbfile
    datastores = parse_recipe(bb_data, bbfile, appends, mc)
  File "/work/openembedded-core/bitbake/lib/bb/cache.py", line 315, in parse_recipe
    bb_data = bb.parse.handle(bbfile, bb_data)
  File "/work/openembedded-core/bitbake/lib/bb/parse/__init__.py", line 113, in handle
    return h['handle'](fn, data, include)
  File "/work/openembedded-core/bitbake/lib/bb/parse/parse_py/BBHandler.py", line 132, in handle
    abs_fn = resolve_file(fn, d)
  File "/work/openembedded-core/bitbake/lib/bb/parse/__init__.py", line 136, in resolve_file
    raise IOError(errno.ENOENT, "file %s not found" % fn)
FileNotFoundError: [Errno 2] file /.../workspace/recipes/multipath-tools/multipath-tools_git.bb not found


The rebase error is expected, the failure to create the upgraded recipe is not ;-}
Comment 1 Paul Eggleton 2017-03-28 04:00:44 UTC
I'm unable to reproduce this issue - and I did rewind meta-oe back to ebef44963866d3231ccfe79d66d21c96ca0b4113 prior to your upgrade getting merged. I do get the conflict warning, but it completes without error here. Can you still reproduce it if you try the same?
Comment 2 Patrick Ohly 2017-03-28 08:32:11 UTC
I can still reproduce it with intel-iot-refkit. I also tried with Poky now and there it indeed doesn't occur.

Updating refkit to use the exact same bitbake as poky master doesn't help, so it must be something related to paths (just a wild guess).

Steps to reproduce from scratch:
git clone --recurse git@github.com:intel/intel-iot-refkit.git
cd intel-iot-refkit/bitbake/
git checkout 921592026c69287cdb40ffd90944d5944f28e2c3
cd ../meta-openembedded/
git checkout 960acf856f49e65ed31e096aa5fb4aeed237172e
cd ../
. refkit-init-build-env 
echo "require conf/distro/include/refkit-development.inc" >>conf/local.conf 
devtool upgrade --srcrev 922421cf795d53d339862bb2d99f1a85c96ad9a3 --version 0.6.4 multipath-tools
Comment 3 Henry Bruce 2017-03-30 18:59:41 UTC
I see a similar issue with mraa

$ devtool upgrade mraa --srcrev 5e34a5cd3be7514cca3e75971a1f271712e01d1c
raise IOError(errno.ENOENT, "file %s not found" % fn)
FileNotFoundError: [Errno 2] file /path/to/intel-iot-refkit/build/workspace/recipes/mraa/mraa_1.5.1.bb not found
Comment 4 Tim Orling 2017-08-26 21:36:15 UTC
$ rm -rf workspace/sources/python-matplotlib && devtool upgrade python-matplotlib --version 1.5.0
NOTE: Starting bitbake server...
NOTE: Starting bitbake server...
Loading cache: 100% |###########################################################################################################| Time: 0:00:00
Loaded 2772 entries from dependency cache.
Parsing recipes: 100% |#########################################################################################################| Time: 0:00:00
Parsing of 1971 .bb files complete (1970 cached, 1 parsed). 2773 targets, 121 skipped, 0 masked, 0 errors.
NOTE: Executing do_fetch...
NOTE: Executing do_unpack...
NOTE: Patching...
NOTE: Executing do_patch...
NOTE: Fetching http://downloads.sourceforge.net/matplotlib/matplotlib-1.5.0.tar.gz...
ERROR: Nothing PROVIDES 'tmp-recipetool-17X9MBZW'
ERROR: Failed to fetch URL http://downloads.sourceforge.net/matplotlib/matplotlib-1.5.0.tar.gz

If I run 'devtool add http://downloads.sourceforge.net/matplotlib/matplotlib-1.5.0.tar.gz' it has no problem fetching.

The problem in this case is in scripts/lib/scriptutils.py
http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/scripts/lib/scriptutils.py#n175
res returns False

Not sure where to go from here to fix it...

[time goes by]

On a hunch, I removed my workspace/ and it worked. I am experimenting with adding the ability to upgrade from a new fetchuri and must have caused some corruption in the metadata...
Comment 5 Paul Eggleton 2017-09-01 00:12:58 UTC
Fix merged to master:

http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=34580ac2872da6ae70d5269eb107d31cd5239ad7

I've spoken to Tim separately about the "Nothing PROVIDES" issue (which I'm fairly sure is unrelated) and he'll report it if it comes up again.