Bug 13404 - 'devtool upgrade' tracebacks when a recipe includes another recipe with PV in the name
Summary: 'devtool upgrade' tracebacks when a recipe includes another recipe with PV in...
Status: RESOLVED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: Scripts and Tools (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 2.8 M2
Assignee: Paul Eggleton
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2019-06-18 11:07 UTC by Alexander Kanavin
Modified: 2019-07-03 20:27 UTC (History)
1 user (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Alexander Kanavin 2019-06-18 11:07:35 UTC
Devtool is not able to handle a situation when a recipe to be upgraded includes another recipe with $PV in its filename (in this case the included recipe filename should be renamed but isn't).


How to reproduce:

With poky at http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=111b7173fee69fe8b01ec2fa6999713841326537

issue

'devtool upgrade libnewt-python'

This will result in

Traceback (most recent call last):
  File "/home/alexander/development/poky/scripts/devtool", line 334, in <module>
    ret = main()
  File "/home/alexander/development/poky/scripts/devtool", line 321, in main
    ret = args.func(args, config, basepath, workspace)
  File "/home/alexander/development/poky/scripts/lib/devtool/upgrade.py", line 551, in upgrade
    rf, copied = _create_new_recipe(args.version, md5, sha256, args.srcrev, srcbranch, srcsubdir1, srcsubdir2, config.workspace_path, tinfoil, rd, license_diff, new_licenses)
  File "/home/alexander/development/poky/scripts/lib/devtool/upgrade.py", line 419, in _create_new_recipe
    rd = tinfoil.parse_recipe_file(fullpath, False)
  File "/home/alexander/development/poky/bitbake/lib/bb/tinfoil.py", line 641, in parse_recipe_file
    dscon = self.run_command('parseRecipeFile', fn, appends, appendlist)
  File "/home/alexander/development/poky/bitbake/lib/bb/tinfoil.py", line 466, in run_command
    raise TinfoilCommandFailed(result[1])
bb.tinfoil.TinfoilCommandFailed: Traceback (most recent call last):
  File "/home/alexander/development/poky/bitbake/lib/bb/parse/parse_py/ConfHandler.py", line 94, in include_single_file
    bb.parse.handle(fn, data, True)
  File "/home/alexander/development/poky/bitbake/lib/bb/parse/__init__.py", line 107, in handle
    return h['handle'](fn, data, include)
  File "/home/alexander/development/poky/bitbake/lib/bb/parse/parse_py/BBHandler.py", line 120, in handle
    abs_fn = resolve_file(fn, d)
  File "/home/alexander/development/poky/bitbake/lib/bb/parse/__init__.py", line 125, in resolve_file
    raise IOError(errno.ENOENT, "file %s not found in %s" % (fn, bbpath))
FileNotFoundError: [Errno 2] file recipes-extended/newt/libnewt_0.52.21.bb not found in /home/alexander/development/poky/build/workspace:/home/alexander/development/poky/meta-poky:/home/alexander/development/poky/build:/home/alexander/development/poky/meta:/home/alexander/development/poky/meta-yocto-bsp:/home/alexander/development/meta-mingw

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/alexander/development/poky/bitbake/lib/bb/command.py", line 74, in runCommand
    result = command_method(self, commandline)
  File "/home/alexander/development/poky/bitbake/lib/bb/command.py", line 562, in parseRecipeFile
    envdata = parser.loadDataFull(fn, appendfiles)
  File "/home/alexander/development/poky/bitbake/lib/bb/cache.py", line 327, in loadDataFull
    bb_data = self.load_bbfile(virtualfn, appends, virtonly=True)
  File "/home/alexander/development/poky/bitbake/lib/bb/cache.py", line 340, in load_bbfile
    datastores = parse_recipe(bb_data, bbfile, appends, mc)
  File "/home/alexander/development/poky/bitbake/lib/bb/cache.py", line 303, in parse_recipe
    bb_data = bb.parse.handle(bbfile, bb_data)
  File "/home/alexander/development/poky/bitbake/lib/bb/parse/__init__.py", line 107, in handle
    return h['handle'](fn, data, include)
  File "/home/alexander/development/poky/bitbake/lib/bb/parse/parse_py/BBHandler.py", line 130, in handle
    statements.eval(d)
  File "/home/alexander/development/poky/bitbake/lib/bb/parse/ast.py", line 23, in eval
    statement.eval(data)
  File "/home/alexander/development/poky/bitbake/lib/bb/parse/ast.py", line 45, in eval
    bb.parse.ConfHandler.include(self.filename, s, self.lineno, data, "include required")
  File "/home/alexander/development/poky/bitbake/lib/bb/parse/parse_py/ConfHandler.py", line 71, in include
    include_single_file(parentfn, fn, lineno, data, error_out)
  File "/home/alexander/development/poky/bitbake/lib/bb/parse/parse_py/ConfHandler.py", line 98, in include_single_file
    raise ParseError("Could not %s file %s" % (error_out, fn), parentfn, lineno)
bb.parse.ParseError: ParseError at /home/alexander/development/poky/build/workspace/recipes/libnewt-python/libnewt-python_0.52.21.bb:1: Could not include required file recipes-extended/newt/libnewt_0.52.21.bb

and the workspace will be left in an inconsistent state:

workspace/recipes will have the half-updated recipe that bitbake is unable to parse (due to missing include)

but workspace/appends will be empty.

Because of that, 'devtool reset' will not be able to remove the offending recipe from the workspace, and it has to be deleted by hand.
Comment 1 Alexander Kanavin 2019-06-19 10:06:37 UTC
A related improvement would be to have a 'devtool reset --force-all' command, which would basically re-initialize the workspace to a completely blank state. 

This wouls allow recovery from similar tracebacks - even if this specific one is fixed, there may be more.
Comment 2 Paul Eggleton 2019-07-02 04:27:19 UTC
I've sent a patch to fix this so it doesn't leave files around after the error by default (as well as generally fixing error handling). I think automatically handling this scenario completely (i.e. upgrading the other recipe) would be overkill, since this is rather unusual.

With regard to changing devtool reset, I'd prefer instead that we fix these kinds of situations as they come up rather than providing a tool to work around them.
Comment 3 Paul Eggleton 2019-07-03 20:27:48 UTC
Fix merged to master:

http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=b52d58b8bb81f5053b9fa8b79a137c37d6ef6fb3

(Note that the recipe originally triggering this was already changed to avoid the issue under http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=0d8357f9f38f529cc03ce2602d5209c9f8dfe878 ).