Bug 14040

Summary: recipetool: creating binary recipe from upstream git repo without explicit PV causes circular reference in SRC_URI
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Chris Laplante <mostthingsweb>
Component: devtools / tool chainAssignee: Yoann Congal <yoann.congal>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium+ CC: meta.mr.watcher, meta.watcher, randy.macleod, richard.purdie, yoann.congal
Version: 3.0.5   
Target Milestone: 4.3 M1   
Hardware: x86   
OS: Multiple   
See Also: https://bugzilla.yoctoproject.org/show_bug.cgi?id=15085
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Chris Laplante 2020-09-03 08:00:07 UTC
Found while attempting this:
devtool add -b abi-dumper-native https://github.com/lvc/abi-dumper

The '-b' flag causes ';workdir=${BP}' to be tacked onto the SRC_URI. 

Since we didn't specify a PV, it takes the default:
PV = "1.0+git${SRCPV}"

But SRCPV is defined in terms of the SRC_URI, which leads to infinite recursion (traceback below). Here are the pertinent variables which cause the recursion:

SRC_URI = "git://github.com/lvc/abi-dumper;protocol=https;subdir=${BP}"
BP = "${BPN}-${PV}"
PV = "1.0+git${SRCPV}"
SRCPV = "${@bb.fetch2.get_srcrev(d)}"

def get_srcrev(d, method_name='sortable_revision'):
    # ... trimmed
    scms = []
    fetcher = Fetch(d.getVar('SRC_URI').split(), d)
    # ... trimmed


Suggested fix: when setting PV in terms of 'SRCPV', ensure that 'subdir=${BPN}' is used instead of 'subdir=${BP}'. Thoughts?



Traceback:

ERROR: Traceback (most recent call last):
  File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/data_smart.py", line 411, in expandWithRefs
    s = __expand_python_regexp__.sub(varparse.python_sub, s)
  File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/data_smart.py", line 110, in python_sub
    if "_remote_data" in self.d:
  File "/usr/lib/python3.6/_collections_abc.py", line 666, in __contains__
    self[key]
  File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/data_smart.py", line 1021, in __getitem__
    value = self.getVar(item, False)
  File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/data_smart.py", line 604, in getVar
    return self.getVarFlag(var, "_content", expand, noweakdefault, parsing)
RecursionError: maximum recursion depth exceeded

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/command.py", line 74, in runCommand
    result = command_method(self, commandline)
  File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/command.py", line 562, in parseRecipeFile
    envdata = parser.loadDataFull(fn, appendfiles)
  File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/cache.py", line 324, in loadDataFull
    bb_data = self.load_bbfile(virtualfn, appends, virtonly=True)
  File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/cache.py", line 337, in load_bbfile
    datastores = parse_recipe(bb_data, bbfile, appends, mc)
  File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/cache.py", line 300, in parse_recipe
    bb_data = bb.parse.handle(bbfile, bb_data)
  File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/parse/__init__.py", line 107, in handle
    return h['handle'](fn, data, include)
  File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/parse/parse_py/BBHandler.py", line 141, in handle
    return ast.multi_finalize(fn, d)
  File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/parse/ast.py", line 386, in multi_finalize
    finalize(fn, d)
  File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/parse/ast.py", line 345, in finalize
    runAnonFuncs(d)
  File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/parse/ast.py", line 329, in runAnonFuncs
    bb.utils.better_exec("\n".join(code), {"d": d})
  File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/utils.py", line 396, in better_exec
    exec(code, get_context(), context)
  File "<code>", line 3, in <module>
  File "/home/laplante/yocto/sources/poky/meta/classes/base.bbclass", line 678, in __anon_706__home_laplante_yocto_sources_poky_meta_classes_base_bbclass
    fpaths = (d.getVar('FILESPATH') or '').split(':')
  File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/data_smart.py", line 604, in getVar
    return self.getVarFlag(var, "_content", expand, noweakdefault, parsing)
  File "/home/laplante/yocto/sources/poky/bitbake/lib/bb/data_smart.py", line 802, in getVarFlag
    parser = self.expandWithRefs(value, cachename)
Comment 1 Randy MacLeod 2020-09-10 07:34:49 UTC
11593 is related.
Comment 2 Randy MacLeod 2020-09-10 07:35:06 UTC
Also 14040.
Comment 3 Randy MacLeod 2020-09-10 07:36:31 UTC
Oops, this is 14040 but it's a circular reference so this error is amusing.
Comment 4 Chris Laplante 2020-09-10 09:25:03 UTC
Classic. I'll be sure to fix 14040 before I fix 14040 ;)
Comment 5 Yoann Congal 2023-03-28 22:46:22 UTC
This was fixed by Saul Wold in https://git.yoctoproject.org/poky/commit/?id=254fbcf614cd57d5a5d4ca36b9765b463eaaf10f in february 2022

Just tested it :

 $ devtool add -b abi-dumper-native https://github.com/lvc/abi-dumper 
INFO: Fetching git://github.com/lvc/abi-dumper;protocol=https;subdir=${BPN};branch=master...

This produce a recipe and the generated SRC_URI does indeed implement the proposed fix "subdir=${BPN}"
Comment 6 Richard Purdie 2023-03-29 12:42:20 UTC
Great to hear it is fixed. Are we missing a test case for this to ensure it doesn't regress again?
Comment 7 Yoann Congal 2023-03-29 13:03:17 UTC
(In reply to Richard Purdie from comment #6)
> Great to hear it is fixed. Are we missing a test case for this to ensure it
> doesn't regress again?

I've looked around a bit and I did not find a testcase for the '-b' flag.
https://git.yoctoproject.org/poky/tree/meta/lib/oeqa/selftest/cases/devtool.py
https://git.yoctoproject.org/poky/tree/meta/lib/oeqa/selftest/cases/recipetool.py
(And no testcase merged with the fix)

I'll create a separate ticket for this.
Comment 8 Yoann Congal 2023-03-29 13:10:24 UTC
=> Bug 15085 - Add a testcase for the devtool add "-b" flag
Comment 9 Richard Purdie 2023-03-29 15:51:19 UTC
Great, thanks!