Bug 13723

Summary: Layer index parsing broken
Product: [Yocto Project Subprojects] Layer Index Reporter: Paul Eggleton <bluelightning>
Component: Layer IndexAssignee: Paul Eggleton <bluelightning>
Status: RESOLVED FIXED QA Contact: apoorv sangal <apoorvsangal>
Severity: major    
Priority: High CC: mark.hatle
Version: unspecified   
Target Milestone: 3.1 M2   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Paul Eggleton 2019-12-23 02:27:34 UTC
Parsing OE-Core (and of course any other layer that uses it) is currently broken:

------------ snip ------------
$ layerindex/update.py -b master -l openembedded-core
INFO: Fetching remote repository git://git.openembedded.org/openembedded-core
INFO: Fetching remote repository git://git.openembedded.org/bitbake
INFO: Collecting data for layer openembedded-core on branch master
NOTE: Starting bitbake server...
ERROR: Unable to start bitbake server (None)
ERROR: Last 60 lines of server log for this session (/tmp/tmp_g7583ey/bitbake-cookerdaemon.log):
  File "Var <AVAILABLE_LICENSES[:=]>", line 1, in <module>
  File "/home/paul/poky/layerindex-web/layers/git___git_openembedded_org_openembedded-core/meta/classes/license.bbclass", line 275, in available_licenses
    licenses += os.listdir(lic_dir)
bb.data_smart.ExpansionError: Failure expanding variable AVAILABLE_LICENSES[:=], expression was ${@' '.join(available_licenses(d))} which triggered exception FileNotFoundError: [Errno 2] No such file or directory: '${COREBASE}/meta/files/common-licenses'
ERROR: Unable to parse Var <AVAILABLE_LICENSES[:=]>
Traceback (most recent call last):
  File "/home/paul/poky/layerindex-web/layers/git___git_openembedded_org_bitbake/lib/bb/data_smart.py", line 411, in expandWithRefs
    s = __expand_python_regexp__.sub(varparse.python_sub, s)
  File "/home/paul/poky/layerindex-web/layers/git___git_openembedded_org_bitbake/lib/bb/data_smart.py", line 136, in python_sub
    value = utils.better_eval(codeobj, DataContext(self.d), {'d' : self.d})
  File "/home/paul/poky/layerindex-web/layers/git___git_openembedded_org_bitbake/lib/bb/utils.py", line 421, in better_eval
    return eval(source, ctx, locals)
  File "Var <AVAILABLE_LICENSES[:=]>", line 1, in <module>
  File "/home/paul/poky/layerindex-web/layers/git___git_openembedded_org_openembedded-core/meta/classes/license.bbclass", line 275, in available_licenses
    licenses += os.listdir(lic_dir)
FileNotFoundError: [Errno 2] No such file or directory: '${COREBASE}/meta/files/common-licenses'
------------ snip ------------

I haven't yet bisected to find out which commit caused this to break, but I have found that it is happening fairly early on - it's within the call to recipeparse.init_parser() in update_layer.py which in the layer index is too early for the layer.conf files to have been parsed, and thus COREBASE is not yet set.
Comment 1 Mark Hatle 2019-12-23 14:31:54 UTC
I ran into a similar issue with another layer.  The code was something like:

${@'${VARIABLE}/something'.function()}

And 'VARIABLE' was not set within the context of the parsing engine (using tinfoil).  So an error was triggered that '${...}' didn't expand.  By simply defining 'VARIABLE' this resolved the issue where I saw it.

But I do wonder if this is really the right behavior or if it the variable usage should be:

${@('%/something' % d.getVar('VARIABLE')).function()}

The issue with the above is that the usage of 'VARIABLE' is now lost and not tracked by the regular hash code that looks for ${...}
Comment 2 Paul Eggleton 2020-01-07 22:50:22 UTC
It turns out that this was directly caused by http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=2a263780814e3b07d48e3812579cfe4696db8413 (or more specifically, COREBASE not being set when the immediate assignment of AVAILABLE_LICENSES is set). I've merged a patch that just sets a value for COREBASE to avoid the problem. Michael has cherry-picked the patch on top of the working branch used for layers.openembedded.org and the update is now running.