Bug 11235 - Improve configuration parser - better error reporting and more lenient
Summary: Improve configuration parser - better error reporting and more lenient
Status: RESOLVED OBSOLETE
Alias: None
Product: AutoBuilder
Classification: Infrastructure
Component: autobuilder (show other bugs)
Version: 2.4
Hardware: x86 Multiple
: Medium enhancement
Target Milestone: 2.5
Assignee: Joshua Lock
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2017-03-24 10:27 UTC by Joshua Lock
Modified: 2018-03-22 08:48 UTC (History)
3 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: Yes (doc changes required)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Joshua Lock 2017-03-24 10:27:19 UTC
The buildset configuration parser is very unforgiving of syntax errors such as a stray comma[1].
Said stray comma results in a long Traceback which doesn't provide many clues for fixing the problem short of digging into the code, i.e. (from[2]):

Traceback (most recent call last):
  File "/media/data/yocto-autobuilder/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 1187, in unwindGenerator
    return _inlineCallbacks(None, gen, Deferred())
  File "/media/data/yocto-autobuilder/lib/python2.7/site-packages/Twisted-12.2.0-py2.7-linux-x86_64.egg/twisted/internet/defer.py", line 1045, in _inlineCallbacks
    result = g.send(result)
  File "/media/data/yocto-autobuilder/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/scripts/upgrade_master.py", line 169, in upgradeMaster
    master_cfg = loadConfig(config, configFile)
  File "/media/data/yocto-autobuilder/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/scripts/upgrade_master.py", line 53, in loadConfig
    config['basedir'], configFileName)
--- <exception caught here> ---
  File "/media/data/yocto-autobuilder/lib/python2.7/site-packages/buildbot-0.8.8-py2.7.egg/buildbot/config.py", line 149, in loadConfig
    exec f in localDict
  File "/media/data/yocto-autobuilder/yocto-controller/controller.cfg", line 94, in <module>
    yocto_buildsets.createBuildsets()
  File "/media/data/yocto-autobuilder/lib/python2.7/site-packages/autobuilder/Autobuilder.py", line 65, in createBuildsets
    self.parseBuildSet(buildset)
  File "/media/data/yocto-autobuilder/lib/python2.7/site-packages/autobuilder/Autobuilder.py", line 102, in parseBuildSet
    self.parseRepos(buildset)
  File "/media/data/yocto-autobuilder/lib/python2.7/site-packages/autobuilder/Autobuilder.py", line 127, in parseRepos
    if layer.iterkeys().next() not in self.repos:
exceptions.AttributeError: 'list' object has no attribute 'iterkeys'
Errors loading configuration:
  error while parsing config file: 'list' object has no attribute 'iterkeys' (traceback in logfile)

The yocto-autobuilder will be friendlier to administrators if it can help them diagnose such issues more easily by providing more informative reporting of failures, such as pointing to the specific file and line that caused the parser failure.

For extra convenience we could:
* make the parser more lenient of stray punctuation (like setLenient() in Android's JsonReader[3])
* switch to using a pure data format, such as JSON, rather than parsing Python AST's. If we were to switch to JSON a user could use their editor of choice's built in linter to detect such issues at editing time.


1. http://git.yoctoproject.org/clean/cgit.cgi/yocto-autobuilder/commit/?id=239eeaa1e2cf3a985e75f8b75dedfe704a7d6991
2. https://lists.yoctoproject.org/pipermail/yocto/2017-March/035274.html
3. https://developer.android.com/reference/android/util/JsonReader.html#setLenient(boolean)
Comment 1 Joshua Lock 2018-03-22 08:48:55 UTC
Effectively resolved by the new infrastructure architecture of helper scripts (yocto-autobuilder-helper) with a more recent buildbot with minimal configuration (yocto-autobuilder2), the configuration parser is not used in the new CI stack.