Bug 12700

Summary: [2.5 M3 rc1][poky-tiny][Centos7][autobuilder2] - Clobber build dir - failed to load utils
Product: [Infrastructure] AutoBuilder Reporter: Aaron Chan <aaron.chun.yew.chan>
Component: autobuilderAssignee: Aaron Chan <aaron.chun.yew.chan>
Status: RESOLVED WONTFIX QA Contact: Aaron Chan <aaron.chun.yew.chan>
Severity: normal    
Priority: Medium+ CC: apoorv.sangal, bluelightning, infras.ab.watcher, Infras.watcher, pidge, richard.purdie, ross.burton
Version: 2.5   
Target Milestone: 2.5 M4   
Hardware: x86   
OS: Multiple   
URL: http://centaurus02.png.intel.com:8010/#/builders/13/builds/3
Whiteboard:
OS type for building Yocto: CentOS Type of Regression: Regression (Found by QA test)
Verified: Documentation change: Don't know
Attachments:
Description Flags
BuildStep - Clobber build dir
none
[yocto-autobuilder-helper] config.json configfile none

Description Aaron Chan 2018-04-20 02:28:53 UTC
Created attachment 4277 [details]
BuildStep - Clobber build dir

=======================================================================================================
BUILDSTEP - Clobber build dir
=======================================================================================================
Traceback (most recent call last):
  File "/home/ab/yocto-autobuilder-helper/janitor/clobberdir", line 20, in <module>
    ourconfig = utils.loadconfig(os.path.dirname(__file__) + "/../scripts/dummy")
  File "/home/ab/yocto-autobuilder-helper/janitor/../scripts/utils.py", line 79, in loadconfig
    ourconfig = json.load(f)
  File "/usr/lib64/python2.7/json/__init__.py", line 290, in load
    **kw)
  File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python2.7/json/decoder.py", line 382, in raw_decode
    obj, end = self.scan_once(s, idx)
ValueError: Expecting object: line 717 column 1 (char 30284)
program finished with exit code 1

There are 2 possible reason for this error.
1. utils.loadconfig unable to source the scripts directory to load utils module.
2. error in Json content @ ourconfig = json.load(f)

It appears that this error is only found on this distribution (centos7) and seems to be consistent over other jobs executed as well.
Need to further investigate this to narrow down the root cause.
Comment 1 Richard Purdie 2018-04-26 07:39:23 UTC
Hi Aaron, This looks like it might be caused by an error in the config.json file. Could you check that and if you can't find the issue, attach the config.json please?
Comment 2 Aaron Chan 2018-04-26 18:45:08 UTC
Created attachment 4288 [details]
[yocto-autobuilder-helper] config.json configfile
Comment 3 Aaron Chan 2018-04-26 18:50:16 UTC
Hi Richard,

Was able to locate and find the syntax error from the config.json file since its a long list.
So what i did was to recopy the entire yocto-autobuilder-helper into the worker home directory.

# Just a proposal that we should give a better error handling when loading the json contents.

http://git.yoctoproject.org/cgit/cgit.cgi/yocto-autobuilder-helper/tree/scripts/utils.py#n79

#
# Helper to load the config.json file for scripts in the scripts directory (pass in __file__)
#
def loadconfig(f):
    scriptsdir = os.path.dirname(os.path.realpath(f))

    try:
        with open(os.path.join(scriptsdir, '..', 'config.json')) as f:
            ourconfig = json.load(f)
    except:
        print "[autobuilder2-helper] has found a syntax error in config.json"
        sys.exit(1)

    # Expand templates in the configuration
    ourconfig = expandtemplates(ourconfig)

    return ourconfig
Comment 4 Richard Purdie 2018-05-03 08:22:26 UTC
*** Bug 12701 has been marked as a duplicate of this bug. ***
Comment 5 Ross Burton 2018-05-03 08:24:00 UTC
The syntax error is that you're missing a trailing }
Comment 6 Richard Purdie 2018-05-03 08:25:04 UTC
Unfortunately this isn't something we can easily fix. We're using the standard upstream python json library and any parser issues would need to be fixed there. Much as I'd like to improve this, its therefore out of scope of what we can do within the codebases we're writing.