| Summary: | npm fetcher failed when the npm package view have brackets in json values | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Build System, Metadata & Runtime] BitBake | Reporter: | Jean-Marie Lemetayer <jeanmarie.lemetayer> | ||||
| Component: | bitbake | Assignee: | Jean-Marie Lemetayer <jeanmarie.lemetayer> | ||||
| Status: | RESOLVED FIXED | QA Contact: | |||||
| Severity: | normal | ||||||
| Priority: | Medium | CC: | poky.bs.watcher, poky.watcher | ||||
| Version: | 2.6.2 | ||||||
| Target Milestone: | 0.0.0 | ||||||
| Hardware: | x86 | ||||||
| OS: | Multiple | ||||||
| Whiteboard: | |||||||
| OS type for building Yocto: | --- | Type of Regression: | --- | ||||
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |||||
| Attachments: |
|
||||||
Analysis -------- The last 'npm view' that we could see in the log.do_fetch is: npm view webpack@'4.29.0' --json --registry http://registry.npmjs.org/ Running it on host PC: { ... "lint-staged": { "*.js|{lib,setup,bin,hot,buildin,tooling,schemas}/**/*.js|test/*.js|{test,examples}/**/webpack.config.js}": [ }, ... } The JSON is valid but the value have one extra '}'. But the '_parse_view' function in bitbake/lib/bb/fetch2/npm.py is manually counting brackets. As a result the output JSON is stopped early, and is no more valid, thus the JSONDecodeError exception. ----- Note that we have the same issue with '@angular-devkit/build-angular' version 0.13.8. But the error is on the 'node-sass' package version 4.11.0 which include the readme (with an extra '}') in the 'npm view'. Proposed fix ------------ Do not parse the 'npm view --json' output manually, but use a JSON parser instead. In his commit (https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?h=thud&id=fbf27c4c5dae4837f399901c6afb4ffb25098e46) Paul Eggleton says that - The output of "npm view dependencies" isn't entirely JSON if there are multiple results - which seems to be fixed. Running 'npm view --json webpack@4.29.0' will output a JSON object '{...}'. And running 'npm view --json webpack'>@4.25.0'' will output a JSON array of object '[{...},{...}]'. For information the nodejs released in the meta-openembedded on the thud branch is 8.12.0 which packed a npm version 6.4.1. --- diff --git a/bitbake/lib/bb/fetch2/npm.py b/bitbake/lib/bb/fetch2/npm.py index 408dfc3d03..b197c2d181 100644 --- a/bitbake/lib/bb/fetch2/npm.py +++ b/bitbake/lib/bb/fetch2/npm.py @@ -150,20 +150,11 @@ class Npm(FetchMethod): Parse the output of npm view --json; the last JSON result is assumed to be the one that we're interested in. ''' - pdata = None - outdeps = {} - datalines = [] - bracelevel = 0 - for line in output.splitlines(): - if bracelevel: - datalines.append(line) - elif '{' in line: - datalines = [] - datalines.append(line) - bracelevel = bracelevel + line.count('{') - line.count('}') - if datalines: - pdata = json.loads('\n'.join(datalines)) - return pdata + pdata = json.loads(output); + try: + return pdata[-1] + except: + return pdata Created attachment 4514 [details]
[PATCH] bitbake: fetch2/npm: fix npw view parsing
Here is the proposed patch.
The patch looks reasonable to me. Could you send a patch to the bitbake-devel mailing list with a commit log which explains the issue. I'd also perhaps like to see some comments in the code explaining what its doing/why and maybe adding some tests to the lib/bb/tests/ directory so that we don't regress this in future. |
Preconditions/Environment ------------------------- Build environment with: - poky on the thud branch - meta-openembedded on the thud branch Triggering Action/Cause ----------------------- devtool add 'npm://registry.npmjs.org;name=@angular-devkit/build-angular;version=0.13.9' Expectation ----------- The recipe is created. Actual Result ------------- ERROR: Error executing a python function in exec_python_func() autogenerated: The stack trace of python calls that resulted in this exception/failure was: File: 'exec_python_func() autogenerated', lineno: 2, function: <module> 0001: *** 0002:base_do_fetch(d) 0003: ... File: '/home/jmlemetayer/development/microbs/yocto/micro-detector/poky/bitbake/lib/bb/fetch2/npm.py', lineno: 165, function: _parse_view 0161: datalines = [] 0162: datalines.append(line) 0163: bracelevel = bracelevel + line.count('{') - line.count('}') 0164: if datalines: *** 0165: pdata = json.loads('\n'.join(datalines)) 0166: return pdata 0167: 0168: def _getdependencies(self, pkg, data, version, d, ud, optional=False, fetchedlist=None): 0169: if fetchedlist is None: ... Exception: json.decoder.JSONDecodeError: Extra data: line 1 column 28 (char 27) ERROR: Function failed: base_do_fetch Reproducibility --------------- 10/10