Bug 13344 - npm fetcher failed when the npm package view have brackets in json values
Summary: npm fetcher failed when the npm package view have brackets in json values
Status: RESOLVED FIXED
Alias: None
Product: BitBake
Classification: Build System, Metadata & Runtime
Component: bitbake (show other bugs)
Version: 2.6.2
Hardware: x86 Multiple
: Medium normal
Target Milestone: 0.0.0
Assignee: Jean-Marie Lemetayer
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2019-05-10 13:05 UTC by Jean-Marie Lemetayer
Modified: 2019-10-02 09:10 UTC (History)
2 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments
[PATCH] bitbake: fetch2/npm: fix npw view parsing (1.84 KB, patch)
2019-05-10 14:30 UTC, Jean-Marie Lemetayer
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Jean-Marie Lemetayer 2019-05-10 13:05:03 UTC
Preconditions/Environment
-------------------------
Build environment with:
 - poky on the thud branch
 - meta-openembedded on the thud branch

Triggering Action/Cause
-----------------------
devtool add 'npm://registry.npmjs.org;name=@angular-devkit/build-angular;version=0.13.9'

Expectation
-----------
The recipe is created.

Actual Result
-------------
ERROR: Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_python_func() autogenerated', lineno: 2, function: <module>
     0001:
 *** 0002:base_do_fetch(d)
     0003:

...

File: '/home/jmlemetayer/development/microbs/yocto/micro-detector/poky/bitbake/lib/bb/fetch2/npm.py', lineno: 165, function: _parse_view
     0161:                datalines = []
     0162:                datalines.append(line)
     0163:            bracelevel = bracelevel + line.count('{') - line.count('}')
     0164:        if datalines:
 *** 0165:            pdata = json.loads('\n'.join(datalines))
     0166:        return pdata
     0167:
     0168:    def _getdependencies(self, pkg, data, version, d, ud, optional=False, fetchedlist=None):
     0169:        if fetchedlist is None:

...

Exception: json.decoder.JSONDecodeError: Extra data: line 1 column 28 (char 27)

ERROR: Function failed: base_do_fetch

Reproducibility
---------------
10/10
Comment 1 Jean-Marie Lemetayer 2019-05-10 13:18:22 UTC
Analysis
--------

The last 'npm view' that we could see in the log.do_fetch is:
npm view webpack@'4.29.0' --json --registry http://registry.npmjs.org/

Running it on host PC:
{
  ...

  "lint-staged": {
    "*.js|{lib,setup,bin,hot,buildin,tooling,schemas}/**/*.js|test/*.js|{test,examples}/**/webpack.config.js}": [
  },

  ...
}

The JSON is valid but the value have one extra '}'. But the '_parse_view' function in bitbake/lib/bb/fetch2/npm.py is manually counting brackets.

As a result the output JSON is stopped early, and is no more valid, thus the JSONDecodeError exception.

-----
Note that we have the same issue with '@angular-devkit/build-angular' version 0.13.8. But the error is on the 'node-sass' package version 4.11.0 which include the readme (with an extra '}') in the 'npm view'.
Comment 2 Jean-Marie Lemetayer 2019-05-10 13:47:18 UTC
Proposed fix
------------

Do not parse the 'npm view --json' output manually, but use a JSON parser instead.

In his commit (https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?h=thud&id=fbf27c4c5dae4837f399901c6afb4ffb25098e46) Paul Eggleton says that  - The output of "npm view dependencies" isn't entirely JSON if there are multiple results - which seems to be fixed.

Running 'npm view --json webpack@4.29.0' will output a JSON object '{...}'. And running 'npm view --json webpack'>@4.25.0'' will output a JSON array of object '[{...},{...}]'.

For information the nodejs released in the meta-openembedded on the thud branch is 8.12.0 which packed a npm version 6.4.1.

---

diff --git a/bitbake/lib/bb/fetch2/npm.py b/bitbake/lib/bb/fetch2/npm.py
index 408dfc3d03..b197c2d181 100644
--- a/bitbake/lib/bb/fetch2/npm.py
+++ b/bitbake/lib/bb/fetch2/npm.py
@@ -150,20 +150,11 @@ class Npm(FetchMethod):
         Parse the output of npm view --json; the last JSON result
         is assumed to be the one that we're interested in.
         '''
-        pdata = None
-        outdeps = {}
-        datalines = []
-        bracelevel = 0
-        for line in output.splitlines():
-            if bracelevel:
-                datalines.append(line)
-            elif '{' in line:
-                datalines = []
-                datalines.append(line)
-            bracelevel = bracelevel + line.count('{') - line.count('}')
-        if datalines:
-            pdata = json.loads('\n'.join(datalines))
-        return pdata
+        pdata = json.loads(output);
+        try:
+            return pdata[-1]
+        except:
+            return pdata
Comment 3 Jean-Marie Lemetayer 2019-05-10 14:30:26 UTC
Created attachment 4514 [details]
[PATCH] bitbake: fetch2/npm: fix npw view parsing

Here is the proposed patch.
Comment 4 Richard Purdie 2019-05-16 14:43:15 UTC
The patch looks reasonable to me. Could you send a patch to the bitbake-devel mailing list with a commit log which explains the issue. I'd also perhaps like to see some comments in the code explaining what its doing/why and maybe adding some tests to the lib/bb/tests/ directory so that we don't regress this in future.