<?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
<!DOCTYPE bugzilla SYSTEM "https://bugzilla.yoctoproject.org/page.cgi?id=bugzilla.dtd">

<bugzilla version="5.0.6"
          urlbase="https://bugzilla.yoctoproject.org/"
          
          maintainer="it-coreprojects-helpdesk@linuxfoundation.org"
>

    <bug>
          <bug_id>13344</bug_id>
          
          <creation_ts>2019-05-10 13:05:03 +0000</creation_ts>
          <short_desc>npm fetcher failed when the npm package view have brackets in json values</short_desc>
          <delta_ts>2019-10-02 09:10:06 +0000</delta_ts>
          <reporter_accessible>1</reporter_accessible>
          <cclist_accessible>1</cclist_accessible>
          <classification_id>7</classification_id>
          <classification>Build System, Metadata &amp; Runtime</classification>
          <product>BitBake</product>
          <component>bitbake</component>
          <version>2.6.2</version>
          <rep_platform>x86</rep_platform>
          <op_sys>Multiple</op_sys>
          <bug_status>RESOLVED</bug_status>
          <resolution>FIXED</resolution>
          
          
          <bug_file_loc></bug_file_loc>
          <status_whiteboard></status_whiteboard>
          <keywords></keywords>
          <priority>Medium</priority>
          <bug_severity>normal</bug_severity>
          <target_milestone>0.0.0</target_milestone>
          
          
          <everconfirmed>1</everconfirmed>
          <reporter name="Jean-Marie Lemetayer">jeanmarie.lemetayer</reporter>
          <assigned_to name="Jean-Marie Lemetayer">jeanmarie.lemetayer</assigned_to>
          <cc>poky.bs.watcher</cc>
    
    <cc>poky.watcher</cc>
          
          
          <cf_os>---</cf_os>
          <cf_regression_type>---</cf_regression_type>
          
          <cf_docchange>No (bug/feature does not impact docs)</cf_docchange>

      

      

      

          <comment_sort_order>oldest_to_newest</comment_sort_order>  
          <long_desc isprivate="0" >
    <commentid>83782</commentid>
    <comment_count>0</comment_count>
    <who name="Jean-Marie Lemetayer">jeanmarie.lemetayer</who>
    <bug_when>2019-05-10 13:05:03 +0000</bug_when>
    <thetext>Preconditions/Environment
-------------------------
Build environment with:
 - poky on the thud branch
 - meta-openembedded on the thud branch

Triggering Action/Cause
-----------------------
devtool add &apos;npm://registry.npmjs.org;name=@angular-devkit/build-angular;version=0.13.9&apos;

Expectation
-----------
The recipe is created.

Actual Result
-------------
ERROR: Error executing a python function in exec_python_func() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: &apos;exec_python_func() autogenerated&apos;, lineno: 2, function: &lt;module&gt;
     0001:
 *** 0002:base_do_fetch(d)
     0003:

...

File: &apos;/home/jmlemetayer/development/microbs/yocto/micro-detector/poky/bitbake/lib/bb/fetch2/npm.py&apos;, lineno: 165, function: _parse_view
     0161:                datalines = []
     0162:                datalines.append(line)
     0163:            bracelevel = bracelevel + line.count(&apos;{&apos;) - line.count(&apos;}&apos;)
     0164:        if datalines:
 *** 0165:            pdata = json.loads(&apos;\n&apos;.join(datalines))
     0166:        return pdata
     0167:
     0168:    def _getdependencies(self, pkg, data, version, d, ud, optional=False, fetchedlist=None):
     0169:        if fetchedlist is None:

...

Exception: json.decoder.JSONDecodeError: Extra data: line 1 column 28 (char 27)

ERROR: Function failed: base_do_fetch

Reproducibility
---------------
10/10</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>83783</commentid>
    <comment_count>1</comment_count>
    <who name="Jean-Marie Lemetayer">jeanmarie.lemetayer</who>
    <bug_when>2019-05-10 13:18:22 +0000</bug_when>
    <thetext>Analysis
--------

The last &apos;npm view&apos; that we could see in the log.do_fetch is:
npm view webpack@&apos;4.29.0&apos; --json --registry http://registry.npmjs.org/

Running it on host PC:
{
  ...

  &quot;lint-staged&quot;: {
    &quot;*.js|{lib,setup,bin,hot,buildin,tooling,schemas}/**/*.js|test/*.js|{test,examples}/**/webpack.config.js}&quot;: [
  },

  ...
}

The JSON is valid but the value have one extra &apos;}&apos;. But the &apos;_parse_view&apos; function in bitbake/lib/bb/fetch2/npm.py is manually counting brackets.

As a result the output JSON is stopped early, and is no more valid, thus the JSONDecodeError exception.

-----
Note that we have the same issue with &apos;@angular-devkit/build-angular&apos; version 0.13.8. But the error is on the &apos;node-sass&apos; package version 4.11.0 which include the readme (with an extra &apos;}&apos;) in the &apos;npm view&apos;.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>83785</commentid>
    <comment_count>2</comment_count>
    <who name="Jean-Marie Lemetayer">jeanmarie.lemetayer</who>
    <bug_when>2019-05-10 13:47:18 +0000</bug_when>
    <thetext>Proposed fix
------------

Do not parse the &apos;npm view --json&apos; output manually, but use a JSON parser instead.

In his commit (https://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?h=thud&amp;id=fbf27c4c5dae4837f399901c6afb4ffb25098e46) Paul Eggleton says that  - The output of &quot;npm view dependencies&quot; isn&apos;t entirely JSON if there are multiple results - which seems to be fixed.

Running &apos;npm view --json webpack@4.29.0&apos; will output a JSON object &apos;{...}&apos;. And running &apos;npm view --json webpack&apos;&gt;@4.25.0&apos;&apos; will output a JSON array of object &apos;[{...},{...}]&apos;.

For information the nodejs released in the meta-openembedded on the thud branch is 8.12.0 which packed a npm version 6.4.1.

---

diff --git a/bitbake/lib/bb/fetch2/npm.py b/bitbake/lib/bb/fetch2/npm.py
index 408dfc3d03..b197c2d181 100644
--- a/bitbake/lib/bb/fetch2/npm.py
+++ b/bitbake/lib/bb/fetch2/npm.py
@@ -150,20 +150,11 @@ class Npm(FetchMethod):
         Parse the output of npm view --json; the last JSON result
         is assumed to be the one that we&apos;re interested in.
         &apos;&apos;&apos;
-        pdata = None
-        outdeps = {}
-        datalines = []
-        bracelevel = 0
-        for line in output.splitlines():
-            if bracelevel:
-                datalines.append(line)
-            elif &apos;{&apos; in line:
-                datalines = []
-                datalines.append(line)
-            bracelevel = bracelevel + line.count(&apos;{&apos;) - line.count(&apos;}&apos;)
-        if datalines:
-            pdata = json.loads(&apos;\n&apos;.join(datalines))
-        return pdata
+        pdata = json.loads(output);
+        try:
+            return pdata[-1]
+        except:
+            return pdata</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>83787</commentid>
    <comment_count>3</comment_count>
      <attachid>4514</attachid>
    <who name="Jean-Marie Lemetayer">jeanmarie.lemetayer</who>
    <bug_when>2019-05-10 14:30:26 +0000</bug_when>
    <thetext>Created attachment 4514
[PATCH] bitbake: fetch2/npm: fix npw view parsing

Here is the proposed patch.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>83822</commentid>
    <comment_count>4</comment_count>
    <who name="Richard Purdie">richard.purdie</who>
    <bug_when>2019-05-16 14:43:15 +0000</bug_when>
    <thetext>The patch looks reasonable to me. Could you send a patch to the bitbake-devel mailing list with a commit log which explains the issue. I&apos;d also perhaps like to see some comments in the code explaining what its doing/why and maybe adding some tests to the lib/bb/tests/ directory so that we don&apos;t regress this in future.</thetext>
  </long_desc><long_desc isprivate="0" >
    <commentid>85141</commentid>
    <comment_count>5</comment_count>
    <who name="Jean-Marie Lemetayer">jeanmarie.lemetayer</who>
    <bug_when>2019-10-02 09:10:06 +0000</bug_when>
    <thetext>Fixed by https://git.openembedded.org/bitbake/commit/?id=3d319c79981811d3cfd4732885057db4fd5afcc2</thetext>
  </long_desc>
      
          <attachment
              isobsolete="0"
              ispatch="1"
              isprivate="0"
          >
            <attachid>4514</attachid>
            <date>2019-05-10 14:30:26 +0000</date>
            <delta_ts>2019-05-10 14:30:26 +0000</delta_ts>
            <desc>[PATCH] bitbake: fetch2/npm: fix npw view parsing</desc>
            <filename>0001-bitbake-fetch2-npm-fix-npw-view-parsing.patch</filename>
            <type>text/plain</type>
            <size>1887</size>
            <attacher name="Jean-Marie Lemetayer">jeanmarie.lemetayer</attacher>
            
              <data encoding="base64">RnJvbSA0OTU5ODA3YmY1Mjk4ZjUwYzZkZjkwOGZkMTk3ZGU4NGEyYjk4OWY1IE1vbiBTZXAgMTcg
MDA6MDA6MDAgMjAwMQpGcm9tOiBKZWFuLU1hcmllIExFTUVUQVlFUiA8amVhbi1tYXJpZS5sZW1l
dGF5ZXJAc2F2b2lyZmFpcmVsaW51eC5jb20+CkRhdGU6IEZyaSwgMTAgTWF5IDIwMTkgMTE6MjM6
MjMgKzAyMDAKU3ViamVjdDogW1BBVENIXSBiaXRiYWtlOiBmZXRjaDIvbnBtOiBmaXggbnB3IHZp
ZXcgcGFyc2luZwoKRml4ZXMgW1lPQ1RPICMxMzM0NF0KCldoZW4gcGFyc2luZyBtYW51YWxseSB0
aGUgJ25wbSB2aWV3IC0tanNvbicgb3VwdXQsIGFuIGV4dHJhIGNsb3NpbmcKYnJhY2tldHMgaW4g
YSBKU09OIHN0cmluZyBjYW4gbGVhZHMgdGhlIGZldGNoZXIgdG8gZmFpbCB3aXRoIGEKSlNPTkRl
Y29kZUVycm9yIGV4Y2VwdGlvbi4KClRoaXMgY29tbWl0IHVzZSB0aGUgSlNPTiBwYXJzZXIgdG8g
ZXh0cmFjdDoKIC0gVGhlIGxhc3Qgb2JqZWN0IGluIHRoZSByZXR1cm5lZCBhcnJheSBpZiB0aGVy
ZSBhcmUgbXVsdGlwbGUgcmVzdWx0cy4KIC0gVGhlIHJldHVybmVkIG9iamVjdCBpZiB0aGVyZSBp
cyBvbmx5IG9uZSByZXN1bHQuCgpTaWduZWQtb2ZmLWJ5OiBKZWFuLU1hcmllIExFTUVUQVlFUiA8
amVhbi1tYXJpZS5sZW1ldGF5ZXJAc2F2b2lyZmFpcmVsaW51eC5jb20+Ci0tLQogYml0YmFrZS9s
aWIvYmIvZmV0Y2gyL25wbS5weSB8IDE5ICsrKysrLS0tLS0tLS0tLS0tLS0KIDEgZmlsZSBjaGFu
Z2VkLCA1IGluc2VydGlvbnMoKyksIDE0IGRlbGV0aW9ucygtKQoKZGlmZiAtLWdpdCBhL2JpdGJh
a2UvbGliL2JiL2ZldGNoMi9ucG0ucHkgYi9iaXRiYWtlL2xpYi9iYi9mZXRjaDIvbnBtLnB5Cmlu
ZGV4IDQwOGRmYzNkMDMuLmIxOTdjMmQxODEgMTAwNjQ0Ci0tLSBhL2JpdGJha2UvbGliL2JiL2Zl
dGNoMi9ucG0ucHkKKysrIGIvYml0YmFrZS9saWIvYmIvZmV0Y2gyL25wbS5weQpAQCAtMTUwLDIw
ICsxNTAsMTEgQEAgY2xhc3MgTnBtKEZldGNoTWV0aG9kKToKICAgICAgICAgUGFyc2UgdGhlIG91
dHB1dCBvZiBucG0gdmlldyAtLWpzb247IHRoZSBsYXN0IEpTT04gcmVzdWx0CiAgICAgICAgIGlz
IGFzc3VtZWQgdG8gYmUgdGhlIG9uZSB0aGF0IHdlJ3JlIGludGVyZXN0ZWQgaW4uCiAgICAgICAg
ICcnJwotICAgICAgICBwZGF0YSA9IE5vbmUKLSAgICAgICAgb3V0ZGVwcyA9IHt9Ci0gICAgICAg
IGRhdGFsaW5lcyA9IFtdCi0gICAgICAgIGJyYWNlbGV2ZWwgPSAwCi0gICAgICAgIGZvciBsaW5l
IGluIG91dHB1dC5zcGxpdGxpbmVzKCk6Ci0gICAgICAgICAgICBpZiBicmFjZWxldmVsOgotICAg
ICAgICAgICAgICAgIGRhdGFsaW5lcy5hcHBlbmQobGluZSkKLSAgICAgICAgICAgIGVsaWYgJ3sn
IGluIGxpbmU6Ci0gICAgICAgICAgICAgICAgZGF0YWxpbmVzID0gW10KLSAgICAgICAgICAgICAg
ICBkYXRhbGluZXMuYXBwZW5kKGxpbmUpCi0gICAgICAgICAgICBicmFjZWxldmVsID0gYnJhY2Vs
ZXZlbCArIGxpbmUuY291bnQoJ3snKSAtIGxpbmUuY291bnQoJ30nKQotICAgICAgICBpZiBkYXRh
bGluZXM6Ci0gICAgICAgICAgICBwZGF0YSA9IGpzb24ubG9hZHMoJ1xuJy5qb2luKGRhdGFsaW5l
cykpCi0gICAgICAgIHJldHVybiBwZGF0YQorICAgICAgICBwZGF0YSA9IGpzb24ubG9hZHMob3V0
cHV0KTsKKyAgICAgICAgdHJ5OgorICAgICAgICAgICAgcmV0dXJuIHBkYXRhWy0xXQorICAgICAg
ICBleGNlcHQ6CisgICAgICAgICAgICByZXR1cm4gcGRhdGEKIAogICAgIGRlZiBfZ2V0ZGVwZW5k
ZW5jaWVzKHNlbGYsIHBrZywgZGF0YSwgdmVyc2lvbiwgZCwgdWQsIG9wdGlvbmFsPUZhbHNlLCBm
ZXRjaGVkbGlzdD1Ob25lKToKICAgICAgICAgaWYgZmV0Y2hlZGxpc3QgaXMgTm9uZToKLS0gCjIu
MjAuMQoK
</data>

          </attachment>
      

    </bug>

</bugzilla>