Bug 13415

Summary: Multiple problems with npm packages
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Kristupas Savickas <kristupas.savickas>
Component: Scripts and ToolsAssignee: Razi Seyyedi <razi.seyyedi>
Status: RESOLVED FIXED QA Contact:
Severity: major    
Priority: Medium CC: jeanmarie.lemetayer, kristupas.savickas, poky.bs.watcher, poky.watcher, randy.macleod, ross.burton, tim.orling
Version: unspecified   
Target Milestone: Future   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: Don't know
Attachments:
Description Flags
do_install
none
do_populate_lic
none
do_compile
none
2019-06-20T10_48_45_724Z-debug.log
none
package.json none

Description Kristupas Savickas 2019-06-20 10:36:32 UTC
I've tried adding a nodejs package that's pulled from git following this section in documentation: https://www.yoctoproject.org/docs/current/dev-manual/dev-manual.html#npm-using-the-npm-projects-method. I've ran into multiple problems while doing this:

After using devtool add I end up with a recipe like this:

> # Recipe created by recipetool
> # This is the basis of a recipe and may need further editing in order to be fully functional.
> # (Feel free to remove these comments when editing.)

> SUMMARY = ""
> # WARNING: the following LICENSE and LIC_FILES_CHKSUM values are best guesses - it is
> # your responsibility to verify that the values are complete and correct.
> LICENSE = "ISC & MIT"
> LIC_FILES_CHKSUM = "file://node_modules/ws/LICENSE;md5=95833e8f03687308b0584a377b9e12b0 \
>                     file://node_modules/ws/node_modules/async-limiter/LICENSE;md5=4b83a79a0c223073786a52b5ece4619d"
> 
> SRC_URI = "git://git@REDACTED.git;protocol=ssh \
>            npm://registry.npmjs.org;name=ws;version=6.2.1;subdir=node_modules/ws \
>            "
> 
> # Modify these as desired
> PV = "1.0.0+git${SRCPV}"
> SRCREV = "975a8bd9bae00a16d82c74b0a8b5d502c5afc637"
> 
> NPM_SHRINKWRAP := "${THISDIR}/${PN}/npm-shrinkwrap.json"
> NPM_LOCKDOWN := "${THISDIR}/${PN}/lockdown.json"
> 
> inherit npm
> 
> # Must be set after inherit npm since that itself sets S
> S = "${WORKDIR}/git"
> LICENSE_${PN}-ws-async-limiter = "MIT"
> LICENSE_${PN}-ws = "MIT"
> LICENSE_${PN} = "ISC"

Running devtool build factory-tunnel results in an error like this:

>DEBUG: Executing python function extend_recipe_sysroot
>NOTE: Direct dependencies are ['/home/kristupas/projects/rpi-factory-worker/poky/meta/recipes-core/musl/musl_git.bb:do_populate_sysroot', 'virtual:native:/home/kristupas/projects/rpi-factory-worker/meta-openembedded/meta-oe/recipes-devtools/nodejs/nodejs_8.12.0.bb:do_populate_sysroot', '/home/kristupas/projects/rpi-factory-worker/poky/meta/recipes-devtools/gcc/gcc-cross_8.2.bb:do_populate_sysroot', 'virtual:native:/home/kristupas/projects/rpi-factory-worker/poky/meta/recipes-devtools/pseudo/pseudo_git.bb:do_populate_sysroot', '/home/kristupas/projects/rpi-factory-worker/poky/meta/recipes-devtools/gcc/gcc-runtime_8.2.bb:do_populate_sysroot']
>NOTE: Installed into sysroot: ['pseudo-native']
>NOTE: Skipping as already exists in sysroot: ['musl', 'nodejs-native', 'gcc-cross-aarch64', 'gcc-runtime', 'linux-libc-headers', 'libssp-nonshared', 'bsd-headers', 'icu-native', 'pkgconfig-native', 'zlib-native', 'quilt-native', 'openssl-native', 'xz-native', 'gmp-native', 'libmpc-native', 'mpfr-native', 'binutils-cross-aarch64', 'gnu-config-native', 'automake-native', 'texinfo-dummy-native', 'libtool-native', 'flex-native', 'autoconf-native', 'libgcc', 'gettext-minimal-native', 'm4-native']
>DEBUG: Python function extend_recipe_sysroot finished
>DEBUG: Executing shell function do_install
>npm notice 
>npm notice 
Comment 1 Kristupas Savickas 2019-06-20 10:41:31 UTC
Created attachment 4529 [details]
do_install

My comment got chopped in half.

I figured something was wrong with the PV field of the recipe. It was looking for factory-tunnel-1.0.0+git999.tgz even though only factory-tunnel-1.0.0.tgz existed. I changed PV to "1.0.0" and everything built as intended using devtool build.
Comment 2 Kristupas Savickas 2019-06-20 10:45:35 UTC
Created attachment 4530 [details]
do_populate_lic

Then there was an issue with the licenses. When I ran bitbake factory-tunnel I got an error on do_populate_lic task (even though devtool build succeeded). Bitbake was looking trying to find the license for async-limiter (which is a dependancy of ws) inside "node_modules/ws/node_modoles/async-limiter", while it was actually in "node_modules/async-limiter" (which is the correct place of it as far as I'm aware). I was able to change the LIC_FILES_CHKSUM to overcome this issue.
Comment 3 Kristupas Savickas 2019-06-20 10:46:39 UTC
Created attachment 4531 [details]
do_compile

Then I ran into this: running bitbake factory-tunnel, or devtool build-image caught an error during do_compile. I was not able to overcome it.
Comment 4 Kristupas Savickas 2019-06-20 10:49:49 UTC
Created attachment 4532 [details]
2019-06-20T10_48_45_724Z-debug.log
Comment 5 Randy MacLeod 2019-06-20 15:03:24 UTC
Since npm isn't part of oe-core, you'll have to use the meta-oe email list
to get support. We'll leave this defect open in case there is an issue with recipetool that we need to fix.

Do you have a public git repo that can be used to reproduce the error.
Comment 6 Kristupas Savickas 2019-06-21 06:59:07 UTC
(In reply to comment #5)

> Do you have a public git repo that can be used to reproduce the error.

Sadly, not at the moment. I'll attach a package.json file, it should be enough to reproduce this.
Comment 7 Kristupas Savickas 2019-06-21 07:01:55 UTC
Created attachment 4534 [details]
package.json
Comment 8 Randy MacLeod 2019-06-27 14:56:19 UTC
See if this is something that Razi can work on.
Comment 9 Jean-Marie Lemetayer 2020-05-17 02:47:15 UTC
This must be fixed in the master.

I can't really test the issue but since it's a pretty simple package.json and all the issues seen in the logs have been fixed, I would say it's fine.