Bug 14236 - npmsw does not support github URLs in the npm-shrinkwrap.json file
Summary: npmsw does not support github URLs in the npm-shrinkwrap.json file
Status: RESOLVED FIXED
Alias: None
Product: BitBake
Classification: Build System, Metadata & Runtime
Component: bitbake (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium+ enhancement
Target Milestone: 4.3 M4
Assignee: Mark
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2021-02-17 13:50 UTC by Paulo Castro
Modified: 2023-02-28 17:53 UTC (History)
4 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Paulo Castro 2021-02-17 13:50:16 UTC
It seems that npmsw does not support github URLs in the following format:

github:balena-io-modules/unbzip2-stream#942fc218013c14adab01cf693b0500cf6ac83193

or even:

balena-io-modules/unbzip2-stream#942fc218013c14adab01cf693b0500cf6ac83193

which are supported by the "standard" npm tool and documented at:

https://docs.npmjs.com/cli/v6/commands/npm-install 

It looks like the relevant poky / bitbake lines of code are around:

https://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/bitbake/lib/bb/fetch2/npmsw.py?id=507a47a4e5077d5f8f76d9629be6b871dfd8eb90#n125

Please add support for those github URL formats. Thanks!

Additional info:

The balena CLI is an open source Node.js / npm project:
https://github.com/balena-io/balena-cli
It includes a fairly large npm-shrinkwrap.json file:
https://github.com/balena-io/balena-cli/blob/master/npm-shrinkwrap.json

One of our users attempted building the balena CLI using Yocto / Bitbake and reported getting errors from Bitbake's npmsw regarding github URLs.

A workaround seems to be to modify the URLs to use the following (older?) format:

git+https://github.com/resin-io-modules/unbzip2-stream.git#942fc218013c14adab01cf693b0500cf6ac83193

The error / bug was originally reported at:
https://forums.balena.io/t/balena-cli-compilation-issues-in-yocto/252703

I have also created a counterpart issue in the balena CLI repo:
https://github.com/balena-io/balena-cli/issues/2192
Comment 1 Paulo Castro 2021-02-17 14:06:30 UTC
It was also reported that npmsw does not support "the from: line to specify branches with commits not merged to master (branch must be specified in the version: line before the commit hash)".

I think it regards the "from" property of shrinkwrap blocks such as:

"multicast-dns": {
  "version": "git+https://github.com/resin-io-modules/multicast-dns.git#a15c63464eb43e8925b187ed5cb9de6892e8aacc",
  "from": "git+https://github.com/resin-io-modules/multicast-dns.git#listen-on-all-interfaces",
  ...
}
Comment 2 Randy MacLeod 2021-02-18 15:42:37 UTC
A valid bug but we don't have anyone to work on it.
Comment 3 Randy MacLeod 2021-12-21 22:24:28 UTC
Paulo, would you like to work on this bug? Bumping to M2.
Comment 4 Mark 2023-02-17 21:44:10 UTC
This has partially been addressed by
---
commit 1d8af6aed0a929f493d2c3e31b8d3ee3a70beb43
Author: Stefan Herbrechtsmeier <stefan.herbrechtsmeier@weidmueller.com>
Date:   Thu Oct 28 14:41:43 2021 +0200

    fetch2: npmsw: Add support for github prefix in npm shrinkwrap version
---

I have added the other shortforms for bitbucket, gist, and gitlab along with some tests. I will send the review out next week.
Comment 5 Mark 2023-02-28 17:53:07 UTC
This was partially fixed by commit 1d8af6aed0a9 (Thu Oct 28 14:41:43 2021 +0200) [fetch2: npmsw: Add support for github prefix in npm shrinkwrap version] which added support for the github prefix.

To complete the fix, commit 0ac6f6cb5d807919ed13a8b7bb3fb551b79c5a71 (Fri Feb 24 15:43:44 2023 -0500) [fetch/npmsw: add more short forms for git operations] was merged which adds support for the gist, gitlab and bitbucket prefixes as well as the gitlab short form which forgoes any prefix at all.