Created attachment 4795 [details] archiver-npm-bug-reproduction.patch Steps to reproduce: 1. Apply the attached archiver-npm-bug-reproduction.patch to an oe-core master tree. 2. Enable meta-oe/meta-oe in bblayers.conf since npm.bbclass needs nodejs. 3. Run: bitbake -c ar_mirror cute-files Expected result: - Only the sources for cute-files and the Node modules it uses end up in ${WORKDIR}/archiver-sources/ Actual result: - The entire contents of my ${DL_DIR} gets copied to ${WORKDIR}/archiver-sources/ This appears to be because when do_ar_mirror calls ud.localpath it gets back an empty string. Fixing that isn't straightforward because FetchMethod expects each URI to only correspond to a single file. NpmShrinkWrap potentially downloads multiple files for each npmsw: URI. It's not clear to me what the fix for this is. I've only come up with two ideas: A. Teach FetchMethod and NpmShrinkWrap to return all the resulting filepaths from the localpaths method, and then make do_ar_mirror use that rather than just getting a single path by calling localpath. B. Teach NpmShrinkWrap to create a mirrortarball.
Leaving as unassigned. We're not sure what the best way to proceed is. Teaching the npm fetcher about mirror tarballs may help.
I had a look at following the suggestion in comment 1 to try option B, but I think that I'm suffering from a lack of understanding of how it ought to work. As far as I can tell, do_fetch downloads all the tarballs for the NPM modules mentioned in the npm-shrinkwrap.json file. Since they are already present in ${DL_DIR} in a suitable form, I don't think there's any need to create new "mirror tarballs" and combining them into a single mirror tarball would appear to provide little benefit either. (In fact, keeping them as separate tarballs is beneficial since they can be shared with other recipes in that form.) I had a go at implementing something like option A in the attached extremely-rough patch to show that it might work. It seemed to fix the situation described in comment 0. However, there are plenty of other places that would need to be updated to cope with localpath returning multiple paths (e.g. get_recipe_local_files) and the return type should really be an array rather than a single space-separated string. All comments gratefully received.
Created attachment 4800 [details] Badly teach npmsw fetcher to return multiple paths for one URI
Mike, I suspect you'll get more comments if you post this patch to the list: bitbake-devel@lists.openembedded.org
Mike, any news on this bug?
> Mike, any news on this bug? I'm afraid that I worked around it for our purposes and haven't looked at it for nearly two years. I did see some discussion about making substantial changes to the way NPM works in oe-core which might have some impact, but I haven't kept up with that. I imagine that I'll get to look at it again next time we try to upgrade to a newer release of oe-core.
Thanks Mike. Upgrade soon! ;-)
Bulk move of 64 bugs to 4.3 M3 after a quick review. If a bug is actually fixed, please add a commit link and resolve it. -- Randy for YP bug team.
Moved to M4.
Bulk move to 5.0 M1. -- Randy
Hello everyone, this bug is resolved, I made a serie of patches for npm fetcher and installer. Furthermore: there is no ar_mirror task anymore, which means we cannot do: bitbake -c ar_mirror cute-files You can check the serie of patches in bitbake and oe-core. In my opinion you can close this bug. Thanks BELOUARGA Mohamed
Bug resolved