Bug 13903

Summary: npmsw fetcher fails if multiple recipes have common npm packages
Product: [Build System, Metadata & Runtime] BitBake Reporter: Jean-Marie Lemetayer <jeanmarie.lemetayer>
Component: bitbakeAssignee: Caner Altinbasak <caner.altinbasak>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium+ CC: caner.altinbasak, mostthingsweb, poky.bs.watcher, poky.watcher, randy.macleod, scott.r.ware, tim.orling
Version: 5.99   
Target Milestone: 3.4 M4   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)
Attachments:
Description Flags
Failing build log
none
successful recipe fetch
none
failing recipe build none

Description Jean-Marie Lemetayer 2020-05-17 12:31:12 UTC 
https://lists.openembedded.org/g/bitbake-devel/topic/73374817#11279

Preconditions/Environment
-------------------------
Create multiple recipe using npm shrinkwrap files with common packages.

Triggering Action/Cause
-----------------------
Start the build.

Expectation
-----------
The fetch task is ok.

Actual Result
-------------
The fetch task fails.

Reproducibility
---------------
Always
Comment 1 Scott Ware 2020-05-19 13:18:27 UTC 
Created attachment 4675 [details]
Failing build log

This is an example of build output during a failed build. When multiple "node" recipes attempt to fetch the same dependency simultaneously failures occur.
Comment 2 Scott Ware 2020-05-19 13:22:06 UTC 
Created attachment 4676 [details]
successful recipe fetch

Example "node" recipe built in isolation fetching successfully.
Comment 3 Scott Ware 2020-05-19 13:28:54 UTC 
Created attachment 4677 [details]
failing recipe build

Example "node" recipe fetch failure when built in parallel with other similar recipes with duplicate dependencies.
Comment 4 Randy MacLeod 2021-04-29 19:12:36 UTC 
I suspect this is fix. If so please paste a link to the fix commit and close he defect.
Comment 5 Randy MacLeod 2021-05-20 15:50:52 UTC 
Jean-Marie, Scott, 
What's the status of this bug?
If there is a fix merged, please mark it as resolved and provide a link to the commit.
Thanks,
../Randy
Comment 6 Scott Ware 2021-06-01 16:03:47 UTC 
As far as I know this isn't fixed, I can't see a relevant commit for the npm fetcher class. My workflow only contains a single node recipe these days so I am no longer hitting the issue.
Comment 7 Chris Laplante 2021-06-02 16:55:58 UTC 
I have a patch on a private branch that fixes this, but it's more of a workaround than an actual fix to the npm fetcher. The patch just adds this line to npm.bbclass:


do_fetch[lockfiles] += "${DL_DIR}/npm2_fetch.lock"


I can submit the patch if people want.
Comment 8 Caner Altinbasak 2021-10-19 10:58:28 UTC 
I've a two stage fix for this. First one fixes the bad checksum problem, because of trying to download the files simultaneously. Second one is about making sure that downloaded check and verification are atomic. When first problem is fixed, one of the recipes might complete the download when other has decided file doesn't exist in download location. It ends up having downloaded false, but verified true, which confuses all the logic. I've sent two patches to e-mail list last week. Here they are:
https://lists.openembedded.org/g/bitbake-devel/message/12764
https://lists.openembedded.org/g/bitbake-devel/message/12765