Bug 13903 - npmsw fetcher fails if multiple recipes have common npm packages
Summary: npmsw fetcher fails if multiple recipes have common npm packages
Status: RESOLVED FIXED
Alias: None
Product: BitBake
Classification: Build System, Metadata & Runtime
Component: bitbake (show other bugs)
Version: 5.99
Hardware: x86 Multiple
: Medium+ normal
Target Milestone: 3.4 M4
Assignee: Caner Altinbasak
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2020-05-17 12:31 UTC by Jean-Marie Lemetayer
Modified: 2021-11-05 14:06 UTC (History)
7 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)

Attachments
Failing build log (18.90 KB, text/x-log)
2020-05-19 13:18 UTC, Scott Ware 		no flags Details
successful recipe fetch (873.93 KB, text/x-log)
2020-05-19 13:22 UTC, Scott Ware 		no flags Details
failing recipe build (16.32 MB, text/x-log)
2020-05-19 13:28 UTC, Scott Ware 		no flags Details
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Jean-Marie Lemetayer 2020-05-17 12:31:12 UTC 
https://lists.openembedded.org/g/bitbake-devel/topic/73374817#11279

Preconditions/Environment
-------------------------
Create multiple recipe using npm shrinkwrap files with common packages.

Triggering Action/Cause
-----------------------
Start the build.

Expectation
-----------
The fetch task is ok.

Actual Result
-------------
The fetch task fails.

Reproducibility
---------------
Always
Comment 1 Scott Ware 2020-05-19 13:18:27 UTC 
Created attachment 4675 [details]
Failing build log

This is an example of build output during a failed build. When multiple "node" recipes attempt to fetch the same dependency simultaneously failures occur.
Comment 2 Scott Ware 2020-05-19 13:22:06 UTC 
Created attachment 4676 [details]
successful recipe fetch

Example "node" recipe built in isolation fetching successfully.
Comment 3 Scott Ware 2020-05-19 13:28:54 UTC 
Created attachment 4677 [details]
failing recipe build

Example "node" recipe fetch failure when built in parallel with other similar recipes with duplicate dependencies.
Comment 4 Randy MacLeod 2021-04-29 19:12:36 UTC 
I suspect this is fix. If so please paste a link to the fix commit and close he defect.
Comment 5 Randy MacLeod 2021-05-20 15:50:52 UTC 
Jean-Marie, Scott, 
What's the status of this bug?
If there is a fix merged, please mark it as resolved and provide a link to the commit.
Thanks,
../Randy
Comment 6 Scott Ware 2021-06-01 16:03:47 UTC 
As far as I know this isn't fixed, I can't see a relevant commit for the npm fetcher class. My workflow only contains a single node recipe these days so I am no longer hitting the issue.
Comment 7 Chris Laplante 2021-06-02 16:55:58 UTC 
I have a patch on a private branch that fixes this, but it's more of a workaround than an actual fix to the npm fetcher. The patch just adds this line to npm.bbclass:


do_fetch[lockfiles] += "${DL_DIR}/npm2_fetch.lock"


I can submit the patch if people want.
Comment 8 Caner Altinbasak 2021-10-19 10:58:28 UTC 
I've a two stage fix for this. First one fixes the bad checksum problem, because of trying to download the files simultaneously. Second one is about making sure that downloaded check and verification are atomic. When first problem is fixed, one of the recipes might complete the download when other has decided file doesn't exist in download location. It ends up having downloaded false, but verified true, which confuses all the logic. I've sent two patches to e-mail list last week. Here they are:
https://lists.openembedded.org/g/bitbake-devel/message/12764
https://lists.openembedded.org/g/bitbake-devel/message/12765