Bug 13235

Summary:	Document parallel fetch task limitation configuration option
Product:	[Build System, Metadata & Runtime] BitBake	Reporter:	Chris Isbell <chris.isbell>
Component:	bitbake	Assignee:	Michael Opdenacker <michael.opdenacker>
Status:	RESOLVED FIXED	QA Contact:
Severity:	minor
Priority:	Medium	CC:	michael.opdenacker, mostthingsweb, poky.bs.watcher, poky.watcher, randy.macleod, richard.purdie, srifenbark
Version:	unspecified
Target Milestone:	4.0
Hardware:	x86
OS:	Multiple
Whiteboard:
OS type for building Yocto:	---	Type of Regression:	---
Verified:		Documentation change:	Yes (doc changes required)

Description Chris Isbell 2019-03-25 11:42:59 UTC

Problem:

Bitbake reports the following error when accessing git repositories over ssh:

ssh_exchange_identification: Connection closed by remote host
fatal: Could not read from remote repository.

The repository that fails changes between builds, and some builds succeed. There is no obvious pattern.

The command "bitbake package-index" was especially prone to failure.

Root Cause:

The builds were being performed on a 24 core processor and Bitbake was starting multiple parallel ssh sessions to access repositories held on an Ubuntu 16.04LTS server on the same LAN.

Builds worked on a 4 core processor.

By default, the ssh server is configured to reject too many parallel sessions from the same host, with a default value of ten sessions.

Partial Solution:

Update the configuration on the git server to allow more parallel sessions.

Edit /etc/ssh/sshd_conf and add lines similar to the following:

MaxSessions 30
MaxStartups 30:60:90

(Only one of these lines may be necessary.)

Suggested Solution:

Bitbake should limit the total number of parallel connections to repositories to a reasonable default value, with the option for user to override this.

Comment 1 Richard Purdie 2019-03-28 14:45:14 UTC

You can limit the number of fetch tasks using:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a0cb1021c04c37b972bbf3d3626c7eef8a9876d1

We don't normally see this issue as we don't have multiple ssh connections to git repos in our default builds. Where you have a large number of projects on a local server like in a software stack development environment, you would see this though and I guess this is what you're seeing.

We also don't want to limit to say 4 fetch processes by default as this would impact performance on multicore builds where the majority of the source is present.

We should therefore perhaps add a commented out version of this to local.conf.sample and put more information in the manual about this. Adding Scott to see if we can do that.

Comment 2 Scott Rifenbark 2019-04-01 20:35:17 UTC

Hi, 

I have a candidate section in the BitBake User Manual where we might consider noting this information - see https://www.yoctoproject.org/docs/2.7/bitbake-user-manual/bitbake-user-manual.html#the-download-fetch.  I think we could update that section to provide information on this issue.

Part of that previous section is the section specific to the the Git Fetcher (https://www.yoctoproject.org/docs/2.7/bitbake-user-manual/bitbake-user-manual.html#git-fetcher).  Would it be better to call out the behavior here instead of the higher level section?

Additionally, this area in the overview & concepts manual talks about getting source files (https://yoctoproject.org/docs/2.7/overview-manual/overview-manual.html#sources-dev-environment).  Do you think it is useful to mention this behavior here?  Or, at least note it and point to the BB manual?

Another question for me.. Is this problem limited to Git repos only? Or, are other SCMs affected? And, it appears to be only SSH? 

Thanks,
Scott

Comment 3 Chris Isbell 2019-04-02 08:50:35 UTC

Hi

Having a commented out option in local.conf.sample would be helpful for naive users (like me).

I am using URLs of the form: "git://git@server/some_repo;protocol=ssh"

(The server is using Gitolite, which is configured to use ssh keys for access control.)

Thanks!

Comment 4 Scott Rifenbark 2019-04-16 23:58:37 UTC

Hi, 

Any comment on my possible documentation solutions mentioned in Comment 2?  Is the commented out option in local.conf.sample sufficient?

Thanks,
Scott

Comment 5 Scott Rifenbark 2019-10-17 17:02:45 UTC

Hi, 

Any feedback on my Comment 2?  I can implement that.

Thanks,
Scott

Comment 6 Chris Laplante 2020-01-24 20:26:59 UTC

FYI another cause of this is a firewall/IPS that detects lots of ssh sessions as an SSH brute force. One workaround is to use SSH multiplexing, e.g. https://www.cyberciti.biz/faq/linux-unix-reuse-openssh-connection/.

Comment 7 Richard Purdie 2020-02-19 21:45:00 UTC

The configuration applies generically to fetch tasks in total, not any specific protocol. I've queued a patch addin a section to local.conf.samople.extended. I'm going to reassign to Mark to take the manual pieces further, it probably belongs in a higher level documentation section rather than protocol specific.

Comment 8 Mark Morton 2020-03-20 16:21:34 UTC

After reading the comments, I'm not really sure what to document here. It seems like we want to provide a workaround if someone experiences this issue, but is there anything else to add?

Any feedback on Scott's suggestions, or are we looking for something new altogether?

Comment 9 Michael Opdenacker 2022-03-09 16:05:07 UTC

Sent a documentation patch to address this issue:
https://lists.yoctoproject.org/g/docs/message/2558

Comment 10 Michael Opdenacker 2022-03-16 15:45:52 UTC

Fixed by this addition to the Yocto Project manual:
https://git.yoctoproject.org/yocto-docs/commit/?h=master-next&id=cf76420c50f13762fb800a19931a8af5ae3567cf