Problem: Bitbake reports the following error when accessing git repositories over ssh: ssh_exchange_identification: Connection closed by remote host fatal: Could not read from remote repository. The repository that fails changes between builds, and some builds succeed. There is no obvious pattern. The command "bitbake package-index" was especially prone to failure. Root Cause: The builds were being performed on a 24 core processor and Bitbake was starting multiple parallel ssh sessions to access repositories held on an Ubuntu 16.04LTS server on the same LAN. Builds worked on a 4 core processor. By default, the ssh server is configured to reject too many parallel sessions from the same host, with a default value of ten sessions. Partial Solution: Update the configuration on the git server to allow more parallel sessions. Edit /etc/ssh/sshd_conf and add lines similar to the following: MaxSessions 30 MaxStartups 30:60:90 (Only one of these lines may be necessary.) Suggested Solution: Bitbake should limit the total number of parallel connections to repositories to a reasonable default value, with the option for user to override this.
You can limit the number of fetch tasks using: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a0cb1021c04c37b972bbf3d3626c7eef8a9876d1 We don't normally see this issue as we don't have multiple ssh connections to git repos in our default builds. Where you have a large number of projects on a local server like in a software stack development environment, you would see this though and I guess this is what you're seeing. We also don't want to limit to say 4 fetch processes by default as this would impact performance on multicore builds where the majority of the source is present. We should therefore perhaps add a commented out version of this to local.conf.sample and put more information in the manual about this. Adding Scott to see if we can do that.
Hi, I have a candidate section in the BitBake User Manual where we might consider noting this information - see https://www.yoctoproject.org/docs/2.7/bitbake-user-manual/bitbake-user-manual.html#the-download-fetch. I think we could update that section to provide information on this issue. Part of that previous section is the section specific to the the Git Fetcher (https://www.yoctoproject.org/docs/2.7/bitbake-user-manual/bitbake-user-manual.html#git-fetcher). Would it be better to call out the behavior here instead of the higher level section? Additionally, this area in the overview & concepts manual talks about getting source files (https://yoctoproject.org/docs/2.7/overview-manual/overview-manual.html#sources-dev-environment). Do you think it is useful to mention this behavior here? Or, at least note it and point to the BB manual? Another question for me.. Is this problem limited to Git repos only? Or, are other SCMs affected? And, it appears to be only SSH? Thanks, Scott
Hi Having a commented out option in local.conf.sample would be helpful for naive users (like me). I am using URLs of the form: "git://git@server/some_repo;protocol=ssh" (The server is using Gitolite, which is configured to use ssh keys for access control.) Thanks!
Hi, Any comment on my possible documentation solutions mentioned in Comment 2? Is the commented out option in local.conf.sample sufficient? Thanks, Scott
Hi, Any feedback on my Comment 2? I can implement that. Thanks, Scott
FYI another cause of this is a firewall/IPS that detects lots of ssh sessions as an SSH brute force. One workaround is to use SSH multiplexing, e.g. https://www.cyberciti.biz/faq/linux-unix-reuse-openssh-connection/.
The configuration applies generically to fetch tasks in total, not any specific protocol. I've queued a patch addin a section to local.conf.samople.extended. I'm going to reassign to Mark to take the manual pieces further, it probably belongs in a higher level documentation section rather than protocol specific.
After reading the comments, I'm not really sure what to document here. It seems like we want to provide a workaround if someone experiences this issue, but is there anything else to add? Any feedback on Scott's suggestions, or are we looking for something new altogether?
Sent a documentation patch to address this issue: https://lists.yoctoproject.org/g/docs/message/2558
Fixed by this addition to the Yocto Project manual: https://git.yoctoproject.org/yocto-docs/commit/?h=master-next&id=cf76420c50f13762fb800a19931a8af5ae3567cf