13235 2019-03-25 11:42:59 +0000 Document parallel fetch task limitation configuration option 2022-03-16 15:45:52 +0000 1 1 7 Build System, Metadata & Runtime BitBake bitbake unspecified x86 Multiple RESOLVED FIXED Medium minor 4.0 1 chris.isbell michael.opdenacker michael.opdenacker mostthingsweb poky.bs.watcher poky.watcher randy.macleod richard.purdie srifenbark --- --- Yes (doc changes required) oldest_to_newest 83310 0 chris.isbell 2019-03-25 11:42:59 +0000 Problem: Bitbake reports the following error when accessing git repositories over ssh: ssh_exchange_identification: Connection closed by remote host fatal: Could not read from remote repository. The repository that fails changes between builds, and some builds succeed. There is no obvious pattern. The command "bitbake package-index" was especially prone to failure. Root Cause: The builds were being performed on a 24 core processor and Bitbake was starting multiple parallel ssh sessions to access repositories held on an Ubuntu 16.04LTS server on the same LAN. Builds worked on a 4 core processor. By default, the ssh server is configured to reject too many parallel sessions from the same host, with a default value of ten sessions. Partial Solution: Update the configuration on the git server to allow more parallel sessions. Edit /etc/ssh/sshd_conf and add lines similar to the following: MaxSessions 30 MaxStartups 30:60:90 (Only one of these lines may be necessary.) Suggested Solution: Bitbake should limit the total number of parallel connections to repositories to a reasonable default value, with the option for user to override this. 83354 1 richard.purdie 2019-03-28 14:45:14 +0000 You can limit the number of fetch tasks using: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=a0cb1021c04c37b972bbf3d3626c7eef8a9876d1 We don't normally see this issue as we don't have multiple ssh connections to git repos in our default builds. Where you have a large number of projects on a local server like in a software stack development environment, you would see this though and I guess this is what you're seeing. We also don't want to limit to say 4 fetch processes by default as this would impact performance on multicore builds where the majority of the source is present. We should therefore perhaps add a commented out version of this to local.conf.sample and put more information in the manual about this. Adding Scott to see if we can do that. 83393 2 srifenbark 2019-04-01 20:35:17 +0000 Hi, I have a candidate section in the BitBake User Manual where we might consider noting this information - see https://www.yoctoproject.org/docs/2.7/bitbake-user-manual/bitbake-user-manual.html#the-download-fetch. I think we could update that section to provide information on this issue. Part of that previous section is the section specific to the the Git Fetcher (https://www.yoctoproject.org/docs/2.7/bitbake-user-manual/bitbake-user-manual.html#git-fetcher). Would it be better to call out the behavior here instead of the higher level section? Additionally, this area in the overview & concepts manual talks about getting source files (https://yoctoproject.org/docs/2.7/overview-manual/overview-manual.html#sources-dev-environment). Do you think it is useful to mention this behavior here? Or, at least note it and point to the BB manual? Another question for me.. Is this problem limited to Git repos only? Or, are other SCMs affected? And, it appears to be only SSH? Thanks, Scott 83406 3 chris.isbell 2019-04-02 08:50:35 +0000 Hi Having a commented out option in local.conf.sample would be helpful for naive users (like me). I am using URLs of the form: "git://git@server/some_repo;protocol=ssh" (The server is using Gitolite, which is configured to use ssh keys for access control.) Thanks! 83562 4 srifenbark 2019-04-16 23:58:37 +0000 Hi, Any comment on my possible documentation solutions mentioned in Comment 2? Is the commented out option in local.conf.sample sufficient? Thanks, Scott 85382 5 srifenbark 2019-10-17 17:02:45 +0000 Hi, Any feedback on my Comment 2? I can implement that. Thanks, Scott 86117 6 mostthingsweb 2020-01-24 20:26:59 +0000 FYI another cause of this is a firewall/IPS that detects lots of ssh sessions as an SSH brute force. One workaround is to use SSH multiplexing, e.g. https://www.cyberciti.biz/faq/linux-unix-reuse-openssh-connection/. 86494 7 richard.purdie 2020-02-19 21:45:00 +0000 The configuration applies generically to fetch tasks in total, not any specific protocol. I've queued a patch addin a section to local.conf.samople.extended. I'm going to reassign to Mark to take the manual pieces further, it probably belongs in a higher level documentation section rather than protocol specific. 86764 8 mark.morton 2020-03-20 16:21:34 +0000 After reading the comments, I'm not really sure what to document here. It seems like we want to provide a workaround if someone experiences this issue, but is there anything else to add? Any feedback on Scott's suggestions, or are we looking for something new altogether? 92748 9 michael.opdenacker 2022-03-09 16:05:07 +0000 Sent a documentation patch to address this issue: https://lists.yoctoproject.org/g/docs/message/2558 92782 10 michael.opdenacker 2022-03-16 15:45:52 +0000 Fixed by this addition to the Yocto Project manual: https://git.yoctoproject.org/yocto-docs/commit/?h=master-next&id=cf76420c50f13762fb800a19931a8af5ae3567cf