To reproduce, build an image, then move away sstate-cache and start "python3 -m http.server" in the moved sstate-cache directory. In the original build directory, add "SSTATE_MIRRORS = "file://.* http://127.0.0.1:8000/PATH" to conf/local.conf. Remove tmp*. Run "bitbake --dry-run -D -D <image name>". Checking the sstate mirror will fail with "urlopen failed: <urlopen error [Errno 9] Bad file descriptor>". Besides the debug messages, there is no user-visible indication that something is wrong with the HTTP server and/or network setup. bitbake simply continues with building from scratch. When there are such "unexpected" networking errors, I'd prefer to have the build aborted or at least get a warning about it. As it is now, we spent a day trying to understand why all builds were done without using sstate. Others had the same problem: https://www.yoctoproject.org/irc/%23yocto.2017-02-27.log.html, search for "urlopen failed" Or perhaps the check code needs to be enhanced to handle this situation better. We actually started to hit this issue when trying to build refkit in the crops Ubuntu 16.04 container, with sstate served by Apache. It works in other containers. See https://github.com/intel/intel-iot-refkit/pull/240 It's still uncertain why it fails under Ubuntu 16.04. My hope is that by enhancing the code so that it works with a minimal HTTP server like http.server, it'll also work under Ubuntu 16.04, or at least additional diagnostics will reveal where the problem is.
After looking at the code I wonder about the number of parallel connections to the same server. At first glance it looks like one connection is established for each HEAD request, with the connections created in parallel using threads. But perhaps I misunderstood the design. Python3 should support keep-alive. Perhaps that in combination with a connection pool that has one open connection per server is a better solution?
I've debugged this some more and have a patch that improves the situation. First, the current code indeed already uses Keep-Alive. Where it fails in combination with SimpleHTTP is that this particular server doesn't respond with "close" in the response header, which wget.py already checks for and closes the cached connection. Instead, the server just closes the TCP connection. This then leads to the client using a socket which isn't connected anymore, thus triggering the "Bad file descriptor" error. I can imagine (but haven't checked) that "close" is just a hint and that the server is allowed to close the connection also without sending that. For example, it might be done with the request and intend to keep the connection open, but then runs out of resources and has to close anyway. So the client code needs to be prepared for this. My patch checks for the error, closes the cached connection and tries again. Works fine with SimpleHTTP *and* under Ubuntu (https://intel-iot-refkit.01.org/job/intel-iot-refkit_pull-requests/34/console), so the initial hope that the same fix would help for both was justified. However, I'd still like to understand why the Ubuntu docker container behaves differently than the OpenSUSE one.
I've posted the "fetch2/wget.py: improve error handling during sstate check" which makes SimpleHTTP as server usable. I had considered whether network issues can be reported as a warning, but the code and the problem are complex, so I don't think there is an easy, reliable way of warning the user when their setup is broken. In our case on Ubuntu, it turned out that the version of Python3 in Ubuntu did not handle no_proxy=.<local domain> correctly (https://bugs.python.org/issue29142), causing HTTP sstate checks to go through a proxy when they should have been direct. This could have been found with tcpdump, so it's probably not necessary to enhance bitbake. I therefore propose to close this issue as resolved after addressing the situation with SimpleHTTP.
SimpleHTTP now works better thanks to "fetch2/wget.py: improve error handling during sstate check". The error refkit turned out to be due to Python3 being sensitive to leading dots in no_proxy.