Created attachment 3579 [details] failure log, notice the ctr-c to exit the hung process. tinfoil.prepare() shows errors when setting the PRSERV_HOST = "localhost:0" in the configuration. This is not failing on previous commits (209a19c and before) to the tinfoil2 changes. Procedure to reproduce the failure: 1. on the poky master branch (dbb247c), init a build environment: poky $ . ./oe-init-build-env build-tf2 2. on the fresh build environment, add the PRSERV_HOST variable to your configuration: build-tf2 $ echo 'PRSERV_HOST = "localhost:0"' >> conf/local.conf 3. in order to execute a test from oe-selftest, add the meta-selftest layer: build-tf2 $ bitbake-layers add-layer ../meta-selftest 4. build a core-image-minimal as it is a pre-requisite for the coming test. It is ok to use sstates if you can otherwise it would just take a bit more time: build-tf2 $ bitbake core-image-minimal 5. The easiest way to get tinfoil.prepare() to run is to execute a test that uses runqemu() which in turn uses tinfoil, such is the case for test executed below: build-tf2 $ oe-selftest -r imagefeatures.ImageFeatures.test_all_users_can_connect_via_ssh_without_password this test should take about 5 minutes to run provided that the previous step already created the image it will boot in runqemu. Expected result: the test executes and finishes with no errors. Actual result: on some systems, a python stack trace is printed indicating that the bitbake server could not be used. See attached. Reproducibility: 10/10 on machines that show the failure. Seen on Fedora 25, Fedora 24, Opensuse 42.1, ubuntu 14.04. Machines from the GDC AB fail while machines from the main AB do not. I reproduced this error manually on F25 and opensuse42.1 outside the autobuilder. Workaround: manage the prserv manually outside bitbake and indicate the given host/port in the PRSERV_HOST variable.
I was able to reproduce the bug with below data: Host Machine : Debian GNU/Linux 8.6 (jessie) Commit : dbb247cac5fbf7b037e4955f9793828451723924
The missing piece of magic needed to make this reproduce is pip3 install xmlrunner; pip3 install unittest-xml-reporting. This means that XML unit test reporting gets enabled and somehow this changes sys.stdout from io.TextIOWrapper to io.StringIO. When the PR server is autoloaded, it tries to deamonise itself and direct stdout for logging. StringIO has no .fileno() method so it crashes. This leaves a zombie process which means oe-selftest doesn't exit. The connection to the crashed PR server eventually times out (ping fails) and the cooker starts but by this time the UI has given up and exitted.
in serv.py: - os.dup2(so.fileno(),sys.stdout.fileno()) + sys.stdout = so - os.dup2(se.fileno(),sys.stderr.fileno()) + sys.stdout = se would seem to avoid the problem but it is suggestive of wider issues in behaviour between the xml and text runners.
see bug 10162, it shows one of the manifestations of the XMLTestRunner vs TextTestRunner.
The XMLTestRunner was enabled before the morty release and the faulty behavior began after. So I see two things here: 1. There is a change that, combined with the XMLTestRunner makes the PRServer crash. 2. There is still integration to be made to be able to use the XMLTestRunner in the same way we use unittest.TextTestRunner and perhaps it provokes bug 10873 as well. Next is to test the AB without the XMLRunner installed to confirm that both bugs 10873 & 10826 are caused by this same thing. In the affirmative case it means we still need to search the commit that began showing this behavior an analyze it and we should remove the XMLTestRunner from test machines until it is properly integrated.
(In reply to comment #2) > The missing piece of magic needed to make this reproduce is pip3 install > xmlrunner; pip3 install unittest-xml-reporting. I believe you just need to do the second one, unittest-xml-reporting. > > This means that XML unit test reporting gets enabled and somehow this > changes sys.stdout from io.TextIOWrapper to io.StringIO. When the PR server > is autoloaded, it tries to deamonise itself and direct stdout for logging. > StringIO has no .fileno() method so it crashes. This leaves a zombie process > which means oe-selftest doesn't exit. This behavior could be enabled by the try: ... Except: pass that solved bug 10162. > > The connection to the crashed PR server eventually times out (ping fails) > and the cooker starts but by this time the UI has given up and exitted.
(In reply to comment #5) > The XMLTestRunner was enabled before the morty release and the faulty > behavior began after. So I see two things here: > > 1. There is a change that, combined with the XMLTestRunner makes the > PRServer crash. > > 2. There is still integration to be made to be able to use the XMLTestRunner > in the same way we use unittest.TextTestRunner and perhaps it provokes bug > 10873 as well. > > Next is to test the AB without the XMLRunner installed to confirm that both > bugs 10873 & 10826 are caused by this same thing. In the affirmative case it > means we still need to search the commit that began showing this behavior an > analyze it and we should remove the XMLTestRunner from test machines until > it is properly integrated. Prior to tinfoil2, the PR server wasn't started as the cooker wasn't run in the same way. After tinfoil2, a more complete cooker is started/shutdown which includes the PR server. Therefore the tinfoil2 patches trigger the issue, what we also needed to understand is what combination of things trigger the problem. We now appear to know that you need tinfoil2 + autoloaded PR server + oe-selftest with xmlrunner. That means we probably can lower the priority of the bug as the main impact is going to be onto QA, normal users are unlikely to use this combination of components.
http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=9f6a1043f68580ed9604e750fd0f993f933bb66e