10866 2016-12-30 23:58:17 +0000 tinfoil2 breaks the prserv functionality 2017-12-06 17:21:54 +0000 1 1 7 Build System, Metadata & Runtime BitBake bitbake unspecified x86 Multiple RESOLVED FIXED https://bugzilla.yoctoproject.org/show_bug.cgi?id=10162 High critical 2.3 M1 10826 1 benjamin.esquivel richard.purdie alejandro.hernandez bluelightning jose.perez.carranza mariano.lopez monserratx.sedeno.bustos poky.bs.watcher poky.watcher sgw --- --- No (bug/feature does not impact docs) oldest_to_newest 69420 0 3579 benjamin.esquivel 2016-12-30 23:58:17 +0000 Created attachment 3579 failure log, notice the ctr-c to exit the hung process. tinfoil.prepare() shows errors when setting the PRSERV_HOST = "localhost:0" in the configuration. This is not failing on previous commits (209a19c and before) to the tinfoil2 changes. Procedure to reproduce the failure: 1. on the poky master branch (dbb247c), init a build environment: poky $ . ./oe-init-build-env build-tf2 2. on the fresh build environment, add the PRSERV_HOST variable to your configuration: build-tf2 $ echo 'PRSERV_HOST = "localhost:0"' >> conf/local.conf 3. in order to execute a test from oe-selftest, add the meta-selftest layer: build-tf2 $ bitbake-layers add-layer ../meta-selftest 4. build a core-image-minimal as it is a pre-requisite for the coming test. It is ok to use sstates if you can otherwise it would just take a bit more time: build-tf2 $ bitbake core-image-minimal 5. The easiest way to get tinfoil.prepare() to run is to execute a test that uses runqemu() which in turn uses tinfoil, such is the case for test executed below: build-tf2 $ oe-selftest -r imagefeatures.ImageFeatures.test_all_users_can_connect_via_ssh_without_password this test should take about 5 minutes to run provided that the previous step already created the image it will boot in runqemu. Expected result: the test executes and finishes with no errors. Actual result: on some systems, a python stack trace is printed indicating that the bitbake server could not be used. See attached. Reproducibility: 10/10 on machines that show the failure. Seen on Fedora 25, Fedora 24, Opensuse 42.1, ubuntu 14.04. Machines from the GDC AB fail while machines from the main AB do not. I reproduced this error manually on F25 and opensuse42.1 outside the autobuilder. Workaround: manage the prserv manually outside bitbake and indicate the given host/port in the PRSERV_HOST variable. 69444 1 jose.perez.carranza 2017-01-03 17:15:10 +0000 I was able to reproduce the bug with below data: Host Machine : Debian GNU/Linux 8.6 (jessie) Commit : dbb247cac5fbf7b037e4955f9793828451723924 69484 2 richard.purdie 2017-01-04 17:55:34 +0000 The missing piece of magic needed to make this reproduce is pip3 install xmlrunner; pip3 install unittest-xml-reporting. This means that XML unit test reporting gets enabled and somehow this changes sys.stdout from io.TextIOWrapper to io.StringIO. When the PR server is autoloaded, it tries to deamonise itself and direct stdout for logging. StringIO has no .fileno() method so it crashes. This leaves a zombie process which means oe-selftest doesn't exit. The connection to the crashed PR server eventually times out (ping fails) and the cooker starts but by this time the UI has given up and exitted. 69485 3 richard.purdie 2017-01-04 17:56:30 +0000 in serv.py: - os.dup2(so.fileno(),sys.stdout.fileno()) + sys.stdout = so - os.dup2(se.fileno(),sys.stderr.fileno()) + sys.stdout = se would seem to avoid the problem but it is suggestive of wider issues in behaviour between the xml and text runners. 69490 4 benjamin.esquivel 2017-01-04 19:13:33 +0000 see bug 10162, it shows one of the manifestations of the XMLTestRunner vs TextTestRunner. 69500 5 benjamin.esquivel 2017-01-04 20:17:56 +0000 The XMLTestRunner was enabled before the morty release and the faulty behavior began after. So I see two things here: 1. There is a change that, combined with the XMLTestRunner makes the PRServer crash. 2. There is still integration to be made to be able to use the XMLTestRunner in the same way we use unittest.TextTestRunner and perhaps it provokes bug 10873 as well. Next is to test the AB without the XMLRunner installed to confirm that both bugs 10873 & 10826 are caused by this same thing. In the affirmative case it means we still need to search the commit that began showing this behavior an analyze it and we should remove the XMLTestRunner from test machines until it is properly integrated. 69501 6 benjamin.esquivel 2017-01-04 20:24:49 +0000 (In reply to comment #2) > The missing piece of magic needed to make this reproduce is pip3 install > xmlrunner; pip3 install unittest-xml-reporting. I believe you just need to do the second one, unittest-xml-reporting. > > This means that XML unit test reporting gets enabled and somehow this > changes sys.stdout from io.TextIOWrapper to io.StringIO. When the PR server > is autoloaded, it tries to deamonise itself and direct stdout for logging. > StringIO has no .fileno() method so it crashes. This leaves a zombie process > which means oe-selftest doesn't exit. This behavior could be enabled by the try: ... Except: pass that solved bug 10162. > > The connection to the crashed PR server eventually times out (ping fails) > and the cooker starts but by this time the UI has given up and exitted. 69506 7 richard.purdie 2017-01-04 22:43:29 +0000 (In reply to comment #5) > The XMLTestRunner was enabled before the morty release and the faulty > behavior began after. So I see two things here: > > 1. There is a change that, combined with the XMLTestRunner makes the > PRServer crash. > > 2. There is still integration to be made to be able to use the XMLTestRunner > in the same way we use unittest.TextTestRunner and perhaps it provokes bug > 10873 as well. > > Next is to test the AB without the XMLRunner installed to confirm that both > bugs 10873 & 10826 are caused by this same thing. In the affirmative case it > means we still need to search the commit that began showing this behavior an > analyze it and we should remove the XMLTestRunner from test machines until > it is properly integrated. Prior to tinfoil2, the PR server wasn't started as the cooker wasn't run in the same way. After tinfoil2, a more complete cooker is started/shutdown which includes the PR server. Therefore the tinfoil2 patches trigger the issue, what we also needed to understand is what combination of things trigger the problem. We now appear to know that you need tinfoil2 + autoloaded PR server + oe-selftest with xmlrunner. That means we probably can lower the priority of the bug as the main impact is going to be onto QA, normal users are unlikely to use this combination of components. 69515 8 richard.purdie 2017-01-05 14:25:31 +0000 http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=9f6a1043f68580ed9604e750fd0f993f933bb66e 3579 2016-12-30 23:58:17 +0000 2016-12-30 23:58:17 +0000 failure log, notice the ctr-c to exit the hung process. file_10866.txt text/plain 4508 benjamin.esquivel YmVzcXVpdmVAeWN0YjA0On4vd29ya2Rpci9wb2t5LXdvcmtpbmcvYnVpbGQtdGVzdHByc2Vydltt YXN0ZXJdJCBvZS1zZWxmdGVzdCAtciBpbWFnZWZlYXR1cmVzLkltYWdlRmVhdHVyZXMudGVzdF9h bGxfdXNlcnNfY2FuX2Nvbm5lY3RfdmlhX3NzaF93aXRob3V0X3Bhc3N3b3JkDQoyMDE2LTEyLTMw IDE1OjU1OjAyLDIzOSAtIHNlbGZ0ZXN0IC0gSU5GTyAtIFJ1bm5pbmcgYml0YmFrZSAtZSB0byBn ZXQgQkJQQVRIDQoyMDE2LTEyLTMwIDE1OjU1OjA0LDIyNiAtIHNlbGZ0ZXN0IC0gSU5GTyAtIENo ZWNraW5nIHRoYXQgZXZlcnl0aGluZyBpcyBpbiBvcmRlciBiZWZvcmUgcnVubmluZyB0aGUgdGVz dHMNCjIwMTYtMTItMzAgMTU6NTU6MDYsMjMwIC0gc2VsZnRlc3QgLSBJTkZPIC0gUnVubmluZyBi aXRiYWtlIC1wDQoyMDE2LTEyLTMwIDE1OjU1OjEzLDk3NSAtIHNlbGZ0ZXN0IC0gSU5GTyAtIExv YWRpbmcgdGVzdHMgZnJvbTogb2VxYS5zZWxmdGVzdC5pbWFnZWZlYXR1cmVzLkltYWdlRmVhdHVy ZXMudGVzdF9hbGxfdXNlcnNfY2FuX2Nvbm5lY3RfdmlhX3NzaF93aXRob3V0X3Bhc3N3b3JkDQoy MDE2LTEyLTMwIDE1OjU1OjE1LDkwMyAtIHNlbGZ0ZXN0IC0gSU5GTyAtIEFkZGluZzogImluY2x1 ZGUgc2VsZnRlc3QuaW5jIiBpbiBsb2NhbC5jb25mDQoyMDE2LTEyLTMwIDE1OjU1OjE1LDkwNCAt IHNlbGZ0ZXN0IC0gSU5GTyAtIEFkZGluZzogImluY2x1ZGUgYmJsYXllcnMuaW5jIiBpbiBiYmxh eWVycy5jb25mDQoNClJ1bm5pbmcgdGVzdHMuLi4NCi0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0NCjIwMTYtMTItMzAg MTU6NTU6MTUgLSAgIHRlc3RfYWxsX3VzZXJzX2Nhbl9jb25uZWN0X3ZpYV9zc2hfd2l0aG91dF9w YXNzd29yZCAob2VxYS5zZWxmdGVzdC5pbWFnZWZlYXR1cmVzLkltYWdlRmVhdHVyZXMpIC4uLiBF UlJPUiAoMTEzLjQ2NXMpDQoNCj09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0NCkVSUk9SIFsxMTMuNDY1c106IHRlc3Rf YWxsX3VzZXJzX2Nhbl9jb25uZWN0X3ZpYV9zc2hfd2l0aG91dF9wYXNzd29yZCAob2VxYS5zZWxm dGVzdC5pbWFnZWZlYXR1cmVzLkltYWdlRmVhdHVyZXMpDQotLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQpUcmFjZWJh Y2sgKG1vc3QgcmVjZW50IGNhbGwgbGFzdCk6DQogIEZpbGUgIi9ob21lL2Jlc3F1aXZlL3dvcmtk aXIvcG9reS13b3JraW5nL2JpdGJha2UvbGliL2JiL21haW4ucHkiLCBsaW5lIDUyMCwgaW4gc2V0 dXBfYml0YmFrZQ0KICAgIHNlcnZlcl9jb25uZWN0aW9uID0gc2VydmVyLmVzdGFibGlzaENvbm5l Y3Rpb24oZmVhdHVyZXNldCkNCiAgRmlsZSAiL2hvbWUvYmVzcXVpdmUvd29ya2Rpci9wb2t5LXdv cmtpbmcvYml0YmFrZS9saWIvYmIvc2VydmVyL3Byb2Nlc3MucHkiLCBsaW5lIDI4NywgaW4gZXN0 YWJsaXNoQ29ubmVjdGlvbg0KICAgIF8sIGVycm9yID0gc2VsZi5jb25uZWN0aW9uLmNvbm5lY3Rp b24ucnVuQ29tbWFuZChbInNldEZlYXR1cmVzIiwgZmVhdHVyZXNldF0pDQogIEZpbGUgIi9ob21l L2Jlc3F1aXZlL3dvcmtkaXIvcG9reS13b3JraW5nL2JpdGJha2UvbGliL2JiL3NlcnZlci9wcm9j ZXNzLnB5IiwgbGluZSA2MSwgaW4gcnVuQ29tbWFuZA0KICAgIGJiLmZhdGFsKCJHYXZlIHVwOyBU b28gbWFueSB0cmllczogdGltZW91dCB3aGlsZSBhdHRlbXB0aW5nIHRvIGNvbW11bmljYXRlIHdp dGggYml0YmFrZSBzZXJ2ZXIiKQ0KICBGaWxlICIvaG9tZS9iZXNxdWl2ZS93b3JrZGlyL3Bva3kt d29ya2luZy9iaXRiYWtlL2xpYi9iYi9fX2luaXRfXy5weSIsIGxpbmUgMTA0LCBpbiBmYXRhbA0K ICAgIHJhaXNlIEJCSGFuZGxlZEV4Y2VwdGlvbigpDQpiYi5CQkhhbmRsZWRFeGNlcHRpb24NCg0K RHVyaW5nIGhhbmRsaW5nIG9mIHRoZSBhYm92ZSBleGNlcHRpb24sIGFub3RoZXIgZXhjZXB0aW9u IG9jY3VycmVkOg0KDQpUcmFjZWJhY2sgKG1vc3QgcmVjZW50IGNhbGwgbGFzdCk6DQogIEZpbGUg Ii9ob21lL2Jlc3F1aXZlL3dvcmtkaXIvcG9reS13b3JraW5nL21ldGEvbGliL29lcWEvdXRpbHMv ZGVjb3JhdG9ycy5weSIsIGxpbmUgMTA5LCBpbiB3cmFwcGVkX2YNCiAgICByZXR1cm4gZnVuYygq YXJncywgKiprd2FyZ3MpDQogIEZpbGUgIi9ob21lL2Jlc3F1aXZlL3dvcmtkaXIvcG9reS13b3Jr aW5nL21ldGEvbGliL29lcWEvc2VsZnRlc3QvaW1hZ2VmZWF0dXJlcy5weSIsIGxpbmUgNTksIGlu IHRlc3RfYWxsX3VzZXJzX2Nhbl9jb25uZWN0X3ZpYV9zc2hfd2l0aG91dF9wYXNzd29yZA0KICAg IHdpdGggcnVucWVtdSgiY29yZS1pbWFnZS1taW5pbWFsIikgYXMgcWVtdToNCiAgRmlsZSAiL3Vz ci9saWI2NC9weXRob24zLjQvY29udGV4dGxpYi5weSIsIGxpbmUgNTksIGluIF9fZW50ZXJfXw0K ICAgIHJldHVybiBuZXh0KHNlbGYuZ2VuKQ0KICBGaWxlICIvaG9tZS9iZXNxdWl2ZS93b3JrZGly L3Bva3ktd29ya2luZy9tZXRhL2xpYi9vZXFhL3V0aWxzL2NvbW1hbmRzLnB5IiwgbGluZSAyMjcs IGluIHJ1bnFlbXUNCiAgICB0aW5mb2lsLnByZXBhcmUoY29uZmlnX29ubHk9RmFsc2UsIHF1aWV0 PVRydWUpDQogIEZpbGUgIi9ob21lL2Jlc3F1aXZlL3dvcmtkaXIvcG9reS13b3JraW5nL2JpdGJh a2UvbGliL2JiL3RpbmZvaWwucHkiLCBsaW5lIDIxMCwgaW4gcHJlcGFyZQ0KICAgIGV4dHJhZmVh dHVyZXMpDQogIEZpbGUgIi9ob21lL2Jlc3F1aXZlL3dvcmtkaXIvcG9reS13b3JraW5nL2JpdGJh a2UvbGliL2JiL21haW4ucHkiLCBsaW5lIDUyMiwgaW4gc2V0dXBfYml0YmFrZQ0KICAgIGJiLmZh dGFsKCJDb3VsZCBub3QgY29ubmVjdCB0byBzZXJ2ZXIgJXM6ICVzIiAlIChjb25maWdQYXJhbXMu cmVtb3RlX3NlcnZlciwgc3RyKGUpKSkNCiAgRmlsZSAiL2hvbWUvYmVzcXVpdmUvd29ya2Rpci9w b2t5LXdvcmtpbmcvYml0YmFrZS9saWIvYmIvX19pbml0X18ucHkiLCBsaW5lIDEwNCwgaW4gZmF0 YWwNCiAgICByYWlzZSBCQkhhbmRsZWRFeGNlcHRpb24oKQ0KYmIuQkJIYW5kbGVkRXhjZXB0aW9u DQoNCi0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0NClJhbiAxIHRlc3QgaW4gMTEzLjQ2N3MNCg0KRkFJTEVEIChlcnJv cnM9MSkNCg0KR2VuZXJhdGluZyBYTUwgcmVwb3J0cy4uLg0KMjAxNi0xMi0zMCAxNTo1NzowOSwz NzUgLSBzZWxmdGVzdCAtIElORk8gLSBGaW5pc2hlZA0KVHJhY2ViYWNrIChtb3N0IHJlY2VudCBj YWxsIGxhc3QpOg0KICBGaWxlICIvaG9tZS9iZXNxdWl2ZS93b3JrZGlyL3Bva3ktd29ya2luZy9z Y3JpcHRzL29lLXNlbGZ0ZXN0IiwgbGluZSA3ODUsIGluIDxtb2R1bGU+DQogICAgcmVtb3ZlX2lu Y19maWxlcygpDQogIEZpbGUgIi9ob21lL2Jlc3F1aXZlL3dvcmtkaXIvcG9reS13b3JraW5nL3Nj cmlwdHMvb2Utc2VsZnRlc3QiLCBsaW5lIDE2OSwgaW4gcmVtb3ZlX2luY19maWxlcw0KICAgIG9z LnJlbW92ZShvcy5wYXRoLmpvaW4ob3MuZW52aXJvbi5nZXQoIkJVSUxERElSIiksICJjb25mL3Nl bGZ0ZXN0LmluYyIpKQ0KICBGaWxlICIvdXNyL2xpYjY0L3B5dGhvbjMuNC9wb3NpeHBhdGgucHki LCBsaW5lIDgyLCBpbiBqb2luDQogICAgcGF0aCArPSBiDQpUeXBlRXJyb3I6IHVuc3VwcG9ydGVk IG9wZXJhbmQgdHlwZShzKSBmb3IgKz06ICdOb25lVHlwZScgYW5kICdzdHInDQoNCl5DDQoNCkVy cm9yIGluIGF0ZXhpdC5fcnVuX2V4aXRmdW5jczoNClRyYWNlYmFjayAobW9zdCByZWNlbnQgY2Fs bCBsYXN0KToNCiAgRmlsZSAiL3Vzci9saWI2NC9weXRob24zLjQvbXVsdGlwcm9jZXNzaW5nL3Bv cGVuX2ZvcmsucHkiLCBsaW5lIDMwLCBpbiBwb2xsDQogICAgcGlkLCBzdHMgPSBvcy53YWl0cGlk KHNlbGYucGlkLCBmbGFnKQ0KS2V5Ym9hcmRJbnRlcnJ1cHQNCldBUk5JTkc6IFRpbWVvdXQgd2hp bGUgYXR0ZW1wdGluZyB0byBjb21tdW5pY2F0ZSB3aXRoIGJpdGJha2Ugc2VydmVyDQpXQVJOSU5H OiBUaW1lb3V0IHdoaWxlIGF0dGVtcHRpbmcgdG8gY29tbXVuaWNhdGUgd2l0aCBiaXRiYWtlIHNl cnZlcg0KV0FSTklORzogVGltZW91dCB3aGlsZSBhdHRlbXB0aW5nIHRvIGNvbW11bmljYXRlIHdp dGggYml0YmFrZSBzZXJ2ZXINCldBUk5JTkc6IFRpbWVvdXQgd2hpbGUgYXR0ZW1wdGluZyB0byBj b21tdW5pY2F0ZSB3aXRoIGJpdGJha2Ugc2VydmVyDQpFUlJPUjogR2F2ZSB1cDsgVG9vIG1hbnkg dHJpZXM6IHRpbWVvdXQgd2hpbGUgYXR0ZW1wdGluZyB0byBjb21tdW5pY2F0ZSB3aXRoIGJpdGJh a2Ugc2VydmVyDQpFUlJPUjogQ291bGQgbm90IGNvbm5lY3QgdG8gc2VydmVyIE5vbmU6IA0KYmVz cXVpdmVAeWN0YjA0On4vd29ya2Rpci9wb2t5LXdvcmtpbmcvYnVpbGQtdGVzdHByc2VydlttYXN0 ZXJdJCA=