8192 2015-08-21 08:53:40 +0000 python3.4.3 os.fork and os.waitpid (prelink related) 2016-12-15 16:25:26 +0000 1 1 7 Build System, Metadata & Runtime OE-Core devtools / tool chain unspecified Other x86 RESOLVED DUPLICATE 9131 Medium minor 2.3 M2 9131 1 multisosnooley edwin.plauchu.camacho alejandro.hernandez anibal.limon benjamin.esquivel bjorn.bosell leonardo.sandoval.gonzalez luke.smith meta.mr.watcher meta.watcher richard.purdie --- --- No (bug/feature does not impact docs) oldest_to_newest 53309 0 multisosnooley 2015-08-21 08:53:40 +0000 Target platform: amd geode oe-core from master Layers: meta meta-yocto meta-yocto-bsp meta-cpc304 (https://github.com/pohmelie/meta-cpc304) Distro: poky BB version: 1.27.1 Build and run just fine. When I try to install something (that required compilation (!) with gcc) with "python3 -m pip" or "python3 setup.py install" then I get "No child processes" error. I have found the "bad" part: it is in distutils os.waitpid. So, I wrote simple script to reproduce the bug. root@cpc304:~# cat yoba.py import os pid = os.fork() if pid: print("parent") os.waitpid(pid, 0) else: print("child") root@cpc304:~# python yoba.py parent child root@cpc304:~# python3 --version Python 3.4.3 root@cpc304:~# python3 yoba.py parent Traceback (most recent call last): File "yoba.py", line 8, in <module> os.waitpid(pid, 0) ChildProcessError: [Errno 10] No child processes root@cpc304:~# And it works on python2 just fine, but fails on python3.4.3. Probably it's because some patches are 3.4.2 oriented (https://github.com/openembedded/oe-core/blob/master/meta/recipes-devtools/python/python3/python-3.3-multilib.patch for example), but version is 3.4.3. Anyway right now I'm building absolutely same image but for x86_64 and will try it out with virtualbox. 53310 1 multisosnooley 2015-08-21 08:56:32 +0000 Oh, code was cutted out. http://pastebin.com/Ui0Rwmfn http://pastebin.com/cvc3TKQV 53327 2 multisosnooley 2015-08-21 14:16:30 +0000 Bug does not reproduced with x86_64 build with virtualbox. Also there is some strange behaviour with multiprocessing lib. Again, only on my board and not with virtualbox. 53330 3 multisosnooley 2015-08-21 14:44:11 +0000 c code with fork works just fine http://pastebin.com/kJQemSzE 53332 4 multisosnooley 2015-08-21 14:45:16 +0000 So, it definetly something "python-side" 53352 5 multisosnooley 2015-08-24 13:54:46 +0000 python 3.3.3 from "fido" branch works just fine on my geode board. 53985 6 alejandro.hernandez 2015-09-11 18:43:23 +0000 I can confirm this isn't working, while I still don't exactly know what the problem is, I know its not related to python3 entirely, I was able to track the issue down to the Glibc upgrade from 2.21 to 2.22 on (which makes sense): http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?id=cb7368c110553dcb972480abc4a59b975677f7df I will keep digging. 54115 7 2734 alejandro.hernandez 2015-09-15 21:38:00 +0000 Created attachment 2734 strace_log_before_glibc_upgrade 54116 8 2735 alejandro.hernandez 2015-09-15 21:39:58 +0000 Created attachment 2735 strace_log_after_glibc_upgrade 54117 9 alejandro.hernandez 2015-09-15 21:40:32 +0000 Following up on this issue; as it is shown on the attachments (trimmed strace logs before and after the upgrade) when using gilbc 2.22 the clone() syscall is never executed, hence why the waitpid() fails afterwards, I quickly skimmed through the modified patches, but didn't see anything at first sight that could be causing this behavior, so it might be something that changed upstream on glibc. 54119 10 multisosnooley 2015-09-15 21:52:45 +0000 (In reply to comment #9) Wow, awesome work! I have no experience with strace and such level of debugging, so I can't help, but I'm glad you can. 54235 11 alejandro.hernandez 2015-09-17 17:57:58 +0000 I'm still trying!, so this also seems to be x86 specific, I tested it on my local PC using Arch Linux (x86-64) with python 3.4 and glibc 2.22 and it worked!, so I also tested it on poky x86-64 and it worked just fine, ARM works fine too, which also may indicate that its something that changed upstream, since glibc implements fork in assembly and its architecture specific. 54907 12 alejandro.hernandez 2015-10-01 17:22:41 +0000 I tested this on Fedora 22 x86 using glibc 2.22 and python 3.4.2 and it works just fine, so I natively built python 3.4.2 on poky x86 linking to poky's glibc 2.22 and it worked fine as well, this means there's an issue with poky's Python3, either during compilation or with the patches 56963 13 alejandro.hernandez 2015-12-02 20:35:34 +0000 I tried building this with GCC 4.9 just to make sure it wasn't a GCC 5.* issue, I had the same result. 57019 14 anibal.limon 2015-12-03 22:41:58 +0000 I tested on Jethro qemux86 python 3.4.3 and glibc 2.22 and works. root@qemux86:~# python3.4 --version Python 3.4.3 root@qemux86:~# cat yoba.py import os pid = os.fork() if pid: print("parent") os.waitpid(pid, 0) else: print("child") root@qemux86:~# python3 python3 python3.4 python3.4m python3-config python3.4-config python3.4m-config root@qemux86:~# python3 yoba.py parent child 57020 15 anibal.limon 2015-12-03 22:55:31 +0000 hi multisosnooley, Can you confirm that is failing on Jethro? Regards, 57021 16 multisosnooley 2015-12-03 22:58:55 +0000 (In reply to comment #15) > Can you confirm that is failing on Jethro? I will try, but this will took couple of days, since board are not with me right now. 57083 17 multisosnooley 2015-12-07 14:15:56 +0000 I've just test jethro on my amd geode board (boot from usb stick) and simple python tests (os.fork and multiprocessing.Process) are ok. So, probably, this bug is covered by updates. I have no ability to play with python a lot (pip and libraries), since I can't install my linux build due some errors and I have no time and motivation to fix it. But if Alejandro Hernandez still can reproduce this bug (AFAIR he can try to), then part(?) of bug is still there. 57087 18 alejandro.hernandez 2015-12-07 14:48:50 +0000 Its very likely, this was fixed by the updates, I will do a bisect to check what fixed it and mark this as solved afterwards 57833 19 alejandro.hernandez 2016-01-14 18:10:49 +0000 I just tested this on master (92cc02fe517bad0af4e493fda3331738cbcc9a26) using qemux86, and it's not working, so we're back were we started. # python3.5 a.py parent Traceback (most recent call last): File "a.py", line 8, in <module> os.waitpid(pid, 0) ChildProcessError: [Errno 10] No child processes 59296 20 benjamin.esquivel 2016-02-19 22:50:40 +0000 moving it to minor since this is a very sporadic issue. Will continue working in it though. 60332 21 bjorn.bosell 2016-03-23 11:42:33 +0000 Hi, We recently ran into a strange issue that seems very similar. Yocto: 2.01 (Jethro) Python: 3.4.3 Platform: imx6 (ARM Cortex-a9) The issue we saw was that we couldn't start new processes using multiprocess.Process in python. We were able to work around this issue in a very odd way. After installing an image on our target, we used smart to remove python3, and then re-installed it using a rpm package repository built from the same tree as the original image. After playing around with installing images/rpms we managed to narrowed the problem down to os.fork() returning something very bad. (In fact, it turned out it returned a pointer to the fork() function from libc) When running the two different scenarios (clean image vs re-installed python3) through gdb we could see that in the case where os.fork() fails, resolve_fork() is invoked instead of fork(). And resolve_fork() returns the pointer to the actual fork() function. We do not know why this is yet, it is still very strange, "posixmodule.c" from the python sources calls fork(), but for some reason, from a fresh install the function invoked is resolve_fork(). 60338 22 bjorn.bosell 2016-03-23 13:40:57 +0000 Sorry, that should be fork_resolve() rather than resolve_fork() 60487 23 bjorn.bosell 2016-03-29 13:42:30 +0000 Looks like this may be the same problem: https://bugzilla.yoctoproject.org/show_bug.cgi?id=9131 62834 24 alejandro.hernandez 2016-06-09 15:33:10 +0000 (In reply to comment #23) > Looks like this may be the same problem: > https://bugzilla.yoctoproject.org/show_bug.cgi?id=9131 I can confirm this is the same problem, disabling pre-linking works , and since it is disabled as default, we shouldn't have a problem for now. 62835 25 alejandro.hernandez 2016-06-09 15:35:54 +0000 Changing the status to NEEDINFO, until the pre linking issue gets fixed. 62946 26 sjolley.yp.pm 2016-06-13 22:29:54 +0000 It shouldn't be moved to NEEDINFO if it is dependent on another bug. I have assigned it to Mark. If is is a duplicate, it should be closed as duplicate. Also moved it to Medium since the comments show it is not a major issue. 68415 27 richard.purdie 2016-11-17 16:29:37 +0000 Could you retest this please? We enabled prelink for x86 again in August with updates to fix some of these issues. 69131 28 edwin.plauchu.camacho 2016-12-14 17:54:39 +0000 I Ran it again Setting LD_USE_LOAD_BIAS=1 when starting Yocto: master Python: 3.5.2 Platform: qemux86 -------------------------------------------------- sh-4.3# export LD_USE_LOAD_BIAS=1 sh-4.3# cat yoba.py import os pid = os.fork() if pid: print("parent") os.waitpid(pid, 0) else: print("child") sh-4.3# python3 yoba.py parent child 69132 29 edwin.plauchu.camacho 2016-12-14 17:58:23 +0000 I've forgotten to add the revision ( 14539203655913b3304bf98d2dd1f2e1fed3d7ca ) 69161 30 edwin.plauchu.camacho 2016-12-15 16:25:26 +0000 We have set LD_USE_LOAD_BIAS with value 1 as per resolution in bug https://bugzilla.yoctoproject.org/show_bug.cgi?id=9131 *** This bug has been marked as a duplicate of bug 9131 *** 2734 2015-09-15 21:38:00 +0000 2015-09-15 21:38:00 +0000 strace_log_before_glibc_upgrade file_8192.txt text/plain 845 alejandro.hernandez MzU2ICAgcmVhZCgzLCAiIyEvdXNyL2Jpbi9weXRob24zXG5pbXBvcnQgb3NcbnBpZCA9IG9zLmZv cmsoKVxuaWYgcGlkOiBcbiAgICBwcmludChcInBhcmVudFwiKVxuICAgIG9zLndhaXRwaWQocGlk LCAwKSAgIFxuZWxzZTpcbiAgICBwcmludChcImNoaWxkXCIpXG4iLCAxMDI0KSA9IDEyNQ0KMzU2 ICAgX2xsc2VlaygzLCAwLCBbMF0sIFNFRUtfU0VUKSAgICAgID0gMA0KMzU2ICAgcmVhZCgzLCAi IyEvdXNyL2Jpbi9weXRob24zXG5pbXBvcnQgb3NcbnBpZCA9IG9zLmZvcmsoKVxuaWYgcGlkOiBc biAgICBwcmludChcInBhcmVudFwiKVxuICAgIG9zLndhaXRwaWQocGlkLCAwKSAgIFxuZWxzZTpc biAgICBwcmludChcImNoaWxkXCIpXG4iLCAxMDI0KSA9IDEyNQ0KMzU2ICAgcmVhZCgzLCAiIiwg MTAyNCkgICAgICAgICAgICAgICAgID0gMA0KMzU2ICAgY2xvc2UoMykgICAgICAgICAgICAgICAg ICAgICAgICAgID0gMDM1NiAgIA0KbXVubWFwKDB4Yjc3ZDYwMDAsIDQwOTYpICAgICAgICAgID0g MA0KMzU2ICAgY2xvbmUoY2hpbGRfc3RhY2s9MCwgZmxhZ3M9Q0xPTkVfQ0hJTERfQ0xFQVJUSUR8 Q0xPTkVfQ0hJTERfU0VUVElEfFNJR0NITEQsIGNoaWxkX3RpZHB0cj0weGI3N2QzNzY4KSA9IDM1 Nw0KMzU3ICAgc2V0X3JvYnVzdF9saXN0KDB4Yjc3ZDM3NzAsIDEyKSAgID0gMA0KMzU3ICAgd3Jp dGUoMSwgImNoaWxkXG4iLCA2IDx1bmZpbmlzaGVkIC4uLj4NCjM1NiAgIHdyaXRlKDEsICJwYXJl bnRcbiIsIDcpICAgICAgICAgICA9IDcNCjM1NiAgIHdhaXRwaWQoMzU3LCAgPHVuZmluaXNoZWQg Li4uPg0KMzU3ICAgPC4uLiB3cml0ZSByZXN1bWVkPiApICAgICAgICAgICAgID0= 2735 2015-09-15 21:39:58 +0000 2015-09-15 21:39:58 +0000 strace_log_after_glibc_upgrade file_8192.txt text/plain 817 alejandro.hernandez MzU3ICAgbW1hcDIoTlVMTCwgNDA5NiwgUFJPVF9SRUFEfFBST1RfV1JJVEUsIE1BUF9QUklWQVRF fE1BUF9BTk9OWU1PVVMsIC0xLCAwKSA9IDB4Yjc3OTkwMDANCjM1NyAgIHJlYWQoMywgIiMhL3Vz ci9iaW4vcHl0aG9uM1xuaW1wb3J0IG9zXG5waWQgPSBvcy5mb3JrKClcbmlmIHBpZDogXG4gICAg cHJpbnQoXCJwYXJlbnRcIilcbiAgICBvcy53YWl0cGlkKHBpZCwgMCkgICBcbmVsc2U6XG4gICAg cHJpbnQoXCJjaGlsZFwiKVxuIiwgMTAyNCkgPSAxMjUNCjM1NyAgIF9sbHNlZWsoMywgMCwgWzBd LCBTRUVLX1NFVCkgICAgICA9IDANCjM1NyAgIHJlYWQoMywgIiMhL3Vzci9iaW4vcHl0aG9uM1xu aW1wb3J0IG9zXG5waWQgPSBvcy5mb3JrKClcbmlmIHBpZDogXG4gICAgcHJpbnQoXCJwYXJlbnRc IilcbiAgICBvcy53YWl0cGlkKHBpZCwgMCkgICBcbmVsc2U6XG4gICAgcHJpbnQoXCJjaGlsZFwi KVxuIiwgMTAyNCkgPSAxMjUgICAgICAgICAgICAgICAgICAgICAgICANCjM1NyAgIHJlYWQoMywg IiIsIDEwMjQpICAgICAgICAgICAgICAgICA9IDANCjM1NyAgIGNsb3NlKDMpICAgICAgICAgICAg ICAgICAgICAgICAgICA9IDANCjM1NyAgIG11bm1hcCgweGI3Nzk5MDAwLCA0MDk2KSAgICAgICAg ICA9IDANCjM1NyAgIHdyaXRlKDEsICJwYXJlbnRcbiIsIDcpICAgICAgICAgICA9IDcNCjM1NyAg IHdhaXRwaWQoMTE4ODA1NTgyNCwgMHhiZmFkMjUxYywgMCkgPSAtMSBFQ0hJTEQgKE5vIGNoaWxk IHByb2Nlc3NlcykNCjM1NyAgIHdyaXRlKDIsICJUcmFjZWJhY2sgKG1vc3QgcmVjZW50IGNhbGwg bGFzdCk6XG4iLCAzNSkgPSAzNQ==