| Summary: | 32-bit buildtools-tarball hangs when using bitbake | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [Runtime] General Runtime | Reporter: | Randy Witt <randy.e.witt> | ||||||||||
| Component: | General Runtime | Assignee: | Juro Bystricky <juro.bystricky> | ||||||||||
| Status: | RESOLVED FIXED | QA Contact: | |||||||||||
| Severity: | major | ||||||||||||
| Priority: | Medium+ | CC: | juro.bystricky, mhalstead, randy.macleod, richard.purdie, ross.burton | ||||||||||
| Version: | 1.8 | ||||||||||||
| Target Milestone: | 1.8.2 | ||||||||||||
| Hardware: | x86 | ||||||||||||
| OS: | Multiple | ||||||||||||
| Whiteboard: | |||||||||||||
| OS type for building Yocto: | --- | Type of Regression: | --- | ||||||||||
| Verified: | Documentation change: | No (bug/feature does not impact docs) | |||||||||||
| Attachments: |
|
||||||||||||
Created attachment 2652 [details]
strace when using debug patch
If you look at line 78842 you can see the Ea1 from the debug and that it subsequently spins on a futex.
(In reply to comment #0) > Created attachment 2651 [details] > Patch with debug. > > Install a buildtools-tarball such as > poky-glibc-i686-buildtools-tarball-core2-64-buildtools-nativesdk-standalone- > 1.8+snapshot-20150810.sh on a 64-bit distro. In this case Fedora 22(haven't > tested other distros). > > source the environment-setup-i686-pokysdk-linux script > > source the oe-init-build-env script > > Run a bitbake command such as "bitbake core-image-minimal" > > > The follow error will be seen and bitbake will hang and must be manually > killed: > ERROR: Timeout while attempting to communicate with bitbake server > ERROR: Could not connect to server False: > > It seems that python is getting stuck in self.event_queue.put() in > bitbake/lib/bb/server/process.py. It ends up spinning on a futex. > > I have attached a patch with some debug and the strace from running with > that debug. Just curious, did you install the tarball into default location (/opt/poky/1.8+snapshot) ? (In reply to comment #2) > (In reply to comment #0) > > Created attachment 2651 [details] > > Patch with debug. > > > > Install a buildtools-tarball such as > > poky-glibc-i686-buildtools-tarball-core2-64-buildtools-nativesdk-standalone- > > 1.8+snapshot-20150810.sh on a 64-bit distro. In this case Fedora 22(haven't > > tested other distros). > > > > source the environment-setup-i686-pokysdk-linux script > > > > source the oe-init-build-env script > > > > Run a bitbake command such as "bitbake core-image-minimal" > > > > > > The follow error will be seen and bitbake will hang and must be manually > > killed: > > ERROR: Timeout while attempting to communicate with bitbake server > > ERROR: Could not connect to server False: > > > > It seems that python is getting stuck in self.event_queue.put() in > > bitbake/lib/bb/server/process.py. It ends up spinning on a futex. > > > > I have attached a patch with some debug and the strace from running with > > that debug. > > Just curious, did you install the tarball into default location > (/opt/poky/1.8+snapshot) ? I did not install to /opt/... (In reply to comment #3) > (In reply to comment #2) > > (In reply to comment #0) > > > Created attachment 2651 [details] > > > Patch with debug. > > > > > > Install a buildtools-tarball such as > > > poky-glibc-i686-buildtools-tarball-core2-64-buildtools-nativesdk-standalone- > > > 1.8+snapshot-20150810.sh on a 64-bit distro. In this case Fedora 22(haven't > > > tested other distros). > > > > > > source the environment-setup-i686-pokysdk-linux script > > > > > > source the oe-init-build-env script > > > > > > Run a bitbake command such as "bitbake core-image-minimal" > > > > > > > > > The follow error will be seen and bitbake will hang and must be manually > > > killed: > > > ERROR: Timeout while attempting to communicate with bitbake server > > > ERROR: Could not connect to server False: > > > > > > It seems that python is getting stuck in self.event_queue.put() in > > > bitbake/lib/bb/server/process.py. It ends up spinning on a futex. > > > > > > I have attached a patch with some debug and the strace from running with > > > that debug. > > > > Just curious, did you install the tarball into default location > > (/opt/poky/1.8+snapshot) ? > > I did not install to /opt/... The strace file shows a lot of /opt/ attempts, i.e.: (stat("/opt/poky/1.8+snapshot/sysroots/x86_64-pokysdk-linux/usr/bin/sh", 0x7ffc15b5f0c0) = -1 ENOENT (No such file or directory) It would be interesting to see if the problem remains if the tarball is installed in the default location. If the problem goes away, it's most likely a relocation issue. So it appears that 32-bit buildtools hangs when running the steps in https://bugzilla.yoctoproject.org/show_bug.cgi?id=8140#c0 on both 64-bit and 32-bit hosts. So an i686 buildtools built on both x86_64 or i686 will hang during bitbake on 32-bit and 64-bit hosts. The manifestation is also the same, spinning on a futex. Preliminary results: At this time the culprit seems to be libpthread-2.21.so. libpthread-2.20.so seems to work fine. Work in progress. Some additional observations: 32 bit versions of libpthread-2.19.so and libpthread-2.20.so both work. libpthread-2.21.so or later do not. Unfortunately, there are too many changes between versions 2.20 and 2.21, so a "compare what has changed" is not that simple. When attaching gdb to a running process (bitbake->python2.7.real), I can see we loop (forever) in sem_wait. The loop will never terminate, as the semaphore value is -2, (which does not make sense). Trying to determine where the value -2 originated. At this time I am pretty sure this is a glibc bug. Deep down in the bowels of nptl, the code path is split two ways, depending on the value of _HAVE_64B_ATOMICS. In particular this affects how the the semaphores are handled. __HAVE_64B_ATOMICS is defined as 1 for x64 and as 0 for x32. This also explains why 64 bit build works and 32 bit build hangs. If _HAVE_64B_ATOMICS is defined as 0, the code gets a bit more convoluted as well. Somewhere in there lies the bug. However, for Pentium and later CPUs it should be possible to define __HAVE_64B_ATOMICS as 1 as well. Once this is done, 32-bit buildtools-tarball does not hang any more and the code is actually more efficient. I attached a patch that can be used to verify this. It should be safe for Yocto, as long we don't build glibc for i386 or i486. Created attachment 2828 [details]
Patch to fix the problem.
I should mention the patch is for glibc 2.21 (Fido). Jethro/master needs similar patch, but for glibc 2.22. TBD Created attachment 2830 [details]
Patch for jethro/master
Patch for glibc 2.22
The patch for glibc2.22 was merged for jethro/master: http://git.yoctoproject.org/cgit/cgit.cgi/poky/commit/?h=jethro&id=c86957aeb8ee30ca4a626c68fd2a70f7706fceac The real cause of the problem is the change of semaphore structure after glibc 2.20. Python uses libraries that create and use semaphores using calls sem_open, sem_init, sem_wait... Combination of calls sem_init/sem_wait has no problems, however calling sem_open and using sem_wait on the semaphore created this way will misbehave badly, as sem_open and sem_wait assume different structures for the semaphore: sem_open assumes the "new" structure, sem_wait is linked to the code using the "old" structure. The patch "fixes" this indirectly, enforcing the same "new" semaphore structure. Nevertheless, it is a good solution as it results in more efficient code as well, it will use atomic instructions in glibc where they were not used before although they existed. The downside is, this will not work on old i386/i486 CPUs not supporting atomic operations. Nobody in their right mind would use a 486 to build. Fixed in oe-core 2b8c7aa51f6ac7f79c4834e04b697c04afc8beaf. |
Created attachment 2651 [details] Patch with debug. Install a buildtools-tarball such as poky-glibc-i686-buildtools-tarball-core2-64-buildtools-nativesdk-standalone-1.8+snapshot-20150810.sh on a 64-bit distro. In this case Fedora 22(haven't tested other distros). source the environment-setup-i686-pokysdk-linux script source the oe-init-build-env script Run a bitbake command such as "bitbake core-image-minimal" The follow error will be seen and bitbake will hang and must be manually killed: ERROR: Timeout while attempting to communicate with bitbake server ERROR: Could not connect to server False: It seems that python is getting stuck in self.event_queue.put() in bitbake/lib/bb/server/process.py. It ends up spinning on a futex. I have attached a patch with some debug and the strace from running with that debug.