Bug 1291

Summary: multilib: File conflicts in eglibc on ia32
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Mark Hatle <mark.hatle>
Component: coreAssignee: Mark Hatle <mark.hatle>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium CC: hjl.tools, meta.mr.watcher, meta.watcher, nitin.a.kamble
Version: unspecified   
Target Milestone: 1.1 M3   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: ---

Description Mark Hatle 2011-07-25 11:37:32 UTC
When building a combination x86 and x86_64 system, the following headers contain different contents:

bits/pthreadtypes.h
bits/semaphore.h
bits/byteswap.h
bits/endian.h
bits/huge_vall.h
bits/link.h
bits/mathdef.h
bits/select.h
bits/setjmp.h
bits/wordsize.h
bits/xtitypes.h
bits/fenv.h
bits/mathinline.h
fpu_control.h
bits/string.h
bits/a.out.h
bits/envrionments.h
bits/fcntl.h
bits/mman.h
bits/msg.h
bits/shm.h
bits/sigcontext.h
bits/stat.h
bits/wchar.h
bits/debugreg.h
bits/epoll.h
bits/io.h
bits/perm.h
bits/procfs.h
bits/reg.h
bits/ucontext.h
bits/sem.h
bits/user.h
bits/syscall.h

In almost all of the above cases, the x86_64 version of the header is the "correct" version.  It contains both support for 32-bit and 64-bit IA32.

In some of the above the conflicts are simply the copyright notices being out of sync between the 32-bit and 64-bit versions.

One case where the x86_64 version seems to be incomplete is the string.h.  There are assembly optimizations present in the 32-bit version that are not in the 64-bit version.

Finally the generation of the syscalls is different between the two versions, so they also contain different information.

This different information causes a conflict to occur when we're installing the software into the build and run-time environments.  The packaging systems compare each files MD5SUM and SHA1SUM, if the results differ, the file contents differ and there is a conflict.

On non-IA32 architectures, the 32-bit and 64-bit builds are shared under a single unified architecture.  This eliminates the problem on PPC, MIPS, and a few others.  On IA32, there are two architecture directories in use, i386 and x86_64 -- contributing the mismatches and problems.
Comment 1 Mark Hatle 2011-07-25 11:42:07 UTC
A workaround for this issue is available is poky-contrib:

http://git.pokylinux.org/cgit.cgi/poky-contrib/commit/?h=mhatle/ml&id=83cf9299565baa359c7c71011d9d7604dc8a9ca1

The workaround was created by merging the x86_64 and i586 header files, with a preference toward the x86_64 version, as they are generally known to be correct for both 32-bit and 64-bit.  In many cases it was observed that the differences were cosmetic in nature -- or that the i586 version simply didn't have knowledge of the x86_64 architecture.

The string.h file was updated in both x86_64 and i586 to include the 32-bit assembly optimizations.  These optimizations are lost if you simply use the x86_64 version of the header.

The syscall.h is generated, and generated differently for both x86_64 and i586.  Simply using the syscall list from one or the other was not adequate to end up with the same generated file.  This was addressed by using the header conflict resolution helper code "oe_multilib_header".  This helper simply renames the header into a 32-bit or 64-bit version and then includes the correct version based on the system WORDSIZE.
Comment 2 H.J. Lu 2011-07-25 14:24:40 UTC
For string.h, there is no difference between compiling with
-march=i686 on ia32 and -march=i686 -m32 on Intelel64.  In
both cases, sysdeps/i386/i486/bits/string.h should be used.
We should do

1, Remove sysdeps/i386/i486/bits/string.h.
2. Add ia32 support to sysdeps/i386/bits/string.h
and sysdeps/x86_64/bits/string.h.
Comment 3 H.J. Lu 2011-07-25 14:28:27 UTC
Basically, for each installed header file, there should be only
one copy for ia32 and we should add ia32 support to Intel64 version.
Comment 4 H.J. Lu 2011-07-25 14:30:54 UTC
Please break the single commit into small ones.
Each commit should only address one header file
since each header file may require a different
fix and not all changes are correct.
Comment 5 Mark Hatle 2011-07-26 11:43:03 UTC
Below is a complete list of the conflicting header files, and the difference between the 32-bit and 64-bit file.  Comments relate to the changes in the 64-bit files.  The files were compared after eglibc was built and installed, so that the final generated versions were compared.


#  bits/a.out.h - Add support for __WORDSIZE = 64
#  bits/byteswap.h - Copyright date mismatch, add support for __WORDSIZE = 64
#  bits/endian.h - Comment mismatch
#  bits/environment.h - add support for __WORDSIZE = 64
#  bits/fcntl.h - Comment/Copyright date mismatch, add support for __WORDSIZE = 64
#  bits/fenv.h - Copyright date mismatch, add support for __WORDSIZE = 64
#  bits/huge_vall.h - Comment/Copyright date mismatch, remove support for older gcc
#  bits/link.h - Function name difference, add x86_64 definitions
#  bits/mathdef.h - Copyright date mismatch, add support for __WORDSIZE = 64
#  bits/mathinline.h - Copyright date mismatch, contributed by mismatch, remove support for older gcc/assembly op$
#  bits/mman.h - Header/Copyright date mismatch, add MAP_32BIT definition
#  bits/msq.h - Copyright date mismatch, add __WORDSIZE = 32 definitions
#  bits/pthread_type.h -- Contributed by added, add support for __WORDSIZE = 64
#  bits/select.h - Copyright date mismatch, add support for __WORDSIZE = 64

#  bits/semaphore.h - Copyright date mismatch, add support for __WORDSIZE = 64
#  bits/sem.h - Copyright date mismatch
#  bits/setjmp.h - Copyrgiht date mismatch, add support for __WORDSIZE = 64
#  bits/shm.h - Copyright date mismatch, add support for __WORDSIZE = 32
#  bits/sigcontext.h - Copyright date mismatch, license wording mismatch, add support for __WORDSIZE = 32
#  bits/stat.h - Copyright date mismatch, add support for __WORDSIZE = 32 and __WORDSIZE = 64
#  bits/string.h - Header/Copyright date mismatch, remove assembly optimizations
#  bits/syscall.h - different order, some different syscalls listed
#  bits/wchar.h - Change the way the definitions are done
#  bits/wordsize.h - Different header, remove license notice, add __x86_64__ support
#  bits/xtitypes.h - Header difference, different typedef format
#  bits/fpu_control.h - header difference, revised comments, updated assembly macros
#  sys/debugreg.h - Copyright date mismatch, new definition and added __WORDSIZE=64 support
#  sys/epoll.h - Copyright date mismatch, slightly different definitions
#  sys/io.h - Copyright date mismatch, slightly different assembly formats
#  sys/perm.h - Copyright date mismatch
#  sys/procfs.h - Copyright date mismatch, support for __WORDSIZE = 32
#  sys/reg.h - Copyright date mismatch, support for __WORDSIZE = 64
#  sys/ucontext.h - Copyright date mismatch, support for __WORDSIZE = 64
#  sys/user.h - Copyright date mismatch, support for __WORDSIZE = 64

As indicated above, a few of the files bits/endian.h, bits/sem.h and bits/perm.h are identical other then the copyright date or other comment.  Most of the remaining files add support for x86_64 with a compatible header file.

bits/string.h, bits/mathinline.h, bits/huge_vall.h, lose any optimizations in the x86-64 version.
Comment 6 Mark Hatle 2011-07-26 11:44:06 UTC
Based on comments from OpenEmbedded, I will be abandoning the patch and simply copying the 64-bit version over the 32-bit version.  Doing one patch per header file is simply not worth the effort at this point.
Comment 7 H.J. Lu 2011-07-26 12:06:43 UTC
(In reply to comment #6)
> Based on comments from OpenEmbedded, I will be abandoning the patch and simply
> copying the 64-bit version over the 32-bit version.  Doing one patch per header
> file is simply not worth the effort at this point.

I think this is a very reasonable approach.
Comment 8 Mark Hatle 2011-08-02 08:18:08 UTC
Fix in oe-core:

commit 019a33236f76aacb989e8f37b09b81599c27f296
Author: Mark Hatle <mark.hatle@windriver.com>
Date:   Tue Jul 26 14:17:11 2011 -0500