Bug 4919

Summary: Non-obvious failure when NO32LIBS=0 but 32-bit toolchain is broken
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Laszlo Papp <lpapp>
Component: coreAssignee: Seebs <seebs>
Status: RESOLVED FIXED QA Contact:
Severity: minor    
Priority: Medium CC: meta.mr.watcher, meta.watcher, net147, ross.burton
Version: unspecified   
Target Milestone: 1.5   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Laszlo Papp 2013-07-25 20:06:44 UTC
Note that, Arch logically uses /usr/lib and /lib for 64 bit on a 64 bit system as that is the majority, right?

It uses /usr/lib32 and /lib32 for the 32 bit minority. This may probably not be handled correctly. I am using the poky master HEAD version with the master HEAD version of meta-sourcery. The configuration is in line with meta-sourcery's README.

bitbake core-image-minimal
WARNING: Host distribution "arch-rolling" has not been validated with this version of the build system; you may possibly experience unexpected failures. It is recommended that you use a tested distribution.
Parsing recipes: 100% |################################################################################################################################################################| ETA:  00:00:00
Parsing of 848 .bb files complete (0 cached, 848 parsed). 1175 targets, 48 skipped, 0 masked, 0 errors.

Build Configuration:
BB_VERSION        = "1.19.1"
BUILD_SYS         = "x86_64-linux"
NATIVELSBSTRING   = "arch-rolling"
TARGET_SYS        = "arm-poky-linux-gnueabi"
MACHINE           = "qemuarm"
DISTRO            = "poky"
DISTRO_VERSION    = "1.4+snapshot-20130725"
TUNE_FEATURES     = "armv5 thumb dsp"
TARGET_FPU        = "soft"
CSL_VER_MAIN      = "2013.05-24"
EXTERNAL_TOOLCHAIN = "/usr"
meta-sourcery     = "master:18bbe5f8ed0f95d2c75b691b92266bc63f832324"
meta              
meta-yocto        
meta-yocto-bsp    = "master:67864ca79da08df752487a3a4e1a975546da123d"

NOTE: Resolving any missing task queue dependencies
NOTE: Preparing runqueue
NOTE: Executing SetScene Tasks
NOTE: Executing RunQueue Tasks
ERROR: Function failed: do_compile (log file is located at /home/lpapp/Projects/poky/build/tmp/work/x86_64-linux/pseudo-native/1.5.1-r3/temp/log.do_compile.2120)
ERROR: Logfile of failure stored in: /home/lpapp/Projects/poky/build/tmp/work/x86_64-linux/pseudo-native/1.5.1-r3/temp/log.do_compile.2120
Log data follows:
| DEBUG: Executing shell function do_compile
| make: *** No rule to make target `distclean'.  Stop.
| WARNING: Guessing architecture CFLAGS '-m32'.
| If you need specific flags, use --cflags.
| SQLite header for version 3007017 found in /home/lpapp/Projects/poky/build/tmp/sysroots/x86_64-linux/usr.
| Enabling in-memory database by default.
| NOTE: make -e MAKEFLAGS= libpseudo
| mkdir -p lib/pseudo/lib
| ./makewrappers
| Checking for old/new clone mechanics...New clone.
| common
| ports/common/wrapfuncs.in: .......
| linux
| ports/linux/wrapfuncs.in: ................................................................
| unix
| ports/unix/wrapfuncs.in: ................................................................
| uids_generic
| ports/uids_generic/wrapfuncs.in: ........................
| linux/newclone
| ports/linux/newclone/wrapfuncs.in: .
| Writing functions... Warning: lchown from linux overriding unix
| done.  Cleaning up.
| ./maketables enums/*.in
| enums/msg_type.in: type: msg_type_t (prefix 'PSEUDO_MSG_ENUM')
|    ping               shutdown           op                 ack
|    nak                fastop
| 
| enums/op.in: type: op_t (prefix 'OP_ENUM')
|   extra column: int wait (default 0)
|    chdir              chmod              chown              chroot
|    close              creat              dup                fchmod
|    fchown             fstat              link               mkdir
|    mknod              open               rename             stat
|    unlink             symlink            exec               may-unlink
|    did-unlink         cancel-unlink
| 
| enums/query_field.in: type: query_field_t (prefix 'PSQF_ENUM')
|    access             client             dev                fd
|    ftype              gid                id                 inode
|    mode               op                 order              path
|    perm               program            result             severity
|    stamp              tag                text               type
|    uid
| 
| enums/query_type.in: type: query_type_t (prefix 'PSQT_ENUM')
|   extra column: const char * sql (default "LITTLE BOBBY TABLES")
|    exact              less               greater            bitand
|    notequal           like               notlike            sqlpat
| 
| enums/res.in: type: res_t (prefix 'RESULT_ENUM')
|    succeed            fail               error
| 
| enums/sev.in: type: sev_t (prefix 'SEVERITY_ENUM')
|    debug              info               warn               error
|    critical
| 
| Writing datatypes... done.  Cleaning up.
| gcc -fno-strict-aliasing -isystem/home/lpapp/Projects/poky/build/tmp/sysroots/x86_64-linux/usr/include -O2 -pipe -pipe -std=gnu99 -Wall -W -Wextra -fPIC -D_LARGEFILE64_SOURCE -D_ATFILE_SOURCE -m32 -DPSEUDO_PREFIX='"/home/lpapp/Projects/poky/build/tmp/sysroots/x86_64-linux/usr"' -DPSEUDO_SUFFIX='""' -DPSEUDO_BINDIR='"bin"' -DPSEUDO_LIBDIR='"lib/pseudo/lib"' -DPSEUDO_LOCALSTATEDIR='"var/pseudo"' -DPSEUDO_VERSION='"1.5.1"' -DUSE_MEMORY_DB -DPSEUDO_FORCE_ASYNC -O2 -g -L/home/lpapp/Projects/poky/build/tmp/sysroots/x86_64-linux/usr/lib -I/home/lpapp/Projects/poky/build/tmp/sysroots/x86_64-linux/usr/include  -D_GNU_SOURCE -c -o pseudo_wrappers.o pseudo_wrappers.c
| In file included from pseudo_wrapfuncs.c:3052:0,
|                  from pseudo_wrappers.c:67:
| pseudo_wrapfuncs.c: In function 'wrap_fchownat':
| ports/unix/guts/fchownat.c:12:6: warning: variable 'doing_link' set but not used [-Wunused-but-set-variable]
|   int doing_link = 0;
|       ^
| gcc -c -isystem/home/lpapp/Projects/poky/build/tmp/sysroots/x86_64-linux/usr/include -isystem/home/lpapp/Projects/poky/build/tmp/sysroots/x86_64-linux/usr/include -O2 -pipe -pipe -std=gnu99 -Wall -W -Wextra -fPIC -D_LARGEFILE64_SOURCE -D_ATFILE_SOURCE -m32 -DPSEUDO_PREFIX='"/home/lpapp/Projects/poky/build/tmp/sysroots/x86_64-linux/usr"' -DPSEUDO_SUFFIX='""' -DPSEUDO_BINDIR='"bin"' -DPSEUDO_LIBDIR='"lib/pseudo/lib"' -DPSEUDO_LOCALSTATEDIR='"var/pseudo"' -DPSEUDO_VERSION='"1.5.1"' -DUSE_MEMORY_DB -DPSEUDO_FORCE_ASYNC -O2 -g -L/home/lpapp/Projects/poky/build/tmp/sysroots/x86_64-linux/usr/lib -I/home/lpapp/Projects/poky/build/tmp/sysroots/x86_64-linux/usr/include  pseudo_client.c
| gcc -c -isystem/home/lpapp/Projects/poky/build/tmp/sysroots/x86_64-linux/usr/include -isystem/home/lpapp/Projects/poky/build/tmp/sysroots/x86_64-linux/usr/include -O2 -pipe -pipe -std=gnu99 -Wall -W -Wextra -fPIC -D_LARGEFILE64_SOURCE -D_ATFILE_SOURCE -m32 -DPSEUDO_PREFIX='"/home/lpapp/Projects/poky/build/tmp/sysroots/x86_64-linux/usr"' -DPSEUDO_SUFFIX='""' -DPSEUDO_BINDIR='"bin"' -DPSEUDO_LIBDIR='"lib/pseudo/lib"' -DPSEUDO_LOCALSTATEDIR='"var/pseudo"' -DPSEUDO_VERSION='"1.5.1"' -DUSE_MEMORY_DB -DPSEUDO_FORCE_ASYNC -O2 -g -L/home/lpapp/Projects/poky/build/tmp/sysroots/x86_64-linux/usr/lib -I/home/lpapp/Projects/poky/build/tmp/sysroots/x86_64-linux/usr/include  pseudo_ipc.c
| gcc -isystem/home/lpapp/Projects/poky/build/tmp/sysroots/x86_64-linux/usr/include -O2 -pipe -pipe -std=gnu99 -Wall -W -Wextra -fPIC -D_LARGEFILE64_SOURCE -D_ATFILE_SOURCE -m32 -DPSEUDO_PREFIX='"/home/lpapp/Projects/poky/build/tmp/sysroots/x86_64-linux/usr"' -DPSEUDO_SUFFIX='""' -DPSEUDO_BINDIR='"bin"' -DPSEUDO_LIBDIR='"lib/pseudo/lib"' -DPSEUDO_LOCALSTATEDIR='"var/pseudo"' -DPSEUDO_VERSION='"1.5.1"' -DUSE_MEMORY_DB -DPSEUDO_FORCE_ASYNC -O2 -g -L/home/lpapp/Projects/poky/build/tmp/sysroots/x86_64-linux/usr/lib -I/home/lpapp/Projects/poky/build/tmp/sysroots/x86_64-linux/usr/include  -c -o pseudo_tables.o pseudo_tables.c
| gcc -c -isystem/home/lpapp/Projects/poky/build/tmp/sysroots/x86_64-linux/usr/include -isystem/home/lpapp/Projects/poky/build/tmp/sysroots/x86_64-linux/usr/include -O2 -pipe -pipe -std=gnu99 -Wall -W -Wextra -fPIC -D_LARGEFILE64_SOURCE -D_ATFILE_SOURCE -m32 -DPSEUDO_PREFIX='"/home/lpapp/Projects/poky/build/tmp/sysroots/x86_64-linux/usr"' -DPSEUDO_SUFFIX='""' -DPSEUDO_BINDIR='"bin"' -DPSEUDO_LIBDIR='"lib/pseudo/lib"' -DPSEUDO_LOCALSTATEDIR='"var/pseudo"' -DPSEUDO_VERSION='"1.5.1"' -DUSE_MEMORY_DB -DPSEUDO_FORCE_ASYNC -O2 -g -L/home/lpapp/Projects/poky/build/tmp/sysroots/x86_64-linux/usr/lib -I/home/lpapp/Projects/poky/build/tmp/sysroots/x86_64-linux/usr/include  pseudo_util.c
| gcc -isystem/home/lpapp/Projects/poky/build/tmp/sysroots/x86_64-linux/usr/include -O2 -pipe -pipe -std=gnu99 -Wall -W -Wextra -fPIC -D_LARGEFILE64_SOURCE -D_ATFILE_SOURCE -m32 -DPSEUDO_PREFIX='"/home/lpapp/Projects/poky/build/tmp/sysroots/x86_64-linux/usr"' -DPSEUDO_SUFFIX='""' -DPSEUDO_BINDIR='"bin"' -DPSEUDO_LIBDIR='"lib/pseudo/lib"' -DPSEUDO_LOCALSTATEDIR='"var/pseudo"' -DPSEUDO_VERSION='"1.5.1"' -DUSE_MEMORY_DB -DPSEUDO_FORCE_ASYNC -O2 -g -L/home/lpapp/Projects/poky/build/tmp/sysroots/x86_64-linux/usr/lib -I/home/lpapp/Projects/poky/build/tmp/sysroots/x86_64-linux/usr/include  -shared -o lib/pseudo/lib/libpseudo.so \
| 	pseudo_client.o pseudo_ipc.o \
| 	pseudo_wrappers.o pseudo_tables.o pseudo_util.o -ldl -lpthread
| /bin/ld: skipping incompatible /usr/lib/gcc/x86_64-unknown-linux-gnu/4.8.1/../../../../lib/libdl.so when searching for -ldl
| /bin/ld: skipping incompatible /usr/lib/gcc/x86_64-unknown-linux-gnu/4.8.1/../../../../lib/libdl.a when searching for -ldl
| /bin/ld: skipping incompatible /lib/../lib/libdl.so when searching for -ldl
| /bin/ld: skipping incompatible /lib/../lib/libdl.a when searching for -ldl
| /bin/ld: skipping incompatible /usr/lib/../lib/libdl.so when searching for -ldl
| /bin/ld: skipping incompatible /usr/lib/../lib/libdl.a when searching for -ldl
| /bin/ld: skipping incompatible /usr/lib/gcc/x86_64-unknown-linux-gnu/4.8.1/../../../libdl.so when searching for -ldl
| /bin/ld: skipping incompatible /usr/lib/gcc/x86_64-unknown-linux-gnu/4.8.1/../../../libdl.a when searching for -ldl
| /bin/ld: skipping incompatible /usr/lib/libdl.so when searching for -ldl
| /bin/ld: skipping incompatible /usr/lib/libdl.a when searching for -ldl
| /bin/ld: cannot find -ldl
| /bin/ld: skipping incompatible /usr/lib/gcc/x86_64-unknown-linux-gnu/4.8.1/../../../../lib/libpthread.so when searching for -lpthread
| /bin/ld: skipping incompatible /usr/lib/gcc/x86_64-unknown-linux-gnu/4.8.1/../../../../lib/libpthread.a when searching for -lpthread
| /bin/ld: skipping incompatible /lib/../lib/libpthread.so when searching for -lpthread
| /bin/ld: skipping incompatible /lib/../lib/libpthread.a when searching for -lpthread
| /bin/ld: skipping incompatible /usr/lib/../lib/libpthread.so when searching for -lpthread
| /bin/ld: skipping incompatible /usr/lib/../lib/libpthread.a when searching for -lpthread
| /bin/ld: skipping incompatible /usr/lib/gcc/x86_64-unknown-linux-gnu/4.8.1/../../../libpthread.so when searching for -lpthread
| /bin/ld: skipping incompatible /usr/lib/gcc/x86_64-unknown-linux-gnu/4.8.1/../../../libpthread.a when searching for -lpthread
| /bin/ld: skipping incompatible /usr/lib/libpthread.so when searching for -lpthread
| /bin/ld: skipping incompatible /usr/lib/libpthread.a when searching for -lpthread
| /bin/ld: cannot find -lpthread
| /bin/ld: skipping incompatible /usr/lib/gcc/x86_64-unknown-linux-gnu/4.8.1/libgcc.a when searching for -lgcc
| /bin/ld: cannot find -lgcc
| /bin/ld: skipping incompatible /usr/lib/gcc/x86_64-unknown-linux-gnu/4.8.1/../../../../lib/libgcc_s.so when searching for -lgcc_s
| /bin/ld: skipping incompatible /lib/../lib/libgcc_s.so when searching for -lgcc_s
| /bin/ld: skipping incompatible /usr/lib/../lib/libgcc_s.so when searching for -lgcc_s
| /bin/ld: skipping incompatible /usr/lib/gcc/x86_64-unknown-linux-gnu/4.8.1/../../../libgcc_s.so when searching for -lgcc_s
| /bin/ld: skipping incompatible /usr/lib/libgcc_s.so when searching for -lgcc_s
| /bin/ld: cannot find -lgcc_s
| /bin/ld: skipping incompatible /usr/lib/gcc/x86_64-unknown-linux-gnu/4.8.1/../../../../lib/libc.so when searching for -lc
| /bin/ld: skipping incompatible /usr/lib/gcc/x86_64-unknown-linux-gnu/4.8.1/../../../../lib/libc.a when searching for -lc
| /bin/ld: skipping incompatible /lib/../lib/libc.so when searching for -lc
| /bin/ld: skipping incompatible /lib/../lib/libc.a when searching for -lc
| /bin/ld: skipping incompatible /usr/lib/../lib/libc.so when searching for -lc
| /bin/ld: skipping incompatible /usr/lib/../lib/libc.a when searching for -lc
| /bin/ld: skipping incompatible /usr/lib/gcc/x86_64-unknown-linux-gnu/4.8.1/../../../libc.so when searching for -lc
| /bin/ld: skipping incompatible /usr/lib/gcc/x86_64-unknown-linux-gnu/4.8.1/../../../libc.a when searching for -lc
| /bin/ld: skipping incompatible /usr/lib/libc.so when searching for -lc
| /bin/ld: skipping incompatible /usr/lib/libc.a when searching for -lc
| /bin/ld: cannot find -lc
| /bin/ld: skipping incompatible /usr/lib/gcc/x86_64-unknown-linux-gnu/4.8.1/libgcc.a when searching for -lgcc
| /bin/ld: cannot find -lgcc
| /bin/ld: skipping incompatible /usr/lib/gcc/x86_64-unknown-linux-gnu/4.8.1/../../../../lib/libgcc_s.so when searching for -lgcc_s
| /bin/ld: skipping incompatible /lib/../lib/libgcc_s.so when searching for -lgcc_s
| /bin/ld: skipping incompatible /usr/lib/../lib/libgcc_s.so when searching for -lgcc_s
| /bin/ld: skipping incompatible /usr/lib/gcc/x86_64-unknown-linux-gnu/4.8.1/../../../libgcc_s.so when searching for -lgcc_s
| /bin/ld: skipping incompatible /usr/lib/libgcc_s.so when searching for -lgcc_s
| /bin/ld: cannot find -lgcc_s
| collect2: error: ld returned 1 exit status
| make: *** [lib/pseudo/lib/libpseudo.so] Error 1
| ERROR: oe_runmake failed
| ERROR: Function failed: do_compile (log file is located at /home/lpapp/Projects/poky/build/tmp/work/x86_64-linux/pseudo-native/1.5.1-r3/temp/log.do_compile.2120)
ERROR: Task 66 (virtual:native:/home/lpapp/Projects/poky/meta/recipes-devtools/pseudo/pseudo_1.5.1.bb, do_compile) failed with exit code '1'
NOTE: Tasks Summary: Attempted 132 tasks of which 37 didn't need to be rerun and 1 failed.
No currently running tasks (131 of 1617)

Summary: 1 task failed:
  virtual:native:/home/lpapp/Projects/poky/meta/recipes-devtools/pseudo/pseudo_1.5.1.bb, do_compile
Summary: There was 1 WARNING message shown.
Summary: There was 1 ERROR message shown, returning a non-zero exit code.
Comment 1 Laszlo Papp 2013-07-25 20:40:58 UTC
Is libpseudo really a blocker for building a distribution? I mean, there are so many distributions out there without it embedded into the core process, and they work.

I believe the rationale would be to tell libpseudo to look into /usr/lib32 and /lib32 rather than /usr/lib and /lib.

Is that possible somehow?
Comment 2 Jonathan Liu 2013-07-26 00:43:09 UTC
It looks like you need to set EXTERNAL_TOOLCHAIN to the path of your Sourcery cross compiler toolchain in conf/local.conf. See usage & instuctions here: https://github.com/MentorEmbedded/meta-sourcery
Comment 3 Laszlo Papp 2013-07-26 01:25:19 UTC
As you can see in the build configuration output, it is already set.
Comment 4 Seebs 2013-07-26 01:58:55 UTC
This is a red herring. All the warnings about skipping incompatible libraries are irrelevant. You can easily verify that these warnings occur normally when specifying an incompatible (and thus irrelevant) directory with -L, and they have no effect, because -L adds on to the basic compiler search path.

$ gcc -L/usr/lib32 -m64 -o h hello.c
/usr/bin/ld: skipping incompatible /usr/lib32/libc.so when searching for -lc
/usr/bin/ld: skipping incompatible /usr/lib32/libc.a when searching for -lc

This link *succeeded*.

So the real problem isn't all the incompatible stuff. It's the THIRD line:

| /bin/ld: skipping incompatible /usr/lib/libpthread.so when searching for -lpthread
| /bin/ld: skipping incompatible /usr/lib/libpthread.a when searching for -lpthread
| /bin/ld: cannot find -lpthread

If there were a suitable -lpthread wherever the 32-bit compiler looks for libraries, it wouldn't produce that third line.

The fact is, we run on many machines which use /usr/lib32 for 32-bit libraries, and /usr/lib for 64-bit libraries, and they all work. Well, that is. They all work *when the development packages for those libraries are also installed*, not just the 32-bit runtimes.
Comment 5 Laszlo Papp 2013-07-26 02:02:15 UTC
No no, it is not related in any way to -L/usb/lib32 as I tested. Besides, you apparently do not know Archlinux. As for what is relevant to you is the fact that Archlinux does not differentiate between development packages and runtime as you seem to think.

Not to mention, pthread is coming with glibc32 if which was not installed, there would be a lot more trouble to it.
Comment 6 Laszlo Papp 2013-07-26 02:26:44 UTC
I have just double checked again, and "libpthread" is definitely a red herring.

Actually, the offending lines were what I suggested, and they can be fixed apparently by intalling gcc-multilib on Arch.

In any case, the tools should warn me if I do not have something necessary installed rather than leaving us in the dark. That would have probably saved a lot of time for at least 3-4 people today...

Anyway, it is 03:30 am'ish here, and I should have better than things to do than Yocto thingie, so see you tomorrow, good night.
Comment 7 Laszlo Papp 2013-07-26 02:28:19 UTC
Please do not close the bug again, the conclusion is that the error diagnostics has to be improved.

Let me know if you have anything to test for that.
Comment 8 Laszlo Papp 2013-07-26 02:47:29 UTC
Please check 4921 for another blocker failure. :(
Comment 9 Seebs 2013-07-26 15:49:44 UTC
That's really a pretty solidly different bug. You spent several hours yesterday insisting that the compiler was looking in the wrong directory. It wasn't. Arch Linux's wiki says to install gcc-multilib to build both 64-bit and 32-bit code.

This bug report was on the Yocto build system "looking in the wrong directory" and not correctly handling systems which use lib32. That doesn't happen, and is not the problem.

The concern that the system does not give useful diagnostics in the case where the host system is misconfigured and can't build 32-bit binaries is a legitimate concern, but it's not the same as the issue originally reported here.

Leaving this open, but documentation bugs like this are usually a lower priority, as we sort of assume the user can discern what's happening from the compiler error messges.
Comment 10 Laszlo Papp 2013-07-26 15:51:41 UTC
Yet, you could not without a _full_ day investigation of you, both.

Good luck to the next person having the same issue.
Comment 11 Laszlo Papp 2013-07-26 21:03:43 UTC
It is not necessarily a documentation bug. I wonder if it was possible to give a descriptive warning or even error on the command line.
Comment 12 Ross Burton 2013-09-26 09:37:28 UTC
FWIW, stock oe-core doesn't have this problem.  meta-sourcery sets NO32LIBS[1] as it only has a 32-bit toolchain.

[1] https://github.com/MentorEmbedded/meta-sourcery/blob/master/conf/distro/include/tcmode-external-sourcery.inc
Comment 13 Ross Burton 2013-09-26 10:08:16 UTC
I attempted to replicate this on my Debian box (64-bit host, no 32-bit toolchain) and noticed that the log has the following warning right at the top:

Log data follows:
| DEBUG: Executing shell function do_compile
| WARNING: Can't find stubs-32.h, but usually need it to build 32-bit libpseudo.
| If the build fails, install 32-bit developer packages.
| If you are using 32-bit binaries, the 32-bit libpseudo is NOT optional.

Laszlo, would you be able to remove gcc-multilib and rebuild pseudo-native to see if this warning is in the log.do_compile?  If it isn't then we could extend the sanity test, otherwise maybe the notice needs to be more obvious (repeated at the end on failure, for example).
Comment 14 Laszlo Papp 2013-09-26 10:10:12 UTC
Thanks.

Unfortunately, my current tasks are not related to Yocto, so it is unlikely I can put energy into helping with this.

Apologies for that.
Comment 15 Ross Burton 2013-09-26 10:42:51 UTC
Patch series to repeat the warning at the end of the log just sent, so it's visible on screen in the summary.  If it's discovered that the basic sanity test in maybe_make32() is passing in the configuration that caused this bug the test can be extended.
Comment 16 Ross Burton 2013-09-26 16:36:40 UTC
Merged into oe-core in 51950fcbe4b98bdbb8b3dde88a8729e540d9609f.