Bug 2496 - Some recipes will build fail when rebuild with different commits and arches
Summary: Some recipes will build fail when rebuild with different commits and arches
Status: VERIFIED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: core (show other bugs)
Version: unspecified
Hardware: x86 Multiple
: Medium normal
Target Milestone: 1.3 M4
Assignee: Radu Moisan
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2012-05-25 03:20 UTC by Hongna Xu
Modified: 2012-11-06 16:47 UTC (History)
13 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Hongna Xu 2012-05-25 03:20:52 UTC
Last week i git clone the poky/master tree and build meta-toolchain with 5 qemu archs, then this week i git pull all new patches and rebuild meta-toolchain, it reported eglibc-nativesdk compile failed and gcc-cross-canadian-{arch} configure failed.
Reproduce steps:
1.git clone poky/master commitID 12b4543ac9f54c0db0ee4a7ee546a71946a051f8, modify conf/local.conf, set MACHINE=qemuarm,PACKAGE_CLASSES ?= "package_ipk",SDKMACHINE ?="x86_64", then bitbake meta-toolchain.
2.After step 1 finished, change MACHINE=qemux86, then bitbake meta-toolchian.
3.Then git pull with the latest master(49694d650c61547d881a782af2a0f24dc39a042f),change MACHINE=qemuarm, also build meta-toolchain(you may build eglibc to save time), it will report eglibc compile fail and gcc-cross-canadian-arm configure fail, so i clean the sstate of thoes two recipes, then continue with meta-toolchai, it will succeed.
4.Change MACHINE=qemux86 and build meta-toolchain with the same build dir, i encounter the same error eglibc-nativesdk compile fail, but now cleaning sstate didn't work. the error info is:
************************************
| i586-poky-linux-gcc  -m32   -march=i586 --sysroot=/home/build/git-work/poky/build/tmp/sysroots/qemux86-tcbootstrap idn-stub.c -c -std=gnu99 -fgnu89-inline -O2 -Wall -Winline -Wwrite-strings -feliminate-unused-debug-types -fmerge-all-constants -g -pipe -Wstrict-prototypes   -fPIC      -I../include -I/home/build/git-work/poky/build/tmp/work/i586-poky-linux/eglibc-2.15-r10+svnr17386/build-i586-poky-linux/libidn -I/home/build/git-work/poky/build/tmp/work/i586-poky-linux/eglibc-2.15-r10+svnr17386/build-i586-poky-linux -I../sysdeps/i386/elf -I../nptl/sysdeps/unix/sysv/linux/i386/i586 -I../nptl/sysdeps/unix/sysv/linux/i386 -I../sysdeps/unix/sysv/linux/i386 -I../nptl/sysdeps/unix/sysv/linux -I../nptl/sysdeps/pthread -I../sysdeps/pthread -I../ports/sysdeps/unix/sysv/linux -I../sysdeps/unix/sysv/linux -I../sysdeps/gnu -I../sysdeps/unix/common -I../sysdeps/unix/mman -I../sysdeps/unix/inet -I../sysdeps/unix/sysv/i386 -I../nptl/sysdeps/unix/sysv -I../ports/sysdeps/unix/sysv -I../sysdeps/unix/sysv -I../sysdeps/unix/i386 -I../nptl/sysdeps/unix -I../ports/sysdeps/unix -I../sysdeps/unix -I../sysdeps/posix -I../nptl/sysdeps/i386/i586 -I../sysdeps/i386/i586 -I../sysdeps/i386/i486 -I../nptl/sysdeps/i386/i486 -I../sysdeps/i386/fpu -I../nptl/sysdeps/i386 -I../sysdeps/i386 -I../sysdeps/wordsize-32 -I../sysdeps/ieee754/ldbl-96 -I../sysdeps/ieee754/dbl-64 -I../sysdeps/ieee754/flt-32 -I../sysdeps/ieee754 -I../sysdeps/generic/elERROR: Function failed: do_compile (see /home/build/git-work/poky/build/tmp/work/i586-poky-linux/eglibc-2.15-r10+svnr17386/temp/log.do_compile.11737 for further information)
| f -I../sysdeps/generic -I../nptl -I../ports  -I.. -I../libio -I. -nostdinc -isystem /home/build/git-work/poky/build/tmp/sysroots/x86_64-linux/usr/lib/i586-poky-linux.gcc-cross-intermediate/gcc/i586-poky-linux/4.7.1/include -isystem /home/build/git-work/poky/build/tmp/sysroots/x86_64-linux/usr/lib/i586-poky-linux.gcc-cross-intermediate/gcc/i586-poky-linux/4.7.1/include-fixed -isystem /home/build/git-work/poky/build/tmp/sysroots/qemux86/usr/include -D_LIBC_REENTRANT -include ../include/libc-symbols.h  -DPIC -DSHARED     -o /home/build/git-work/poky/build/tmp/work/i586-poky-linux/eglibc-2.15-r10+svnr17386/build-i586-poky-linux/libidn/idn-stub.os -MD -MP -MF /home/build/git-work/poky/build/tmp/work/i586-poky-linux/eglibc-2.15-r10+svnr17386/build-i586-poky-linux/libidn/idn-stub.os.dt -MT /home/build/git-work/poky/build/tmp/work/i586-poky-linux/eglibc-2.15-r10+svnr17386/build-i586-poky-linux/libidn/idn-stub.os
| idn-stub.c: In function 'load_dso':
| idn-stub.c:51:11: error: 'LIBCIDN_SO' undeclared (first use in this function)
| idn-stub.c:51:11: note: each undeclared identifier is reported only once for each function it appears in
| make[2]: *** [/home/build/git-work/poky/build/tmp/work/i586-poky-linux/eglibc-2.15-r10+svnr17386/build-i586-poky-linux/libidn/idn-stub.o] Error 1
| make[2]: *** Waiting for unfinished jobs....
| idn-stub.c: In function 'load_dso':
| idn-stub.c:51:11: error: 'LIBCIDN_SO' undeclared (first use in this function)
| idn-stub.c:51:11: note: each undeclared identifier is reported only once for each function it appears in
| make[2]: *** [/home/build/git-work/poky/build/tmp/work/i586-poky-linux/eglibc-2.15-r10+svnr17386/build-i586-poky-linux/libidn/idn-stub.os] Error 1
| make[2]: Leaving directory `/home/build/git-work/poky/build/tmp/work/i586-poky-linux/eglibc-2.15-r10+svnr17386/eglibc-2_15/libc/libidn'
| make[1]: *** [libidn/subdir_lib] Error 2
| make[1]: Leaving directory `/home/build/git-work/poky/build/tmp/work/i586-poky-linux/eglibc-2.15-r10+svnr17386/eglibc-2_15/libc'
| make: *** [all] Error 2
| ERROR: oe_runmake failed
NOTE: package eglibc-2.15-r10+svnr17386: task do_compile: Failed
ERROR: Task 164 (/home/build/git-work/poky/meta/recipes-core/eglibc/eglibc_2.15.bb, do_compile) failed with exit code '1'
Waiting for 1 running tasks to finish:
0: eglibc-nativesdk-2.15-r10+svnr17386 do_compile (pid 14843)
NOTE: package eglibc-nativesdk-2.15-r10+svnr17386: task do_compile: Succeeded
NOTE: Tasks Summary: Attempted 658 tasks of which 651 didn't need to be rerun and 1 failed.

Summary: 1 task failed:
  /home/build/git-work/poky/meta/recipes-core/eglibc/eglibc_2.15.bb, do_compile
Summary: There was 1 ERROR message shown, returning a non-zero exit code.
*************************************
Comment 1 liang li 2012-05-25 10:47:18 UTC
when use newest git master(commit :"6558abda67956e4fab1304b0cdf03c65f18fabd5") to build qemuarm and qemux86 meta-toolchain in on build dir ,then change git to commit :"12b4543ac9f54c0db0ee4a7ee546a71946a051f8", and rebuild qemuarm and qemux86 meta-toolchian,the same errors are found.
--------------
when i use command "bitbake gcc-cross-canadian-i586 -ccleanall" (for qemux86) or "bitbake gcc-cross-canadian-arm -ccleanall"(for qemuarm), then  rerun"bitbake meta-toolchain",the errors are not found again and meta-toolchain can be built successfully.
Comment 2 Radu Moisan 2012-06-01 07:22:45 UTC
what's the point of tring to reproduce some behavior that's only reproducible on some specific snapshot of the repository? I have begun testing with the master branch (as of yesterday) but could not reproduce yet. I still have a build in progress - I'll update when it finshes.
Comment 3 Radu Moisan 2012-06-01 09:53:49 UTC
last build on arm failed.
What I tried then was "MACHINE=qemuarm bitbake gcc-cross-canadian -c cleanall" which failed with ERROR: Nothing PROVIDES 'gcc-cross-canadian'
Similar result I found with "MACHINE=qemux86 bitbake gcc-cross-canadian -c cleanall"
However as liang li suggested "MACHINE=qemux86 bitbake gcc-cross-canadian-i586 -c cleanall" succeds. I'll follow this lead, package name should not be arch dependent.
Comment 4 Lianhao Lu 2012-06-01 09:58:43 UTC
(In reply to comment #3)
> last build on arm failed.
> What I tried then was "MACHINE=qemuarm bitbake gcc-cross-canadian -c
> cleanall" which failed with ERROR: Nothing PROVIDES 'gcc-cross-canadian'
> Similar result I found with "MACHINE=qemux86 bitbake gcc-cross-canadian -c
> cleanall"
> However as liang li suggested "MACHINE=qemux86 bitbake
> gcc-cross-canadian-i586 -c cleanall" succeds. I'll follow this lead, package
> name should not be arch dependent.

The package name was changed to target arch dependent so that we can install both gcc-cross-canadian-i586 and gcc-cross-canadian-arm from the same repository.
Comment 5 Radu Moisan 2012-06-01 12:35:55 UTC
I've done a clean build and rebuild meta-toolchain four times for qemuarm alternatively with qemux86, and it build successfully every time.
Comment 6 Radu Moisan 2012-06-01 13:26:32 UTC
Following meta-toolchain I've run:
MACHINE=qemuarm bitbake gcc-cross-canadian-arm -c cleanall
MACHINE=qemuarm bitbake gcc-cross-canadian-arm
MACHINE=qemux86 bitbake gcc-cross-canadian-i586 -c cleanall
MACHINE=qemux86 bitbake gcc-cross-canadian-i586

without any problems.
Comment 7 Hongna Xu 2012-06-04 02:16:18 UTC
I mean that, our project should not build fail when rebuild some recipes on different commit and arches, even though it could be fixed by "cleanall" command sometimes. Because every weekly testing,I must build meta-toolchain's ipk packages for five arches with two SDKMACHINEs, it means i must build it for ten times with different configuration, cause the bug exist, i also run many times "cleanall", it cost me much time to keep an eye on it. so could we fix it? Maybe some recipe trigger it happen, we don't meet this issue before, I don't remember it begin from which milestone, it exist for several months.(we don't file it cause it can't be reproduce before).
Comment 8 Radu Moisan 2012-06-05 07:32:36 UTC
I also tried:

MACHINE=qemux86 bitbake meta-toolchian
MACHINE=qemuarm bitbake meta-toolchian
MACHINE=qemux86 bitbake meta-toolchian
MACHINE=qemuarm bitbake meta-toolchian

and still no problem. Thus the steps to repoduce this problem may be trickier then it might apper at first. However, I don't argure we should address this issue. In my opinion this bug shuld be closed, and anothor one should be raised targetting this specific problem (with an appropriate title). As this bug title is now, it reffers to eglibc, which, IMHO, I don't see it ralated to.

If I missunderstood your problem or failed to reproduce it because of various reasons please have patience with me, I'm still on the learning curve :) Please provide further information if so.
Comment 9 Radu Moisan 2012-06-05 07:35:56 UTC
waiting feedback from Hongna Xu
Comment 10 Hongna Xu 2012-06-05 08:19:55 UTC
  (In reply to comment #6)
> Following meta-toolchain I've run:
> MACHINE=qemuarm bitbake gcc-cross-canadian-arm -c cleanall
> MACHINE=qemuarm bitbake gcc-cross-canadian-arm
> MACHINE=qemux86 bitbake gcc-cross-canadian-i586 -c cleanall
> MACHINE=qemux86 bitbake gcc-cross-canadian-i586
> 
> without any problems.

There is no need to run "cleanall" in general, shall we?  because it is no need to run cleanall before to generate meta-toolchain tarballs. meanwhile, it consume me much time to pay an eye on it, cause i must build for ten times meta-toolchain ervery building. It is also for robustness for our project.
Comment 11 Hongna Xu 2012-06-05 08:24:27 UTC
(In reply to comment #8)
> I also tried:
> 
> MACHINE=qemux86 bitbake meta-toolchian
> MACHINE=qemuarm bitbake meta-toolchian
> MACHINE=qemux86 bitbake meta-toolchian
> MACHINE=qemuarm bitbake meta-toolchian
> 
> and still no problem. Thus the steps to repoduce this problem may be
> trickier then it might apper at first. However, I don't argure we should
> address this issue. In my opinion this bug shuld be closed, and anothor one
> should be raised targetting this specific problem (with an appropriate
> title). As this bug title is now, it reffers to eglibc, which, IMHO, I don't
> see it ralated to.
> 
> If I missunderstood your problem or failed to reproduce it because of
> various reasons please have patience with me, I'm still on the learning
> curve :) Please provide further information if so.

I mean build on different master commits. Ok, Let's lay it aside for one week or more, please keep your build environment, a few days later, git pull the patches and in the same build dir to build the meta-toolchain.
Comment 12 Jiajun Xu 2012-06-05 13:12:50 UTC
(In reply to comment #6)
> Following meta-toolchain I've run:
MACHINE=qemuarm bitbake
> gcc-cross-canadian-arm -c cleanall
MACHINE=qemuarm bitbake
> gcc-cross-canadian-arm
MACHINE=qemux86 bitbake gcc-cross-canadian-i586 -c
> cleanall
MACHINE=qemux86 bitbake gcc-cross-canadian-i586

without any
> problems.

Hi Radu,
I think you could follow the steps Hongna mentioned in bug description to reproduce the bug. The issue is that when you have a build environment with old commit and then build latest commit under the same build environment, you will meet unexpected error.

Reproduce steps:
1.git clone poky/master commitID 12b4543ac9f54c0db0ee4a7ee546a71946a051f8, modify conf/local.conf, set MACHINE=qemuarm,PACKAGE_CLASSES ?= "package_ipk",SDKMACHINE ?="x86_64", then bitbake meta-toolchain.

2.After step 1 finished, change MACHINE=qemux86, then bitbake meta-toolchian.

3.Then git pull with the latest master(49694d650c61547d881a782af2a0f24dc39a042f),change MACHINE=qemuarm, also build meta-toolchain(you may build eglibc to save time), it will report eglibc compile fail and gcc-cross-canadian-arm configure fail, so i clean the sstate of thoes two recipes, then continue with meta-toolchai, it will succeed.

4.Change MACHINE=qemux86 and build meta-toolchain with the same build dir, i encounter the same error eglibc-nativesdk compile fail
Comment 13 Radu Moisan 2012-06-05 13:34:15 UTC
ok, I understand now the scenario, the key part is to have different commits in my tree. Although I understand your need for a fix for this asap, is seems to me more like a build system bug and a little far from my current competency. I will, however, get in sync with Alex and Richard about this issue to see how will we proceed. For now I will postpone it until a decision is made.
Comment 14 Alexandru Georgescu 2012-08-16 10:16:58 UTC
Hi Radu,
Is there any update on this?

Regards,
Alex.
Comment 15 Radu Moisan 2012-08-16 11:09:39 UTC
(In reply to comment #14)
> Hi Radu,
> Is there any update on this?
> 
> Regards,
> Alex.

no updates.
Comment 16 Richard Purdie 2012-09-24 12:34:07 UTC
It will not be possible to reproduce this particular bug with master after revision: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=8b9b1ab42e6e3226fc06818ec3dd186b6f81acad (since eglibc's build directory will get cleaned) so I'm marking this as resolved whenever the sstate checksum changes.

If someone can reproduce failures after that change please repoen the issue.
Comment 17 Laurentiu Serban 2012-11-06 16:47:54 UTC
Setting this to verified for danny