Bug 417 - gcc compile in sdk image failed with qemuppc
Summary: gcc compile in sdk image failed with qemuppc
Status: RESOLVED DUPLICATE of bug 780
Alias: None
Product: Meta-yocto
Classification: Build System, Metadata & Runtime
Component: meta-yocto (show other bugs)
Version: 1.0
Hardware: x86 ppc
: Medium normal
Target Milestone: 1.0 M3
Assignee: Dexuan Cui
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2010-10-08 23:36 UTC by xianchao zhang
Modified: 2011-02-28 22:32 UTC (History)
13 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description xianchao zhang 2010-10-08 23:36:35 UTC
Testing Env:    
Host: Ubuntu 10.04 LTS
Qemuppc images: sdk from nightly build on autobuilder,
http://autobuilder.pokylinux.org/nightly/20101007-1/

Steps:

1. download the sdk image from autobuilder
2. boot up qemuppc with sdk image
3. run command 'gcc test.c -o test', there will arise the following error:

root@qemuppc:/opt/my_test# gcc test.c -o test
In file included from /usr/include/stdio.h:75:0,
                 from test.c:1:
/usr/include/libio.h:338:1: internal compiler error: Segmentation fault
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://gcc.gnu.org/bugs.html> for instructions.
Comment 1 Ke Yu 2010-10-10 22:59:39 UTC
Dexuan is investigate it.
Comment 2 Dexuan Cui 2010-10-11 00:04:25 UTC
The bug is somewhat weird: it doesn't always happen every time:

root@qemuppc:~# cat mytest.c
int main() { return 0; }

root@qemuppc:~# gcc mytest.c
root@qemuppc:~# gcc mytest.c
root@qemuppc:~# gcc mytest.c
root@qemuppc:~# gcc mytest.c
<built-in>:0:0: internal compiler error: Segmentation fault
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://gcc.gnu.org/bugs.html> for instructions.
root@qemuppc:~# gcc mytest.c
<built-in>:0:0: internal compiler error: Segmentation fault
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://gcc.gnu.org/bugs.html> for instructions.
root@qemuppc:~# gcc mytest.c
root@qemuppc:~# gcc mytest.c
root@qemuppc:~# gcc mytest.c
root@qemuppc:~# gcc mytest.c
root@qemuppc:~# gcc mytest.c
<built-in>:0:0: internal compiler error: Segmentation fault
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://gcc.gnu.org/bugs.html> for instructions.
root@qemuppc:~# gcc mytest.c
<built-in>:0:0: internal compiler error: Segmentation fault
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://gcc.gnu.org/bugs.html> for instructions.
root@qemuppc:~# gcc mytest.c
root@qemuppc:~# gcc mytest.c
<built-in>:0:0: internal compiler error: Segmentation fault
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://gcc.gnu.org/bugs.html> for instructions.
root@qemuppc:~# gcc mytest.c
<built-in>:0:0: internal compiler error: Segmentation fault
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://gcc.gnu.org/bugs.html> for instructions.
root@qemuppc:~# gcc mytest.c
root@qemuppc:~# gcc mytest.c
<built-in>:0:0: internal compiler error: Segmentation fault
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://gcc.gnu.org/bugs.html> for instructions.
root@qemuppc:~# gcc mytest.c
root@qemuppc:~# gcc mytest.c
root@qemuppc:~# gcc mytest.c
<built-in>:0:0: internal compiler error: Segmentation fault
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://gcc.gnu.org/bugs.html> for instructions.
root@qemuppc:~# gcc mytest.c
Comment 3 Dexuan Cui 2010-10-11 01:34:07 UTC
(In reply to comment #2)
> The bug is somewhat weird: it doesn't always happen every time:
> root@qemuppc:~# cat mytest.c
> int main() { return 0; }

Actually it's the c preprocessor cpp that gets the SegFault (not every time):

root@qemuppc:~# echo -n > k.c
root@qemuppc:~# /usr/bin/powerpc-poky-linux-cpp  k.c
# 1 "k.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 1 "k.c"
root@qemuppc:~# /usr/bin/powerpc-poky-linux-cpp  k.c
# 1 "k.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 1 "k.c"
root@qemuppc:~# /usr/bin/powerpc-poky-linux-cpp  k.c
# 1 "k.c"
<built-in>:0:0: internal compiler error: Segmentation fault
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://gcc.gnu.org/bugs.html> for instructions.

So this is definitely a gcc's issue.

Dongxiao saild on mips when he upgraded gcc to 4.5, he didn't see the bug. So looks like something later causes the bug.
I'm trying to locate the culprit commit.
Comment 4 Dexuan Cui 2010-10-11 22:30:57 UTC
(In reply to comment #3)
> (In reply to comment #2)
> > The bug is somewhat weird: it doesn't always happen every time:
> > root@qemuppc:~# cat mytest.c
> > int main() { return 0; }
> Actually it's the c preprocessor cpp that gets the SegFault (not every time):
> root@qemuppc:~# echo -n > k.c
> root@qemuppc:~# /usr/bin/powerpc-poky-linux-cpp  k.c
> # 1 "k.c"
> # 1 "<built-in>"
> # 1 "<command-line>"
> # 1 "k.c"
> root@qemuppc:~# /usr/bin/powerpc-poky-linux-cpp  k.c
> # 1 "k.c"
> # 1 "<built-in>"
> # 1 "<command-line>"
> # 1 "k.c"
> root@qemuppc:~# /usr/bin/powerpc-poky-linux-cpp  k.c
> # 1 "k.c"
> <built-in>:0:0: internal compiler error: Segmentation fault
> Please submit a full bug report,
> with preprocessed source if appropriate.
> See <http://gcc.gnu.org/bugs.html> for instructions.
> So this is definitely a gcc's issue.
> Dongxiao saild on mips when he upgraded gcc to 4.5, he didn't see the bug. So
> looks like something later causes the bug.
> I'm trying to locate the culprit commit.

Using the nightly images, I can narrow down to commits of Oct 6:
http://autobuilder.pokylinux.org/nightly/20101005-1/qemu/qemuppc/ is OK, but
http://autobuilder.pokylinux.org/nightly/20101006-1/qemu/qemuppc/ is NOT ok.

I looked at the commits of Oct 5 and Oct6, but didn't find a culprit commit. :-(
Comment 5 Dexuan Cui 2010-10-12 01:26:48 UTC
(In reply to comment #4)
> Using the nightly images, I can narrow down to commits of Oct 6:
> http://autobuilder.pokylinux.org/nightly/20101005-1/qemu/qemuppc/ is OK, but
> http://autobuilder.pokylinux.org/nightly/20101006-1/qemu/qemuppc/ is NOT ok.
> I looked at the commits of Oct 5 and Oct6, but didn't find a culprit commit.
A surprising thing is: 
http://autobuilder.pokylinux.org/nightly/20101003-1/qemu/qemuppc/is also NOT ok -- can't understand why Oct05 is ok... looks there is some kind of tricky thing?

I tried gdb inside the target qempppc, but what's odd is: gdb can't catch the " Segmentation fault" the target ppc's cpp reports.
strace also doesn't show useful info.

gcc 4.5 was upgraded for ppc on Sep 22. I had a look at all the commits after  that, but didn't find an obvious suspecct. glibc->eglibc might be one and I'll be trying that.

BTW: I also looked at gcc's bugzilla and didn't find similar bugs.

Currently I'm buiding qemuppc based on today's latest master and hope I can reproduce the issue on it and try to get new findings.

Could somebody give some hints? Thanks!
Comment 6 Dexuan Cui 2010-10-12 17:37:15 UTC
(In reply to comment #5)
> (In reply to comment #4)
> > Using the nightly images, I can narrow down to commits of Oct 6:
> > http://autobuilder.pokylinux.org/nightly/20101005-1/qemu/qemuppc/ is OK, but
> > http://autobuilder.pokylinux.org/nightly/20101006-1/qemu/qemuppc/ is NOT ok.
> > I looked at the commits of Oct 5 and Oct6, but didn't find a culprit commit.
> A surprising thing is: 
> http://autobuilder.pokylinux.org/nightly/20101003-1/qemu/qemuppc/is also NOT ok
> -- can't understand why Oct05 is ok... looks there is some kind of tricky
> thing?
> I tried gdb inside the target qempppc, but what's odd is: gdb can't catch the "
> Segmentation fault" the target ppc's cpp reports.
> strace also doesn't show useful info.
> gcc 4.5 was upgraded for ppc on Sep 22. I had a look at all the commits after 
> that, but didn't find an obvious suspecct. glibc->eglibc might be one and I'll
> be trying that.
> BTW: I also looked at gcc's bugzilla and didn't find similar bugs.
> Currently I'm buiding qemuppc based on today's latest master and hope I can
> reproduce the issue on it and try to get new findings.
Unfortunately, I can NOT reproduce the issue on (yesterday's) latest master (commit 04ace838). :-(  Looks I'll have to try to further look into the old "buggy" nightly images.


Thanks,
-- Dexuan
Comment 7 xianchao zhang 2010-10-12 21:18:42 UTC
(In reply to comment #0)
> Testing Env:    
> Host: Ubuntu 10.04 LTS
> Qemuppc images: sdk from nightly build on autobuilder,
> http://autobuilder.pokylinux.org/nightly/20101007-1/
> 
> Steps:
> 
> 1. download the sdk image from autobuilder
> 2. boot up qemuppc with sdk image
> 3. run command 'gcc test.c -o test', there will arise the following error:
> 
> root@qemuppc:/opt/my_test# gcc test.c -o test
> In file included from /usr/include/stdio.h:75:0,
>                  from test.c:1:
> /usr/include/libio.h:338:1: internal compiler error: Segmentation fault
> Please submit a full bug report,
> with preprocessed source if appropriate.
> See <http://gcc.gnu.org/bugs.html> for instructions.

With the milestone build 10/11, this issue has not arise. seems fixed?
Comment 8 Jessica 2010-10-12 21:22:06 UTC
It's not fixed but as Dexuan mentioned can't be reproduced with the latest build.
Comment 9 Dexuan Cui 2010-10-13 20:21:45 UTC
(In reply to comment #6)
> (In reply to comment #5)
> > (In reply to comment #4)
> > > Using the nightly images, I can narrow down to commits of Oct 6:
> > > http://autobuilder.pokylinux.org/nightly/20101005-1/qemu/qemuppc/ is OK, but
> > > http://autobuilder.pokylinux.org/nightly/20101006-1/qemu/qemuppc/ is NOT ok.
> > > I looked at the commits of Oct 5 and Oct6, but didn't find a culprit commit.
> > A surprising thing is: 
> > http://autobuilder.pokylinux.org/nightly/20101003-1/qemu/qemuppc/is also NOT ok
> > -- can't understand why Oct05 is ok... looks there is some kind of tricky
> > thing?
> > I tried gdb inside the target qempppc, but what's odd is: gdb can't catch the "
> > Segmentation fault" the target ppc's cpp reports.
> > strace also doesn't show useful info.
> > gcc 4.5 was upgraded for ppc on Sep 22. I had a look at all the commits after 
> > that, but didn't find an obvious suspecct. glibc->eglibc might be one and I'll
> > be trying that.
> > BTW: I also looked at gcc's bugzilla and didn't find similar bugs.
> > Currently I'm buiding qemuppc based on today's latest master and hope I can
> > reproduce the issue on it and try to get new findings.
> Unfortunately, I can NOT reproduce the issue on (yesterday's) latest master
> (commit 04ace838). :-(  Looks I'll have to try to further look into the old
> "buggy" nightly images.

With Qing's great help (previously gdb can't catch the SIGSEGV of cpp, Qing uses the LD_PRELOAD trick to work this around and can catch the signal now -- my understanding is: Qing re-sets the signal handling of the signal in an own .so), we managerd to use gdb to get the place where the SegFault happens:

Loaded symbols for /lib/ld.so.1
Core was generated by `/usr/libexec/gcc/powerpc-poky-linux/4.5.0/cc1 -E -quiet -D__unix__ -D__gnu_linu'.
Program terminated with signal 11, Segmentation fault.
#0  0x1099de88 in ?? ()
(gdb) bt
#0  0x1099de88 in ?? ()
#1  0x101bc3b4 in mul_double_with_sign ()
#2  0x101c055c in int_const_binop ()

Due to the lackness of furthe debug info, looks it's hard to further debug. :-(

disassembing the faulting IP 0x1099de88 shows ppc-specific code. Qing helped to guess a register mighe be clobbered somehow...

BTW: using my build, I can't reproduce the bug on latest master and builds/milestone.
I tried nightly images: the images of Sep 25~Oct1, Oct 5, Oct8 are ok, while Oct 3, Oct6 and Oct are bad (Unluckily autobuild didn't save enought debug info). Looks nobody met with the issue in own build...
Comment 10 Jessica 2010-10-13 20:46:45 UTC
Thanks for the update and hard work on trying to root cause the issue.  I'd say let's put the bug on the back-burner till we can reproduce it again.
Comment 11 Qing He 2010-10-13 21:02:50 UTC
(In reply to comment #9)
> previously gdb can't catch the SIGSEGV of cpp, Qing
> uses the LD_PRELOAD trick to work this around and can catch the signal now --
> my understanding is: Qing re-sets the signal handling of the signal in an own
> .so), we managerd to use gdb to get the place where the SegFault happens:

explain here a little for further reference:
cc1 (which is finally called by cpp/gcc and do most of the work) has a signal handler of its own to intercept SIGSEGV, that means it bypassed normal signal handling and there won't be a core file dumped on segfault.

The trick is to replace the signal() function (using LD_PRELOAD) so the action of setting own handler of cc1 is ignored.

---
$ cat > b.c <<EOF
typedef void (*sighandler_t)(int);
sighandler_t signal(int sig, sighandler_t handler) {return handler;}
EOF

$ gcc b.c -o b.so -shared
$ LD_PRELOAD=./b.so cpp a.c
Comment 12 Dexuan Cui 2010-10-13 21:53:06 UTC
(In reply to comment #10)
> Thanks for the update and hard work on trying to root cause the issue.  I'd say
> let's put the bug on the back-burner till we can reproduce it again.

We have asked Scott to save more debug info in the next builds.
At present let's keep a close eye on the new builds and see if autobuilder can reproduce the issue again.
Comment 13 Dexuan Cui 2010-10-20 00:38:09 UTC
(In reply to comment #12)
> (In reply to comment #10)
> > Thanks for the update and hard work on trying to root cause the issue.  I'd say
> > let's put the bug on the back-burner till we can reproduce it again.
> We have asked Scott to save more debug info in the next builds.
> At present let's keep a close eye on the new builds and see if autobuilder can
> reproduce the issue again.

I still can't reproduce the issue on recent nightly builds, like builds of Oct 11, 18:
http://autobuilder.pokylinux.org/milestone/20101011/qemu/qemuppc/
http://autobuilder.pokylinux.org/milestone/20101018/qemu/qemuppc/
So looks this bug was somehow hidden or fixed?
Comment 14 Dexuan Cui 2010-10-26 23:15:10 UTC
(In reply to comment #13)
> (In reply to comment #12)
> > (In reply to comment #10)
> > > Thanks for the update and hard work on trying to root cause the issue.  I'd say
> > > let's put the bug on the back-burner till we can reproduce it again.
> > We have asked Scott to save more debug info in the next builds.
> > At present let's keep a close eye on the new builds and see if autobuilder can
> > reproduce the issue again.
> I still can't reproduce the issue on recent nightly builds, like builds of Oct
> 11, 18:
> http://autobuilder.pokylinux.org/milestone/20101011/qemu/qemuppc/
> http://autobuilder.pokylinux.org/milestone/20101018/qemu/qemuppc/
> So looks this bug was somehow hidden or fixed?

Still can not reproduce the issue on recent builds:
http://autobuilder.pokylinux.org/milestone/20101024/qemu/qemuppc/
Comment 15 Dexuan Cui 2010-11-15 01:05:01 UTC
(In reply to comment #14)
> (In reply to comment #13)
> > (In reply to comment #12)
> > > (In reply to comment #10)
> > > > Thanks for the update and hard work on trying to root cause the issue.  I'd say
> > > > let's put the bug on the back-burner till we can reproduce it again.
> > > We have asked Scott to save more debug info in the next builds.
> > > At present let's keep a close eye on the new builds and see if autobuilder can
> > > reproduce the issue again.
> > I still can't reproduce the issue on recent nightly builds, like builds of Oct
> > 11, 18:
> > http://autobuilder.pokylinux.org/milestone/20101011/qemu/qemuppc/
> > http://autobuilder.pokylinux.org/milestone/20101018/qemu/qemuppc/
> > So looks this bug was somehow hidden or fixed?
> Still can not reproduce the issue on recent builds:
> http://autobuilder.pokylinux.org/milestone/20101024/qemu/qemuppc/

Now I can reproduce the issue in latest nightly images (about ~50% probability):
http://autobuilder.pokylinux.org/nightly/20101113-1/qemu/qemuppc/
We have to figiure it out!

BTW: looks it's related to ldconfig:
After running ldconfig in target, I can't reproduce the issue any longer.

I'm trying to figure out what causes the occasional SegFault.
Comment 16 Dexuan Cui 2011-02-24 22:54:11 UTC
I tried the images http://autobuilder.pokylinux.org/nightly/ and I can't reproduce the issue any longer.

The last time I reproduced the issue was about 3.5 months ago and since then we did many big changes to Poky. 
I think the issue might somehow become even more difficult to occur even if it hasn't been fixed. 

So let me mark the bug as WORKSFORME.
Comment 17 Dexuan Cui 2011-02-24 22:55:41 UTC
(In reply to comment #16)
> I tried the images http://autobuilder.pokylinux.org/nightly/ and I can't
> reproduce the issue any longer.
> The last time I reproduced the issue was about 3.5 months ago and since then we
> did many big changes to Poky. 
> I think the issue might somehow become even more difficult to occur even if it
> hasn't been fixed. 
> So let me mark the bug as WORKSFORME.

So let me mark the bug as WORKSFORME.
Comment 18 Dexuan Cui 2011-02-28 22:30:37 UTC
Looks we do have a rather tricky and hard-to-reproduce bug on ppc here..
Till now, I haven't reproduced it locally.
I even tried to manually build ppc on autobuilder, but still couldn't reproduce it.
However, autobuilder ppc images still have the issue *occassionally*, and with a "bad" image, I estimiate we have a 30%~50% probability to reproduce the issue.
Comment 19 Dexuan Cui 2011-02-28 22:32:49 UTC
(In reply to comment #18)
> Looks we do have a rather tricky and hard-to-reproduce bug on ppc here..
> Till now, I haven't reproduced it locally.
> I even tried to manually build ppc on autobuilder, but still couldn't reproduce
> it.
> However, autobuilder ppc images still have the issue *occassionally*, and with
> a "bad" image, I estimiate we have a 30%~50% probability to reproduce the
> issue.

Bug 780 is a duplicate of this one.

*** This bug has been marked as a duplicate of bug 780 ***