Bug 2074

(In reply to comment #0)
> I attempted to reproduce this failure with the current Poky master.  I did NOT
> receive a failure following the steps that lead to one in Edison.  However, I
> got a different failure.  I don't believe it's related, but will include it and
> the steps here just in case it is.

Ignore the first paragraph, that was supposed to be part of a different bug report.

I have tried to reproduce this using 2 users on the same machine with a sstate-cache and then using SSTATE_MIRROR, using own-mirrors and PREMIRROR setup.

I build core-image-minimal for qemuppc as first user, then built it again as
the second user, both of these completed correctly, not except the final image build duplicated, everything used sstate.

I then built psplash in the second user's build area and it built correctly.  

In a different set of build areas I attempted to do the same thing with a patch that Khem did to remove the local-prefix, these also build correctly with no failures.

I wanted to confirm if you are using any toolchain builds, I need to test that next.

I did see one build that had the linux-yocto kernel did a fetch instead of using sstate, still need to track down what happened there (see seperate thread)

The failure is due to the toolchain still looking in the previous config/install location for files.

The error does not occur if the 2nd user has permissions to look at the target directory area.  If the files exist, they will be used -- if they don't exit the compiler will ignore that location.

The error only exists when the 2nd user does -not- have permissions to look into the user 1 directory structure.  (Thats the cause of the permission denied message, it's still looking in the old location and EPERM is a failure.. but it existing or not isn't.)

There was a related defect to this one that has the exact steps required to do the build and replicate the problem.

I re-tried this with the removal of the build user1's tmp directory (sgw_ab/poky/build).  this is done as 2 different users.

My retry also succeeded both the core-image-minimal and the psplash build
I confirmed the compiler did not have permission to read the user1 sysroot
directory since it was removed, I also verified that the compiler was looking
for user1's sysroot via strings and strace.

I actually see it trying this pattern:

19909 stat("/srv/ssd/sgw_ab/sgw1_build/poky/build/tmp/sysroots/qemuppc/srv/ssd/sgw_ab/poky/build/tmp/sysroots/qemuppc/usr/include", 0x7fffe3113c60) = -1 ENOENT (No such file or directory)

It does ultimately find it correctly under sgw1_build/poky/build/tmp/sysroots/qemuppc/... here:

19909 stat("/srv/ssd/sgw_ab/sgw1_build/poky/build/tmp/sysroots/qemuppc/usr/include", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0


Hints on getting a good reproducer are always welcome!

http://git.openembedded.org/openembedded-core-contrib/commit/?h=kraj/gcc-4.7&id=2275888181dc1892cf130c0fcf67be182c19b2d5

should help

to reporoduce make usr1 dirs readable for usr1 only such that other can not even read it. the rebuild from sstate using usr2

Note that I think this issue only occurs if the files exist but are unreadable. If they don't exist before, that hits different error code paths and has different behaviour.

As Khem mentioned, you have to keep the old directory but make it unreadable to the second user.

-ENOENT goes through a different error path then the -ENOPERM.  The later causes the failure condition.

Bug 2082 has the -exact- steps to reproduce this issue.  The (proposed) test case was shown to reproduce the problem on other peoples systems.

The local-prefix patch did -not- resolve the problem for me.

(I have not tested this within the last couple of weeks, so another patch may have resolved it.)

(In reply to comment #8)
> Bug 2082 has the -exact- steps to reproduce this issue.  The (proposed) test
> case was shown to reproduce the problem on other peoples systems.
> 
> The local-prefix patch did -not- resolve the problem for me.
> 

I presumed so since we also use --with-headers option. there could be other
static references that I have not run into yet. I am working on cleaning
this up but its turning out to be a big pile of changes. Even if it works
it will be too much of a change for 1.2

> (I have not tested this within the last couple of weeks, so another patch may
> have resolved it.)

Removing the with-local-prefix only appeared to stop the cross compiler from looking in:

<path>/sysroot/.../usr/local/include

I'm not sure that removing that from the cross compiler is even a good idea.  Since some folks may want to populate the local/include  (even though we should never be at the system level)

The -EPERM was happening to me on the sysroot/.../usr/include and later on the sysroot/.../lib and usr/lib directories..  so every time it checked if a directory existed it failed..

In the case where something doesn't exist it falls back and appears to replace the first path of the path with the sysroot information that was passed in -- thus allowing the access of the directories.  It really looks to me like there is some kind of bug (feautre?) where it ignores the --sysroot parameter passed to glibc and tries to use the original locations first, then evaluates the sysroot..

I was unable to reproduce this on CentOS with a mips target, but was able to replicate this on an FC13 system w/ an x86_64 target.

See: http://gate.crashing.org/~fray/yocto/psplash-strace.log.bz2 (8 MB) log of information from the FC13 system.

Centos Works for both users

http://gate.crashing.org/~fray/yocto/centos-out.bz2

runs of just the configure step:


http://gate.crashing.org/~fray/yocto/fc13-configure-out.bz2
http://gate.crashing.org/~fray/yocto/centos-configure-out.bz2

(In reply to comment #13)
> runs of just the configure step:
> 

Correction:

> http://gate.crashing.org/~fray/yocto/fc13-configure-out.bz2
> http://gate.crashing.org/~fray/yocto/centos62-configure-out.bz2

From the logs, the 

execve("/home/usr2/oe-core/build-ia32-2/tmp-eglibc/sysroots/x86_64-linux/usr/bin/x86_64-oe-linux/../../libexec/x86_64-oe-linux/gcc/x86_64-oe-linux/4.6.4/cc1", ["/home/usr2/oe-core/build-ia32-2/tmp-eglibc/sysroots/x86_64-linux/usr/bin/x86_64-oe-linux/../../libexec/x86_64-oe-linux/gcc/x86_64-oe-linux/4.6.4/cc1", "-fpreprocessed", "/home/usr2/oe-core/build-ia32-2/tmp-eglibc/ccache/x86_64-oe-linux/psplash/conftest.tmp.msp-mhatle-lx2.6733.i", "-quiet", "-dumpbase", "conftest.tmp.msp-mhatle-lx2.6733.i", "-m64", "-mtune=generic", "-march=x86-64", "-auxbase-strip", "/home/usr2/oe-core/build-ia32-2/tmp-eglibc/ccache/x86_64-oe-linux/psplash/tmp.hash.msp-mhatle-lx2.6733.o", "-g", "-O2", "-feliminate-unused-debug-types", "-o", "-"] 

made me suspicious contrasted to 

execve("/home/usr2/oe-core/build-ia32-2/tmp-eglibc/sysroots/x86_64-linux/usr/bin/x86_64-oe-linux/../../libexec/x86_64-oe-linux/gcc/x86_64-oe-linux/4.6.4/cc1", ["/home/usr2/oe-core/build-ia32-2/tmp-eglibc/sysroots/x86_64-linux/usr/bin/x86_64-oe-linux/../../libexec/x86_64-oe-linux/gcc/x86_64-oe-linux/4.6.4/cc1", "-quiet", "-iprefix", "/home/usr2/oe-core/build-ia32-2/tmp-eglibc/sysroots/x86_64-linux/usr/bin/x86_64-oe-linux/../../lib/x86_64-oe-linux/gcc/x86_64-oe-linux/4.6.4/", "-isysroot", "/home/usr2/oe-core/build-ia32-2/tmp-eglibc/sysroots/qemux86-64", "conftest.c", "-quiet", "-dumpbase", "conftest.c", "-m64", "-mtune=generic", "-march=x86-64", "-auxbase", "conftest", "-g", "-O2", "-feliminate-unused-debug-types", "-o", "-"]

Mark confirmed that removing ccache causes the configure to complete successfully. Its a start to know where to look at least and starts to explain the behavior difference.

I have a new way to reproduce what I think is the same issue on Ubuntu 11.10 64 bit. I ran a build in directory A, machine qemux86 with a sstate cache generated.

I then did "sudo apt-get install ccache", chmod a-r and chown root.root on the tmp/sysroots/qemux86 directory for build A. 

I then setup a new build directory, B for machine qemux86, pointing at build A's sstate cache.

bitbake psplash -c cleansstate; bitbake psplash leads to a failure in eglibc-initial, "cc1: error: /media/build1/poky/build/tmp/sysroots/qemux86/usr/include: Permission denied" which is a reference to build A.

(eglibc-initial is rebuilding since it noticed the addition of ccache)

This should at least make the issue more debuggable for people without FC13.

Further distilled down to their being some problem with our toolchain when provided with precompiled source:

richard@dax:/media/build1/poky/build3/tmp/work/i586-poky-linux/eglibc-initial-2.13-r23+svnr15508/build-i586-poky-linux$ /media/build1/poky/build3/tmp/sysroots/x86_64-linux/usr/bin/i586-poky-linux.gcc-cross-initial/i586-poky-linux-gcc -m32 -march=i586 --sysroot=/media/build1/poky/build3/tmp/sysroots/qemux86-tcbootstrap -c -I/media/build1/poky/build3/tmp/sysroots/qemux86 -o /home/richard/.ccache/b/7/bbc769bc628827f1d23de1a53d35d3-348.o.tmp.dax.11809 test.c

works

richard@dax:/media/build1/poky/build3/tmp/work/i586-poky-linux/eglibc-initial-2.13-r23+svnr15508/build-i586-poky-linux$ /media/build1/poky/build3/tmp/sysroots/x86_64-linux/usr/bin/i586-poky-linux.gcc-cross-initial/i586-poky-linux-gcc -m32 -march=i586 --sysroot=/media/build1/poky/build3/tmp/sysroots/qemux86-tcbootstrap -c -I/media/build1/poky/build3/tmp/sysroots/qemux86 -o /home/richard/.ccache/b/7/bbc769bc628827f1d23de1a53d35d3-348.o.tmp.dax.11809 test.i
cc1: error: /media/build1/poky/build/tmp/sysroots/qemux86/media/build1/poky/build/tmp/sysroots/qemux86/usr/include: Permission denied
cc1: error: /media/build1/poky/build/tmp/sysroots/qemux86/usr/include: Permission denied

does not

So its not ccache at fault, it just changes the workflow splitting the precompiled source out and this breaks.

(In reply to comment #17)
> Further distilled down to their being some problem with our toolchain when
> provided with precompiled source:

*preprocessed* source

(In reply to comment #18)
> (In reply to comment #17)
> > Further distilled down to their being some problem with our toolchain when
> > provided with precompiled source:
> 
> *preprocessed* source

hmmm interesting. Can you find out the cmdline how ccache is invoking the compiler in turn ?

(In reply to comment #17)
> Further distilled down to their being some problem with our toolchain when
> provided with precompiled source:
> 
> richard@dax:/media/build1/poky/build3/tmp/work/i586-poky-linux/eglibc-initial-2.13-r23+svnr15508/build-i586-poky-linux$
> /media/build1/poky/build3/tmp/sysroots/x86_64-linux/usr/bin/i586-poky-linux.gcc-cross-initial/i586-poky-linux-gcc
> -m32 -march=i586
> --sysroot=/media/build1/poky/build3/tmp/sysroots/qemux86-tcbootstrap -c
> -I/media/build1/poky/build3/tmp/sysroots/qemux86 -o
> /home/richard/.ccache/b/7/bbc769bc628827f1d23de1a53d35d3-348.o.tmp.dax.11809
> test.c
> 
> works
> 
> richard@dax:/media/build1/poky/build3/tmp/work/i586-poky-linux/eglibc-initial-2.13-r23+svnr15508/build-i586-poky-linux$
> /media/build1/poky/build3/tmp/sysroots/x86_64-linux/usr/bin/i586-poky-linux.gcc-cross-initial/i586-poky-linux-gcc
> -m32 -march=i586
> --sysroot=/media/build1/poky/build3/tmp/sysroots/qemux86-tcbootstrap -c
> -I/media/build1/poky/build3/tmp/sysroots/qemux86 -o
> /home/richard/.ccache/b/7/bbc769bc628827f1d23de1a53d35d3-348.o.tmp.dax.11809
> test.i
> cc1: error:
> /media/build1/poky/build/tmp/sysroots/qemux86/media/build1/poky/build/tmp/sysroots/qemux86/usr/include:
> Permission denied
> cc1: error: /media/build1/poky/build/tmp/sysroots/qemux86/usr/include:
> Permission denied
> 
> does not
> 
> So its not ccache at fault, it just changes the workflow splitting the
> precompiled source out and this breaks.


Ah Interesting. I think the difference is .i and .c extention.
gcc driver when invoked with .i extention automatically passed -fpreprocessed

I bet the problem is due to line control direcitves which look like 

# 126 "/usr/include/strings.h" 2 3 4

and preprocessor has to read them and pass them on to compiler so compiler can
do better job at diagnostics and I think the paths has already been resolved
to point to previous sysroot.

RP

Can you paste the .i file somewhere for me to look at ?

actually I see three paths which are hardcoded to paths where compiler was configured ( i.e. first sysroot)

usr/local/include
usr/include
x86_64-linux/usr/lib/armv5te-angstrom-linux-gnueabi/gcc/arm-angstrom-linux-gnueabi/specs

The first one is because of local-prefix being not sysroot relative
so one solution we have is either we nullify it by using a /tmp/ or /var 
for it during configure or we fix gcc to make it relative to sysroot but
thats a big change in behavior

The second one could be because of includedir or with-headers option that
we use during configure I have not verified yet

Third one I have no idea yet how that could be fixed.

I've been poking around gcc but its not my area of expertise. I think the problem might "disappear" if we add %I to the cpp-output spec macro so it reads:

  {"@cpp-output",
   "%{!M:%{!MM:%{!E:cc1 -fpreprocessed %i %I %(cc1_options) %{!fsyntax-only:%(invoke_as)}}}}", 0, 0, 0},

(in gcc.c) since this means the -isysroot option gets preserved.

test.i contents:
# 1 "test.c"
# 1 "<built-in>"
# 1 "<command-line>"
# 1 "test.c"
# 10 "test.c"
int
main ()
{

  ;
  return 0;
}

Simplest test case I have at the moment is to add a line like:

fprintf(stderr, "Here 2 %s %d %s\n", sysroot, stdinc, iprefix);

to incpath.c register_include_chains()

and then "gcc -m32 -march=i586 --sysroot=/media/build1/poky/build3/tmp/sysroots/qemux86-tcbootstrap -c -I/media/build1/poky/build3/tmp/sysroots/qemux86 -o test.o test.i". With .i files, sysroot is set to the gcc internal default, in the .c case it sets set to the commandline --sysroot option.

Somehow the --sysroot option isn't preserved for .i files.

(In reply to comment #24)
> I've been poking around gcc but its not my area of expertise. I think the
> problem might "disappear" if we add %I to the cpp-output spec macro so it
> reads:
> 
>   {"@cpp-output",
>    "%{!M:%{!MM:%{!E:cc1 -fpreprocessed %i %I %(cc1_options)
> %{!fsyntax-only:%(invoke_as)}}}}", 0, 0, 0},
> 
> (in gcc.c) since this means the -isysroot option gets preserved.

yes this will work. I committed it for gcc-4.7 here
http://git.openembedded.org/openembedded-core-contrib/commit/?h=kraj/gcc-4.7&id=a197cf9e3ac732d545ec2a36c9a50ed4dd5ac01f

the problem with specs file still remain though. It still accesses it
now I dont know if it solves the issue at hand or we are going to fail on
accessing specs file now that includes are straightened out.

I've confirmed this fixes the test case I was able to reproduce on Ubuntu 11.10 and have posted a patch for review

Further testing revealed the same problem for .ii files and c++. I've updated the patch on the mailing list and this resolved the issue I saw in testing for that case too.

Fixed in master: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=22aac28f28fe1e703fb66e304a41a31f7b600e8d

(In reply to comment #30)
> Fixed in master:
> http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=22aac28f28fe1e703fb66e304a41a31f7b600e8d

cool. There is still reference to specs file left into build time install path
hopefully we are not hitting it

The original issue described in this bug is resolved. Yes, the toolchain does try and open a specs file at the original location however we don't generate or use a specs file and this failed file open does not in any way adversely affect the build.

I'd open a separate bug for the spec file issue as it has very different severity and priority characteristics.