2074 2012-03-12 20:36:47 +0000 Unable to use compiler generated as one user with another user via the sstate-cache 2012-04-13 15:43:46 +0000 1 1 7 Build System, Metadata & Runtime Meta-yocto meta-yocto unspecified x86 Multiple RESOLVED FIXED (1.2) Patch out for review Medium normal 1.2 M4 1 mark.hatle richard.purdie jessica.zhang poky.bs.watcher poky.watcher raj.khem sgw --- --- --- oldest_to_newest 18970 0 mark.hatle 2012-03-12 20:36:47 +0000 I attempted to reproduce this failure with the current Poky master. I did NOT receive a failure following the steps that lead to one in Edison. However, I got a different failure. I don't believe it's related, but will include it and the steps here just in case it is. Build as user "test1", in /home/test1/poky/build. Run core-image-minimal... Store off the downloads and sstate-cache for use by another user -- remove the poky (and build) directories. Log in as user "test2". Using PREMIRRORS and SSTATE_MIRROR setup the new build to the copied downloads and sstate-cache location. Run a core-image-minimal build -- (it will succeed, unlike edison). Then run a "psplash" build, it will fail with: configure:3038: ccache i586-poky-linux-gcc -m32 -march=i586 --sysroot=/home/test2/poky/build/tmp/sysroots/qemux86 -c -O2 -pipe -g -feliminate-unused-debug-types conftest.c >&5 cc1: error: /home/test1/poky/build/tmp/sysroots/qemux86/home/test1/poky/build/tmp/sysroots/qemux86/usr/include: Permission denied cc1: error: /home/test1/poky/build/tmp/sysroots/qemux86/usr/include: Permission denied ---- The compiler is still looking into the older location(s) for items tat should only be coming from the new user code. 18972 1 mark.hatle 2012-03-12 20:38:18 +0000 (In reply to comment #0) > I attempted to reproduce this failure with the current Poky master. I did NOT > receive a failure following the steps that lead to one in Edison. However, I > got a different failure. I don't believe it's related, but will include it and > the steps here just in case it is. Ignore the first paragraph, that was supposed to be part of a different bug report. 20027 2 sgw 2012-04-06 22:49:52 +0000 I have tried to reproduce this using 2 users on the same machine with a sstate-cache and then using SSTATE_MIRROR, using own-mirrors and PREMIRROR setup. I build core-image-minimal for qemuppc as first user, then built it again as the second user, both of these completed correctly, not except the final image build duplicated, everything used sstate. I then built psplash in the second user's build area and it built correctly. In a different set of build areas I attempted to do the same thing with a patch that Khem did to remove the local-prefix, these also build correctly with no failures. I wanted to confirm if you are using any toolchain builds, I need to test that next. I did see one build that had the linux-yocto kernel did a fetch instead of using sstate, still need to track down what happened there (see seperate thread) 20029 3 mark.hatle 2012-04-06 23:27:25 +0000 The failure is due to the toolchain still looking in the previous config/install location for files. The error does not occur if the 2nd user has permissions to look at the target directory area. If the files exist, they will be used -- if they don't exit the compiler will ignore that location. The error only exists when the 2nd user does -not- have permissions to look into the user 1 directory structure. (Thats the cause of the permission denied message, it's still looking in the old location and EPERM is a failure.. but it existing or not isn't.) There was a related defect to this one that has the exact steps required to do the build and replicate the problem. 20031 4 sgw 2012-04-07 21:32:11 +0000 I re-tried this with the removal of the build user1's tmp directory (sgw_ab/poky/build). this is done as 2 different users. My retry also succeeded both the core-image-minimal and the psplash build I confirmed the compiler did not have permission to read the user1 sysroot directory since it was removed, I also verified that the compiler was looking for user1's sysroot via strings and strace. I actually see it trying this pattern: 19909 stat("/srv/ssd/sgw_ab/sgw1_build/poky/build/tmp/sysroots/qemuppc/srv/ssd/sgw_ab/poky/build/tmp/sysroots/qemuppc/usr/include", 0x7fffe3113c60) = -1 ENOENT (No such file or directory) It does ultimately find it correctly under sgw1_build/poky/build/tmp/sysroots/qemuppc/... here: 19909 stat("/srv/ssd/sgw_ab/sgw1_build/poky/build/tmp/sysroots/qemuppc/usr/include", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 Hints on getting a good reproducer are always welcome! 20032 5 raj.khem 2012-04-07 22:19:34 +0000 http://git.openembedded.org/openembedded-core-contrib/commit/?h=kraj/gcc-4.7&id=2275888181dc1892cf130c0fcf67be182c19b2d5 should help to reporoduce make usr1 dirs readable for usr1 only such that other can not even read it. the rebuild from sstate using usr2 20035 6 richard.purdie 2012-04-08 09:08:33 +0000 Note that I think this issue only occurs if the files exist but are unreadable. If they don't exist before, that hits different error code paths and has different behaviour. 20062 7 mark.hatle 2012-04-09 15:25:27 +0000 As Khem mentioned, you have to keep the old directory but make it unreadable to the second user. -ENOENT goes through a different error path then the -ENOPERM. The later causes the failure condition. 20063 8 mark.hatle 2012-04-09 15:27:32 +0000 Bug 2082 has the -exact- steps to reproduce this issue. The (proposed) test case was shown to reproduce the problem on other peoples systems. The local-prefix patch did -not- resolve the problem for me. (I have not tested this within the last couple of weeks, so another patch may have resolved it.) 20065 9 raj.khem 2012-04-09 15:42:03 +0000 (In reply to comment #8) > Bug 2082 has the -exact- steps to reproduce this issue. The (proposed) test > case was shown to reproduce the problem on other peoples systems. > > The local-prefix patch did -not- resolve the problem for me. > I presumed so since we also use --with-headers option. there could be other static references that I have not run into yet. I am working on cleaning this up but its turning out to be a big pile of changes. Even if it works it will be too much of a change for 1.2 > (I have not tested this within the last couple of weeks, so another patch may > have resolved it.) 20066 10 mark.hatle 2012-04-09 16:01:13 +0000 Removing the with-local-prefix only appeared to stop the cross compiler from looking in: <path>/sysroot/.../usr/local/include I'm not sure that removing that from the cross compiler is even a good idea. Since some folks may want to populate the local/include (even though we should never be at the system level) The -EPERM was happening to me on the sysroot/.../usr/include and later on the sysroot/.../lib and usr/lib directories.. so every time it checked if a directory existed it failed.. In the case where something doesn't exist it falls back and appears to replace the first path of the path with the sysroot information that was passed in -- thus allowing the access of the directories. It really looks to me like there is some kind of bug (feautre?) where it ignores the --sysroot parameter passed to glibc and tries to use the original locations first, then evaluates the sysroot.. 20154 11 mark.hatle 2012-04-10 22:25:57 +0000 I was unable to reproduce this on CentOS with a mips target, but was able to replicate this on an FC13 system w/ an x86_64 target. See: http://gate.crashing.org/~fray/yocto/psplash-strace.log.bz2 (8 MB) log of information from the FC13 system. 20242 12 sgw 2012-04-12 15:10:19 +0000 Centos Works for both users http://gate.crashing.org/~fray/yocto/centos-out.bz2 20247 13 mark.hatle 2012-04-12 16:25:52 +0000 runs of just the configure step: http://gate.crashing.org/~fray/yocto/fc13-configure-out.bz2 http://gate.crashing.org/~fray/yocto/centos-configure-out.bz2 20252 14 mark.hatle 2012-04-12 16:54:05 +0000 (In reply to comment #13) > runs of just the configure step: > Correction: > http://gate.crashing.org/~fray/yocto/fc13-configure-out.bz2 > http://gate.crashing.org/~fray/yocto/centos62-configure-out.bz2 20258 15 richard.purdie 2012-04-12 18:47:19 +0000 From the logs, the execve("/home/usr2/oe-core/build-ia32-2/tmp-eglibc/sysroots/x86_64-linux/usr/bin/x86_64-oe-linux/../../libexec/x86_64-oe-linux/gcc/x86_64-oe-linux/4.6.4/cc1", ["/home/usr2/oe-core/build-ia32-2/tmp-eglibc/sysroots/x86_64-linux/usr/bin/x86_64-oe-linux/../../libexec/x86_64-oe-linux/gcc/x86_64-oe-linux/4.6.4/cc1", "-fpreprocessed", "/home/usr2/oe-core/build-ia32-2/tmp-eglibc/ccache/x86_64-oe-linux/psplash/conftest.tmp.msp-mhatle-lx2.6733.i", "-quiet", "-dumpbase", "conftest.tmp.msp-mhatle-lx2.6733.i", "-m64", "-mtune=generic", "-march=x86-64", "-auxbase-strip", "/home/usr2/oe-core/build-ia32-2/tmp-eglibc/ccache/x86_64-oe-linux/psplash/tmp.hash.msp-mhatle-lx2.6733.o", "-g", "-O2", "-feliminate-unused-debug-types", "-o", "-"] made me suspicious contrasted to execve("/home/usr2/oe-core/build-ia32-2/tmp-eglibc/sysroots/x86_64-linux/usr/bin/x86_64-oe-linux/../../libexec/x86_64-oe-linux/gcc/x86_64-oe-linux/4.6.4/cc1", ["/home/usr2/oe-core/build-ia32-2/tmp-eglibc/sysroots/x86_64-linux/usr/bin/x86_64-oe-linux/../../libexec/x86_64-oe-linux/gcc/x86_64-oe-linux/4.6.4/cc1", "-quiet", "-iprefix", "/home/usr2/oe-core/build-ia32-2/tmp-eglibc/sysroots/x86_64-linux/usr/bin/x86_64-oe-linux/../../lib/x86_64-oe-linux/gcc/x86_64-oe-linux/4.6.4/", "-isysroot", "/home/usr2/oe-core/build-ia32-2/tmp-eglibc/sysroots/qemux86-64", "conftest.c", "-quiet", "-dumpbase", "conftest.c", "-m64", "-mtune=generic", "-march=x86-64", "-auxbase", "conftest", "-g", "-O2", "-feliminate-unused-debug-types", "-o", "-"] Mark confirmed that removing ccache causes the configure to complete successfully. Its a start to know where to look at least and starts to explain the behavior difference. 20259 16 richard.purdie 2012-04-12 18:54:34 +0000 I have a new way to reproduce what I think is the same issue on Ubuntu 11.10 64 bit. I ran a build in directory A, machine qemux86 with a sstate cache generated. I then did "sudo apt-get install ccache", chmod a-r and chown root.root on the tmp/sysroots/qemux86 directory for build A. I then setup a new build directory, B for machine qemux86, pointing at build A's sstate cache. bitbake psplash -c cleansstate; bitbake psplash leads to a failure in eglibc-initial, "cc1: error: /media/build1/poky/build/tmp/sysroots/qemux86/usr/include: Permission denied" which is a reference to build A. (eglibc-initial is rebuilding since it noticed the addition of ccache) This should at least make the issue more debuggable for people without FC13. 20260 17 richard.purdie 2012-04-12 19:29:48 +0000 Further distilled down to their being some problem with our toolchain when provided with precompiled source: richard@dax:/media/build1/poky/build3/tmp/work/i586-poky-linux/eglibc-initial-2.13-r23+svnr15508/build-i586-poky-linux$ /media/build1/poky/build3/tmp/sysroots/x86_64-linux/usr/bin/i586-poky-linux.gcc-cross-initial/i586-poky-linux-gcc -m32 -march=i586 --sysroot=/media/build1/poky/build3/tmp/sysroots/qemux86-tcbootstrap -c -I/media/build1/poky/build3/tmp/sysroots/qemux86 -o /home/richard/.ccache/b/7/bbc769bc628827f1d23de1a53d35d3-348.o.tmp.dax.11809 test.c works richard@dax:/media/build1/poky/build3/tmp/work/i586-poky-linux/eglibc-initial-2.13-r23+svnr15508/build-i586-poky-linux$ /media/build1/poky/build3/tmp/sysroots/x86_64-linux/usr/bin/i586-poky-linux.gcc-cross-initial/i586-poky-linux-gcc -m32 -march=i586 --sysroot=/media/build1/poky/build3/tmp/sysroots/qemux86-tcbootstrap -c -I/media/build1/poky/build3/tmp/sysroots/qemux86 -o /home/richard/.ccache/b/7/bbc769bc628827f1d23de1a53d35d3-348.o.tmp.dax.11809 test.i cc1: error: /media/build1/poky/build/tmp/sysroots/qemux86/media/build1/poky/build/tmp/sysroots/qemux86/usr/include: Permission denied cc1: error: /media/build1/poky/build/tmp/sysroots/qemux86/usr/include: Permission denied does not So its not ccache at fault, it just changes the workflow splitting the precompiled source out and this breaks. 20261 18 richard.purdie 2012-04-12 19:33:12 +0000 (In reply to comment #17) > Further distilled down to their being some problem with our toolchain when > provided with precompiled source: *preprocessed* source 20265 19 raj.khem 2012-04-12 21:04:43 +0000 (In reply to comment #18) > (In reply to comment #17) > > Further distilled down to their being some problem with our toolchain when > > provided with precompiled source: > > *preprocessed* source hmmm interesting. Can you find out the cmdline how ccache is invoking the compiler in turn ? 20266 20 raj.khem 2012-04-12 21:09:43 +0000 (In reply to comment #17) > Further distilled down to their being some problem with our toolchain when > provided with precompiled source: > > richard@dax:/media/build1/poky/build3/tmp/work/i586-poky-linux/eglibc-initial-2.13-r23+svnr15508/build-i586-poky-linux$ > /media/build1/poky/build3/tmp/sysroots/x86_64-linux/usr/bin/i586-poky-linux.gcc-cross-initial/i586-poky-linux-gcc > -m32 -march=i586 > --sysroot=/media/build1/poky/build3/tmp/sysroots/qemux86-tcbootstrap -c > -I/media/build1/poky/build3/tmp/sysroots/qemux86 -o > /home/richard/.ccache/b/7/bbc769bc628827f1d23de1a53d35d3-348.o.tmp.dax.11809 > test.c > > works > > richard@dax:/media/build1/poky/build3/tmp/work/i586-poky-linux/eglibc-initial-2.13-r23+svnr15508/build-i586-poky-linux$ > /media/build1/poky/build3/tmp/sysroots/x86_64-linux/usr/bin/i586-poky-linux.gcc-cross-initial/i586-poky-linux-gcc > -m32 -march=i586 > --sysroot=/media/build1/poky/build3/tmp/sysroots/qemux86-tcbootstrap -c > -I/media/build1/poky/build3/tmp/sysroots/qemux86 -o > /home/richard/.ccache/b/7/bbc769bc628827f1d23de1a53d35d3-348.o.tmp.dax.11809 > test.i > cc1: error: > /media/build1/poky/build/tmp/sysroots/qemux86/media/build1/poky/build/tmp/sysroots/qemux86/usr/include: > Permission denied > cc1: error: /media/build1/poky/build/tmp/sysroots/qemux86/usr/include: > Permission denied > > does not > > So its not ccache at fault, it just changes the workflow splitting the > precompiled source out and this breaks. Ah Interesting. I think the difference is .i and .c extention. gcc driver when invoked with .i extention automatically passed -fpreprocessed 20267 21 raj.khem 2012-04-12 21:27:41 +0000 I bet the problem is due to line control direcitves which look like # 126 "/usr/include/strings.h" 2 3 4 and preprocessor has to read them and pass them on to compiler so compiler can do better job at diagnostics and I think the paths has already been resolved to point to previous sysroot. 20269 22 raj.khem 2012-04-12 22:08:22 +0000 RP Can you paste the .i file somewhere for me to look at ? 20270 23 raj.khem 2012-04-12 23:35:31 +0000 actually I see three paths which are hardcoded to paths where compiler was configured ( i.e. first sysroot) usr/local/include usr/include x86_64-linux/usr/lib/armv5te-angstrom-linux-gnueabi/gcc/arm-angstrom-linux-gnueabi/specs The first one is because of local-prefix being not sysroot relative so one solution we have is either we nullify it by using a /tmp/ or /var for it during configure or we fix gcc to make it relative to sysroot but thats a big change in behavior The second one could be because of includedir or with-headers option that we use during configure I have not verified yet Third one I have no idea yet how that could be fixed. 20272 24 richard.purdie 2012-04-12 23:49:51 +0000 I've been poking around gcc but its not my area of expertise. I think the problem might "disappear" if we add %I to the cpp-output spec macro so it reads: {"@cpp-output", "%{!M:%{!MM:%{!E:cc1 -fpreprocessed %i %I %(cc1_options) %{!fsyntax-only:%(invoke_as)}}}}", 0, 0, 0}, (in gcc.c) since this means the -isysroot option gets preserved. 20273 25 richard.purdie 2012-04-12 23:50:41 +0000 test.i contents: # 1 "test.c" # 1 "<built-in>" # 1 "<command-line>" # 1 "test.c" # 10 "test.c" int main () { ; return 0; } 20274 26 richard.purdie 2012-04-12 23:53:38 +0000 Simplest test case I have at the moment is to add a line like: fprintf(stderr, "Here 2 %s %d %s\n", sysroot, stdinc, iprefix); to incpath.c register_include_chains() and then "gcc -m32 -march=i586 --sysroot=/media/build1/poky/build3/tmp/sysroots/qemux86-tcbootstrap -c -I/media/build1/poky/build3/tmp/sysroots/qemux86 -o test.o test.i". With .i files, sysroot is set to the gcc internal default, in the .c case it sets set to the commandline --sysroot option. Somehow the --sysroot option isn't preserved for .i files. 20279 27 raj.khem 2012-04-13 04:40:18 +0000 (In reply to comment #24) > I've been poking around gcc but its not my area of expertise. I think the > problem might "disappear" if we add %I to the cpp-output spec macro so it > reads: > > {"@cpp-output", > "%{!M:%{!MM:%{!E:cc1 -fpreprocessed %i %I %(cc1_options) > %{!fsyntax-only:%(invoke_as)}}}}", 0, 0, 0}, > > (in gcc.c) since this means the -isysroot option gets preserved. yes this will work. I committed it for gcc-4.7 here http://git.openembedded.org/openembedded-core-contrib/commit/?h=kraj/gcc-4.7&id=a197cf9e3ac732d545ec2a36c9a50ed4dd5ac01f the problem with specs file still remain though. It still accesses it now I dont know if it solves the issue at hand or we are going to fail on accessing specs file now that includes are straightened out. 20304 28 richard.purdie 2012-04-13 11:57:03 +0000 I've confirmed this fixes the test case I was able to reproduce on Ubuntu 11.10 and have posted a patch for review 20308 29 richard.purdie 2012-04-13 13:19:30 +0000 Further testing revealed the same problem for .ii files and c++. I've updated the patch on the mailing list and this resolved the issue I saw in testing for that case too. 20312 30 richard.purdie 2012-04-13 13:27:25 +0000 Fixed in master: http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=22aac28f28fe1e703fb66e304a41a31f7b600e8d 20321 31 raj.khem 2012-04-13 15:24:43 +0000 (In reply to comment #30) > Fixed in master: > http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=22aac28f28fe1e703fb66e304a41a31f7b600e8d cool. There is still reference to specs file left into build time install path hopefully we are not hitting it 20322 32 richard.purdie 2012-04-13 15:43:46 +0000 The original issue described in this bug is resolved. Yes, the toolchain does try and open a specs file at the original location however we don't generate or use a specs file and this failed file open does not in any way adversely affect the build. I'd open a separate bug for the spec file issue as it has very different severity and priority characteristics.