Bug 13825

Summary: bison-native build race
Product: [Build System, Metadata & Runtime] OE-Core Reporter: Richard Purdie <richard.purdie>
Component: coreAssignee: Adrian Bunk <bunk>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: Medium+ CC: bunk, meta.mr.watcher, meta.watcher, mingli.yu, raj.khem, randy.macleod
Version: 3.0   
Target Milestone: 3.2 M1   
Hardware: x86   
OS: Multiple   
Whiteboard:
OS type for building Yocto: --- Type of Regression: ---
Verified: Documentation change: No (bug/feature does not impact docs)

Description Richard Purdie 2020-03-06 08:45:59 UTC 
There appears to be a build race in bison-native:

https://autobuilder.yoctoproject.org/typhoon/#/builders/92/builds/2205

Log is truncated but what we have:

| AnnotationList.c:(.text+0x7ad): undefined reference to `rpl_fprintf'
| src/bison-AnnotationList.o:AnnotationList.c:(.text+0x7fa): more undefined references to `rpl_fprintf' follow
| collect2: error: ld returned 1 exit status
| Makefile:3905: recipe for target 'src/bison' failed
| make: *** [src/bison] Error 1
| ERROR: oe_runmake failed
| WARNING: exit code 1 from a shell command.
| ERROR: Execution of '/home/pokybuild/build-perf-test/build-901d68d-20200305174828/tmp/work/x86_64-linux/bison-native/3.5.2-r0/temp/run.do_compile.7351' failed with exit code 1:
| gcc    -isystem/home/pokybuild/build-perf-test/build-901d68d-20200305174828/tmp/work/x86_64-linux/bison-native/3.5.2-r0/recipe-sysroot-native/usr/include -O2 -pipe  -L/home/pokybuild/build-perf-test/build-901d68d-20200305174828/tmp/work/x86_64-linux/bison-native/3.5.2-r0/recipe-sysroot-native/usr/lib                         -L/home/pokybuild/build-perf-test/build-901d68d-20200305174828/tmp/work/x86_64-linux/bison-native/3.5.2-r0/recipe-sysroot-native/lib                         -Wl,--enable-new-dtags                         -Wl,-rpath-link,/home/pokybuild/build-perf-test/build-901d68d-20200305174828/tmp/work/x86_64-linux/bison-native/3.5.2-r0/recipe-sysroot-native/usr/lib                         -Wl,-rpath-link,/home/pokybuild/build-perf-test/build-901d68d-20200305174828/tmp/work/x86_64-linux/bison-native/3.5.2-r0/recipe-sysroot-native/lib                         -Wl,-rpath,/home/pokybuild/build-perf-test/build-901d68d-20200305174828/tmp/work/x86_64-linux/bison-native/3.5.2-r0/recipe-sysroot-native/usr/lib                         -Wl,-rpath,/home/pokybuild/build-perf-test/build-901d68d-20200305174828/tmp/work/x86_64-linux/bison-native/3.5.2-r0/recipe-sysroot-native/lib                         -Wl,-O1 -Wl,--allow-shlib-undefined -Wl,--dynamic-linker=/home/pokybuild/build-perf-test/build-901d68d-20200305174828/tmp/sysroots-uninative/x86_64-linux/lib/ld-linux-x86-64.so.2 -o src/bison src/bison-AnnotationList.o src/bison-InadequacyList.o src/bison-Sbitset.o src/bison-assoc.o src/bison-closure.o src/bison-complain.o src/bison-conflicts.o src/bison-derives.o src/bison-files.o src/bison-fixits.o src/bison-getargs.o src/bison-gram.o src/bison-graphviz.o src/bison-ielr.o src/bison-lalr.o src/bison-location.o src/bison-lr0.o src/bison-main.o src/bison-muscle-tab.o src/bison-named-ref.o src/bison-nullable.o src/bison-output.o src/bison-parse-gram.o src/bison-print-graph.o src/bison-print-xml.o src/bison-print.o src/bison-reader.o src/bison-reduce.o src/bison-relation.o src/bison-scan-code-c.o src/bison-scan-gram-c.o src/bison-scan-skel-c.o src/bison-state.o src/bison-symlist.o src/bison-symtab.o src/bison-tables.o src/bison-uniqstr.o           lib/libbison.a
| src/bison-AnnotationList.o: In function `AnnotationList__debug':
| AnnotationList.c:(.text+0x658): undefined reference to `rpl_fprintf'
| AnnotationList.c:(.text+0x704): undefined reference to `rpl_fprintf'
| AnnotationList.c:(.text+0x736): undefined reference to `rpl_fprintf'
| AnnotationList.c:(.text+0x786): undefined reference to `rpl_fprintf'
| AnnotationList.c:(.text+0x7ad): undefined reference to `rpl_fprintf'
| src/bison-AnnotationList.o:AnnotationList.c:(.text+0x7fa): more undefined references to `rpl_fprintf' follow
| collect2: error: ld returned 1 exit status
| Makefile:3905: recipe for target 'src/bison' failed
| make: *** [src/bison] Error 1
| WARNING: exit code 1 from a shell command.
| 
NOTE: recipe bison-native-3.5.2-r0: task do_compile: Failed
ERROR: Task (virtual:native:/home/pokybuild/yocto-worker/buildperf-ubuntu1604/build/meta/recipes-devtools/bison/bison_3.5.2.bb:do_compile) failed with exit code '1'

rerunning the build passed. Once it failed it was persistent in the build that failed so files appear corrupted.
Comment 1 Adrian Bunk 2020-03-08 11:03:05 UTC 
(In reply to comment #0)
> | AnnotationList.c:(.text+0x658): undefined reference to `rpl_fprintf'

rpl_fprintf is the gnulib replacement from the fprintf-posix module.

> rerunning the build passed. Once it failed it was persistent in the build
> that failed so files appear corrupted.

Is this re-using configure results from a build host running a kernel with a different set of security features enabled?

The fprintf-posix replacement is used for me on Ubuntu 18.04 due to:
...
checking whether snprintf fully supports the 'n' directive... no
...
configure:14698: ./conftest
*** %n in writable segment detected ***
../bison-3.5.2/configure: line 3041: 31456 Aborted                 
...

Workaround for bison-native:
  EXTRA_OECONF += "gl_cv_func_printf_directive_n=yes"
Comment 2 Adrian Bunk 2020-03-08 18:50:07 UTC 
Two updates on that:
1. The relevant difference is not in the kernel, it is whether the host gcc defaults to _FORTIFY_SOURCE=2 (like in recent Ubuntu releases).
2. Correct would be EXTRA_OECONF += "gl_cv_func_printf_directive_n=no", bison is supposed to use the gnulib replacement functions on glibc systems with _FORTIFY_SOURCE > 0. Always adding -U_FORTIFY_SOURCE when building with the host gcc might be an alternative (untested).
Comment 3 Richard Purdie 2020-03-10 21:29:44 UTC 
Thanks Adrian, that is really helpful and hints what kind of issue we're looking for.

It was on our Ubuntu build performance worker. What puzzles me a bit is those are sstate isolated and would always build from scratch so why would this happen in one build but not any others? :/
Comment 4 Khem Raj 2020-03-16 23:55:08 UTC 
Richard, does setting gl_cv_func_printf_directive_n=no solves the race ?

on debian10/archlinux it gets gl_cv_func_printf_directive_n=yes but ubuntu 16/18/20 all get it set to 'no', I think to disable it explicitly will solve this problem perhaps at some added performance cost of using gnulib emulation wrapper.

another way to look is to add CPPFLAGS += "-Drpl_fprintf=fprintf"
Comment 5 Mingli Yu 2020-03-24 01:41:23 UTC 
As a race issue, is there any hints how to reproduce the issue?