Bug 7634 - recipetool: extend autotools recipe creation
Summary: recipetool: extend autotools recipe creation
Status: RESOLVED FIXED
Alias: None
Product: OE-Core
Classification: Build System, Metadata & Runtime
Component: core (show other bugs)
Version: 1.8
Hardware: All Multiple
: Medium enhancement
Target Milestone: 2.1
Assignee: Paul Eggleton
QA Contact:
URL:
Whiteboard:
Depends on:
Blocks:
 
Reported: 2015-04-21 09:00 UTC by Paul Eggleton
Modified: 2016-03-02 21:33 UTC (History)
4 users (show)

See Also:
OS type for building Yocto: ---
Type of Regression: ---
Verified:
Documentation change: No (bug/feature does not impact docs)


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Paul Eggleton 2015-04-21 09:00:09 UTC
recipetool has the ability to create recipes for software that uses autoconf/automake with a fairly basic ability to extract dependencies and other information from configure.ac; however there are macros and syntax that this doesn't currently handle. Someone with a better understanding of autotools needs to extend it; we may need to properly parse the files instead of the keyword searching we are doing now.

The implementation for this is in scripts/lib/recipetool/create_buildsys.py.
Comment 1 Christopher Larson 2015-07-07 16:18:56 UTC
I once had some prototyped autotools.bbclass changes which pulled e.g. the gettext macro from a call to autoconf rather than grep, we could probably do something similar here.

E.g. this old snippet:
+	if [ -n "`autoconf -t AC_CANONICAL_BUILD`" ]; then
+		macrodir="`autoconf -t AC_CONFIG_MACRO_DIR|cut -d: -f4|tail -n 1`"
+		if [ -z "$macrodir" ]; then
+			macrodir="."
+		fi

Admittedly running autoconf repeatedly wasn't ideal, but you get the idea -- you can use autoconf tracing to determine if a macro is called, and if so, with what arguments:

       -t, --trace=MACRO[:FORMAT]
              report the list of calls to MACRO
Comment 2 Christopher Larson 2015-07-20 17:32:48 UTC
$ cd bash
$ autoconf -t AC_CHECK_LIB
configure.ac:416:AC_CHECK_LIB:cposix:strerror:LIBS="$LIBS -lcposix"
configure.ac:543:AC_CHECK_LIB:termcap:tgetent:bash_cv_termcap_lib=libtermcap:AC_CHECK_LIB(tinfo, tgetent, bash_cv_termcap_lib=libtinfo, [AC_CHECK_LIB(curses, tgetent, bash_cv_termcap_lib=libcurses, [AC_CHECK_LIB(ncurses, tgetent, bash_cv_termcap_lib=libncurses, bash_cv_termcap_lib=gnutermcap)])])
configure.ac:543:AC_CHECK_LIB:tinfo:tgetent:bash_cv_termcap_lib=libtinfo:AC_CHECK_LIB(curses, tgetent, bash_cv_termcap_lib=libcurses, [AC_CHECK_LIB(ncurses, tgetent, bash_cv_termcap_lib=libncurses, bash_cv_termcap_lib=gnutermcap)])
configure.ac:543:AC_CHECK_LIB:curses:tgetent:bash_cv_termcap_lib=libcurses:AC_CHECK_LIB(ncurses, tgetent, bash_cv_termcap_lib=libncurses, bash_cv_termcap_lib=gnutermcap)
configure.ac:543:AC_CHECK_LIB:ncurses:tgetent:bash_cv_termcap_lib=libncurses:bash_cv_termcap_lib=gnutermcap
configure.ac:855:AC_CHECK_LIB:dl:dlopen
configure.ac:870:AC_CHECK_LIB:sun:getpwent
configure.ac:875:AC_CHECK_LIB:socket:getpeername:bash_cv_have_socklib=yes:bash_cv_have_socklib=no:-lnsl
configure.ac:875:AC_CHECK_LIB:nsl:t_open:bash_cv_have_libnsl=yes:bash_cv_have_libnsl=no
configure.ac:1033:AC_CHECK_LIB:termcap:tgetent:bash_cv_termcap_lib=libtermcap:AC_CHECK_LIB(tinfo, tgetent, bash_cv_termcap_lib=libtinfo, [AC_CHECK_LIB(curses, tgetent, bash_cv_termcap_lib=libcurses, [AC_CHECK_LIB(ncurses, tgetent, bash_cv_termcap_lib=libncurses, bash_cv_termcap_lib=gnutermcap)])])
configure.ac:1033:AC_CHECK_LIB:tinfo:tgetent:bash_cv_termcap_lib=libtinfo:AC_CHECK_LIB(curses, tgetent, bash_cv_termcap_lib=libcurses, [AC_CHECK_LIB(ncurses, tgetent, bash_cv_termcap_lib=libncurses, bash_cv_termcap_lib=gnutermcap)])
configure.ac:1033:AC_CHECK_LIB:curses:tgetent:bash_cv_termcap_lib=libcurses:AC_CHECK_LIB(ncurses, tgetent, bash_cv_termcap_lib=libncurses, bash_cv_termcap_lib=gnutermcap)
configure.ac:1033:AC_CHECK_LIB:ncurses:tgetent:bash_cv_termcap_lib=libncurses:bash_cv_termcap_lib=gnutermcap
Comment 3 Christopher Larson 2015-09-21 17:53:19 UTC
First, a note. You can pass any number of -t <macro> arguments to one autoconf execution, to get everything we need from one, and parse the output.

Next, from my personal todo, added here for safe-keeping:

- [ ] Use autoconf -t to reproduce the current capability for dependencies (AC_CHECK_LIB, PKG_CHECK_MODULES, AM_GNU_GETTEXT, AC_CHECK_PROG, AC_PATH_PROG, AC_PATH_X)
- [ ] Possibly also handle AC_SEARCH_LIBS
- [ ] Possibly also handle other prog variants: AC_CHECK_PROG, AC_CHECK_PROGS, AC_CHECK_TARGET_TOOL, AC_CHECK_TOOL, AC_CHECK_TARGET_TOOLS, AC_CHECK_TOOLS, AC_PATH_PROGS, AC_PATH_PROGS_FEATURE_CHECK, AC_PATH_TARGET_TOOL, AC_PATH_TOOL
- [ ] Possibly also check specific tool macros: AC_PROG_INSTALL, AC_PROG_AWK, AC_PROG_GREP, AC_PROG_EGREP, AC_PROG_FGREP, AC_PROG_MKDIR_P, AC_PROG_LEX, AC_PROG_LN_S, AC_PROG_RANLIB, AC_PROG_SED, AC_PROG_YACC (yacc & lex being the important ones, for the flex-native/bison-native deps)
- [ ] Consider: checking for cross-compilation issues. This is tougher, since -t doesn't tell us if the macro is only called inside a if block for cross_compilation=<yes,no>, or if the macro is called inside an AC_CACHE_CHECK (though we might be able to figure that out, either through line number or examining the AC_CACHE_CHECK arguments). We could warn about *possible* issues for AC_CHECK_FILE, AC_CHECK_FILES, AC_RUN_IFELSE, and catch AC_CACHE_CHECK to try to reduce the false positives. We could also warn about passing the cross-compilation fallback argument of AC_RUN_IFELSE, as that value isn't generally target specific, so could well be wrong.
Comment 4 Christopher Larson 2015-09-21 17:55:30 UTC
Example output with multiple -t's, with the macros we likely want to handle initially, run against GNU nano svn trunk:

$ autoconf -t AC_CHECK_LIB -t AC_SEARCH_LIBS -t PKG_CHECK_MODULES -t AC_CHECK_PROG -t AC_CHECK_PROGS -t AC_PATH_PROG -t AC_PATH_PROGS -t AC
_PATH_PROGS_FEATURE_CHECK -t AM_GNU_GETTEXT -t AC_PATH_X -t AC_PROG_YACC -t AC_PROG_LEX
configure.ac:35:AC_CHECK_PROG:CC:${ac_tool_prefix}gcc:${ac_tool_prefix}gcc::
configure.ac:35:AC_CHECK_PROG:CC:${ac_tool_prefix}cc:${ac_tool_prefix}cc
configure.ac:35:AC_CHECK_PROG:CC:cc:cc:::/usr/ucb/cc
configure.ac:35:AC_CHECK_PROG:CC:$ac_tool_prefix$ac_prog:$ac_tool_prefix$ac_prog::
configure.ac:35:AC_CHECK_PROGS:ac_ct_CC:cl.exe::
configure.ac:35:AC_CHECK_PROG:ac_ct_CC:$ac_prog:$ac_prog::
configure.ac:36:AC_CHECK_PROG:CC:${ac_tool_prefix}gcc:${ac_tool_prefix}gcc::
configure.ac:36:AC_CHECK_PROG:CC:${ac_tool_prefix}cc:${ac_tool_prefix}cc
configure.ac:36:AC_CHECK_PROG:CC:cc:cc:::/usr/ucb/cc
configure.ac:36:AC_CHECK_PROG:CC:$ac_tool_prefix$ac_prog:$ac_tool_prefix$ac_prog::
configure.ac:36:AC_CHECK_PROGS:ac_ct_CC:cl.exe::
configure.ac:36:AC_CHECK_PROG:ac_ct_CC:$ac_prog:$ac_prog::
configure.ac:38:AC_SEARCH_LIBS:strerror:cposix
configure.ac:243:AC_CHECK_LIB:${termlib}:tputs:tcap="-l$termlib"
configure.ac:243:AC_CHECK_LIB:${termlib}:tputs:tcap="-l$termlib"
configure.ac:491:AC_CHECK_PROG:NCURSESW_CONFIG:${ac_tool_prefix}ncursesw5-config:${ac_tool_prefix}ncursesw5-config::
configure.ac:496:AC_CHECK_LIB:ncursesw:get_wch:CURSES_LIB_NAME=ncursesw CURSES_LIB_WIDE=yes
configure.ac:498:AC_CHECK_LIB:ncursesw:get_wch:CURSES_LIB="-lncursesw" CURSES_LIB_NAME=ncursesw CURSES_LIB_WIDE=yes
configure.ac:504:AC_CHECK_LIB:ncurses:initscr:CURSES_LIB="-lncurses" CURSES_LIB_NAME=ncurses
configure.ac:512:AC_CHECK_LIB:curses:get_wch:CURSES_LIB="-lcurses" CURSES_LIB_NAME=curses CURSES_LIB_WIDE=yes
configure.ac:516:AC_CHECK_LIB:curses:initscr:CURSES_LIB="-lcurses" CURSES_LIB_NAME=curses
configure.ac:529:AC_CHECK_LIB:$CURSES_LIB_NAME:use_default_colors: $as_echo "#define HAVE_USE_DEFAULT_COLORS 1" >>confdefs.h
configure.ac:658:AC_CHECK_LIB:magic:magic_open
configure.ac:658:AC_CHECK_LIB:z:inflate
configure.ac:680:AC_CHECK_PROG:haveit:makeinfo:yes:no
Comment 5 Paul Eggleton 2016-01-19 18:01:35 UTC
I've made a number of improvements to recipetool's ability to extract dependencies from configure.ac; it's still not perfect, but it's better.

I didn't end up going the route of using autoconf -t because it didn't really make things easier - I also found it didn't help with situations such as where other macros had been defined and then used within a statement; the code we have now is able to expand those.

I'm not sure where we want to go with this bug. We're still stuck with situations such as where environment variables get used since they aren't really intended to be expanded until the configure script actually becomes a shell script; beyond actually implementing a proper M4 parser I can't see any practical way to improve this other than incrementally.
Comment 6 Ross Burton 2016-01-21 22:25:19 UTC
A number of relevant patches were merged into oe-core as of c58669fb0977f7f0cb79f252484d5c5ef0dfb7e4.
Comment 7 Paul Eggleton 2016-03-02 21:33:40 UTC
I think I'm going to call this done - if we need to extend the capabilities or fix issues then we can create more specific bugs.