Revert to 2.69-compatible behavior in AC_PROG_LEX (#110346)

Commit 29ede6b96feee29c0c477d1659081bbdb82cd8b3 caused AC_PROG_LEX to
stop looking for a library that provides yywrap.  This broke several
packages in a Debian archive rebuild.

Revert all the way to the 2.69 behavior, which was to set LEXLIB to
-ll or -lfl if that library defines yywrap, but allow AC_PROG_LEX to
succeed if neither -ll nor -lfl exists on the system, even if a lex
program that doesn't define yywrap would need it.
(This behavior was a bug, but people have come to depend on it.
See https://savannah.gnu.org/support/index.php?110269 and the
thread starting from
https://lists.gnu.org/r/autoconf-patches/2020-07/msg00013.html
for gory details.)

To provide a path away from bug-compatibility, AC_PROG_LEX now takes
one argument, documented as a whitespace-separated list of options.
Two options are defined: ‘yywrap’ means to look for yywrap and behave
as if lex is unavailable if it isn’t found; ‘noyywrap’ means to not
look for yywrap at all.  These are mutually exclusive.

Fixes bug #110346.

* lib/autoconf/programs.m4 (AC_PROG_LEX): Add an argument which
  can be either ‘yywrap’, meaning to look for yywrap in -ll, or
  ‘noyywrap’, meaning to not look for yywrap at all.  In the
  absence of either option, issue an obsoletion warning and
  revert to behavior bug-compatible with 2.69.

* tests/semantics.at: Add more tests of AC_PROG_LEX.
* tests/mktests.sh: Exclude AC_PROG_LEX from autogenerated tests.

* doc/autoconf.texi: Update documentation of AC_PROG_LEX.
* NEWS: Update notes on AC_PROG_LEX.

5 files changed, 272 insertions, 42 deletions

diff --git a/NEWS b/NEWS
index 2608cc2e..dded117e 100644
--- a/NEWS
+++ b/NEWS
@@ -139,14 +139,6 @@ GNU Autoconf NEWS - User visible changes.
   AC_CHECK_FUNCS_ONCE, AC_CHECK_HEADERS, AC_CHECK_HEADERS_ONCE,
   AC_CONFIG_MACRO_DIRS, AC_CONFIG_SUBDIRS, and AC_REPLACE_FUNCS.
 
-*** AC_PROG_LEX no longer looks for ‘yywrap’.
-
-  LEXLIB will only be set to ‘-lfl’ or ‘-ll’ if a scanner that defines
-  both ‘main’ and ‘yywrap’ itself still needs something else from that
-  library.
-
-  We recommend scanners define yywrap themselves, or use %noyywrap.
-
 *** AC_FUNC_VFORK no longer ignores a signal-handling bug in Solaris 2.4.
 
   This bug was being ignored because Emacs wanted to use ‘vfork’ on
@@ -245,6 +237,24 @@ GNU Autoconf NEWS - User visible changes.
   specify the aux files you actually need, so that the check can be
   effective.
 
+*** AC_PROG_LEX has an option to not look for yywrap.
+
+  AC_PROG_LEX now takes one argument, which may be either ‘yywrap’ or
+  ‘noyywrap’.  If it is ‘noyywrap’, AC_PROG_LEX will only set LEXLIB
+  to ‘-lfl’ or ‘-ll’ if a scanner that defines both main and yywrap
+  itself still needs something else from that library.  On the other
+  hand, if it is ‘yywrap’, AC_PROG_LEX will fail (setting LEX to ‘:’
+  and LEXLIB to nothing) if it can’t find a library that defines yywrap.
+
+  In the absence of arguments, AC_PROG_LEX’s behavior is bug-compatible
+  with 2.69, which did neither of the above things (see the manual for
+  details).  This mode is deprecated.
+
+  We encourage all programs that use AC_PROG_LEX to use the new
+  ‘noyywrap’ mode, and to define yywrap themselves, or use %noyywrap.
+  The yywrap function in lib(f)l is trivial, and self-contained
+  scanners are easier to work with.
+
 ** Obsolete features and new warnings
 
 *** Use of the long-deprecated name ‘configure.in’ for the autoconf
diff --git a/doc/autoconf.texi b/doc/autoconf.texi
index 2422bd67..3a026871 100644
--- a/doc/autoconf.texi
+++ b/doc/autoconf.texi
@@ -4195,37 +4195,73 @@ The result of the test can be overridden by setting the variable
 @end defmac
 
 @anchor{AC_PROG_LEX}
-@defmac AC_PROG_LEX
+@defmac AC_PROG_LEX (@var{options})
 @acindex{PROG_LEX}
 @ovindex LEX
 @ovindex LEXLIB
 @cvindex YYTEXT_POINTER
 @ovindex LEX_OUTPUT_ROOT
 @caindex prog_LEX
-If @code{flex} is found, set output variable @code{LEX} to @samp{flex}.
-Otherwise set @code{LEX} to @samp{lex}, if found.
-If neither variant is available, set @code{LEX} to @samp{:};
+Search for a lexical analyzer generator, preferring @code{flex}
+to plain @code{lex}.  Output variable @code{LEX} is set to whichever
+program is available.  If neither program is available, @code{LEX}
+is set to @samp{:};
 for packages that ship the generated @file{file.yy.c}
 alongside the source @file{file.l}, this default allows users without a
 lexer generator to still build the package even if the timestamp for
 @file{file.l} is inadvertently changed.
 
-The result of this test can be influenced by setting the variable
-@code{LEX} or the cache variable @code{ac_cv_prog_LEX}.
+The name of the program to use can be overridden by setting the
+output variable @code{LEX} or the cache variable @code{ac_cv_prog_LEX}
+when running @command{configure}.
+
+If a lexical analyzer generator is found, this macro performs additional
+checks for common portability pitfalls.  If these additional checks
+fail, @code{LEX} is reset to @samp{:}; otherwise the following
+additional macros and variables are provided.
 
-If either @code{flex} or @code{lex} is found, this macro performs
-additional checks for common portability pitfalls.  If these additional
-checks fail, @code{LEX} is also set to @samp{:}.
+Preprocessor macro @code{YYTEXT_POINTER} is defined if the lexer
+skeleton, by default, declares @code{yytext} as a @samp{@w{char *}}
+rather than a @samp{@w{char []}}.
 
 Output variable @code{LEX_OUTPUT_ROOT} is set to the base of the file
 name that the lexer generates; this is usually either @file{lex.yy} or
-@file{lexyy}.  Preprocessor macro @code{YYTEXT_POINTER} is defined if
-the lexer skeleton, by default, declares @code{yytext} as a @samp{char
-*} rather than a @samp{char []}.  Finally, if generated lexers need a
-library to work, set output variable @code{LEXLIB} to an option for that
-library (e.g., @option{-ll}); otherwise define @code{LEXLIB} to empty.
-A library that merely defines @code{main} and @code{yywrap} placeholder
-functions is not considered to be needed.
+@file{lexyy}.
+
+If generated lexers need a library to work, output variable
+@code{LEXLIB} is set to a link option for that library (e.g.,
+@option{-ll}), otherwise it is set to empty.
+
+The @var{options} argument modifies the behavior of @code{AC_PROG_LEX}.
+It should be a whitespace-separated list of options.  Currently there
+are only two options, and they are mutually exclusive:
+
+@table @code
+@item yywrap
+Indicate that the library in @code{LEXLIB} needs to define the function
+@code{yywrap}.  If a library that defines this function cannot be found,
+@code{LEX} will be reset to @samp{:}.
+
+@item noyywrap
+Indicate that the library in @code{LEXLIB} does not need to define the
+function @code{yywrap}.  @command{configure} will not search for it at
+all.
+@end table
+
+Prior to Autoconf 2.70, @code{AC_PROG_LEX} did not take any arguments,
+and its behavior was different from either of the above possibilities:
+it would search for a library that defines @code{yywrap}, and would set
+@code{LEXLIB} to that library if it finds one.  However, if a library
+that defines this function could not be found, @code{LEXLIB} would be
+left empty and @code{LEX} would @emph{not} be reset.  This behavior was
+due to a bug, but several packages came to depend on it, so
+@code{AC_PROG_LEX} still does this if neither the @code{yywrap} nor the
+@code{noyywrap} option is given.
+
+Usage of @code{AC_PROG_LEX} without choosing one of the @code{yywrap}
+or @code{noyywrap} options is deprecated.  It is usually better to
+use @code{noyywrap} and define the @code{yywrap} function yourself,
+as this almost always renders the @code{LEXLIB} unnecessary.
 
 @strong{Caution:} As a side-effect of the test, this macro may delete
 any file in the configure script's current working directory named
diff --git a/lib/autoconf/programs.m4 b/lib/autoconf/programs.m4
index a895e59a..e289a018 100644
--- a/lib/autoconf/programs.m4
+++ b/lib/autoconf/programs.m4
@@ -706,18 +706,45 @@ AC_MSG_RESULT([$MKDIR_P])
 AN_MAKEVAR([LEX],  [AC_PROG_LEX])
 AN_PROGRAM([lex],  [AC_PROG_LEX])
 AN_PROGRAM([flex], [AC_PROG_LEX])
-AC_DEFUN_ONCE([AC_PROG_LEX],
-[AC_CHECK_PROGS(LEX, flex lex, :)
-if test "x$LEX" != "x:"; then
-  _AC_PROG_LEX_YYTEXT_DECL
-fi])
+AC_DEFUN([AC_PROG_LEX],
+[m4_case($#,
+  [0], [],
+  [1], [],
+  [m4_fatal([too many arguments to $0])])]dnl
+[_$0(m4_normalize([$1]))])
+
+AC_DEFUN([_AC_PROG_LEX],
+[m4_case([$1],
+  [yywrap], [],
+  [noyywrap], [],
+  [yywrap noyywrap],
+    [m4_fatal([AC_PROG_LEX: yywrap and noyywrap are mutually exclusive])],
+  [noyywrap yywrap],
+    [m4_fatal([AC_PROG_LEX: yywrap and noyywrap are mutually exclusive])],
+  [],
+    [m4_warn([obsolete],
+      [AC_PROG_LEX without either yywrap or noyywrap is obsolete])],
+  [m4_fatal([AC_PROG_LEX: unrecognized argument: $1])])]dnl
+dnl We can't use AC_DEFUN_ONCE because this macro takes arguments.
+dnl Silently skip a second invocation if the options match;
+dnl warn if they don't.
+[m4_ifdef([_AC_PROG_LEX_options],
+  [m4_if([$1], m4_defn([_AC_PROG_LEX_options]), [],
+    [m4_warn([syntax], [AC_PROG_LEX used twice with mismatched options])])],
+[dnl
+dnl _AC_PROG_LEX_options not defined: first use
+m4_define([_AC_PROG_LEX_options], [$1])dnl
+AC_CHECK_PROGS(LEX, flex lex, :)
+  if test "x$LEX" != "x:"; then
+    _AC_PROG_LEX_YYTEXT_DECL([$1])
+fi])])
 
 
 # _AC_PROG_LEX_YYTEXT_DECL
 # ------------------------
 # Check for the Lex output root, the Lex library, and whether Lex
 # declares yytext as a char * by default.
-m4_define([_AC_PROG_LEX_YYTEXT_DECL],
+AC_DEFUN([_AC_PROG_LEX_YYTEXT_DECL],
 [cat >conftest.l <<_ACEOF[
 %%
 a { ECHO; }
@@ -763,17 +790,21 @@ AC_SUBST([LEX_OUTPUT_ROOT], [$ac_cv_prog_lex_root])dnl
 
 AS_VAR_SET_IF([LEXLIB], [], [
   AC_CACHE_CHECK([for lex library], [ac_cv_lib_lex], [
-    ac_save_LIBS=$LIBS
+    ac_save_LIBS="$LIBS"
+    ac_found=false
     for ac_cv_lib_lex in 'none needed' -lfl -ll 'not found'; do
-      case $ac_cv_lib_lex in
-	'none needed') ;;
-	'not found') break;;
-	*) LIBS="$ac_cv_lib_lex $ac_save_LIBS";;
-      esac
+      AS_CASE([$ac_cv_lib_lex],
+        ['none needed'], [],
+        ['not found'],   [break],
+        [*],             [LIBS="$ac_cv_lib_lex $ac_save_LIBS"])
+
       AC_LINK_IFELSE([AC_LANG_DEFINES_PROVIDED[`cat $LEX_OUTPUT_ROOT.c`]],
-	[break])
+	[ac_found=:])
+      if $ac_found; then
+        break
+      fi
     done
-    LIBS=$ac_save_LIBS
+    LIBS="$ac_save_LIBS"
   ])
   AS_IF(
      [test "$ac_cv_lib_lex" = 'not found'],
@@ -782,22 +813,49 @@ AS_VAR_SET_IF([LEXLIB], [], [
      [test "$ac_cv_lib_lex" = 'none needed'],
         [LEXLIB=''],
 	[LEXLIB=$ac_cv_lib_lex])
+dnl
+dnl For compatibility with autoconf 2.69 and prior, if $1 is not `noyywrap',
+dnl and we didn't already set LEXLIB to -ll or -lfl, see if one of those
+dnl libraries provides yywrap and set LEXLIB to it if so.  If $1 is `yywrap',
+dnl and we don't find a library that provides yywrap, we fail.
+  m4_case([$1],
+    [noyywrap],
+      [],
+    [yywrap],
+      [ac_save_LIBS="$LIBS"
+      AS_IF([test -n "$LEXLIB"],
+        [LIBS="$LEXLIB"
+        AC_CHECK_FUNC([yywrap],
+          [:],
+          [AC_MSG_WARN([$LEXLIB does not contain yywrap; giving up on $LEX])
+          LEX=: LEXLIB=])
+        ],
+        [LIBS=
+        AC_SEARCH_LIBS([yywrap], [fl l], [LEXLIB="$LIBS"])
+        AS_IF([test x"$ac_cv_search_yywrap" = xno],
+          [AC_MSG_WARN([yywrap not found; giving up on $LEX])
+          LEX=: LEXLIB=])])
+      LIBS="$ac_save_LIBS"],
+    [],
+      [ac_save_LIBS="$LIBS"
+      LIBS=
+      AC_SEARCH_LIBS([yywrap], [fl l], [LEXLIB="$LIBS"])
+      LIBS="$ac_save_LIBS"])dnl
 ])
 AC_SUBST(LEXLIB)
 
+dnl This test is done last so that we don't define YYTEXT_POINTER if
+dnl any of the above tests gave up on lex.
 AS_IF([test "$LEX" != :], [
 AC_CACHE_CHECK(whether yytext is a pointer, ac_cv_prog_lex_yytext_pointer,
 [# POSIX says lex can declare yytext either as a pointer or an array; the
 # default is implementation-dependent.  Figure out which it is, since
 # not all implementations provide the %pointer and %array declarations.
 ac_cv_prog_lex_yytext_pointer=no
-ac_save_LIBS=$LIBS
-LIBS="$LEXLIB $ac_save_LIBS"
-AC_LINK_IFELSE([AC_LANG_DEFINES_PROVIDED
+AC_COMPILE_IFELSE([AC_LANG_DEFINES_PROVIDED
   [#define YYTEXT_POINTER 1
 `cat $LEX_OUTPUT_ROOT.c`]],
   [ac_cv_prog_lex_yytext_pointer=yes])
-LIBS=$ac_save_LIBS
 ])
 dnl
 if test $ac_cv_prog_lex_yytext_pointer = yes; then
diff --git a/tests/mktests.sh b/tests/mktests.sh
index 849e4c12..3ce0c2b4 100755
--- a/tests/mktests.sh
+++ b/tests/mktests.sh
@@ -134,6 +134,10 @@ ac_exclude_list='
 	/^AC_(PROG_CC|C_CONST|C_VOLATILE)$/ {next}
 	/^AC_PATH_XTRA$/ {next}
 
+        # Use without an argument is obsolete.
+        # Checked in semantics.
+        /^AC_PROG_LEX$/ {next}
+
 	# Requires a working C++ compiler, which is not a given.
 	/^AC_PROG_CXX_C_O$/ {next}
 
diff --git a/tests/semantics.at b/tests/semantics.at
index ab3cea7c..d14e5713 100644
--- a/tests/semantics.at
+++ b/tests/semantics.at
@@ -1044,3 +1044,125 @@ entered d_blurf/configure
 ]])
 
 AT_CLEANUP
+
+
+## ------------------------------------- ##
+## AC_PROG_LEX with and without yywrap.  ##
+## ------------------------------------- ##
+
+# We don't currently have a good way of verifying that each mode
+# does what it's supposed to, but we can at least put them through
+# their paces as much as the autogenerated AT_CHECK_MACRO invocation
+# used to, back when AC_PROG_LEX took no arguments.
+
+AT_CHECK_MACRO([AC_PROG_LEX with yywrap], [AC_PROG_LEX([yywrap])])
+AT_CHECK_MACRO([AC_PROG_LEX with noyywrap], [AC_PROG_LEX([noyywrap])])
+
+AT_SETUP([AC_PROG_LEX in legacy mode])
+
+AT_CONFIGURE_AC([[AC_PROG_LEX]])
+AT_CHECK_AUTOHEADER([], [ignore])
+AT_CHECK_AUTOCONF([], [], [],
+[[configure.ac:4: warning: AC_PROG_LEX without either yywrap or noyywrap is obsolete
+programs.m4: _AC_PROG_LEX is expanded from...
+programs.m4: AC_PROG_LEX is expanded from...
+configure.ac:4: the top level
+]])
+
+AT_CHECK_CONFIGURE
+
+AT_CLEANUP
+
+
+## ---------------------------------- ##
+## Invalid arguments to AC_PROG_LEX.  ##
+## ---------------------------------- ##
+
+AT_SETUP([Invalid arguments to AC_PROG_LEX])
+
+AT_CONFIGURE_AC(
+[[AC_PROG_LEX([nonsense])
+]])
+AT_CHECK_AUTOCONF([], [1], [],
+[[configure.ac:4: error: AC_PROG_LEX: unrecognized argument: nonsense
+programs.m4: _AC_PROG_LEX is expanded from...
+programs.m4: AC_PROG_LEX is expanded from...
+configure.ac:4: the top level
+autom4te: error: m4 failed with exit status: 1
+]])
+
+AT_CONFIGURE_AC(
+[[AC_PROG_LEX([too],[many])
+]])
+AT_CHECK_AUTOCONF([], [1], [],
+[[configure.ac:4: error: too many arguments to AC_PROG_LEX
+programs.m4: AC_PROG_LEX is expanded from...
+configure.ac:4: the top level
+autom4te: error: m4 failed with exit status: 1
+]])
+
+AT_CONFIGURE_AC(
+[[AC_PROG_LEX([yywrap noyywrap])
+]])
+AT_CHECK_AUTOCONF([], [1], [],
+[[configure.ac:4: error: AC_PROG_LEX: yywrap and noyywrap are mutually exclusive
+programs.m4: _AC_PROG_LEX is expanded from...
+programs.m4: AC_PROG_LEX is expanded from...
+configure.ac:4: the top level
+autom4te: error: m4 failed with exit status: 1
+]])
+
+AT_CONFIGURE_AC(
+[[AC_PROG_LEX([noyywrap yywrap])
+]])
+AT_CHECK_AUTOCONF([], [1], [],
+[[configure.ac:4: error: AC_PROG_LEX: yywrap and noyywrap are mutually exclusive
+programs.m4: _AC_PROG_LEX is expanded from...
+programs.m4: AC_PROG_LEX is expanded from...
+configure.ac:4: the top level
+autom4te: error: m4 failed with exit status: 1
+]])
+
+AT_CONFIGURE_AC(
+[[AC_PROG_LEX([yywrap nonsense])
+]])
+AT_CHECK_AUTOCONF([], [1], [],
+[[configure.ac:4: error: AC_PROG_LEX: unrecognized argument: yywrap nonsense
+programs.m4: _AC_PROG_LEX is expanded from...
+programs.m4: AC_PROG_LEX is expanded from...
+configure.ac:4: the top level
+autom4te: error: m4 failed with exit status: 1
+]])
+
+AT_CONFIGURE_AC(
+[[AC_PROG_LEX([nonsense noyywrap])
+]])
+AT_CHECK_AUTOCONF([], [1], [],
+[[configure.ac:4: error: AC_PROG_LEX: unrecognized argument: nonsense noyywrap
+programs.m4: _AC_PROG_LEX is expanded from...
+programs.m4: AC_PROG_LEX is expanded from...
+configure.ac:4: the top level
+autom4te: error: m4 failed with exit status: 1
+]])
+
+# A double invocation with matching arguments should be accepted
+# without complaint.  FIXME: verify that it runs the test only once.
+AT_CONFIGURE_AC(
+[[AC_PROG_LEX([noyywrap])
+AC_PROG_LEX([noyywrap])
+]])
+AT_CHECK_AUTOCONF
+
+# A double invocation with matching arguments should trigger a warning.
+AT_CONFIGURE_AC(
+[[AC_PROG_LEX([yywrap])
+AC_PROG_LEX([noyywrap])
+]])
+AT_CHECK_AUTOCONF([], [0], [],
+[[configure.ac:5: warning: AC_PROG_LEX used twice with mismatched options
+programs.m4: _AC_PROG_LEX is expanded from...
+programs.m4: AC_PROG_LEX is expanded from...
+configure.ac:5: the top level
+]])
+
+AT_CLEANUP