AS_INIT: try to ensure fds 0, 1, 2 are open

A patch was recently proposed for GNU libc to make *all* processes
start up with file descriptors 0, 1, and 2 guaranteed to be open.
Part of the rationale for this patch was that configure scripts fail
catastrophically if these fds are closed, even if you just want to run
--help or --version, e.g.

   $ ./configure --version <&-; echo $?
   ./configure: line 555: 0: Bad file descriptor
   1

Obviously configure scripts cannot rely on behavior specific to GNU
libc, so even if that patch gets committed, it might make sense for
us to try to make configure scripts robust against being started up
with closed stdin/stdout/stderr.  This patch is a first attempt at
adding this robustness.

Detecting whether an fd is closed, readable, writable, or both, is
trivial in C.  In shell, on the other hand, I have not been able to
find a technique that works to my satisfaction on a broad variety of
Unixes. This patch uses /proc/<pid>/fdinfo when available (only on
Linux, AFAIK).  Otherwise, it relies on ‘exec 3>&n’, which can only
detect whether an fd is *open*, not whether it is readable or
writable.  Worse, in some old shells (e.g. Solaris 10 /bin/sh) this
construct will succeed regardless of whether fd ‘n’ is closed.
(A questionably reliable old beard on Stack Overflow told me that this
is a known bug in “the” Bourne shell, by which he presumably means the
original implementation.)  The second patch in this set adds logic to
try using the ‘lsof’ and/or ‘fstat’ commands when available, which
helps, but brings other problems and is possibly not worth the hassle.

I’m asking for feedback on the code, suggestions for additional
techniques to try, and opinions whether this is worth trying to
do at all.  The patchset is available as the ‘zack/ensure-standard-fds’
branch in git, if anyone would like to experiment with it.  I’ve gotten
clean testsuite runs on Debian unstable, Solaris 10, NetBSD 9, and
FreeBSD 11 (the latter two are somewhat unusual installations)
except as noted in the second patch.

* lib/m4sugar/m4sh.m4 (_AS_ENSURE_STANDARD_FDS): New macro.
  (_AS_SHELL_SANITIZE): Move the “Unset variables that we do not need”
  and “NLS nuisances” blocks immediately after setting IFS; merge the
  unsetting of CDPATH into the main unsetting loop; move invocation of
  _AS_PATH_SEPARATOR_PREPARE to immediately above the “Find who we are”
  block; invoke _AS_ENSURE_STANDARD_FDS immediately before
  _AS_PATH_SEPARATOR_PREPARE.

* tests/base.at (configure with closed standard fds): New test.
* tests/torture.at (--help and --version in unwritable directory): New test.

3 files changed, 194 insertions, 21 deletions

diff --git a/lib/m4sugar/m4sh.m4 b/lib/m4sugar/m4sh.m4
index c9e86246..3b1e9015 100644
--- a/lib/m4sugar/m4sh.m4
+++ b/lib/m4sugar/m4sh.m4
@@ -299,6 +299,79 @@ dnl code inserted by AS_REQUIRE_SHELL_FN will appear _after_ this point.
 dnl We shouldn't have to worry about any traps being active at this point.
 exit 255])# _AS_REEXEC_WITH_SHELL
 
+# _AS_ENSURE_STANDARD_FDS
+# -----------------------
+# Ensure that file descriptor 0 is open and readable, and file
+# descriptors 1 and 2 are open and writable.  This is a defensive
+# measure against weird environments that run configure scripts
+# with these descriptors closed.
+#
+# There is no universally portable way to test the readability
+# or writability of a file descriptor from a shell script, but there
+# are several semi-portable techniques which we try in descending order
+# of how likely we are to be on a system where they will work, how efficient
+# they are, and whether they can tell whether an fd is open for *reading*.
+#
+# This code runs before _AS_DETECT_BETTER_SHELL and must therefore be
+# extra careful about using only portable shell constructs.  Also, the
+# tests must not actually read or write any data on the fds being
+# tested, and must take care not to do anything that could temporarily
+# open fds 0, 1, or 2 in the parent shell.  For instance, command
+# substitutions `...` may be implemented with a pipe to the parent
+# shell; if fd 0 was closed to begin with, the read end of this pipe
+# could temporarily occupy fd 0 and invalidate the test.
+m4_defun([_AS_ENSURE_STANDARD_FDS], [dnl
+# If possible ensure that fds 0, 1, and 2 are open in an appropriate way.
+# If Linux's /proc/<pid>/fdinfo directory is available, we can use it
+# to do an accurate test without any forks.
+if test -d /proc/$$/fdinfo; then
+  for as_fd in 0 1 2; do
+    as_fd_ok=false
+    if test -f /proc/$$/fdinfo/$as_fd; then
+      # `while read ...; do ...; done < file` might execute the loop
+      # in a subshell, preventing it from setting as_fd_ok.
+      exec 3</proc/$$/fdinfo/$as_fd
+      while read as_tag as_value <&3; do
+        if test x"$as_tag" = "xflags:"; then
+          # $as_value is the number that fcntl($as_fd, F_GETFL, 0)
+          # would return, in octal. Since /proc/<pid>/fdinfo is
+          # already Linux-specific, we can assume that O_ACCMODE == 3,
+          # O_RDONLY == 0, O_WRONLY == 1, O_RDWR == 2. Rather than
+          # trying to mask out the top bit of the lowest (octal) digit
+          # we include both possibilities in each case pattern.
+          if test "$as_fd" = 0; then #(
+            case "$as_value" in *0 | *4 | *2 | *6) as_fd_ok=true;; esac
+          else #(
+            case "$as_value" in *1 | *5 | *2 | *6) as_fd_ok=true;; esac
+          fi
+          break
+        fi
+      done
+      exec 3<&-
+    fi
+    if test "$as_fd_ok" = "false"; then
+      if test "$as_fd" = 0; then
+        eval "exec $as_fd</dev/null"
+      else
+        eval "exec $as_fd>/dev/null"
+      fi
+    fi
+  done
+
+else
+  # Shell redirection operations can only tell whether an fd is open,
+  # not whether it is readable or writable, so this is a last resort.
+  # `exec >&n` fails in POSIX sh when fd N is closed, but succeeds
+  # regardless of whether fd N is open in some old shells, e.g. Solaris
+  # /bin/sh.  We can live with that; at least it never fails when fd N
+  # is *open*.
+  if (exec 3>&0) 2>/dev/null; then :; else exec 0</dev/null; fi
+  if (exec 3>&1) 2>/dev/null; then :; else exec 1>/dev/null; fi
+  if (exec 3>&2)            ; then :; else exec 2>/dev/null; fi
+
+fi
+])
+
 
 # _AS_PREPARE
 # -----------
@@ -456,7 +529,6 @@ m4_defun([_AS_SHELL_SANITIZE],
 [m4_text_box([M4sh Initialization.])
 
 AS_BOURNE_COMPATIBLE
-_AS_PATH_SEPARATOR_PREPARE
 
 # IFS
 # We need space, tab and new line, in precisely that order.  Quoting is
@@ -468,6 +540,27 @@ as_nl='
 export as_nl
 IFS=" ""	$as_nl"
 
+# Unset variables that we do not need and which cause bugs (e.g. in
+# pre-3.0 UWIN ksh).  But do not cause bugs in bash 2.01; the "|| exit 1"
+# suppresses any "Segmentation fault" message there.  '((' could
+# trigger a bug in pdksh 5.2.14.
+for as_var in BASH_ENV ENV MAIL MAILPATH CDPATH
+do eval test \${$as_var+y} \
+  && ( (unset $as_var) || exit 1) >/dev/null 2>&1 && unset $as_var || :
+done
+PS1='$ '
+PS2='> '
+PS4='+ '
+
+# NLS nuisances.
+LC_ALL=C
+export LC_ALL
+LANGUAGE=C
+export LANGUAGE
+
+_AS_ENSURE_STANDARD_FDS
+_AS_PATH_SEPARATOR_PREPARE
+
 # Find who we are.  Look in the path if we contain no directory separator.
 as_myself=
 case $[0] in @%:@((
@@ -486,26 +579,6 @@ if test ! -f "$as_myself"; then
   AS_EXIT
 fi
 
-# Unset variables that we do not need and which cause bugs (e.g. in
-# pre-3.0 UWIN ksh).  But do not cause bugs in bash 2.01; the "|| exit 1"
-# suppresses any "Segmentation fault" message there.  '((' could
-# trigger a bug in pdksh 5.2.14.
-for as_var in BASH_ENV ENV MAIL MAILPATH
-do eval test \${$as_var+y} \
-  && ( (unset $as_var) || exit 1) >/dev/null 2>&1 && unset $as_var || :
-done
-PS1='$ '
-PS2='> '
-PS4='+ '
-
-# NLS nuisances.
-LC_ALL=C
-export LC_ALL
-LANGUAGE=C
-export LANGUAGE
-
-# CDPATH.
-(unset CDPATH) >/dev/null 2>&1 && unset CDPATH
 _m4_popdef([AS_EXIT])])# _AS_SHELL_SANITIZE
 
 
diff --git a/tests/base.at b/tests/base.at
index 4042a8aa..6894990a 100644
--- a/tests/base.at
+++ b/tests/base.at
@@ -846,3 +846,76 @@ FOO
 ]])
 
 AT_CLEANUP
+
+## ----------------------------------- ##
+## configure with closed standard fds  ##
+## ----------------------------------- ##
+
+AT_SETUP([configure with closed standard fds])
+AT_KEYWORDS([AS@&t@_INIT])
+
+# Create a configure script that runs a relatively complicated but
+# self-contained test.  Run it.
+AT_CONFIGURE_AC([[AC_PROG_CC]])
+AT_CHECK_AUTOCONF
+AT_CHECK_AUTOHEADER
+AT_CHECK_CONFIGURE([], [], [stdout], [stderr])
+AT_CHECK_ENV
+
+mv stdout stdout-expected
+mv stderr stderr-expected
+mv state-env.after state-env-expected
+mv config.status config-status-expected
+mv config.h config-h-expected
+
+# Run it again with stdin (fd 0) closed.
+# There should be no change to the stdout or stderr output and thoe
+# result of configuration should be the same.
+
+AT_CHECK_CONFIGURE([ 0<&- ], [], [stdout], [stderr])
+AT_CHECK_ENV
+AT_CMP([stdout-expected], [stdout])
+AT_CMP([stderr-expected], [stderr])
+AT_CONFIG_CMP([state-env-expected], [state-env.after])
+
+mv stdout stdout-closed-0
+mv stderr stderr-closed-0
+mv state-env.after state-env-closed-0
+mv config.status config-status-closed-0
+mv config.h config-h-closed-0
+
+# Run it again with stdout (fd 1) closed.
+# There should be no change to the stderr output and the
+# result of configuration should be the same.  (Any output
+# that would have gone to stdout, of course, is lost.)
+
+AT_CHECK_CONFIGURE([ 1>&- ], [], [stdout], [stderr])
+AT_CHECK_ENV
+AT_CHECK([test -f stdout && test ! -s stdout])
+AT_CMP([stderr-expected], [stderr])
+AT_CONFIG_CMP([state-env-expected], [state-env.after])
+
+mv stdout stdout-closed-1
+mv stderr stderr-closed-1
+mv state-env.after state-env-closed-1
+mv config.status config-status-closed-1
+mv config.h config-h-closed-1
+
+# Run it again with stderr (fd 2) closed.
+# There should be no change to the stdout output and the
+# result of configuration should be the same.  (Any output
+# that would have gone to stderr, of course, is lost.)
+
+AT_CHECK_CONFIGURE([ 2>&- ], [], [stdout], [stderr])
+AT_CHECK_ENV
+AT_CMP([stdout-expected], [stdout])
+AT_CHECK([test -f stderr && test ! -s stderr])
+AT_CONFIG_CMP([state-env-expected], [state-env.after])
+
+mv stdout stdout-closed-2
+mv stderr stderr-closed-2
+mv state-env.after state-env-closed-2
+mv config.status config-status-closed-2
+mv config.h config-h-closed-2
+
+AT_CLEANUP
diff --git a/tests/torture.at b/tests/torture.at
index 616e051c..53859f6b 100644
--- a/tests/torture.at
+++ b/tests/torture.at
@@ -493,6 +493,33 @@ AT_CLEANUP
 
 
 
+## ---------------------- ##
+## --help and --version.  ##
+## ---------------------- ##
+
+AT_SETUP([--help and --version in unwritable directory])
+
+AS_MKDIR_P([inner])
+AT_DATA([inner/configure.ac],
+[[AC_INIT
+AC_OUTPUT
+]])
+
+AT_CHECK_M4([(cd inner && autoconf --force)])
+AT_CHECK([test -s inner/configure])
+if test "$SHELL_N" != none; then
+  AT_CHECK_SHELL_SYNTAX([inner/configure])
+fi
+
+chmod a-w inner
+AT_CHECK([(cd inner && ./configure --help)], 0, [ignore], [ignore])
+AT_CHECK([(cd inner && ./configure --version)], 0, [ignore], [ignore])
+chmod u+w inner
+
+AT_CLEANUP
+
+
+
 ## -------------------------------------------- ##
 ## Check that `#define' templates are honored.  ##
 ## -------------------------------------------- ##