libtool.m4: fix nm BSD flag detection

Libtool needs to get BSD-format (or MS-format) output out of the system
nm, so that it can scan generated object files for symbol names for
-export-symbols-regex support.  Some nms need specific flags to turn on
BSD-formatted output, so libtool checks for this in its AC_PATH_NM.
Unfortunately the code to do this has a pair of interlocking flaws:

 - it runs the test by doing an nm of /dev/null.  Some platforms
   reasonably refuse to do an nm on a device file, but before now this
   has only been worked around by assuming that the error message has a
   specific textual form emitted by Tru64 nm, and that getting this
   error means this is Tru64 nm and that nm -B would work to produce
   BSD-format output, even though the test never actually got anything
   but an error message out of nm -B.  This is fixable by nm'ing *nm
   itself* (since we necessarily have a path to it).

 - the test is entirely skipped if NM is set in the environment, on the
   grounds that the user has overridden the test: but the user cannot
   reasonably be expected to know that libtool wants not only nm but
   also flags forcing BSD-format output.  Worse yet, one such "user" is
   the top-level Cygnus configure script, which neither tests for
   nor specifies any BSD-format flags.  So platforms needing BSD-format
   flags always fail to set them when run in a Cygnus tree, breaking
   -export-symbols-regex on such platforms.  Libtool also needs to
   augment $LD on some platforms, but this is done unconditionally,
   augmenting whatever the user specified: the nm check should do the
   same.

   One wrinkle: if the user has overridden $NM, a path might have been
   provided: so we use the user-specified path if there was one, and
   otherwise do the path search as usual.  (If the nm specified doesn't
   work, this might lead to a few extra pointless path searches -- but
   the test is going to fail anyway, so that's not a problem.)

(Tested with NM unset, and set to nm, /usr/bin/nm, my-nm where my-nm is a
symlink to /usr/bin/nm on the PATH, and /not-on-the-path/my-nm where
*that* is a symlink to /usr/bin/nm.)

ChangeLog
2021-09-27  Nick Alcock  <nick.alcock@oracle.com>

	PR libctf/27967
	* libtool.m4 (LT_PATH_NM): Try BSDization flags with a user-provided
	NM, if there is one.  Run nm on itself, not on /dev/null, to avoid
	errors from nms that refuse to work on non-regular files.  Remove
	other workarounds for this problem.  Strip out blank lines from the
	nm output.

1 files changed, 9 insertions, 0 deletions

diff --git a/ChangeLog b/ChangeLog
index dd1ba3d66b5..afa6629abd1 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,6 +1,15 @@
 2021-09-27  Nick Alcock  <nick.alcock@oracle.com>
 
 	PR libctf/27967
+	* libtool.m4 (LT_PATH_NM): Try BSDization flags with a user-provided
+	NM, if there is one.  Run nm on itself, not on /dev/null, to avoid
+	errors from nms that refuse to work on non-regular files.  Remove
+	other workarounds for this problem.  Strip out blank lines from the
+	nm output.
+
+2021-09-27  Nick Alcock  <nick.alcock@oracle.com>
+
+	PR libctf/27967
 	* libtool.m4 (lt_cv_sys_global_symbol_pipe): Augment symcode for
 	Solaris 11.