Tweak the handling of $GDBHISTSIZE edge cases [PR gdb/16999]

When GDB reads a nonsensical value for the GDBHISTSIZE environment
variable, i.e. one that is non-numeric or negative, GDB then sets its
history size to 0.  This behavior is annoying and also inconsistent
with the behavior of bash.

This patch makes the behavior of invalid GDBHISTSIZE consistent with how
bash handles HISTSIZE.  When we encounter a null or out-of-range
GDBHISTSIZE (outside of [0, INT_MAX]) we now set the history size to
unlimited instead of 0.  When we encounter a non-numeric GDBHISTSIZE we
do nothing.

gdb/ChangeLog:

	PR gdb/16999
	* NEWS: Mention new GDBHISTSIZE behavior.
	* top.c (init_history): For null or out-of-range GDBHISTSIZE,
	set history size to unlimited.  Ignore non-numeric GDBHISTSIZE.

gdb/doc/ChangeLog:

	PR gdb/16999
	* gdb.texinfo (Command History): Mention new GDBHISTSIZE
	behavior.

gdb/testsuite/ChangeLog:

	PR gdb/16999
	* gdb.base/gdbhistsize-history.exp: New test.

1 files changed, 24 insertions, 14 deletions

diff --git a/gdb/top.c b/gdb/top.c
index 8ff20394cc9..6ae84ab4499 100644
--- a/gdb/top.c
+++ b/gdb/top.c
@@ -1684,21 +1684,31 @@ init_history (void)
   tmpenv = getenv ("GDBHISTSIZE");
   if (tmpenv)
     {
-      int var;
-
-      var = atoi (tmpenv);
-      if (var < 0)
-	{
-	  /* Prefer ending up with no history rather than overflowing
-	     readline's history interface, which uses signed 'int'
-	     everywhere.  */
-	  var = 0;
-	}
-
-      history_size_setshow_var = var;
+      long var;
+      char *endptr;
+
+      tmpenv = skip_spaces (tmpenv);
+      var = strtol (tmpenv, &endptr, 10);
+      endptr = skip_spaces (endptr);
+
+      /* If GDBHISTSIZE is non-numeric then ignore it.  If GDBHISTSIZE is the
+	 empty string, a negative number or a huge positive number (larger than
+	 INT_MAX) then set the history size to unlimited.  Otherwise set our
+	 history size to the number we have read.  This behavior is consistent
+	 with how bash handles HISTSIZE.  */
+      if (*endptr != '\0')
+	;
+      else if (*tmpenv == '\0'
+	       || var < 0
+	       || var > INT_MAX)
+	history_size_setshow_var = -1;
+      else
+	history_size_setshow_var = var;
     }
-  /* If the init file hasn't set a size yet, pick the default.  */
-  else if (history_size_setshow_var == -2)
+
+  /* If neither the init file nor GDBHISTSIZE has set a size yet, pick the
+     default.  */
+  if (history_size_setshow_var == -2)
     history_size_setshow_var = 256;
 
   set_readline_history_size (history_size_setshow_var);