glib: ensure consistent abort-on-OOM with g_vasprintf & its callers

The g_vasprintf method is called by g_strdup_vprintf, g_strdup_printf,
g_string_append_vprintf and more. It has three different implementations
depending on what the build target platform supports:

  1. The gnulib impl appears to use the system malloc, but a
     '#define malloc g_malloc' causes it to use GLib's wrapper
     and thus abort on OOM. This mostly gets used on Windows
     platforms or UNIX platforms with broken printf formatting.

  2. The main impl mostly used on modern Linux/UNIX calls the
     system vasprintf which uses the system malloc and does not
     abort on OOM.

  3. The final impl used on remaining platforms calls system
     vsprintf on a buffer allocated by g_new, and thus always
     aborts on OOM.

Of note is that impl 2 (using vasprintf) historically could abort on
OOM, if the application had installed a non-system malloc impl with
GLib. This was because the code would g_strndup the result from
vasprintf() in that scenario. This was removed in:

  commit a3660532535f92cfac136435579ed4f23231f48c
  Author: Dan Winship <danw@gnome.org>
  Date:   Fri Aug 7 09:46:49 2015 -0400

    glib: remove deprecated g_mem_is_system_malloc() check in gprintf.c

Having inconsistent OOM behaviour for the three impls is undesirable and
aborting on OOM is normal pratice for GLib APIs. Thus we must thus ensure
this happens in all impls of g_vasprintf.

Fixes #1622

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>

1 files changed, 13 insertions, 3 deletions

diff --git a/glib/gprintf.c b/glib/gprintf.c
index fc0a02a3b..d4d0b3e0c 100644
--- a/glib/gprintf.c
+++ b/glib/gprintf.c
@@ -20,6 +20,7 @@
 #include <stdarg.h>
 #include <stdlib.h>
 #include <stdio.h>
+#include <errno.h>
 
 #include "gprintf.h"
 #include "gprintfint.h"
@@ -327,9 +328,18 @@ g_vasprintf (gchar      **string,
 
 #elif defined (HAVE_VASPRINTF)
 
-  len = vasprintf (string, format, args);
-  if (len < 0)
-    *string = NULL;
+  {
+    int saved_errno;
+    len = vasprintf (string, format, args);
+    saved_errno = errno;
+    if (len < 0)
+      {
+        if (saved_errno == ENOMEM)
+          g_error ("%s: failed to allocate memory", G_STRLOC);
+        else
+          *string = NULL;
+      }
+  }
 
 #else