Treat EPERM error from mmap as an OOM (#7500)

Linux can give back EPERM from an mmap call when a user program attempts
to map pages near `mmap_min_addr`, which is a kernel security measure to
prevent people from mapping pages at address 0. We may do this when we
hint to mmap what address to map the pages to.

However, it's theoretically possible we're not actually out of memory -
we could have continuously mapped pages at some other place far away
from `mmap_min_addr` and succeeded instead. So as an added precaution,
if mmap for a given addr gives us EPERM, we'll also attempt to map
*again*, but without the address hint. Maybe the kernel can do the right
thing.

However, while testing #7500, the amount of free address space we could
have otherwise used only turns out to be about 139MB. Which isn't really
a lot. So, given that, we *also* otherwise treat EPERM as an out of
memory error.

This fixes #7500.

Signed-off-by: Austin Seipp <aseipp@pobox.com>

1 files changed, 21 insertions, 0 deletions

diff --git a/rts/posix/OSMem.c b/rts/posix/OSMem.c
index 26aebc2b44..5486a15616 100644
--- a/rts/posix/OSMem.c
+++ b/rts/posix/OSMem.c
@@ -112,6 +112,27 @@ my_mmap (void *addr, W_ size)
     } else {
 	vm_protect(mach_task_self(),(vm_address_t)ret,size,FALSE,VM_PROT_READ|VM_PROT_WRITE);
     }
+#elif linux_HOST_OS
+    ret = mmap(addr, size, PROT_READ | PROT_WRITE,
+               MAP_ANON | MAP_PRIVATE, -1, 0);
+    if (ret == (void *)-1 && errno == EPERM) {
+        // Linux may return EPERM if it tried to give us
+        // a chunk of address space below mmap_min_addr,
+        // See Trac #7500.
+        if (addr != 0) {
+            // Try again with no hint address.
+            // It's not clear that this can ever actually help,
+            // but since our alternative is to abort, we may as well try.
+            ret = mmap(0, size, PROT_READ | PROT_WRITE,
+                       MAP_ANON | MAP_PRIVATE, -1, 0);
+        }
+        if (ret == (void *)-1 && errno == EPERM) {
+            // Linux is not willing to give us any mapping,
+            // so treat this as an out-of-memory condition
+            // (really out of virtual address space).
+            errno = ENOMEM;
+        }
+    }
 #else
     ret = mmap(addr, size, PROT_READ | PROT_WRITE, 
                MAP_ANON | MAP_PRIVATE, -1, 0);