hwasan: Use system allocator to realloc and free untagged pointers in interceptor mode.

The Android dynamic loader has a non-standard feature that allows
libraries such as the hwasan runtime to interpose symbols even after
the symbol already has a value. The new value of the symbol is used to
relocate libraries loaded after the interposing library, but existing
libraries keep the old value. This behaviour is activated by the
DF_1_GLOBAL flag in DT_FLAGS_1, which is set by passing -z global to
the linker, which is what we already do to link the hwasan runtime.

What this means in practice is that if we have .so files that depend
on interceptor-mode hwasan without the main executable depending on
it, some of the libraries in the process will be using the hwasan
allocator and some will be using the system allocator, and these
allocators need to interact somehow. For example, if an instrumented
library calls a function such as strdup that allocates memory on
behalf of the caller, the instrumented library can reasonably expect
to be able to call free to deallocate the memory.

We can handle that relatively easily with hwasan by using tag 0 to
represent allocations from the system allocator. If hwasan's realloc
or free functions are passed a pointer with tag 0, the system allocator
is called.

One limitation is that this scheme doesn't work in reverse: if an
instrumented library allocates memory, it must free the memory itself
and cannot pass ownership to a system library. In a future change,
we may want to expose an API for calling the system allocator so
that instrumented libraries can safely transfer ownership of memory
to system libraries.

Differential Revision: https://reviews.llvm.org/D55986

git-svn-id: https://llvm.org/svn/llvm-project/compiler-rt/trunk@350427 91177308-0d34-0410-b5e6-96231b3b80d8

2 files changed, 55 insertions, 1 deletions

diff --git a/test/hwasan/TestCases/Posix/system-allocator-fallback.cc b/test/hwasan/TestCases/Posix/system-allocator-fallback.cc
new file mode 100644
index 000000000..0e44aae7d
--- /dev/null
+++ b/test/hwasan/TestCases/Posix/system-allocator-fallback.cc
@@ -0,0 +1,50 @@
+// RUN: %clangxx %s -o %t -ldl
+// RUN: %clangxx_hwasan -shared %s -o %t.so -DSHARED_LIB -shared-libsan -Wl,-rpath,%compiler_rt_libdir
+// RUN: %env_hwasan_opts=disable_allocator_tagging=0 %run %t
+
+#include <stddef.h>
+
+// Test that allocations made by the system allocator can be realloc'd and freed
+// by the hwasan allocator.
+
+typedef void run_test_fn(void *(*system_malloc)(size_t size));
+
+#ifdef SHARED_LIB
+
+// Call the __sanitizer_ versions of these functions so that the test
+// doesn't require the Android dynamic loader.
+extern "C" void *__sanitizer_realloc(void *ptr, size_t size);
+extern "C" void __sanitizer_free(void *ptr);
+
+extern "C" run_test_fn run_test;
+void run_test(void *(*system_malloc)(size_t size)) {
+  void *mem = system_malloc(64);
+  mem = __sanitizer_realloc(mem, 128);
+  __sanitizer_free(mem);
+}
+
+#else
+
+#include <dlfcn.h>
+#include <stdlib.h>
+#include <string>
+
+int main(int argc, char **argv) {
+  std::string path = argv[0];
+  path += ".so";
+  void *lib = dlopen(path.c_str(), RTLD_NOW);
+  if (!lib) {
+    printf("error in dlopen(): %s\n", dlerror());
+    return 1;
+  }
+
+  auto run_test = reinterpret_cast<run_test_fn *>(dlsym(lib, "run_test"));
+  if (!run_test) {
+    printf("failed dlsym\n");
+    return 1;
+  }
+
+  run_test(malloc);
+}
+
+#endif
diff --git a/test/hwasan/lit.cfg b/test/hwasan/lit.cfg
index 3ebba51d0..66008a632 100644
--- a/test/hwasan/lit.cfg
+++ b/test/hwasan/lit.cfg
@@ -9,14 +9,18 @@ config.name = 'HWAddressSanitizer' + getattr(config, 'name_suffix', 'default')
 config.test_source_root = os.path.dirname(__file__)
 
 # Setup default compiler flags used with -fsanitize=memory option.
-clang_hwasan_cflags = ["-fsanitize=hwaddress", "-mllvm", "-hwasan-generate-tags-with-calls", config.target_cflags] + config.debug_info_flags
+clang_cflags = [config.target_cflags] + config.debug_info_flags
+clang_cxxflags = config.cxx_mode_flags + clang_cflags
+clang_hwasan_cflags = ["-fsanitize=hwaddress", "-mllvm", "-hwasan-generate-tags-with-calls"] + clang_cflags
 clang_hwasan_cxxflags = config.cxx_mode_flags + clang_hwasan_cflags
 
 def build_invocation(compile_flags):
   return " " + " ".join([config.clang] + compile_flags) + " "
 
+config.substitutions.append( ("%clangxx ", build_invocation(clang_cxxflags)) )
 config.substitutions.append( ("%clang_hwasan ", build_invocation(clang_hwasan_cflags)) )
 config.substitutions.append( ("%clangxx_hwasan ", build_invocation(clang_hwasan_cxxflags)) )
+config.substitutions.append( ("%compiler_rt_libdir", config.compiler_rt_libdir) )
 
 default_hwasan_opts_str = ':'.join(['disable_allocator_tagging=1', 'random_tags=0'] + config.default_sanitizer_opts)
 if default_hwasan_opts_str: