NIFs and port driver code run inside the Erlang VM OS process (the "Beam"). To maximize performance the code is called directly by the same threads executing Erlang beam code and has full access to all the memory of the OS process. A buggy NIF/driver can thus make severe damage by corrupting memory.
In a best case scenario such memory corruption is detected immediately causing the Beam to crash generating a core dump file which can be analyzed to find the bug. However, it is very common for memory corruption bugs to not be immediately detected when the faulty write happens, but instead much later, for example when the calling Erlang process is garbage collected. When that happens it can be very hard to find the root cause of the memory corruption by analysing the core dump. All traces that could have indicated which specific buggy NIF/driver that caused the corruption may be long gone.
One way to make debugging easier is to run an emulator built with target
Increase probability of detecting bugs earlier. It contains a lot more runtime checks to ensure correct use of internal interfaces and data structures.
Generate a core dump that is easier to analyze. Compiler optimizations are turned off, which stops the compiler from "optimizing away" variables, thus making it easier/possible to inspect their state.
Detect lock order violations. A runtime lock checker will
verify that the locks in the
In fact, we recommend to use the debug emulator as default during development of NIFs and drivers, regardless if you are troubleshooting bugs or not. Some subtle bugs may not be detected by the normal emulator and just happen to work anyway by chance. However, another version of the emulator, or even different circumstances within the same emulator, may cause the bug to later provoke all kinds of problems.
The main disadvantage of the
If the
> erl -emu_type debug Erlang/OTP 25 [erts-13.0.2] ... [type-assertions] [debug-compiled] [lock-checking] Eshell V13.0.2 (abort with ^G) 1>
If the
> $ERL_TOP/bin/cerl -debug Erlang/OTP 25 [erts-13.0.2] ... [type-assertions] [debug-compiled] [lock-checking] Eshell V13.0.2 (abort with ^G) 1>
The
> $ERL_TOP/bin/cerl -debug -core core.12345 or > $ERL_TOP/bin/cerl -debug -rcore core.12345
The first variant starts Emacs and runs
Similar to the
To get full effect you should compile both your own NIF/driver code as
well as the Erlang emulator with AddressSanitizer instrumentation. Compile
your own code by passing option
Build and run the emulator with AddressSanitizer support by using the same
procedure as for the debug emulator, except use the
If you run the
> export ASAN_LOG_DIR=/my/asan/log/dir > $ERL_TOP/bin/cerl -asan Erlang/OTP 25 [erts-13.0.2] ... [address-sanitizer] Eshell V13.0.2 (abort with ^G) 1>
You may however also want to set
If you run the
> export ASAN_OPTIONS="log_path=/my/asan/log/file"
To avoid false positive memory leak reports from the emulator
itself set
> export LSAN_OPTIONS="suppressions=$ERL_TOP/erts/emulator/asan/suppress"
The
Memory corruption errors are reported by AddressSanitizer when they happen, but memory leaks are only checked and reported by default then the emulator terminates.
An even more heavy weight debugging tool is
Valgrind is much slower than
Valgrind runs as a virtual machine itself, emulating execution of hardware machine instructions. This means you can run almost any program unchanged on valgrind. However, we have found that the beam executable benefits from being compiled with special adaptions for running on valgrind.
Build the emulator with
Run the
> export VALGRIND_LOG_DIR=/my/valgrind/log/dir > $ERL_TOP/bin/cerl -valgrind Erlang/OTP 25 [erts-13.0.2] ... [valgrind-compiled] Eshell V13.0.2 (abort with ^G) 1>
Last but not least, the fantastic interactive debugging tool
Considering its powerful utility,
Here is an example of a typical session. First we catch the crash in an rr recording session:
> $ERL_TOP/bin/cerl -debug -rr rr: Saving execution to trace directory /home/foobar/.local/share/rr/beam.debug.smp-1. Erlang/OTP 25 [erts-13.0.2] Eshell V13.0.2 (abort with ^G) 1> mymod:buggy_nif(). Segmentation fault
Now we can replay that session with
> rr replay GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2 : (rr) continue : Thread 2 received signal SIGSEGV, Segmentation fault. (rr) backtrace
You get the call stack at the moment of the crash. Bad luck, it is
somewhere deep down in the garbage collection of the beam. But you manage
to figure out that variable
Set a watch point on that memory position and resume execution
backwards. The debugger will then stop at the exact position when
that memory position
(rr) watch -l *hp Hardware watchpoint 1: -location *hp (rr) reverse-continue Continuing. Thread 2 received signal SIGSEGV, Segmentation fault.
This is a quirk to be aware about. We started by executing forward until it crashed with SIGSEGV. We are now executing backwards from that point, so we are hitting the same SIGSEGV again but from the other direction. Just continue backwards once more to move past it.
(rr) reverse-continue Continuing. Thread 2 hit Hardware watchpoint 1: -location *hp Old value = 42 New value = 0
And here we are at the position when someone wrote a broken term on the process heap. Note that "Old value" and "New value" are reversed when we execute backwards. In this case the value 42 was written on the heap. Let's see who the guilty one is:
(rr) backtrace