diff options
328 files changed, 12552 insertions, 4008 deletions
diff --git a/HOWTO/INSTALL-ANDROID.md b/HOWTO/INSTALL-ANDROID.md index 24e8b0d658..7d5af0b0ec 100644 --- a/HOWTO/INSTALL-ANDROID.md +++ b/HOWTO/INSTALL-ANDROID.md @@ -27,22 +27,76 @@ to generate the configure scripts. $ ./otp_build autoconf -Use the following when compiling a 64-bit version. +Use the following commands when compiling a 64-bit version. $ export NDK_ABI_PLAT=android21 # When targeting Android 5.0 Lollipop + + + $ # Either without OpenSSL support: + $ $ ./otp_build configure \ --xcomp-conf=./xcomp/erl-xcomp-arm64-android.conf \ --without-ssl -Use the following instead when compiling a 32-bit version. + $ # Or with OpenSSL linked statically: + $ + $ cd /path/to/OpenSSL/source/dir/built/for/android-arm64 + $ # First follow the NOTES.ANDROID build instructions from OpenSSL + $ + $ # Then to avoid the full installation of this cross-compiled build, + $ # manually create a 'lib' directory at the root of the OpenSSL directory + $ # (at the same level as 'include') and link 'libcrypto.a' inside it. + $ + $ mkdir lib + $ ln -s ../libcrypto.a lib/libcrypto.a + $ cd - # Return to the Erlang/OTP directory + $ + $ # This previous step is needed for the OpenSSL static linking to work as + $ # the --with-ssl option expects a path with both the 'lib' and 'include' + $ # directories. Otherwise the Erlang/OTP build will fallback to dynamic + $ # linking if it doesn't find 'libcrypto.a' in its expected location. + $ ./otp_build configure \ + --xcomp-conf=./xcomp/erl-xcomp-arm64-android.conf \ + --with-ssl=/path/to/OpenSSL/source/dir/built/for/android-arm64 \ + --disable-dynamic-ssl-lib + + +Use the following commands instead when compiling a 32-bit version. $ export NDK_ABI_PLAT=androideabi16 # When targeting Android 4.1 Jelly Bean + + + $ # Either without OpenSSL support: + $ $ ./otp_build configure \ --xcomp-conf=./xcomp/erl-xcomp-arm-android.conf \ --without-ssl + $ # Or with OpenSSL linked statically: + $ + $ cd /path/to/OpenSSL/source/dir/built/for/android-arm + $ # First follow the NOTES.ANDROID build instructions from OpenSSL + $ + $ # Then to avoid the full installation of this cross-compiled build, + $ # manually create a 'lib' directory at the root of the OpenSSL directory + $ # (at the same level as 'include') and link 'libcrypto.a' inside it. + $ + $ mkdir lib + $ ln -s ../libcrypto.a lib/libcrypto.a + $ cd - # Return to the Erlang/OTP directory + $ + $ # This previous step is needed for the OpenSSL static linking to work as + $ # the --with-ssl option expects a path with both the 'lib' and 'include' + $ # directories. Otherwise the Erlang/OTP build will fallback to dynamic + $ # linking if it doesn't find 'libcrypto.a' in its expected location. + $ ./otp_build configure \ + --xcomp-conf=./xcomp/erl-xcomp-arm-android.conf \ + --with-ssl=/path/to/OpenSSL/source/dir/built/for/android-arm \ + --disable-dynamic-ssl-lib + + ### Compile Erlang/OTP ### $ make noboot [-j4] diff --git a/HOWTO/INSTALL.md b/HOWTO/INSTALL.md index c39ff38b22..fab973e5e4 100644 --- a/HOWTO/INSTALL.md +++ b/HOWTO/INSTALL.md @@ -607,7 +607,7 @@ using the similar steps just described. $ (cd $ERL_TOP/erts/emulator && make $TYPE) -where `$TYPE` is `opt`, `gcov`, `gprof`, `debug`, `valgrind`, or `lcnt`. +where `$TYPE` is `opt`, `gcov`, `gprof`, `debug`, `valgrind`, `asan` or `lcnt`. These different beam types are useful for debugging and profiling purposes. diff --git a/HOWTO/TESTING.md b/HOWTO/TESTING.md index 020be0309c..7a7f6982f2 100644 --- a/HOWTO/TESTING.md +++ b/HOWTO/TESTING.md @@ -185,6 +185,52 @@ examine the results so far for the currently executing test suite (in R14B02 and later you want to open the `release/tests/test_server/all_runs.html` file to get to the currently running test) + +Run tests with Address Sanitizer +-------------------------------- + +First build emulator with `asan` build target. +See [$ERL_TOP/HOWTO/INSTALL.md][]. + +Set environment variable `ASAN_LOG_DIR` to the directory +where the error logs will be generated. + + export ASAN_LOG_DIR=$TESTROOT/test_server/asan_logs + mkdir $ASAN_LOG_DIR + +Set environment variable `TS_RUN_EMU` to `asan`. + + export TS_RUN_EMU=asan + +Then run the tests you want with `ts:run` as described above. Either +inspect the log files directly or use the script at +`$ERL_TOP/erts/emulator/asan/asan_logs_to_html` to read all log files +in `$ASAN_LOG_DIR` and distill them into one html page +`asan_summary.html`. Repeated reports from the same memory leak will +for example be ignored by the script and make it easier to analyze. + + +Run tests with Valgrind +----------------------- + +First make sure [valgrind][] is installed, then build OTP from source +and build the emulator with `valgrind` build target. See +[$ERL_TOP/HOWTO/INSTALL.md][]. + +Set environment variable `VALGRIND_LOG_DIR` to the directory +where the valgrind error logs will be generated. + + export VALGRIND_LOG_DIR=$TESTROOT/test_server/vg_logs + mkdir $VALGRIND_LOG_DIR + +Set environment variable `TS_RUN_EMU` to `valgrind`. + + export TS_RUN_EMU=valgrind + +Then run the tests you want with `ts:run` as described above and +inspect the log file(s) in `$VALGRIND_LOG_DIR`. + + [ct_run]: http://www.erlang.org/doc/man/ct_run.html [ct hook]: http://www.erlang.org/doc/apps/common_test/ct_hooks_chapter.html [$ERL_TOP/HOWTO/INSTALL.md]: INSTALL.md @@ -192,5 +238,6 @@ get to the currently running test) [common_test]: http://www.erlang.org/doc/man/ct.html [data_dir]: http://www.erlang.org/doc/apps/common_test/write_test_chapter.html#data_priv_dir [configuring the tests]: #configuring-the-test-environment + [valgrind]: https://valgrind.org [?TOC]: true diff --git a/OTP_VERSION b/OTP_VERSION index 19ec304d74..a9a57c8226 100644 --- a/OTP_VERSION +++ b/OTP_VERSION @@ -1 +1 @@ -23.2.7.1 +23.3.1 diff --git a/bootstrap/bin/no_dot_erlang.boot b/bootstrap/bin/no_dot_erlang.boot Binary files differindex 607a341cd5..1434a96636 100644 --- a/bootstrap/bin/no_dot_erlang.boot +++ b/bootstrap/bin/no_dot_erlang.boot diff --git a/bootstrap/bin/start.boot b/bootstrap/bin/start.boot Binary files differindex 607a341cd5..1434a96636 100644 --- a/bootstrap/bin/start.boot +++ b/bootstrap/bin/start.boot diff --git a/bootstrap/bin/start_clean.boot b/bootstrap/bin/start_clean.boot Binary files differindex 607a341cd5..1434a96636 100644 --- a/bootstrap/bin/start_clean.boot +++ b/bootstrap/bin/start_clean.boot diff --git a/bootstrap/lib/compiler/ebin/beam_asm.beam b/bootstrap/lib/compiler/ebin/beam_asm.beam Binary files differindex e0574f3124..047f014022 100644 --- a/bootstrap/lib/compiler/ebin/beam_asm.beam +++ b/bootstrap/lib/compiler/ebin/beam_asm.beam diff --git a/bootstrap/lib/compiler/ebin/beam_validator.beam b/bootstrap/lib/compiler/ebin/beam_validator.beam Binary files differindex 6b1b04e153..48dc4f8096 100644 --- a/bootstrap/lib/compiler/ebin/beam_validator.beam +++ b/bootstrap/lib/compiler/ebin/beam_validator.beam diff --git a/bootstrap/lib/compiler/ebin/compiler.app b/bootstrap/lib/compiler/ebin/compiler.app index bae37d87d1..a85ab33c24 100644 --- a/bootstrap/lib/compiler/ebin/compiler.app +++ b/bootstrap/lib/compiler/ebin/compiler.app @@ -19,7 +19,7 @@ {application, compiler, [{description, "ERTS CXC 138 10"}, - {vsn, "7.6.5"}, + {vsn, "7.6.6"}, {modules, [ beam_a, beam_asm, diff --git a/bootstrap/lib/kernel/ebin/inet_dns.beam b/bootstrap/lib/kernel/ebin/inet_dns.beam Binary files differindex 66003e4474..ecabcd9569 100644 --- a/bootstrap/lib/kernel/ebin/inet_dns.beam +++ b/bootstrap/lib/kernel/ebin/inet_dns.beam diff --git a/bootstrap/lib/kernel/ebin/kernel.app b/bootstrap/lib/kernel/ebin/kernel.app index 80f68e62d6..ac5d0dbc49 100644 --- a/bootstrap/lib/kernel/ebin/kernel.app +++ b/bootstrap/lib/kernel/ebin/kernel.app @@ -22,7 +22,7 @@ {application, kernel, [ {description, "ERTS CXC 138 10"}, - {vsn, "7.1"}, + {vsn, "7.2"}, {modules, [application, application_controller, application_master, diff --git a/bootstrap/lib/stdlib/ebin/erl_parse.beam b/bootstrap/lib/stdlib/ebin/erl_parse.beam Binary files differindex 5eddd13cfc..489c7e65b6 100644 --- a/bootstrap/lib/stdlib/ebin/erl_parse.beam +++ b/bootstrap/lib/stdlib/ebin/erl_parse.beam diff --git a/bootstrap/lib/stdlib/ebin/otp_internal.beam b/bootstrap/lib/stdlib/ebin/otp_internal.beam Binary files differindex 9ef9945d43..90850f2828 100644 --- a/bootstrap/lib/stdlib/ebin/otp_internal.beam +++ b/bootstrap/lib/stdlib/ebin/otp_internal.beam diff --git a/bootstrap/lib/stdlib/ebin/stdlib.app b/bootstrap/lib/stdlib/ebin/stdlib.app index 62ad6044ba..11754bf301 100644 --- a/bootstrap/lib/stdlib/ebin/stdlib.app +++ b/bootstrap/lib/stdlib/ebin/stdlib.app @@ -20,7 +20,7 @@ %% {application, stdlib, [{description, "ERTS CXC 138 10"}, - {vsn, "3.13.2"}, + {vsn, "3.14"}, {modules, [array, base64, beam_lib, diff --git a/erts/aclocal.m4 b/erts/aclocal.m4 index d237f7ae08..fae4254ae3 100644 --- a/erts/aclocal.m4 +++ b/erts/aclocal.m4 @@ -1374,28 +1374,55 @@ AC_DEFUN(ETHR_CHK_GCC_ATOMIC_OPS, ETHR_CHK_GCC_ATOMIC_OP__(__atomic_compare_exchange_n) ethr_have_gcc_native_atomics=no - ethr_arm_dbm_instr_val=0 + ethr_arm_dbm_sy_instr_val=0 + ethr_arm_dbm_st_instr_val=0 + ethr_arm_dbm_ld_instr_val=0 case "$GCC-$host_cpu" in - yes-arm*) - AC_CACHE_CHECK([for ARM DMB instruction], ethr_cv_arm_dbm_instr, + yes-arm*|yes-aarch*) + AC_CACHE_CHECK([for ARM 'dmb sy' instruction], ethr_cv_arm_dbm_sy_instr, [ - ethr_cv_arm_dbm_instr=no + ethr_cv_arm_dbm_sy_instr=no AC_TRY_LINK([], [ __asm__ __volatile__("dmb sy" : : : "memory"); - __asm__ __volatile__("dmb st" : : : "memory"); ], - [ethr_cv_arm_dbm_instr=yes]) + [ethr_cv_arm_dbm_sy_instr=yes]) ]) - if test $ethr_cv_arm_dbm_instr = yes; then - ethr_arm_dbm_instr_val=1 + if test $ethr_cv_arm_dbm_sy_instr = yes; then + ethr_arm_dbm_sy_instr_val=1 test $ethr_cv_64bit___atomic_compare_exchange_n = yes && ethr_have_gcc_native_atomics=yes + fi + AC_CACHE_CHECK([for ARM 'dmb st' instruction], ethr_cv_arm_dbm_st_instr, + [ + ethr_cv_arm_dbm_st_instr=no + AC_TRY_LINK([], + [ + __asm__ __volatile__("dmb st" : : : "memory"); + ], + [ethr_cv_arm_dbm_st_instr=yes]) + ]) + if test $ethr_cv_arm_dbm_st_instr = yes; then + ethr_arm_dbm_st_instr_val=1 + fi + AC_CACHE_CHECK([for ARM 'dmb ld' instruction], ethr_cv_arm_dbm_ld_instr, + [ + ethr_cv_arm_dbm_ld_instr=no + AC_TRY_LINK([], + [ + __asm__ __volatile__("dmb ld" : : : "memory"); + ], + [ethr_cv_arm_dbm_ld_instr=yes]) + ]) + if test $ethr_cv_arm_dbm_ld_instr = yes; then + ethr_arm_dbm_ld_instr_val=1 fi;; *) ;; esac - AC_DEFINE_UNQUOTED([ETHR_HAVE_GCC_ASM_ARM_DMB_INSTRUCTION], [$ethr_arm_dbm_instr_val], [Define as a boolean indicating whether you have a gcc compatible compiler capable of generating the ARM DMB instruction, and are compiling for an ARM processor with ARM DMB instruction support, or not]) + AC_DEFINE_UNQUOTED([ETHR_HAVE_GCC_ASM_ARM_DMB_INSTRUCTION], [$ethr_arm_dbm_sy_instr_val], [Define as a boolean indicating whether you have a gcc compatible compiler capable of generating the ARM 'dmb sy' instruction, and are compiling for an ARM processor with ARM DMB instruction support, or not]) + AC_DEFINE_UNQUOTED([ETHR_HAVE_GCC_ASM_ARM_DMB_ST_INSTRUCTION], [$ethr_arm_dbm_st_instr_val], [Define as a boolean indicating whether you have a gcc compatible compiler capable of generating the ARM 'dmb st' instruction, and are compiling for an ARM processor with ARM DMB instruction support, or not]) + AC_DEFINE_UNQUOTED([ETHR_HAVE_GCC_ASM_ARM_DMB_LD_INSTRUCTION], [$ethr_arm_dbm_ld_instr_val], [Define as a boolean indicating whether you have a gcc compatible compiler capable of generating the ARM 'dmb ld' instruction, and are compiling for an ARM processor with ARM DMB instruction support, or not]) test $ethr_cv_32bit___sync_val_compare_and_swap = yes && ethr_have_gcc_native_atomics=yes test $ethr_cv_64bit___sync_val_compare_and_swap = yes && @@ -1512,6 +1539,33 @@ AC_ARG_WITH(with_sparc_memory_order, AS_HELP_STRING([--with-sparc-memory-order=TSO|PSO|RMO], [specify sparc memory order (defaults to RMO)])) +AC_ARG_ENABLE(ppc-lwsync-instruction, +AS_HELP_STRING([--enable-ppc-lwsync-instruction], [enable use of powerpc lwsync instruction]) +AS_HELP_STRING([--disable-ppc-lwsync-instruction], [disable use of powerpc lwsync instruction]), +[ case "$enableval" in + no) enable_lwsync=no ;; + *) enable_lwsync=yes ;; + esac ], +[ + AC_CHECK_SIZEOF(void *) + case $host_cpu-$ac_cv_sizeof_void_p in + macppc-8|powerpc-8|ppc-8|powerpc64-8|ppc64-8|powerpc64le-8|ppc64le-8|"Power Macintosh"-8) + enable_lwsync=yes;; + *) + enable_lwsync=undefined;; + esac ]) + +case $enable_lwsync in + no) + AC_DEFINE(ETHR_PPC_HAVE_NO_LWSYNC, [1], [Define if you do not have the powerpc lwsync instruction]) + ;; + yes) + AC_DEFINE(ETHR_PPC_HAVE_LWSYNC, [1], [Define if you have the powerpc lwsync instruction]) + ;; + *) + ;; +esac + LM_CHECK_THR_LIB ERL_INTERNAL_LIBS @@ -2823,6 +2877,8 @@ AC_DEFUN([LM_HARDWARE_ARCH], [ ppc) ARCH=ppc;; ppc64) ARCH=ppc64;; ppc64le) ARCH=ppc64le;; + powerpc64) ARCH=ppc64;; + powerpc64le) ARCH=ppc64le;; "Power Macintosh") ARCH=ppc;; arm64) ARCH=arm64;; armv5b) ARCH=arm;; @@ -2833,6 +2889,9 @@ AC_DEFUN([LM_HARDWARE_ARCH], [ armv6hl) ARCH=arm;; armv7l) ARCH=arm;; armv7hl) ARCH=arm;; + armv8*) ARCH=arm;; + aarch64) ARCH=arm64;; + aarch*) ARCH=arm;; tile) ARCH=tile;; e2k) ARCH=e2k;; *) ARCH=noarch;; @@ -2870,8 +2929,8 @@ AC_DEFUN([LM_HARDWARE_ARCH], [ ARCH=ppc64 ;; arm-8) - AC_MSG_RESULT(yes: adjusting ARCH=arm to ARCH=noarch) - ARCH=noarch + AC_MSG_RESULT(yes: adjusting ARCH=arm to ARCH=arm64) + ARCH=arm64 ;; *) AC_MSG_RESULT(no: ARCH is $ARCH) diff --git a/erts/configure.in b/erts/configure.in index 12dfb9b19b..1b54c39ddc 100644 --- a/erts/configure.in +++ b/erts/configure.in @@ -2151,14 +2151,33 @@ AC_CHECK_FUNCS([getipnodebyname getipnodebyaddr gethostbyname2]) AC_CHECK_FUNCS([ieee_handler fpsetmask finite isnan isinf res_gethostbyname dlopen \ pread pwrite memmove strerror strerror_r strncasecmp \ - gethrtime localtime_r gmtime_r inet_pton mprotect madvise posix_madvise \ + gethrtime localtime_r gmtime_r mprotect madvise posix_madvise \ mmap mremap memcpy mallopt sbrk _sbrk __sbrk brk _brk __brk \ flockfile fstat strlcpy strlcat setsid posix2time time2posix \ setlocale nl_langinfo poll mlockall ppoll vsyslog]) +## We have a special check for inet_pton as AC_CHECK_FUCNS does not work +## on windows 32-bit as there a macro is used to rename the symbol... +AC_MSG_CHECKING([for inet_pton]) +AC_TRY_LINK([ +#ifdef WIN32 +#include <ws2tcpip.h> +#else +#include <arpa/inet.h> +#endif +],[inet_pton(2,"",(void*)0)], have_inet_pton=yes, have_inet_pton=no) + +if test $have_inet_pton = yes; then + AC_DEFINE(HAVE_INET_PTON,[1], + [Define to 1 if you have the `inet_pton' function.]) + AC_MSG_RESULT(yes) +else + AC_MSG_RESULT(no) +fi + AC_MSG_CHECKING([for isfinite]) AC_TRY_LINK([#include <math.h>], - [isfinite(0);], have_isfinite=yes, have_isfinite=no), + [isfinite(0);], have_isfinite=yes, have_isfinite=no) if test $have_isfinite = yes; then AC_DEFINE(HAVE_ISFINITE,[1], @@ -3551,6 +3570,26 @@ AH_BOTTOM([ #endif ]) + +dnl ---------------------------------------------------------------------- +dnl Check for GCC diagnostic ignored "-Waddress-of-packed-member" +dnl ---------------------------------------------------------------------- +saved_CFLAGS="$CFLAGS" +CFLAGS="-Werror $CFLAGS" +AC_TRY_COMPILE([], + [_Pragma("GCC diagnostic push") + _Pragma("GCC diagnostic ignored \"-Waddress-of-packed-member\"") + _Pragma("GCC diagnostic pop") + ], + AC_DEFINE(HAVE_GCC_DIAG_IGNORE_WADDRESS_OF_PACKED_MEMBER,[1], + [define if compiler support _Pragma('GCC diagnostic ignored '-Waddress-of-packed-member'')])) +CFLAGS="$saved_CFLAGS" + + +dnl ---------------------------------------------------------------------- +dnl Enable any -Werror flags +dnl ---------------------------------------------------------------------- + if test "x$GCC" = xyes; then CFLAGS="$WERRORFLAGS $CFLAGS" fi diff --git a/erts/doc/src/erl_cmd.xml b/erts/doc/src/erl_cmd.xml index eb1e9e2de7..372aee0380 100644 --- a/erts/doc/src/erl_cmd.xml +++ b/erts/doc/src/erl_cmd.xml @@ -371,7 +371,7 @@ the default. In <c><![CDATA[embedded]]></c> mode modules are not auto loaded. The latter is recommended when the boot script preloads all modules, as conventionally happens in OTP releases. See - <seeerl marker="kernel:code"><c>code(3)</c></seeerl></p>. + <seeerl marker="kernel:code"><c>code(3)</c></seeerl>.</p> </item> <tag><marker id="name"/><c><![CDATA[-name Name]]></c></tag> <item> @@ -824,9 +824,9 @@ </item> <tag><marker id="+hmqd"/><c>+hmqd off_heap|on_heap</c></tag> <item> - <p>Sets the default value for process flag <c>message_queue_data</c>. + <p>Sets the default value of the <c>message_queue_data</c> process flag. Defaults to <c>on_heap</c>. If <c>+hmqd</c> is not - passed, <c>on_heap</c> will be the default. For more information, see + passed, <c>on_heap</c> will be the default. For more information, see <seeerl marker="erlang#process_flag_message_queue_data"> <c>process_flag(message_queue_data, MQD)</c></seeerl>.</p> </item> @@ -1406,8 +1406,8 @@ <note> <p>This feature has been introduced as a temporary workaround for long-executing native code, and native code that does not - bump reductions properly in OTP. When these bugs have be fixed, - this flag will be removed.</p> + bump reductions properly in OTP. When these bugs have been + fixed, this flag will be removed.</p> </note> </item> <tag><marker id="+spp"/><c>+spp Bool</c></tag> @@ -1424,7 +1424,7 @@ <seeerl marker="erlang#open_port_parallelism"> <c>parallelism</c></seeerl> to <seemfa marker="erlang#open_port/2"> - <c>erlang:open_port/2</c></seemfa></p>. + <c>erlang:open_port/2</c></seemfa>.</p> </item> <tag><marker id="sched_thread_stack_size"/> <c><![CDATA[+sss size]]></c></tag> diff --git a/erts/doc/src/erl_dist_protocol.xml b/erts/doc/src/erl_dist_protocol.xml index 8cb8e09615..ae3de0a31c 100644 --- a/erts/doc/src/erl_dist_protocol.xml +++ b/erts/doc/src/erl_dist_protocol.xml @@ -430,9 +430,6 @@ io:format("old/unused name ~ts at port ~p, fd = ~p ~n", <p>where n = <c>Length</c> - 1.</p> - <p>The current implementation of Erlang does not care if the connection - to the EPMD is broken.</p> - <p>The response for a <c>STOP_REQ</c> is as follows:</p> <table align="left"> @@ -974,7 +971,11 @@ DiB == gen_digest(ChA, ICA)? <section> <marker id="dflags"/> <title>Distribution Flags</title> - <p>The following capability flags are defined:</p> + <p>Early in the distribution handshake the two participating nodes + exchange capability flags. This is done in order to determine how the + communication between the two nodes should be performed. The intersection + of the capabilities presented by the two nodes defines the capabilities + that will be used. The following capability flags are defined:</p> <taglist> <tag><c>-define(DFLAG_PUBLISHED,16#1).</c></tag> <item> @@ -1096,6 +1097,14 @@ DiB == gen_digest(ChA, ICA)? <p>The node supports the new connection setup handshake (version 6) introduced in OTP 23.</p> </item> + <tag><marker id="DFLAG_UNLINK_ID"/><c>-define(DFLAG_UNLINK_ID, 16#2000000).</c></tag> + <item> + <p>Use the <seeguide marker="#new_link_protocol">new link protocol</seeguide>.</p> + <note><p>This flag will become mandatory in OTP 26.</p></note> + <p>Unless both nodes have set the <c>DFLAG_UNLINK_ID</c> flag, the + <seeguide marker="#old_link_protocol">old link protocol</seeguide> + will be used as a fallback.</p> + </item> <tag><marker id="DFLAG_SPAWN"/><c>-define(DFLAG_SPAWN, (1 bsl 32)).</c></tag> <item> <p>Set if the <seeguide marker="#SPAWN_REQUEST"><c>SPAWN_REQUEST</c></seeguide>, @@ -1221,9 +1230,12 @@ DiB == gen_digest(ChA, ICA)? which distributed operation it encodes:</p> <taglist> - <tag><c>LINK</c></tag> + <tag><marker id="LINK"/><c>LINK</c></tag> <item> <p><c>{1, FromPid, ToPid}</c></p> + <p>This signal is sent by <c>FromPid</c> in order + to create a link between <c>FromPid</c> and + <c>ToPid</c>.</p> </item> <tag><c>SEND</c></tag> <item> @@ -1236,9 +1248,18 @@ DiB == gen_digest(ChA, ICA)? <p><c>{3, FromPid, ToPid, Reason}</c></p> <p>This signal is sent when a link has been broken</p> </item> - <tag><c>UNLINK</c></tag> + <tag><marker id="UNLINK"/><c>UNLINK</c> (deprecated)</tag> <item> <p><c>{4, FromPid, ToPid}</c></p> + <p>This signal is sent by <c>FromPid</c> in order to remove + a link between <c>FromPid</c> and <c>ToPid</c>, when using the + <seeguide marker="#old_link_protocol">old link + protocol</seeguide>.</p> + <warning><p>This signal has been deprecated and will not + be supported in OTP 26. For more information see the + documentation of the + <seeguide marker="#new_link_protocol">new link protocol</seeguide>. + </p></warning> </item> <tag><c>NODE_LINK</c></tag> <item> @@ -1300,9 +1321,8 @@ DiB == gen_digest(ChA, ICA)? <c>Reason</c> = exit reason for the monitored process</p> </item> </taglist> - </section> - <section> + <section> <title>New Ctrlmessages for Erlang/OTP 21</title> <taglist> <tag><c>SEND_SENDER</c></tag> @@ -1513,6 +1533,247 @@ DiB == gen_digest(ChA, ICA)? has been passed. </p> </item> + <tag><marker id="UNLINK_ID"/><c>UNLINK_ID</c></tag> + <item> + <p><c>{35, Id, FromPid, ToPid}</c></p> + <p>This signal is sent by <c>FromPid</c> in order to remove a + link between <c>FromPid</c> and <c>ToPid</c>. This unlink signal + replaces the <seeguide marker="#UNLINK"><c>UNLINK</c></seeguide> + signal. Besides process identifiers of the sender and receiver + the <c>UNLINK_ID</c> signal also contains an integer identifier + <c>Id</c>. Valid range of <c>Id</c> is <c>[1, (1 bsl 64) - 1]</c>. + <c>Id</c> is to be passed back to the sender by the receiver in an + <seeguide marker="#UNLINK_ID_ACK"><c>UNLINK_ID_ACK</c></seeguide> + signal. <c>Id</c> must uniquely identify the <c>UNLINK_ID</c> signal + among all not yet acknowledged <c>UNLINK_ID</c> signals from + <c>FromPid</c> to <c>ToPid</c>.</p> + <p> + This signal is only passed when the + <seeguide marker="#new_link_protocol">new link protocol</seeguide> + has been negotiated using the + <seeguide marker="erl_dist_protocol#DFLAG_UNLINK_ID"><c>DFLAG_UNLINK_ID</c></seeguide> + <seeguide marker="erl_dist_protocol#dflags">distribution flag</seeguide>. + </p> + </item> + <tag><marker id="UNLINK_ID_ACK"/><c>UNLINK_ID_ACK</c></tag> + <item> + <p><c>{36, Id, FromPid, ToPid}</c></p> + <p>An unlink acknowledgement signal. This signal is sent as an + acknowledgement of the reception of an + <seeguide marker="#UNLINK_ID"><c>UNLINK_ID</c></seeguide> + signal. The <c>Id</c> element should be the same <c>Id</c> + as present in the <c>UNLINK_ID</c> signal. <c>FromPid</c> + identifies the sender of the <c>UNLINK_ID_ACK</c> signal and + <c>ToPid</c> identifies the sender of the <c>UNLINK_ID</c> + signal.</p> + <p> + This signal is only passed when the + <seeguide marker="#new_link_protocol">new link protocol</seeguide> + has been negotiated using the + <seeguide marker="erl_dist_protocol#DFLAG_UNLINK_ID"><c>DFLAG_UNLINK_ID</c></seeguide> + <seeguide marker="erl_dist_protocol#dflags">distribution flag</seeguide>. + </p> + </item> </taglist> </section> + <section> + <marker id="link_protocol"/> + <title>Link Protocol</title> + + <section> + <marker id="new_link_protocol"/> + <title>New Link Protocol</title> + + <p> + The new link protocol will be used when both nodes flag that + they understand it using the + <seeguide marker="#DFLAG_UNLINK_ID"><c>DFLAG_UNLINK_ID</c></seeguide> + <seeguide marker="#dflags">distribution flag</seeguide>. If + one of the nodes does not understand the new link protocol, the + <seeguide marker="#old_link_protocol">old link protocol</seeguide> + will be used as a fallback. + </p> + + <p> + The new link protocol introduces two new signals, + <seeguide marker="#UNLINK_ID"><c>UNLINK_ID</c></seeguide> and + <seeguide marker="#UNLINK_ID"><c>UNLINK_ID_ACK</c></seeguide>, + which replace the old + <seeguide marker="#UNLINK"><c>UNLINK</c></seeguide> + signal. The old <seeguide marker="#LINK"><c>LINK</c></seeguide> + signal is still sent in order to set up a link, but handled + differently upon reception. + </p> + + <p> + In order to set up a link, a <c>LINK</c> signal is sent, from + the process initiating the operation, to the process that it + wants to link to. In order to remove a link, an + <c>UNLINK_ID</c> signal is sent, from the process initiating + the operation, to the linked process. The receiver of an + <c>UNLINK_ID</c> signal responds with an <c>UNLINK_ID_ACK</c> + signal. Upon reception of an <c>UNLINK_ID</c> signal, the + corresponding <c>UNLINK_ID_ACK</c> signal <em>must</em> be + sent before any other signals are sent to the sender of the + <c>UNLINK_ID</c> signal. Together with + <seeguide marker="erts:communication#passing-of-signals">the + signal ordering guarantees</seeguide> of Erlang this makes it + possible for the sender of the <c>UNLINK_ID</c> signal to know + the order of other signals which is essential for the protocol. + The <c>UNLINK_ID_ACK</c> signal should contain the same + <c>Id</c> as the <c>Id</c> contained in the <c>UNLINK_ID</c> + signal being acknowledged. + </p> + + <p> + Processes also need to maintain process local information about + links. The state of this process local information is changed + when the signals above are sent and received. This process + local information also determines if a signal should be sent + when a process calls + <seemfa marker="erlang#link/1"><c>link/1</c></seemfa> or + <seemfa marker="erlang#unlink/1"><c>unlink/1</c></seemfa>. + A <c>LINK</c> signal is only sent if there does not currently + exist an active link between the processes according to the + process local information and an <c>UNLINK_ID</c> signal is + only sent if there currently exists an active link between the + processes according to the process local information. + </p> + + <p> + The process local information about a link contains: + </p> + <taglist> + <tag>Pid</tag> + <item> + Process identifier of the linked process. + </item> + <tag>Active Flag</tag> + <item> + If set, the link is active and the process will react on + <seeguide marker="system/reference_manual:processes#receiving_exit_signals">incoming + exit signals</seeguide> issued due to the link. If not set, + the link is inactive and incoming exit signals, issued due + to the link, will be ignored. That is, the processes are + considered as <em>not</em> linked. + </item> + <tag>Unlink Id</tag> + <item> + Identifier of an outstanding unlink operation. That is, + an unlink operation that has not yet been acknowledged. + This information is only used when the active flag is not + set. + </item> + </taglist> + + <p> + A process is only considered linked to another process + if it has process local information about the link + containing the process identifier of the other process and + with the active flag set. + </p> + + <p> + The process local information about a link is updated as + follows: + </p> + <taglist> + <tag>A <c>LINK</c> signal is sent</tag> + <item> + Link information is created if not already existing. The + active flag is set, and unlink id is cleared. That is, + if we had an outstanding unlink operation we will ignore + the result of that operation and enable the link. + </item> + <tag>A <c>LINK</c> signal is received</tag> + <item> + If no link information already exists, it is created, the + active flag is set and unlink id is cleared. If the link + information already exists, the signal is silently ignored, + regardless of whether the active flag is set or not. + That is, if we have an outstanding unlink operation we will + <em>not</em> activate the link. In this scenario, the sender + of the <c>LINK</c> signal has not yet sent an + <c>UNLINK_ID_ACK</c> signal corresponding to our + <c>UNLINK_ID</c> signal which means that it will receive + our <c>UNLINK_ID</c> signal after it sent its + <c>LINK</c> signal. This in turn means that both processes + in the end will agree that there is no link between them. + </item> + <tag>An <c>UNLINK_ID</c> signal is sent</tag> + <item> + Link information already exists and the active flag is set + (otherwise the signal would not be sent). The active flag + is unset, and the unlink id of the signal is saved in the + link information. + </item> + <tag>An <c>UNLINK_ID</c> signal is received</tag> + <item> + If the active flag is set, information about the link + is removed. If the active flag is not set (that is, we have + an outstanding unlink operation), the information about the + link is left unchanged. + </item> + <tag>An <c>UNLINK_ID_ACK</c> signal is sent</tag> + <item> + This is done when an <c>UNLINK_ID</c> signal is received and + causes no further changes of the link information. + </item> + <tag>An <c>UNLINK_ID_ACK</c> signal is received</tag> + <item> + If information about the link exists, the active flag is not + set, and the unlink id in the link information equals the + <c>Id</c> in the signal, the link information is removed; + otherwise, the signal is ignored. + </item> + </taglist> + + <p> + When a process receives an exit signal due to a link, the + process will first react to the exit signal if the link + is active and then remove the process local information about + the link. + </p> + <p> + In case the connection is lost between two nodes, exit signals + with exit reason <c>noconnection</c> are sent to all processes + with links over the connection. This will cause all process + local information about links over the connection to be + removed. + </p> + <p> + Exactly the same link protocol is also used internally on an + Erlang node. The signals however have different formats since + they do not have to be sent over the wire. + </p> + </section> + + <section> + <marker id="old_link_protocol"/> + <title>Old Link Protocol</title> + + <p> + The old link protocol utilize two signals + <seeguide marker="#LINK"><c>LINK</c></seeguide>, and + <seeguide marker="#UNLINK"><c>UNLINK</c></seeguide>. The + <c>LINK</c> signal informs the other process that a link + should be set up, and the <c>UNLINK</c> signal informs the + other process that a link should be removed. This protocol + is however a bit too naive. If both processes operate on the + link simultaneously, the link may end up in an inconsistent + state where one process thinks it is linked while the other + thinks it is not linked. + </p> + <p> + This protocol is deprecated and support for it will be removed + in OTP 26. Until then, it will be used as fallback when + communicating with old nodes that do not understand the + <seeguide marker="#new_link_protocol">new link + protocol</seeguide>. + </p> + </section> + + </section> + </section> + </chapter> diff --git a/erts/doc/src/erl_nif.xml b/erts/doc/src/erl_nif.xml index d2a1581e35..57bff58db4 100644 --- a/erts/doc/src/erl_nif.xml +++ b/erts/doc/src/erl_nif.xml @@ -140,6 +140,21 @@ $> erl However, unused local stub functions will be optimized away by the compiler, causing loading of the NIF library to fail.</p> </note> + <warning> + <p> + There is a known limitation for Erlang fallback functions of NIFs. Avoid + functions involved in traversal of binaries by matching and + recursion. If a NIF is loaded over such function, binary arguments to + the NIF may get corrupted and cause VM crash or other misbehavior. + </p> + <p>Example of such bad fallback function:</p> + <code type="none"> +skip_until(Byte, <<Byte, Rest/binary>>) -> + Rest; +skip_until(Byte, <<_, Rest/binary>>) -> + skip_until(Byte, Rest). +</code> + </warning> </section> <section> diff --git a/erts/doc/src/erlang.xml b/erts/doc/src/erlang.xml index 97f6f7e83d..dcefdafcaf 100644 --- a/erts/doc/src/erlang.xml +++ b/erts/doc/src/erlang.xml @@ -237,7 +237,7 @@ <datatype> <name name="message_queue_data"></name> <desc> - <p>Process message queue data configuration. For more info see + <p>Process message queue data configuration. For more information, see <seeerl marker="#process_flag_message_queue_data"><c>process_flag(message_queue_data, MQD)</c></seeerl> </p> </desc> @@ -2696,39 +2696,85 @@ false</code> <func> <name name="link" arity="1" since=""/> - <fsummary>Create a link to another process (or port).</fsummary> + <fsummary>Create a link to another process or port.</fsummary> <desc> - <p>Creates a link between the calling process and another process (or - port) <c><anno>PidOrPort</anno></c>. If the link already exists or a - process attempts to create a link to itself, nothing is done. Returns - <c>true</c> if the link is set up.</p> - - <p>If <c><anno>PidOrPort</anno></c> does not exist and checking it is - cheap, a <c>noproc</c> error is raised. Currently, checking is cheap - if the <c><anno>PidOrPort</anno></c> is local and the caller does not - trap exits (see <seemfa marker="#process_flag/2"><c>process_flag/2 - </c></seemfa>).</p> - - <p>Apart from any exit signals from the linked process itself, two - special exit signals may be sent to the calling process:</p> - - <list type="bulleted"> + <p> + Sets up and activates a link between the calling process and + another process or a port identified by + <c><anno>PidOrPort</anno></c>. We will from here on call the + identified process or port linkee. If the linkee is a port, it + must reside on the same node as the caller. + </p> - <item><p><c>noproc</c> is sent immediately if - <c><anno>PidOrPort</anno></c> does not exist at the time of linking - (if the caller is trapping exits or <c><anno>PidOrPort</anno></c> is - remote).</p></item> + <p> + If one of the participants of a link terminates, it will + <seeguide marker="system/reference_manual:processes#sending_exit_signals">send + an exit signal</seeguide> to the other participant. The exit + signal will contain the + <seeguide marker="system/reference_manual:processes#link_exit_signal_reason">exit + reason</seeguide> of the terminated participant. Other cases when + exit signals are triggered due to a link are when no linkee exist + (<c>noproc</c> exit reason) and when the connection between linked + processes on different nodes is lost or cannot be established + (<c>noconnection</c> exit reason). + </p> - <item><p><c>noconnection</c> if <c><anno>PidOrPort</anno></c> is - remote and a connection between the nodes could not be established - or was severed.</p></item> + <p> + An existing link can be removed by calling + <seemfa marker="#unlink/1"><c>unlink/1</c></seemfa>. + For more information on links and exit signals due to links, see + the <i>Processes</i> chapter in the <i>Erlang Reference Manual</i>: + </p> + <list> + <item> + <seeguide marker="system/reference_manual:processes#links">Links</seeguide> + </item> + <item> + <seeguide marker="system/reference_manual:processes#sending_exit_signals">Sending + Exit Signals</seeguide> + </item> + <item> + <seeguide marker="system/reference_manual:processes#receiving_exit_signals">Receiving + Exit Signals</seeguide> + </item> + </list> - </list> + <p> + For historical reasons, <c>link/1</c> has a strange + semi-synchronous behavior when it is "cheap" to check if the + linkee exists or not, and the caller does not + <seeerl marker="erlang#process_flag_trap_exit">trap exits</seeerl>. + If the above is true and the linkee does not exist, <c>link/1</c> + will raise a <c>noproc</c> error <em>exception</em>. The expected + behavior would instead have been that <c>link/1</c> returned + <c>true</c>, and the caller later was sent an exit signal + with <c>noproc</c> exit reason, but this is unfortunately not the + case. The <c>noproc</c> + <seeguide marker="system/reference_manual:errors#exceptions"> + exception</seeguide> is not to be confused with an + <seeguide marker="system/reference_manual:processes#sending_exit_signals">exit + signal</seeguide> with exit reason <c>noproc</c>. Currently it is "cheap" + to check if the linkee exists when it is supposed to reside on + the same node as the calling process. + </p> - <p>See <seeguide marker="system/reference_manual:processes#links">Processes - âžœ Links</seeguide> in the Erlang Reference Manual for more details.</p> + <p> + The link setup and activation is performed asynchronously. If the + link already exists, or if the caller attempts to create a link + to itself, nothing is done. A detailed description of the + <seeguide marker="erts:erl_dist_protocol#link_protocol">link + protocol</seeguide> can be found in the <i>Distribution Protocol</i> + chapter of the <i>ERTS User's Guide</i>. + </p> + <p>Failure:</p> + <list> + <item><c>badarg</c> if <c><anno>PidOrPort</anno></c> does not identify + a process or a node local port.</item> + <item><c>noproc</c> linkee does not exist and it is "cheap" to check + if it exists as described above.</item> + </list> </desc> </func> @@ -4955,7 +5001,8 @@ RealSystem = system + MissedSystem</code> </func> <func> - <name name="process_flag" arity="2" clause_i="1" since=""/> + <name name="process_flag" arity="2" clause_i="1" + anchor="process_flag_trap_exit" since=""/> <fsummary>Set process flag trap_exit for the calling process.</fsummary> <desc> <p>When <c>trap_exit</c> is set to <c>true</c>, exit signals @@ -5088,37 +5135,37 @@ RealSystem = system + MissedSystem</code> </fsummary> <type name="message_queue_data"/> <desc> - <p>This flag determines how messages in the message queue + <p>Determines how messages in the message queue are stored, as follows:</p> <taglist> <tag><c>off_heap</c></tag> <item> <p><em>All</em> messages in the message queue will be stored - outside of the process heap. This implies that <em>no</em> + outside the process heap. This implies that <em>no</em> messages in the message queue will be part of a garbage collection of the process.</p> </item> <tag><c>on_heap</c></tag> <item> <p>All messages in the message queue will eventually be - placed on heap. They can however temporarily be stored - off heap. This is how messages always have been stored + placed on the process heap. They can, however, be temporarily + stored off the heap. This is how messages have always been stored up until ERTS 8.0.</p> </item> </taglist> - <p>The default <c>message_queue_data</c> process flag is determined - by command-line argument <seecom marker="erl#+hmqd"> + <p>The default value of the <c>message_queue_data</c> process flag is + determined by the command-line argument <seecom marker="erl#+hmqd"> <c>+hmqd</c></seecom> in <c>erl(1)</c>.</p> - <p>If the process potentially can get many messages in its queue, - you are advised to set the flag to <c>off_heap</c>. This - because a garbage collection with many messages placed on - the heap can become extremely expensive and the process can - consume large amounts of memory. Performance of the - actual message passing is however generally better when not - using flag <c>off_heap</c>.</p> - <p>When changing this flag messages will be moved. This work - has been initiated but not completed when this function - call returns.</p> + <p>If the process may potentially accumulate a large number of messages + in its queue it is recommended to set the flag value to <c>off_heap</c>. + This is due to the fact that the garbage collection of a process that + has a large number of messages stored on the heap can become extremely + expensive and the process can consume large amounts of memory. + The performance of the actual message passing is, however, generally + better when the flag value is <c>on_heap</c>.</p> + <p>Changing the flag value causes any existing messages to be moved. + The move operation is initiated, but not necessarily completed, + by the time the function returns.</p> <p>Returns the old value of the flag.</p> </desc> </func> @@ -5386,9 +5433,10 @@ RealSystem = system + MissedSystem</code> removed without prior notice. In the current implementation <c><anno>BinInfo</anno></c> is a list of tuples. The tuples contain; <c>BinaryId</c>, <c>BinarySize</c>, <c>BinaryRefcCount</c>.</p> - <p>The message queue is on the heap depending on the - process flag <seeerl marker="#process_flag_message_queue_data"> - <c>message_queue_data</c></seeerl>.</p> + <p>Depending on the value of the + <seeerl marker="#process_flag_message_queue_data"> + <c>message_queue_data</c></seeerl> process flag the message queue + may be stored on the heap.</p> </item> <tag><c>{catchlevel, <anno>CatchLevel</anno>}</c></tag> <item> @@ -5542,8 +5590,8 @@ RealSystem = system + MissedSystem</code> </item> <tag><c>{message_queue_data, <anno>MQD</anno>}</c></tag> <item> - <p>Returns the current state of process flag - <c>message_queue_data</c>. <c><anno>MQD</anno></c> is either + <p><c><anno>MQD</anno></c> is the current value of the + <c>message_queue_data</c> process flag, which can be either <c>off_heap</c> or <c>on_heap</c>. For more information, see the documentation of <seeerl marker="#process_flag_message_queue_data"> @@ -6531,11 +6579,11 @@ true</pre> </item> <tag><c>{message_queue_data, <anno>MQD</anno>}</c></tag> <item> - <p>Sets the state of the <c>message_queue_data</c> process - flag. <c><anno>MQD</anno></c> is to be either <c>off_heap</c> - or <c>on_heap</c>. The default + <p>Sets the value of the <c>message_queue_data</c> process + flag. <c><anno>MQD</anno></c> can be either <c>off_heap</c> + or <c>on_heap</c>. The default value of the <c>message_queue_data</c> process flag is determined by - command-line argument <seecom marker="erl#+hmqd"> + the command-line argument <seecom marker="erl#+hmqd"> <c>+hmqd</c></seecom> in <c>erl(1)</c>. For more information, see the documentation of <seeerl marker="#process_flag_message_queue_data"> @@ -8812,11 +8860,10 @@ Metadata = #{ pid => pid(), <c>message_queue_data</c></tag> <item> <p>Returns the default value of the <c>message_queue_data</c> - process flag, which is either <c>off_heap</c> or <c>on_heap</c>. - This default is set by command-line argument + process flag, which can be either <c>off_heap</c> or <c>on_heap</c>. + The default value is set by the command-line argument <seecom marker="erl#+hmqd"><c>+hmqd</c></seecom> in - <c>erl(1)</c>. For more information on the - <c>message_queue_data</c> process flag, see documentation of + <c>erl(1)</c>. For more information, see the documentation of <seeerl marker="#process_flag_message_queue_data"> <c>process_flag(message_queue_data, MQD)</c></seeerl>.</p> </item> @@ -11833,27 +11880,50 @@ timestamp() -> <name name="unlink" arity="1" since=""/> <fsummary>Remove a link to another process or port.</fsummary> <desc> - <p>Removes the link, if there is one, between the calling - process and the process or port referred to by - <c><anno>Id</anno></c>.</p> - <p>Returns <c>true</c> and does not fail, even if there is no - link to <c><anno>Id</anno></c>, or if <c><anno>Id</anno></c> - does not exist.</p> - <p>Once <c>unlink(<anno>Id</anno>)</c> has returned, - it is guaranteed that - the link between the caller and the entity referred to by - <c><anno>Id</anno></c> has no effect on the caller - in the future (unless - the link is setup again). If the caller is trapping exits, an - <c>{'EXIT', <anno>Id</anno>, _}</c> message from the link - can have been placed in the caller's message queue before - the call.</p> - <p>Notice that the <c>{'EXIT', <anno>Id</anno>, _}</c> - message can be the - result of the link, but can also be the result of <c>Id</c> - calling <c>exit/2</c>. Therefore, it <em>can</em> be - appropriate to clean up the message queue when trapping exits - after the call to <c>unlink(<anno>Id</anno>)</c>, as follows:</p> + <p> + Removes a link between the calling process and another process + or a port identified by <c><anno>Id</anno></c>. We will from + here on call the identified process or port unlinkee. + </p> + + <p> + A link can be set up using the + <seemfa marker="#link/1"><c>link/1</c></seemfa> BIF. For more + information on links and exit signals due to links, see the + <i>Processes</i> chapter in the <i>Erlang Reference Manual</i>: + </p> + <list> + <item> + <seeguide marker="system/reference_manual:processes#links">Links</seeguide> + </item> + <item> + <seeguide marker="system/reference_manual:processes#sending_exit_signals">Sending + Exit Signals</seeguide> + </item> + <item> + <seeguide marker="system/reference_manual:processes#receiving_exit_signals">Receiving + Exit Signals</seeguide> + </item> + </list> + + <p> + Once <c>unlink(<anno>Id</anno>)</c> has returned, it is + guaranteed that the link between the caller and the unlinkee + has no effect on the caller in the future (unless the link is + setup again). Note that if the caller is + <seeerl marker="erts:erlang#process_flag_trap_exit">trapping + exits</seeerl>, an <c>{'EXIT', <anno>Id</anno>, ExitReason}</c> + message due to the link may have been placed in the message + queue of the caller before the <c>unlink(<anno>Id</anno>)</c> + call completed. Also note that the <c>{'EXIT', <anno>Id</anno>, + ExitReason}</c> message may be the result of the link, but + may also be the result of the unlikee sending the caller an + exit signal by calling the + <seemfa marker="#exit/2"><c>exit/2</c></seemfa> BIF. + Therefore, it may or may not be appropriate to clean up + the message queue after a call to <c>unlink(<anno>Id</anno>)</c> + as follows, when trapping exits: + </p> <code type="none"> unlink(Id), receive @@ -11862,16 +11932,19 @@ receive after 0 -> true end</code> - <note> - <p>Before Erlang/OTP R11B (ERTS 5.5) <c>unlink/1</c> - behaved completely asynchronously, that is, the link was active - until the "unlink signal" reached the linked entity. This - had an undesirable effect, as you could never know when - you were guaranteed <em>not</em> to be effected by the link.</p> - <p>The current behavior can be viewed as two combined operations: - asynchronously send an "unlink signal" to the linked entity - and ignore any future results of the link.</p> - </note> + + <p> + The link removal is performed asynchronously. If such a link + does not exist, nothing is done. A detailed description of the + <seeguide marker="erts:erl_dist_protocol#link_protocol">link + protocol</seeguide> can be found in the <i>Distribution Protocol</i> + chapter of the <i>ERTS User's Guide</i>. + </p> + + <p> + Failure: <c>badarg</c> if <c>Id</c> does not identify a process + or a node local port. + </p> </desc> </func> diff --git a/erts/doc/src/notes.xml b/erts/doc/src/notes.xml index 29f33ba2d8..de11e50ec2 100644 --- a/erts/doc/src/notes.xml +++ b/erts/doc/src/notes.xml @@ -31,6 +31,159 @@ </header> <p>This document describes the changes made to the ERTS application.</p> +<section><title>Erts 11.2</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Fix memory leak of about 6 words when + <c>erlang:process_flag/3</c> is called with a pid of an + already dead process. Bug exists since OTP-21.0.</p> + <p> + Own Id: OTP-17081 Aux Id: PR-2930 </p> + </item> + <item> + <p> + Fixed small memory leak in <c>erl_drv_send_term</c> and + <c>erl_drv_output_term</c> when failing due to the term + being invalid.</p> + <p> + Own Id: OTP-17089 Aux Id: PR-2934 </p> + </item> + <item> + <p> + The DTrace/SystemTap <c>process_heap_grow</c> probe is + now called with valid the heap and stack pointers for the + process in question.</p> + <p> + Own Id: OTP-17096 Aux Id: PR-2932 </p> + </item> + <item> + <p>Fixed a performance issue in memory allocation for + Linux kernels that didn't support <c>MADV_FREE</c>.</p> + <p> + Own Id: OTP-17124</p> + </item> + <item> + <p> + A <seeguide + marker="erts:erl_dist_protocol#new_link_protocol">new + link protocol</seeguide> has been introduced which + prevents links from ending up in an inconsistent state + where one participant considers itself linked while the + other doesn't. This bug has always existed in the + distributed case, but has since OTP 21 also existed in + the node local case since the distributed link protocol + then was adopted also for node local links. The bug + could, however, only trigger if both participants + operated on the link simultaneously.</p> + <p> + Own Id: OTP-17127</p> + </item> + <item> + <p> + Fix memory leak when receiving sigchld from port program + to already dead port.</p> + <p> + Own Id: OTP-17163</p> + </item> + <item> + <p> + Fix bug where complex seq_trace tokens (that is lists, + tuples, maps etc) could becomes corrupted by the GC. The + bug was introduced in OTP-21.</p> + <p> + Own Id: OTP-17209 Aux Id: PR-3039 </p> + </item> + <item> + <p> + Fixed WSLPATH environment variable addition to PATH on + windows, the last character was lost.</p> + <p> + Own Id: OTP-17229</p> + </item> + <item> + <p> + Fixed a bug in the timer implementation which could cause + timers that were set to more than 37.25 hours in the + future to be delayed. This could occur if there were + multiple timers scheduled to be triggered very close in + time, but still at different times, and the scheduler + thread handling the timers was not able to handle them + quickly enough. Delayed timers were in this case + triggered when another unrelated timer was triggered.</p> + <p> + Own Id: OTP-17253</p> + </item> + <item> + <p> + Fixed small memory leak in <c>erlang:trace/3</c> if + option <c>{tracer,_}</c> is included and the option list + is invalid or the call races with a concurrent trace or + code change operation.</p> + <p> + Own Id: OTP-17265 Aux Id: PR-4596 </p> + </item> + <item> + <p> + Fix configure check for <c>inet_pton</c> on 32-bit + windows. The failure of this check would cause epmd to be + built without ipv6 support.</p> + <p> + Own Id: OTP-17283</p> + </item> + </list> + </section> + + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + Various address sanitizer support.</p> + <p> + Own Id: OTP-16959 Aux Id: PR-2965 </p> + </item> + <item> + <p>The emulator will now honor <c>cgroup2</c> CPU + quotas.</p> + <p> + Own Id: OTP-17002</p> + </item> + <item> + <p> + Improved memory barrier usage on ARMv8 hardware, and + specifically on Apple silicon.</p> + <p> + Own Id: OTP-17195 Aux Id: PR-4505, PR-4538 </p> + </item> + <item> + <p> + Improved memory barrier usage on 64-bit POWER hardware.</p> + <p> + Own Id: OTP-17200 Aux Id: PR-4510 </p> + </item> + <item> + <p> + Fix a file descriptor leak when using sendfile and the + remote side closes the connection. This bug has been + present since OTP-21.0.</p> + <p> + Own Id: OTP-17244</p> + </item> + <item> + <p> + Refinement of the documentation of the + <c>message_queue_data</c> process flag.</p> + <p> + Own Id: OTP-17252 Aux Id: PR-4568 </p> + </item> + </list> + </section> + +</section> + <section><title>Erts 11.1.8</title> <section><title>Fixed Bugs and Malfunctions</title> @@ -1216,6 +1369,111 @@ </section> +<section><title>Erts 10.7.2.9</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Fixed a bug in the timer implementation which could cause + timers that were set to more than 37.25 hours in the + future to be delayed. This could occur if there were + multiple timers scheduled to be triggered very close in + time, but still at different times, and the scheduler + thread handling the timers was not able to handle them + quickly enough. Delayed timers were in this case + triggered when another unrelated timer was triggered.</p> + <p> + Own Id: OTP-17253</p> + </item> + <item> + <p> + Fix bug in call_time tracing (used by eprof) that could + cause VM crash. Bug exists since OTP-22.2 (but not in + OTP-23).</p> + <p> + Own Id: OTP-17290 Aux Id: GH-4635 </p> + </item> + </list> + </section> + + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + Fix a file descriptor leak when using sendfile and the + remote side closes the connection. This bug has been + present since OTP-21.0.</p> + <p> + Own Id: OTP-17244</p> + </item> + </list> + </section> + +</section> + +<section><title>Erts 10.7.2.8</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p>Fixed a bug that could cause some work scheduled for + execution on scheduler threads to be delayed until other + similar work appeared. Beside delaying various cleanup of + internal data structures also the following could be + delayed:</p> <list> <item>Termination of a distribution + controller process</item> <item>Disabling of the + distribution on a node</item> <item>Gathering of memory + allocator information using the <c>instrument</c> + module</item> <item>Enabling, disabling, and gathering of + <c>msacc</c> information</item> <item>Delivery of + <c>'CHANGE'</c> messages when time offset is + monitored</item> <item>A call to + <c>erlang:cancel_timer()</c></item> <item>A call to + <c>erlang:read_timer()</c></item> <item>A call to + <c>erlang:statistics(io | garbage_collection | + scheduler_wall_time)</c></item> <item>A call to + <c>ets:all()</c></item> <item>A call to + <c>erlang:memory()</c></item> <item>A call to + <c>erlang:system_info({allocator | allocator_sizes, + _})</c></item> <item>A call to + <c>erlang:trace_delivered()</c></item> </list> <p>The bug + existed on runtime systems running on all types of + hardware except for x86/x86_64.</p> + <p> + Own Id: OTP-17185</p> + </item> + </list> + </section> + +</section> + +<section><title>Erts 10.7.2.7</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + The <c>suspend_process()</c> and <c>resume_process()</c> + BIFs did not check their arguments properly which could + cause an emulator crash.</p> + <p> + Own Id: OTP-17080</p> + </item> + <item> + <p> + The runtime system would get into an infinite loop if the + runtime system was started with more than 1023 file + descriptors already open.</p> + <p> + Own Id: OTP-17088 Aux Id: ERIERL-580 </p> + </item> + </list> + </section> + +</section> + <section><title>Erts 10.7.2.6</title> <section><title>Fixed Bugs and Malfunctions</title> @@ -3175,6 +3433,111 @@ </section> +<section><title>Erts 10.3.5.17</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Fixed a bug in the timer implementation which could cause + timers that were set to more than 37.25 hours in the + future to be delayed. This could occur if there were + multiple timers scheduled to be triggered very close in + time, but still at different times, and the scheduler + thread handling the timers was not able to handle them + quickly enough. Delayed timers were in this case + triggered when another unrelated timer was triggered.</p> + <p> + Own Id: OTP-17253</p> + </item> + </list> + </section> + + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + Fix a file descriptor leak when using sendfile and the + remote side closes the connection. This bug has been + present since OTP-21.0.</p> + <p> + Own Id: OTP-17244</p> + </item> + </list> + </section> + +</section> + +<section><title>Erts 10.3.5.16</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p>Fixed a bug that could cause some work scheduled for + execution on scheduler threads to be delayed until other + similar work appeared. Beside delaying various cleanup of + internal data structures also the following could be + delayed:</p> <list> <item>Termination of a distribution + controller process</item> <item>Disabling of the + distribution on a node</item> <item>Gathering of memory + allocator information using the <c>instrument</c> + module</item> <item>Enabling, disabling, and gathering of + <c>msacc</c> information</item> <item>Delivery of + <c>'CHANGE'</c> messages when time offset is + monitored</item> <item>A call to + <c>erlang:cancel_timer()</c></item> <item>A call to + <c>erlang:read_timer()</c></item> <item>A call to + <c>erlang:statistics(io | garbage_collection | + scheduler_wall_time)</c></item> <item>A call to + <c>ets:all()</c></item> <item>A call to + <c>erlang:memory()</c></item> <item>A call to + <c>erlang:system_info({allocator | allocator_sizes, + _})</c></item> <item>A call to + <c>erlang:trace_delivered()</c></item> </list> <p>The bug + existed on runtime systems running on all types of + hardware except for x86/x86_64.</p> + <p> + Own Id: OTP-17185</p> + </item> + </list> + </section> + +</section> + +<section><title>Erts 10.3.5.15</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Fixed rare distribution bug in race between received + signal (link/monitor/spawn_request/spawn_reply) and + disconnection. Symptom: VM crash. Since: OTP 21.0.</p> + <p> + Own Id: OTP-16869 Aux Id: ERL-1337 </p> + </item> + <item> + <p> + The <c>suspend_process()</c> and <c>resume_process()</c> + BIFs did not check their arguments properly which could + cause an emulator crash.</p> + <p> + Own Id: OTP-17080</p> + </item> + <item> + <p> + The runtime system would get into an infinite loop if the + runtime system was started with more than 1023 file + descriptors already open.</p> + <p> + Own Id: OTP-17088 Aux Id: ERIERL-580 </p> + </item> + </list> + </section> + +</section> + <section><title>Erts 10.3.5.14</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/erts/emulator/Makefile.in b/erts/emulator/Makefile.in index 59bc1eecd2..98dd6ea669 100644 --- a/erts/emulator/Makefile.in +++ b/erts/emulator/Makefile.in @@ -137,6 +137,14 @@ TYPE_FLAGS = $(DEBUG_CFLAGS) -DVALGRIND -DNO_JUMP_TABLE ENABLE_ALLOC_TYPE_VARS += valgrind else +ifeq ($(TYPE),asan) +PURIFY = +TYPEMARKER = .asan +TYPE_FLAGS = $(DEBUG_CFLAGS) -fsanitize=address -fsanitize-recover=address -fno-omit-frame-pointer -DADDRESS_SANITIZER +LDFLAGS += -fsanitize=address +ENABLE_ALLOC_TYPE_VARS += asan +else + ifeq ($(TYPE),gprof) PURIFY = TYPEMARKER = .gprof @@ -181,6 +189,7 @@ endif endif endif endif +endif LIBS += $(TYPE_LIBS) diff --git a/erts/emulator/asan/asan_logs_to_html b/erts/emulator/asan/asan_logs_to_html new file mode 100755 index 0000000000..14c9b7fcde --- /dev/null +++ b/erts/emulator/asan/asan_logs_to_html @@ -0,0 +1,453 @@ +#!/usr/bin/env escript +%% -*- erlang -*- + +%% Parse address sanitizer log files generated from test runs with +%% with environment variables ASAN_LOG_DIR and TS_RUN_EMU=asan set. + +%% Repeated leak reports are ignored and additional leaks of same type +%% as seen before are identified as such. + +-mode(compile). + +main([]) -> + help(); +main(["--help"]) -> + help(); +main([OutDir]) -> + case os:getenv("ASAN_LOG_DIR") of + false -> + io:format(standard_error, + "\nMissing asan log directory argument and environment\n" + "variable ASAN_LOG_DIR is not set.\n\n",[]), + help(); + InDir -> + run(OutDir, InDir) + end; +main([OutDir, InDir]) -> + run(OutDir, InDir). + + +help() -> + io:format("\nSyntax: asan_log_to_html OutDir [InDir]\n" + "\nParses all address-sanetizer log files in InDir\n" + "and generates a summary file OutDir/asan_summary.html.\n" + "Environment variable ASAN_LOG_DIR is used if InDir\n" + "is not specified\n\n", []). + +-record(logacc, {srcfile, % full path of current log file + did_output = false, % output contribution from srcfile + obuf = [], % output buffer + app = none, % current application + app_err = 0, % nr of reports from application + tc_err = 0, % nr of reports from srcfile (test case) + app_stat_bytes = 0, % total leaked bytes from app + app_stat_blocks = 0, % total leaked blocks from app + app_stat_errors = 0}). % total errors from app + +run(OutDir, InDir) -> + StartTime = erlang:monotonic_time(millisecond), + + {ok, InFilesUS} = file:list_dir(InDir), + InFiles = lists:sort(InFilesUS), + + OutFile = filename:join(OutDir, "asan_summary.html"), + {ok, Out} = file:open(OutFile, [write]), + + ok = file:write(Out, <<"<!DOCTYPE html>\n" + "<html>\n" + "<head><title>Address Sanitizer</title>\n">>), + ok = file:write(Out, style_block()), + ok = file:write(Out, <<"</head><body>\n" + "<h1>Address Sanitizer</h1>\n">>), + + {_, _, LogAcc2} = + lists:foldl(fun(File, {LM, RegEx, LogAcc}) -> + analyze_log_file(Out, filename:join(InDir,File), + {LM, RegEx, LogAcc}) + end, + {#{}, none, #logacc{}}, + InFiles), + + LogAcc3 = app_end(Out, LogAcc2), + try_delete_srcfile(LogAcc3), + + Time = calendar:system_time_to_rfc3339(erlang:system_time(second), + [{time_designator, 32}]), + %%{_, _, ThisFile} = code:get_object_code(?MODULE), + ThisFile = escript:script_name(), + User = string:trim(os:cmd("whoami")), + {ok, Host} = inet:gethostname(), + Seconds = (erlang:monotonic_time(millisecond) - StartTime) / 1000, + ok = io:format(Out, "\n<hr><p><small>This page was generated ~s\n" + " by <tt>~s</tt>\n" + " run by ~s@~s in ~.1f seconds.</small></p>\n", + [Time, ThisFile, User, Host, Seconds]), + + ok = file:write(Out, script_block()), + ok = file:write(Out, <<"</body>\n</html>\n">>), + ok = file:close(Out), + io:format("Generated file ~s\n", [OutFile]), + ok. + +analyze_log_file(Out, SrcFile, {LeakMap0, RegEx0, LogAcc0}=Acc) -> + + #logacc{app=PrevApp} = LogAcc0, + + case filelib:is_regular(SrcFile) of + false -> + Acc; + true -> + FileName = filename:basename(SrcFile), + %%io:format("analyze ~s\n", [FileName]), + + %% Is it a new application? + LogAcc2 = case string:lexemes(FileName, "-") of + [_Exe, PrevApp | _] -> + try_delete_srcfile(LogAcc0), + LogAcc0#logacc{srcfile=SrcFile, + tc_err=0, + did_output=false}; + [_Exe, NewApp | _] -> + LogAcc1 = app_end(Out, LogAcc0), + try_delete_srcfile(LogAcc1), + LogAcc1#logacc{srcfile=SrcFile, + obuf=[], + app=NewApp, + app_err=0, + tc_err=0, + did_output=false, + app_stat_bytes=0, + app_stat_blocks=0, + app_stat_errors=0} + end, + + case LogAcc2#logacc.app_err of + truncated -> + {LeakMap0, RegEx0, LogAcc2}; + _ -> + {ok, Bin} = file:read_file(SrcFile), + match_loop(Out, Bin, RegEx0, LogAcc2, 0, [], LeakMap0) + end + end. + +-define(APP_ERR_LIMIT, 200). + +match_loop(Out, _, RegEx, #logacc{app_err=AppErr}=LogAcc0, _, _, LM) + when AppErr >= ?APP_ERR_LIMIT -> + + Txt = [io_format("<h2>WARNING!!! Log truncated for application ~p," + " more than ~p errors found.</h2>\n", + [LogAcc0#logacc.app, ?APP_ERR_LIMIT])], + LogAcc1 = log_error(Out, LogAcc0, Txt), + {LM, RegEx, LogAcc1#logacc{app_err=truncated}}; + +match_loop(Out, Bin, RegEx0, LogAcc0, PrevEnd, Unmatched0, LM0) -> + {Match, RegEx1} = + run_regex(Bin, RegEx0, + %% LeakReport + "(?:(Direct|Indirect) leak of ([0-9]+) byte\\(s\\) " + "in ([0-9]+) object\\(s\\) allocated from:\n" + "((?:[ \t]*#[0-9]+.+\n)+))" % Call stack + "|" + %% ErrorReport + "(?:(==ERROR: AddressSanitizer:.*\n" + "(?:.*\n)+?)" % any lines (non-greedy) + "^(?:==|--))" % stop at line begining with == or -- + "|" + %% Skipped + "(?:^[=-]+$)" % skip lines consisting only of = or - + "|" + "Objects leaked above:\n" % if LSAN_OPTIONS="report_objects=1" + "(?:0x.+\n)+" + "|" + "^\n", % empty lines + [multiline], + [{offset, PrevEnd}, {capture, all, index}]), + + + BP = fun(PartIx) -> binary:part(Bin, PartIx) end, + + case Match of + [ErrorReport, {-1,0}, {-1,0}, {-1,0}, {-1,0}, Captured] -> + {Start,MatchLen} = ErrorReport, + Txt = [io_format("<pre~s>\n", [style(error)]), + file_write(BP(Captured)), + io_format("</pre>\n", [])], + LogAcc1 = log_error(Out, LogAcc0, Txt), + Unmatched1 = [BP({PrevEnd, Start-PrevEnd}) | Unmatched0], + End = Start + MatchLen, + match_loop(Out, Bin, RegEx1, app_stats(LogAcc1,0,0,1), + End, Unmatched1, LM0); + + [LeakReport, TypeIx, BytesIx, BlocksIx, StackIx | _] -> + {Start, MatchLen} = LeakReport, + Bytes = binary_to_integer(BP(BytesIx)), + Blocks = binary_to_integer(BP(BlocksIx)), + End = Start + MatchLen, + Unmatched1 = [BP({PrevEnd, Start-PrevEnd})|Unmatched0], + TypeBin = BP(TypeIx), + + %% We indentify a leak by its type (direct or indirect) + %% and its full call stack. + Key = {TypeBin, BP(StackIx)}, + {LogAcc2, LM2} = + case lookup_leak(LM0, Key) of + undefined -> + %% A new leak + LM1 = insert_leak(LM0, Key, Bytes, Blocks), + Txt = [io_format("<pre~s>\n", [style(new, TypeBin)]), + file_write(BP(LeakReport)), + io_format("</pre>\n", [])], + LogAcc1 = log_error(Out, LogAcc0, Txt), + {app_stats(LogAcc1,Bytes,Blocks,0), LM1}; + + {Bytes, Blocks} -> + %% Exact same leak(s) repeated, ignore + {LogAcc0, LM0}; + + {OldBytes, OldBlocks} -> + %% More leaked bytes/blocks of same type&stack as before + LM1 = insert_leak(LM0, Key, Bytes, Blocks), + ByteDiff = Bytes - OldBytes, + BlockDiff = Blocks - OldBlocks, + Txt = [io_format("<pre~s>\n", [style(more, TypeBin)]), + io_format("More ~s leak of ~w(~w) byte(s) " + "in ~w(~w) object(s) allocated from:\n", + [TypeBin, ByteDiff, Bytes, BlockDiff, Blocks]), + file_write(BP(StackIx)), + io_format("</pre>\n", [])], + LogAcc1 = log_error(Out, LogAcc0, Txt), + {app_stats(LogAcc1, ByteDiff, BlockDiff, 0), LM1} + end, + match_loop(Out, Bin, RegEx1, LogAcc2, End, Unmatched1, LM2); + + [SkipLine] -> + {Start, MatchLen} = SkipLine, + %%nomatch = binary:match(BP(SkipLine), <<"\n">>), % Assert single line + End = Start + MatchLen, + Unmatched1 = [BP({PrevEnd, Start-PrevEnd})|Unmatched0], + match_loop(Out, Bin, RegEx1, LogAcc0, End, Unmatched1, LM0); + + nomatch -> + Unmatched1 = [BP({PrevEnd, byte_size(Bin)-PrevEnd}) | Unmatched0], + + LogAcc1 = + case iolist_size(Unmatched1) > 500 of + true -> + Txt = [io_format("<h2>WARNING!!! May be unmatched error reports" + " in file ~s:</h2>\n<pre>~s</pre>", + [LogAcc0#logacc.srcfile, Unmatched1])], + log_error(Out, LogAcc0, Txt); + false -> + LogAcc0 + end, + {LM0, RegEx1, LogAcc1} + end. + +lookup_leak(LeakMap, Key) -> + maps:get(Key, LeakMap, undefined). + +insert_leak(LeakMap, Key, Bytes, Blocks) -> + LeakMap#{Key => {Bytes, Blocks}}. + +log_error(_Out, #logacc{app_err=AppErr, tc_err=TcErr}=LogAcc, Txt0) -> + {DidTc, Txt1} = + case TcErr of + 0 -> + %% First error in test case, print test case header + SrcFile = LogAcc#logacc.srcfile, + TcFile = filename:basename(SrcFile), + Hdr = case string:lexemes(TcFile, "-") of + [_Exe, App, _Rest] -> + io_format("<h3>Before first test case of ~s</h3>\n", + [App]); + [_Exe, _App, "tc", Num, Mod, Rest] -> + [Func | _] = string:lexemes(Rest, "."), + io_format("<h3>Test case #~s ~s:~s</h3>\n", + [Num, Mod, Func]); + _ -> + io_format("<h3>Strange log file name '~s'</h3>\n", + [SrcFile]) + end, + {true, [Hdr | Txt0]}; + _ -> + {false, Txt0} + end, + LogAcc#logacc{app_err=AppErr+1, tc_err=TcErr+1, + obuf = [Txt1 | LogAcc#logacc.obuf], + did_output = (LogAcc#logacc.did_output or DidTc)}. + +app_stats(#logacc{}=LogAcc, Bytes, Blocks, Errors) -> + LogAcc#logacc{app_stat_bytes = LogAcc#logacc.app_stat_bytes + Bytes, + app_stat_blocks = LogAcc#logacc.app_stat_blocks + Blocks, + app_stat_errors = LogAcc#logacc.app_stat_errors + Errors}. + + +app_end(Out, LogAcc) -> + case LogAcc of + #logacc{app=none} -> + LogAcc; + #logacc{app_err=0} -> + ok = io:format(Out, "<button class=\"app_ok\" disabled>~s</button>\n", + [LogAcc#logacc.app]), + LogAcc#logacc{did_output = true}; + #logacc{} -> + %% Print red clickable app button with stats + %% and all the buffered logs. + ok = io:format(Out, "<button type=\"button\" class=\"app_err\">~s" + "<span class=\"stats\">Leaks: ~p bytes in ~p blocks</span>", + [LogAcc#logacc.app, + LogAcc#logacc.app_stat_bytes, + LogAcc#logacc.app_stat_blocks]), + case LogAcc#logacc.app_stat_errors of + 0 -> ignore; + _ -> + ok = io:format(Out, "<span class=\"stats\">Errors: ~p</span>", + [LogAcc#logacc.app_stat_errors]) + end, + ok = io:format(Out, "</button>\n" + "<div class=\"content\">", []), + + flush_obuf(Out, LogAcc#logacc.obuf), + + ok = io:format(Out, "<button type=\"button\" " + "class=\"app_err_end\">" + "end of ~s</button>\n", [LogAcc#logacc.app]), + ok = io:format(Out, "</div>", []), + LogAcc + end. + +flush_obuf(Out, Obuf) -> + flush_obuf_rev(Out, lists:reverse(Obuf)). + +flush_obuf_rev(_Out, []) -> ok; +flush_obuf_rev(Out, [Txt | T]) -> + [OutFun(Out) || OutFun <- Txt], + flush_obuf_rev(Out, T). + +io_format(Frmt, List) -> + fun(Out) -> io:format(Out, Frmt, List) end. + +file_write(Bin) -> + fun(Out) -> file:write(Out, Bin) end. + +style(error) -> + " style=\"background-color:Tomato;\"". + +style(new, <<"Direct">>) -> + " style=\"background-color:orange;\""; +style(new, <<"Indirect">>) -> + ""; +style(more, _) -> + " style=\"background-color:yellow;\"". + + +run_regex(Bin, none, RegExString, CompileOpts, RunOpts) -> + {ok, RegEx} = re:compile(RegExString, CompileOpts), + run_regex(Bin, RegEx, none, none, RunOpts); +run_regex(Bin, RegEx, _, _, RunOpts) -> + case re:run(Bin, RegEx, RunOpts) of + nomatch -> + {nomatch, RegEx}; + {match, Match} -> + {Match, RegEx} + end. + +try_delete_srcfile(LogAcc) -> + case LogAcc of + #logacc{srcfile=undefined} -> + ignore; + #logacc{did_output=false} -> + %% This file did not contribute any output. + %% Optimize future script invokations by removing it. + delete_file(LogAcc#logacc.srcfile); + _ -> + keep + end. + +delete_file(File) -> + case filelib:is_regular(File) of + true -> + io:format("Delete file ~p\n", [File]), + Dir = filename:dirname(File), + Name = filename:basename(File), + Trash = filename:join([Dir, "DELETED", Name]), + ok = filelib:ensure_dir(Trash), + ok = file:rename(File, Trash); + false -> + ignore + end. + +style_block() -> + <<"<style> + +.app_err, .app_err_end, .app_ok { + color: white; + padding: 10px; + /*border: none;*/ + text-align: left; + /*outline: none;*/ + font-size: 15px; +} + +.app_err { + width: 100%; + background-color: #D11; + cursor: pointer; +} +.app_err:hover { + background-color: #F11; +} +.app_err_end { + background-color: #D11; + cursor: pointer; +} +.app_err_end:hover { + background-color: #F11; +} + +.app_ok { + width: 100%; + background-color: #292; +} + +.stats { + font-style: italic; + margin-left: 50px; +} + +.content { + padding: 0 18px; + display: none; + overflow: hidden; + background-color: #f1f1f1; +} +</style> +">>. + +script_block() -> + <<"<script> +var app_err = document.getElementsByClassName(\"app_err\"); +var i; + +for (i = 0; i < app_err.length; i++) { + app_err[i].addEventListener(\"click\", function() { + var content = this.nextElementSibling; + if (content.style.display === \"block\") { + content.style.display = \"none\"; + } else { + content.style.display = \"block\"; + } + }); +} + +var app_err_end = document.getElementsByClassName(\"app_err_end\"); +for (i = 0; i < app_err_end.length; i++) { + app_err_end[i].addEventListener(\"click\", function() { + var content = this.parentElement; + content.style.display = \"none\"; + }); +} + +</script> +">>. diff --git a/erts/emulator/asan/suppress b/erts/emulator/asan/suppress new file mode 100644 index 0000000000..5625938f37 --- /dev/null +++ b/erts/emulator/asan/suppress @@ -0,0 +1,18 @@ +leak:erts_alloc_permanent_cache_aligned + +# Harmless leak of ErtsThrPrgrData from async threads in exiting emulator +leak:erts_thr_progress_register_unmanaged_thread + +# Block passed to sigaltstack() +leak:sys_thread_init_signal_stack + +#Copied from valgrind/suppress.standard: +#Crypto internal... loading gives expected errors when curves are tried. +#But including <openssl/err.h> and removing them triggers compiler errors on Windows +#fun:valid_curve +#fun:init_curves +leak:init_curve_types +#fun:init_algorithms_types +#fun:initialize +#fun:load +#fun:erts_load_nif diff --git a/erts/emulator/beam/bif.c b/erts/emulator/beam/bif.c index 4f1f114eba..4e71600fa8 100644 --- a/erts/emulator/beam/bif.c +++ b/erts/emulator/beam/bif.c @@ -106,8 +106,7 @@ BIF_RETTYPE link_1(BIF_ALIST_1) if (is_internal_pid(BIF_ARG_1)) { int created; - ErtsLinkData *ldp; - ErtsLink *lnk; + ErtsLink *lnk, *rlnk; if (BIF_P->common.id == BIF_ARG_1) BIF_RET(am_true); @@ -115,56 +114,61 @@ BIF_RETTYPE link_1(BIF_ALIST_1) if (!erts_proc_lookup(BIF_ARG_1)) goto res_no_proc; - lnk = erts_link_tree_lookup_create(&ERTS_P_LINKS(BIF_P), - &created, - ERTS_LNK_TYPE_PROC, - BIF_P->common.id, - BIF_ARG_1); - if (!created) - BIF_RET(am_true); + lnk = erts_link_internal_tree_lookup_create(&ERTS_P_LINKS(BIF_P), + &created, + ERTS_LNK_TYPE_PROC, + BIF_ARG_1); + if (!created) { + ErtsILink *ilnk = (ErtsILink *) lnk; + if (!ilnk->unlinking) + BIF_RET(am_true); + ilnk->unlinking = 0; + } - ldp = erts_link_to_data(lnk); - + rlnk = erts_link_internal_create(ERTS_LNK_TYPE_PROC, BIF_P->common.id); - if (erts_proc_sig_send_link(BIF_P, BIF_ARG_1, &ldp->b)) + if (erts_proc_sig_send_link(BIF_P, BIF_ARG_1, rlnk)) BIF_RET(am_true); erts_link_tree_delete(&ERTS_P_LINKS(BIF_P), lnk); - erts_link_release_both(ldp); + erts_link_internal_release(lnk); + erts_link_internal_release(rlnk); goto res_no_proc; } if (is_internal_port(BIF_ARG_1)) { int created; - ErtsLinkData *ldp; - ErtsLink *lnk; + ErtsLink *lnk, *rlnk; Eterm ref; Eterm *refp; Port *prt = erts_port_lookup(BIF_ARG_1, (erts_port_synchronous_ops ? ERTS_PORT_SFLGS_INVALID_DRIVER_LOOKUP : ERTS_PORT_SFLGS_INVALID_LOOKUP)); - if (!prt) { + if (!prt) goto res_no_proc; - } - lnk = erts_link_tree_lookup_create(&ERTS_P_LINKS(BIF_P), - &created, - ERTS_LNK_TYPE_PORT, - BIF_P->common.id, - BIF_ARG_1); - if (!created) - BIF_RET(am_true); + lnk = erts_link_internal_tree_lookup_create(&ERTS_P_LINKS(BIF_P), + &created, + ERTS_LNK_TYPE_PORT, + BIF_ARG_1); + if (!created) { + ErtsILink *ilnk = (ErtsILink *) lnk; + if (!ilnk->unlinking) + BIF_RET(am_true); + ilnk->unlinking = 0; + } - ldp = erts_link_to_data(lnk); + rlnk = erts_link_internal_create(ERTS_LNK_TYPE_PROC, BIF_P->common.id); refp = erts_port_synchronous_ops ? &ref : NULL; - switch (erts_port_link(BIF_P, prt, &ldp->b, refp)) { - case ERTS_PORT_OP_DROPPED: + switch (erts_port_link(BIF_P, prt, rlnk, refp)) { case ERTS_PORT_OP_BADARG: + erts_link_internal_release(rlnk); erts_link_tree_delete(&ERTS_P_LINKS(BIF_P), lnk); - erts_link_release_both(ldp); + erts_link_internal_release(lnk); goto res_no_proc; + case ERTS_PORT_OP_DROPPED: case ERTS_PORT_OP_SCHEDULED: if (refp) { ASSERT(is_internal_ordinary_ref(ref)); @@ -181,10 +185,10 @@ BIF_RETTYPE link_1(BIF_ALIST_1) } if (is_external_pid(BIF_ARG_1)) { - ErtsLinkData *ldp; - int created; + ErtsELink *elnk, *relnk, *pelnk; + int created, replace; DistEntry *dep; - ErtsLink *lnk; + ErtsLink *lnk, *rlnk; int code; ErtsDSigSendContext ctx; @@ -192,16 +196,39 @@ BIF_RETTYPE link_1(BIF_ALIST_1) if (dep == erts_this_dist_entry) goto res_no_proc; - lnk = erts_link_tree_lookup_create(&ERTS_P_LINKS(BIF_P), - &created, - ERTS_LNK_TYPE_DIST_PROC, - BIF_P->common.id, - BIF_ARG_1); + lnk = erts_link_external_tree_lookup_create(&ERTS_P_LINKS(BIF_P), + &created, + ERTS_LNK_TYPE_DIST_PROC, + BIF_P->common.id, + BIF_ARG_1); + + elnk = erts_link_to_elink(lnk); - if (!created) - BIF_RET(am_true); /* Already present... */ + if (created) { + pelnk = NULL; + relnk = NULL; + rlnk = NULL; + } + else { + if (!elnk->unlinking) + BIF_RET(am_true); /* Already present... */ + /* + * We need to replace the link if the connection has changed. + * Prepare a link... + */ + pelnk = (ErtsELink *) erts_link_external_create(ERTS_LNK_TYPE_DIST_PROC, + BIF_P->common.id, + BIF_ARG_1); + ASSERT(eq(pelnk->ld.proc.other.item, BIF_ARG_1)); + ASSERT(pelnk->ld.dist.other.item == BIF_P->common.id); + /* Release pelnk if not used as replacement... */ + relnk = pelnk; + rlnk = &pelnk->ld.proc; + } + replace = 0; - ldp = erts_link_to_data(lnk); + ASSERT(eq(elnk->ld.proc.other.item, BIF_ARG_1)); + ASSERT(elnk->ld.dist.other.item == BIF_P->common.id); code = erts_dsig_prepare(&ctx, dep, BIF_P, ERTS_PROC_LOCK_MAIN, @@ -209,31 +236,75 @@ BIF_RETTYPE link_1(BIF_ALIST_1) switch (code) { case ERTS_DSIG_PREP_NOT_ALIVE: case ERTS_DSIG_PREP_NOT_CONNECTED: - erts_link_set_dead_dist(&ldp->b, dep->sysname); - erts_proc_sig_send_link_exit(NULL, BIF_ARG_1, &ldp->b, + if (created || elnk->unlinking) { + if (elnk->unlinking) { + /* + * Currently unlinking an old link from an old connection; replace + * old link with the prepared one... + */ + relnk = NULL; + rlnk = lnk; + elnk = pelnk; + replace = !0; + } + erts_link_set_dead_dist(&elnk->ld.dist, dep->sysname); + } + erts_proc_sig_send_link_exit(NULL, THE_NON_VALUE, &elnk->ld.dist, am_noconnection, NIL); - BIF_RET(am_true); + break; case ERTS_DSIG_PREP_PENDING: case ERTS_DSIG_PREP_CONNECTED: { /* - * We have (pending) connection. + * We have a connection (or a pending connection). * Setup link and enqueue link signal. */ - int inserted = erts_link_dist_insert(&ldp->b, dep->mld); - ASSERT(inserted); (void)inserted; + if (created + || (elnk->unlinking + && elnk->dist->connection_id != ctx.connection_id)) { + int inserted; + if (!created) { + /* + * Currently unlinking an old link from an old connection; replace + * old link with the prepared one... + */ + rlnk = lnk; + if (erts_link_dist_delete(&elnk->ld.dist)) + relnk = elnk; + else + relnk = NULL; + elnk = pelnk; + replace = !0; + } + inserted = erts_link_dist_insert(&elnk->ld.dist, dep->mld); + ASSERT(inserted); (void)inserted; + } + erts_de_runlock(dep); code = erts_dsig_send_link(&ctx, BIF_P->common.id, BIF_ARG_1); if (code == ERTS_DSIG_SEND_YIELD) ERTS_BIF_YIELD_RETURN(BIF_P, am_true); ASSERT(code == ERTS_DSIG_SEND_OK); - BIF_RET(am_true); break; } default: ERTS_ASSERT(! "Invalid dsig prepare result"); } + + if (replace) { + ASSERT(pelnk); + erts_link_tree_replace(&ERTS_P_LINKS(BIF_P), rlnk, &pelnk->ld.proc); + } + + if (relnk) + erts_link_release_both(&relnk->ld); + else if (rlnk) + erts_link_release(rlnk); + + elnk->unlinking = 0; + + BIF_RET(am_true); } BIF_ERROR(BIF_P, BADARG); @@ -915,38 +986,52 @@ BIF_RETTYPE unlink_1(BIF_ALIST_1) } if (is_internal_pid(BIF_ARG_1)) { - ErtsLink *lnk = erts_link_tree_lookup(ERTS_P_LINKS(BIF_P), BIF_ARG_1); - if (lnk) { - erts_link_tree_delete(&ERTS_P_LINKS(BIF_P), lnk); - erts_proc_sig_send_unlink(BIF_P, lnk); + ErtsILink *ilnk; + ilnk = (ErtsILink *) erts_link_tree_lookup(ERTS_P_LINKS(BIF_P), + BIF_ARG_1); + if (ilnk && !ilnk->unlinking) { + Uint64 id = erts_proc_sig_send_unlink(BIF_P, + BIF_P->common.id, + &ilnk->link); + if (id) + ilnk->unlinking = id; + else { + erts_link_tree_delete(&ERTS_P_LINKS(BIF_P), &ilnk->link); + erts_link_internal_release(&ilnk->link); + } } BIF_RET(am_true); } if (is_internal_port(BIF_ARG_1)) { - ErtsLink *lnk = erts_link_tree_lookup(ERTS_P_LINKS(BIF_P), BIF_ARG_1); + ErtsILink *ilnk; + ilnk = (ErtsILink *) erts_link_tree_lookup(ERTS_P_LINKS(BIF_P), + BIF_ARG_1); - if (lnk) { + if (ilnk && !ilnk->unlinking) { Eterm ref; Eterm *refp = erts_port_synchronous_ops ? &ref : NULL; ErtsPortOpResult res = ERTS_PORT_OP_DROPPED; Port *prt; - erts_link_tree_delete(&ERTS_P_LINKS(BIF_P), lnk); - /* Send unlink signal */ prt = erts_port_lookup(BIF_ARG_1, ERTS_PORT_SFLGS_DEAD); - if (prt) { + if (!prt) { + erts_link_tree_delete(&ERTS_P_LINKS(BIF_P), &ilnk->link); + erts_link_internal_release(&ilnk->link); + } + else { + ErtsSigUnlinkOp *sulnk; + + sulnk = erts_proc_sig_make_unlink_op(BIF_P, BIF_P->common.id); + ilnk->unlinking = sulnk->id; #ifdef DEBUG ref = NIL; #endif - res = erts_port_unlink(BIF_P, prt, lnk, refp); - + res = erts_port_unlink(BIF_P, prt, sulnk, refp); } - if (res == ERTS_PORT_OP_DROPPED) - erts_link_release(lnk); - else if (refp && res == ERTS_PORT_OP_SCHEDULED) { + if (refp && res == ERTS_PORT_OP_SCHEDULED) { ASSERT(is_internal_ordinary_ref(ref)); BIF_TRAP3(await_port_send_result_trap, BIF_P, ref, am_true, am_true); } @@ -956,9 +1041,10 @@ BIF_RETTYPE unlink_1(BIF_ALIST_1) } if (is_external_pid(BIF_ARG_1)) { - ErtsLink *lnk, *dlnk; - ErtsLinkData *ldp; + ErtsLink *lnk; + ErtsELink *elnk; DistEntry *dep; + Uint64 unlink_id; int code; ErtsDSigSendContext ctx; @@ -970,13 +1056,13 @@ BIF_RETTYPE unlink_1(BIF_ALIST_1) if (!lnk) BIF_RET(am_true); - erts_link_tree_delete(&ERTS_P_LINKS(BIF_P), lnk); - dlnk = erts_link_to_other(lnk, &ldp); + elnk = erts_link_to_elink(lnk); - if (erts_link_dist_delete(dlnk)) - erts_link_release_both(ldp); - else - erts_link_release(lnk); + if (elnk->unlinking) + BIF_RET(am_true); + + unlink_id = erts_proc_sig_new_unlink_id(BIF_P); + elnk->unlinking = unlink_id; code = erts_dsig_prepare(&ctx, dep, BIF_P, ERTS_PROC_LOCK_MAIN, ERTS_DSP_NO_LOCK, 0, 1, 0); @@ -986,9 +1072,16 @@ BIF_RETTYPE unlink_1(BIF_ALIST_1) BIF_RET(am_true); case ERTS_DSIG_PREP_PENDING: case ERTS_DSIG_PREP_CONNECTED: - code = erts_dsig_send_unlink(&ctx, BIF_P->common.id, BIF_ARG_1); - if (code == ERTS_DSIG_SEND_YIELD) - ERTS_BIF_YIELD_RETURN(BIF_P, am_true); + /* + * Do not send unlink signal on another connection than + * the one which the link was set up on. + */ + if (elnk->dist->connection_id == ctx.connection_id) { + code = erts_dsig_send_unlink(&ctx, BIF_P->common.id, BIF_ARG_1, + unlink_id); + if (code == ERTS_DSIG_SEND_YIELD) + ERTS_BIF_YIELD_RETURN(BIF_P, am_true); + } break; default: ASSERT(! "Invalid dsig prepare result"); @@ -1779,8 +1872,10 @@ BIF_RETTYPE erts_internal_process_flag_3(BIF_ALIST_3) exec_process_flag_3, (void *) pf3a); - if (is_non_value(res)) + if (is_non_value(res)) { + erts_free(ERTS_ALC_T_PF3_ARGS, pf3a); BIF_RET(am_badarg); + } return res; } diff --git a/erts/emulator/beam/dist.c b/erts/emulator/beam/dist.c index 945764d552..c79c0834f6 100644 --- a/erts/emulator/beam/dist.c +++ b/erts/emulator/beam/dist.c @@ -156,6 +156,10 @@ static char *erts_dop_to_string(enum dop dop) { return "SPAWN_REQUEST_TT"; if (dop == DOP_SPAWN_REPLY) return "SPAWN_REPLY"; + if (dop == DOP_UNLINK_ID) + return "UNLINK_ID"; + if (dop == DOP_UNLINK_ID_ACK) + return "UNLINK_ID_ACK"; ASSERT(0); return "UNKNOWN"; } @@ -1288,9 +1292,56 @@ erts_dsig_send_link(ErtsDSigSendContext *ctx, Eterm local, Eterm remote) } int -erts_dsig_send_unlink(ErtsDSigSendContext *ctx, Eterm local, Eterm remote) +erts_dsig_send_unlink(ErtsDSigSendContext *ctx, Eterm local, Eterm remote, Uint64 id) { - Eterm ctl = TUPLE3(&ctx->ctl_heap[0], make_small(DOP_UNLINK), local, remote); + Eterm big_heap[ERTS_MAX_UINT64_HEAP_SIZE]; + Eterm unlink_id; + Eterm ctl; + if (ctx->dflags & DFLAG_UNLINK_ID) { + if (IS_USMALL(0, id)) + unlink_id = make_small(id); + else { + Eterm *hp = &big_heap[0]; + unlink_id = erts_uint64_to_big(id, &hp); + } + ctl = TUPLE4(&ctx->ctl_heap[0], make_small(DOP_UNLINK_ID), + unlink_id, local, remote); + } + else { + /* + * A node that isn't capable of talking the new link protocol. + * + * Send an old unlink op, and send ourselves an unlink-ack. We may + * end up in an inconsistent state as we could before the new link + * protocol was introduced... + */ + erts_proc_sig_send_dist_unlink_ack(ctx->c_p, ctx->dep, ctx->connection_id, + remote, local, id); + ctl = TUPLE3(&ctx->ctl_heap[0], make_small(DOP_UNLINK), local, remote); + } + return dsig_send_ctl(ctx, ctl); +} + +int +erts_dsig_send_unlink_ack(ErtsDSigSendContext *ctx, Eterm local, Eterm remote, Uint64 id) +{ + Eterm big_heap[ERTS_MAX_UINT64_HEAP_SIZE]; + Eterm unlink_id; + Eterm ctl; + + if (!(ctx->dflags & DFLAG_UNLINK_ID)) { + /* Receiving node does not understand it, so drop it... */ + return ERTS_DSIG_SEND_OK; + } + + if (IS_USMALL(0, id)) + unlink_id = make_small(id); + else { + Eterm *hp = &big_heap[0]; + unlink_id = erts_uint64_to_big(id, &hp); + } + ctl = TUPLE4(&ctx->ctl_heap[0], make_small(DOP_UNLINK_ID_ACK), + unlink_id, local, remote); return dsig_send_ctl(ctx, ctl); } @@ -2012,16 +2063,16 @@ int erts_net_message(Port *prt, /* old incarnation of node; reply noproc... */ } else if (is_internal_pid(to)) { - ErtsLinkData *ldp = erts_link_create(ERTS_LNK_TYPE_DIST_PROC, - from, to); - ASSERT(ldp->a.other.item == to); - ASSERT(eq(ldp->b.other.item, from)); + ErtsLinkData *ldp = erts_link_external_create(ERTS_LNK_TYPE_DIST_PROC, + to, from); + ASSERT(ldp->dist.other.item == to); + ASSERT(eq(ldp->proc.other.item, from)); - code = erts_link_dist_insert(&ldp->a, ede.mld); - if (erts_proc_sig_send_link(NULL, to, &ldp->b)) { + code = erts_link_dist_insert(&ldp->dist, ede.mld); + if (erts_proc_sig_send_link(NULL, to, &ldp->proc)) { if (!code) { /* Race: connection already down => send link exit */ - erts_proc_sig_send_link_exit(NULL, THE_NON_VALUE, &ldp->a, + erts_proc_sig_send_link_exit(NULL, THE_NON_VALUE, &ldp->dist, am_noconnection, NIL); } break; /* Done */ @@ -2029,7 +2080,7 @@ int erts_net_message(Port *prt, /* Failed to send signal; cleanup and reply noproc... */ if (code) { - code = erts_link_dist_delete(&ldp->a); + code = erts_link_dist_delete(&ldp->dist); ASSERT(code); } erts_link_release_both(ldp); @@ -2044,12 +2095,29 @@ int erts_net_message(Port *prt, break; } - case DOP_UNLINK: { - if (tuple_arity != 3) { + case DOP_UNLINK_ID: { + Eterm *element; + Uint64 id; + if (tuple_arity != 4) goto invalid_message; - } - from = tuple[2]; - to = tuple[3]; + + element = &tuple[2]; + if (!term_to_Uint64(*(element++), &id)) + goto invalid_message; + + if (id == 0) + goto invalid_message; + + if (0) { + case DOP_UNLINK: + if (tuple_arity != 3) + goto invalid_message; + element = &tuple[2]; + id = 0; + } + + from = *(element++); + to = *element; if (is_not_external_pid(from)) goto invalid_message; if (dep != external_pid_dist_entry(from)) @@ -2062,10 +2130,37 @@ int erts_net_message(Port *prt, if (is_not_internal_pid(to)) goto invalid_message; - erts_proc_sig_send_dist_unlink(dep, from, to); + erts_proc_sig_send_dist_unlink(dep, conn_id, from, to, id); break; } + case DOP_UNLINK_ID_ACK: { + Uint64 id; + if (tuple_arity != 4) + goto invalid_message; + + if (!term_to_Uint64(tuple[2], &id)) + goto invalid_message; + + from = tuple[3]; + to = tuple[4]; + if (is_not_external_pid(from)) + goto invalid_message; + if (dep != external_pid_dist_entry(from)) + goto invalid_message; + + if (is_external_pid(to) + && erts_this_dist_entry == external_pid_dist_entry(from)) + break; + + if (is_not_internal_pid(to)) + goto invalid_message; + + erts_proc_sig_send_dist_unlink_ack(NULL, dep, conn_id, + from, to, id); + break; + } + case DOP_MONITOR_P: { /* A remote process wants to monitor us, we get: {DOP_MONITOR_P, Remote pid, local pid or name, ref} */ @@ -2674,12 +2769,12 @@ int erts_net_message(Port *prt, if (flags & ERTS_DIST_SPAWN_FLAG_LINK) { /* Successful spawn-link... */ - ldp = erts_link_create(ERTS_LNK_TYPE_DIST_PROC, - result, parent); - ASSERT(ldp->a.other.item == parent); - ASSERT(eq(ldp->b.other.item, result)); - link_inserted = erts_link_dist_insert(&ldp->a, ede.mld); - lnk = &ldp->b; + ldp = erts_link_external_create(ERTS_LNK_TYPE_DIST_PROC, + parent, result); + ASSERT(ldp->dist.other.item == parent); + ASSERT(eq(ldp->proc.other.item, result)); + link_inserted = erts_link_dist_insert(&ldp->dist, ede.mld); + lnk = &ldp->proc; } } @@ -2702,7 +2797,7 @@ int erts_net_message(Port *prt, if (lnk) { if (link_inserted) { - code = erts_link_dist_delete(&ldp->a); + code = erts_link_dist_delete(&ldp->dist); ASSERT(code); } erts_link_release_both(ldp); @@ -2719,7 +2814,7 @@ int erts_net_message(Port *prt, } } else if (lnk && !link_inserted) { - erts_proc_sig_send_link_exit(NULL, THE_NON_VALUE, &ldp->a, + erts_proc_sig_send_link_exit(NULL, THE_NON_VALUE, &ldp->dist, am_noconnection, NIL); } @@ -4369,6 +4464,7 @@ static int doit_print_link_info(ErtsLink *lnk, void *vptdp, Sint reds) { struct print_to_data *ptdp = vptdp; ErtsLink *lnk2 = erts_link_to_other(lnk, NULL); + ASSERT(lnk->flags & ERTS_ML_FLG_EXTENDED); erts_print(ptdp->to, ptdp->arg, "Remote link: %T %T\n", lnk2->other.item, lnk->other.item); return 1; diff --git a/erts/emulator/beam/dist.h b/erts/emulator/beam/dist.h index 170bb569f5..0730fb3662 100644 --- a/erts/emulator/beam/dist.h +++ b/erts/emulator/beam/dist.h @@ -50,7 +50,8 @@ #define DFLAG_EXIT_PAYLOAD ((Uint64)0x400000) #define DFLAG_FRAGMENTS ((Uint64)0x800000) #define DFLAG_HANDSHAKE_23 ((Uint64)0x1000000) -#define DFLAG_RESERVED 0xfe000000 +#define DFLAG_UNLINK_ID ((Uint64)0x2000000) +#define DFLAG_RESERVED ((Uint64)0xfc000000) /* * As the old handshake only support 32 flag bits, we reserve the remaining * bits in the lower 32 for changes in the handshake protocol or potentially @@ -58,6 +59,7 @@ */ #define DFLAG_SPAWN (((Uint64)0x1) << 32) #define DFLAG_NAME_ME (((Uint64)0x2) << 32) +#define DFLAG_NAME_ME (((Uint64)0x2) << 32) /* Mandatory flags for distribution */ @@ -76,7 +78,8 @@ | DFLAG_BIT_BINARIES \ | DFLAG_DIST_MONITOR \ | DFLAG_DIST_MONITOR_NAME \ - | DFLAG_SPAWN) + | DFLAG_SPAWN \ + | DFLAG_UNLINK_ID) /* Our preferred set of flags. Used for connection setup handshake */ #define DFLAG_DIST_DEFAULT (DFLAG_DIST_MANDATORY | DFLAG_DIST_HOPEFULLY \ @@ -92,7 +95,8 @@ | DFLAG_EXIT_PAYLOAD \ | DFLAG_FRAGMENTS \ | DFLAG_HANDSHAKE_23 \ - | DFLAG_SPAWN) + | DFLAG_SPAWN \ + | DFLAG_UNLINK_ID) /* Flags addable by local distr implementations */ #define DFLAG_DIST_ADDABLE DFLAG_DIST_DEFAULT @@ -149,7 +153,9 @@ enum dop { DOP_SPAWN_REQUEST = 29, DOP_SPAWN_REQUEST_TT = 30, DOP_SPAWN_REPLY = 31, - DOP_SPAWN_REPLY_TT = 32 + DOP_SPAWN_REPLY_TT = 32, + DOP_UNLINK_ID = 35, + DOP_UNLINK_ID_ACK = 36 }; #define ERTS_DIST_SPAWN_FLAG_LINK (1 << 0) @@ -376,7 +382,8 @@ extern int erts_dsig_send_msg(ErtsDSigSendContext*, Eterm, Eterm); extern int erts_dsig_send_reg_msg(ErtsDSigSendContext*, Eterm, Eterm, Eterm); extern int erts_dsig_send_link(ErtsDSigSendContext *, Eterm, Eterm); extern int erts_dsig_send_exit_tt(ErtsDSigSendContext *, Eterm, Eterm, Eterm, Eterm); -extern int erts_dsig_send_unlink(ErtsDSigSendContext *, Eterm, Eterm); +extern int erts_dsig_send_unlink(ErtsDSigSendContext *, Eterm, Eterm, Uint64); +extern int erts_dsig_send_unlink_ack(ErtsDSigSendContext *, Eterm, Eterm, Uint64); extern int erts_dsig_send_group_leader(ErtsDSigSendContext *, Eterm, Eterm); extern int erts_dsig_send_exit(ErtsDSigSendContext *, Eterm, Eterm, Eterm); extern int erts_dsig_send_exit2(ErtsDSigSendContext *, Eterm, Eterm, Eterm); diff --git a/erts/emulator/beam/erl_alloc.c b/erts/emulator/beam/erl_alloc.c index 47150b2aea..cda8855150 100644 --- a/erts/emulator/beam/erl_alloc.c +++ b/erts/emulator/beam/erl_alloc.c @@ -66,7 +66,7 @@ #define ERTS_ALC_DEFAULT_MAX_THR_PREF ERTS_MAX_NO_OF_SCHEDULERS -#if defined(SMALL_MEMORY) || defined(PURIFY) || defined(VALGRIND) +#if defined(SMALL_MEMORY) || defined(PURIFY) || defined(VALGRIND) || defined(ADDRESS_SANITIZER) #define AU_ALLOC_DEFAULT_ENABLE(X) 0 #else #define AU_ALLOC_DEFAULT_ENABLE(X) (X) @@ -289,7 +289,11 @@ static void set_default_literal_alloc_opts(struct au_init *ip) { SET_DEFAULT_ALLOC_OPTS(ip); +#ifdef ADDRESS_SANITIZER + ip->enable = 0; +#else ip->enable = 1; +#endif ip->thr_spec = 0; ip->disable_allowed = 0; ip->thr_spec_allowed = 0; @@ -647,7 +651,7 @@ erts_alloc_init(int *argc, char **argv, ErtsAllocInitOpts *eaiop) fix_type_sizes[ERTS_ALC_FIX_TYPE_IX(ERTS_ALC_T_MONITOR)] = sizeof(ErtsMonitorDataHeap); fix_type_sizes[ERTS_ALC_FIX_TYPE_IX(ERTS_ALC_T_LINK)] - = sizeof(ErtsLinkData); + = sizeof(ErtsILink); fix_type_sizes[ERTS_ALC_FIX_TYPE_IX(ERTS_ALC_T_DRV_SEL_D_STATE)] = sizeof(ErtsDrvSelectDataState); fix_type_sizes[ERTS_ALC_FIX_TYPE_IX(ERTS_ALC_T_NIF_SEL_D_STATE)] diff --git a/erts/emulator/beam/erl_alloc.h b/erts/emulator/beam/erl_alloc.h index c13cf3f5b0..831e7ab0a7 100644 --- a/erts/emulator/beam/erl_alloc.h +++ b/erts/emulator/beam/erl_alloc.h @@ -358,24 +358,11 @@ erts_alloc_get_verify_unused_temp_alloc(Allctr_t **allctr); #define ERTS_ALC_CACHE_LINE_ALIGN_SIZE(SZ) \ (((((SZ) - 1) / ERTS_CACHE_LINE_SIZE) + 1) * ERTS_CACHE_LINE_SIZE) +#if !defined(VALGRIND) && !defined(ADDRESS_SANITIZER) + #define ERTS_QUALLOC_IMPL(NAME, TYPE, PASZ, ALCT) \ ERTS_QUICK_ALLOC_IMPL(NAME, TYPE, PASZ, ALCT, (void) 0, (void) 0, (void) 0) -#define ERTS_TS_QUALLOC_IMPL(NAME, TYPE, PASZ, ALCT) \ -ERTS_QUALLOC_IMPL(NAME, TYPE, PASZ, ALCT) - -#define ERTS_TS_PALLOC_IMPL(NAME, TYPE, PASZ) \ -static erts_spinlock_t NAME##_lck; \ -ERTS_PRE_ALLOC_IMPL(NAME, TYPE, PASZ, \ - erts_spinlock_init(&NAME##_lck, #NAME "_alloc_lock", NIL, \ - ERTS_LOCK_FLAGS_CATEGORY_ALLOCATOR),\ - erts_spin_lock(&NAME##_lck), \ - erts_spin_unlock(&NAME##_lck)) - - -#define ERTS_PALLOC_IMPL(NAME, TYPE, PASZ) \ - ERTS_TS_PALLOC_IMPL(NAME, TYPE, PASZ) - #define ERTS_QUICK_ALLOC_IMPL(NAME, TYPE, PASZ, ALCT, ILCK, LCK, ULCK) \ ERTS_PRE_ALLOC_IMPL(NAME##_pre, TYPE, PASZ, ILCK, LCK, ULCK) \ @@ -606,6 +593,69 @@ NAME##_free(TYPE *p) \ (char *) p); \ } +#else /* !defined(VALGRIND) && !defined(ADDRESS_SANITIZER) */ + +/* + * For VALGRIND and ADDRESS_SANITIZER we short circuit all preallocation + * with dummy wrappers around malloc and free. + */ + +#define ERTS_QUALLOC_IMPL(NAME, TYPE, PASZ, ALCT) \ + ERTS_QUICK_ALLOC_IMPL(NAME, TYPE, PASZ, ALCT, (void) 0, (void) 0, (void) 0) + +#define ERTS_QUICK_ALLOC_IMPL(NAME, TYPE, PASZ, ALCT, ILCK, LCK, ULCK) \ +static void init_##NAME##_alloc(void) \ +{ \ +} \ +static ERTS_INLINE TYPE* NAME##_alloc(void) \ +{ \ + return malloc(sizeof(TYPE)); \ +} \ +static ERTS_INLINE void NAME##_free(TYPE *p) \ +{ \ + free((void *) p); \ +} + +#define ERTS_SCHED_PREF_PALLOC_IMPL(NAME, TYPE, PASZ) \ + ERTS_SCHED_PREF_PRE_ALLOC_IMPL(NAME, TYPE, PASZ) + +#define ERTS_SCHED_PREF_AUX(NAME, TYPE, PASZ) \ +ERTS_SCHED_PREF_PRE_ALLOC_IMPL(NAME##_pre, TYPE, PASZ) + +#define ERTS_SCHED_PREF_QUICK_ALLOC_IMPL(NAME, TYPE, PASZ, ALCT) \ + ERTS_QUALLOC_IMPL(NAME, TYPE, PASZ, ALCT) + +#define ERTS_THR_PREF_QUICK_ALLOC_IMPL(NAME, TYPE, PASZ, ALCT) \ +void erts_##NAME##_pre_alloc_init_thread(void) \ +{ \ +} \ +static void init_##NAME##_alloc(int nthreads) \ +{ \ +} \ +static ERTS_INLINE TYPE* NAME##_alloc(void) \ +{ \ + return malloc(sizeof(TYPE)); \ +} \ +static ERTS_INLINE void NAME##_free(TYPE *p) \ +{ \ + free(p); \ +} + +#define ERTS_SCHED_PREF_PRE_ALLOC_IMPL(NAME, TYPE, PASZ) \ +static void init_##NAME##_alloc(void) \ +{ \ +} \ +static TYPE* NAME##_alloc(void) \ +{ \ + return (TYPE *) malloc(sizeof(TYPE)); \ +} \ +static int NAME##_free(TYPE *p) \ +{ \ + free(p); \ + return 1; \ +} + +#endif /* VALGRIND || ADDRESS_SANITIZER */ #ifdef DEBUG #define ERTS_ALC_DBG_BLK_SZ(PTR) (*(((UWord *) (PTR)) - 2)) diff --git a/erts/emulator/beam/erl_bif_info.c b/erts/emulator/beam/erl_bif_info.c index d65aa71085..1086fdd057 100644 --- a/erts/emulator/beam/erl_bif_info.c +++ b/erts/emulator/beam/erl_bif_info.c @@ -62,8 +62,11 @@ #endif #ifdef VALGRIND -#include <valgrind/valgrind.h> -#include <valgrind/memcheck.h> +# include <valgrind/valgrind.h> +# include <valgrind/memcheck.h> +#endif +#ifdef ADDRESS_SANITIZER +# include <sanitizer/lsan_interface.h> #endif static Export* alloc_info_trap = NULL; @@ -127,6 +130,9 @@ static char erts_system_version[] = ("Erlang/OTP " ERLANG_OTP_RELEASE #ifdef VALGRIND " [valgrind-compiled]" #endif +#ifdef ADDRESS_SANITIZER + " [address-sanitizer]" +#endif #ifdef ERTS_FRMPTR " [frame-pointer]" #endif @@ -351,6 +357,7 @@ make_monitor_list(Process *p, int tree, ErtsMonitor *root, Eterm tail) -record(erl_link, { type, % process | port | dist_process pid, % Process or port + state, % linked | unlinking id % (address) }). */ @@ -359,13 +366,18 @@ static int calc_lnk_size(ErtsLink *lnk, void *vpsz, Sint reds) { Uint *psz = vpsz; Uint sz = 0; - ErtsLinkData *ldp = erts_link_to_data(lnk); + UWord addr; + + if (lnk->type == ERTS_LNK_TYPE_DIST_PROC) + addr = (UWord) erts_link_to_elink(lnk); + else + addr = (UWord) lnk; - (void) erts_bld_uword(NULL, &sz, (UWord) ldp); + (void) erts_bld_uword(NULL, &sz, (UWord) addr); *psz += sz; *psz += is_immed(lnk->other.item) ? 0 : size_object(lnk->other.item); - *psz += 7; /* CONS + 4-tuple */ + *psz += 8; /* CONS + 5-tuple */ return 1; } @@ -379,10 +391,23 @@ typedef struct { static int make_one_lnk_element(ErtsLink *lnk, void * vpllc, Sint reds) { LnkListContext *pllc = vpllc; - Eterm tup, t, pid, id; - ErtsLinkData *ldp = erts_link_to_data(lnk); + Eterm tup, t, pid, id, state; + UWord addr; + ERTS_DECL_AM(linked); + ERTS_DECL_AM(unlinking); + + if (lnk->type == ERTS_LNK_TYPE_DIST_PROC) { + ErtsELink *elnk = erts_link_to_elink(lnk); + state = elnk->unlinking ? AM_unlinking : AM_linked; + addr = (UWord) elnk; + } + else { + ErtsILink *ilnk = (ErtsILink *) lnk; + state = ilnk->unlinking ? AM_unlinking : AM_linked; + addr = (UWord) ilnk; + } - id = erts_bld_uword(&pllc->hp, NULL, (UWord) ldp); + id = erts_bld_uword(&pllc->hp, NULL, (UWord) addr); if (is_immed(lnk->other.item)) pid = lnk->other.item; @@ -409,8 +434,8 @@ static int make_one_lnk_element(ErtsLink *lnk, void * vpllc, Sint reds) break; } - tup = TUPLE4(pllc->hp, pllc->tag, t, pid, id); - pllc->hp += 5; + tup = TUPLE5(pllc->hp, pllc->tag, t, pid, state, id); + pllc->hp += 6; pllc->res = CONS(pllc->hp, tup, pllc->res); pllc->hp += 2; return 1; @@ -525,6 +550,15 @@ do { \ static int collect_one_link(ErtsLink *lnk, void *vmicp, Sint reds) { MonitorInfoCollection *micp = vmicp; + if (lnk->type != ERTS_LNK_TYPE_DIST_PROC) { + if (((ErtsILink *) lnk)->unlinking) + return 1; + } + else { + ErtsELink *elnk = erts_link_to_elink(lnk); + if (elnk->unlinking) + return 1; + } EXTEND_MONITOR_INFOS(micp); micp->mi[micp->mi_i].entity.term = lnk->other.item; micp->sz += 2 + NC_HEAP_SIZE(lnk->other.item); @@ -2120,6 +2154,28 @@ current_stacktrace(Process *p, ErtsHeapFactory *hfact, Process* rp, return res; } +#if defined(VALGRIND) || defined(ADDRESS_SANITIZER) +static int iolist_to_tmp_buf(Eterm iolist, char** bufp) +{ + ErlDrvSizeT buf_size = 1024; /* Try with 1KB first */ + char *buf = erts_alloc(ERTS_ALC_T_TMP, buf_size); + ErlDrvSizeT r = erts_iolist_to_buf(iolist, (char*) buf, buf_size - 1); + if (ERTS_IOLIST_TO_BUF_FAILED(r)) { + erts_free(ERTS_ALC_T_TMP, (void *) buf); + if (erts_iolist_size(iolist, &buf_size)) { + return 0; + } + buf_size++; + buf = erts_alloc(ERTS_ALC_T_TMP, buf_size); + r = erts_iolist_to_buf(iolist, (char*) buf, buf_size - 1); + ASSERT(r == buf_size - 1); + } + buf[buf_size - 1 - r] = '\0'; + *bufp = buf; + return 1; +} +#endif + /* * This function takes care of calls to erlang:system_info/1 when the argument * is a tuple. @@ -2182,59 +2238,45 @@ info_1_tuple(Process* BIF_P, /* Pointer to current process. */ goto badarg; ERTS_BIF_PREP_TRAP1(ret, erts_format_cpu_topology_trap, BIF_P, res); return ret; -#if defined(PURIFY) || defined(VALGRIND) - } else if (ERTS_IS_ATOM_STR("error_checker", sel) -#if defined(PURIFY) - || sel == am_purify -#elif defined(VALGRIND) - || ERTS_IS_ATOM_STR("valgrind", sel) + } else if (ERTS_IS_ATOM_STR("memory_checker", sel)) { + if (arity == 2 && ERTS_IS_ATOM_STR("test_leak", *tp)) { +#if defined(VALGRIND) || defined(ADDRESS_SANITIZER) + erts_alloc(ERTS_ALC_T_HEAP , 100); #endif - ) { - if (*tp == am_memory) { -#if defined(PURIFY) - BIF_RET(erts_make_integer(purify_new_leaks(), BIF_P)); -#elif defined(VALGRIND) -# ifdef VALGRIND_DO_ADDED_LEAK_CHECK + BIF_RET(am_ok); + } + else if (arity == 2 && ERTS_IS_ATOM_STR("test_overflow", *tp)) { + static int test[2]; + BIF_RET(make_small(test[2])); + } +#if defined(VALGRIND) || defined(ADDRESS_SANITIZER) + if (arity == 2 && *tp == am_running) { +# if defined(VALGRIND) + if (RUNNING_ON_VALGRIND) + BIF_RET(ERTS_MAKE_AM("valgrind")); +# elif defined(ADDRESS_SANITIZER) + BIF_RET(ERTS_MAKE_AM("asan")); +# endif + } + else if (arity == 2 && ERTS_IS_ATOM_STR("check_leaks", *tp)) { +# if defined(VALGRIND) +# ifdef VALGRIND_DO_ADDED_LEAK_CHECK VALGRIND_DO_ADDED_LEAK_CHECK; -# else +# else VALGRIND_DO_LEAK_CHECK; +# endif + BIF_RET(am_ok); +# elif defined(ADDRESS_SANITIZER) + __lsan_do_recoverable_leak_check(); + BIF_RET(am_ok); # endif - BIF_RET(make_small(0)); -#endif - } else if (*tp == am_fd) { -#if defined(PURIFY) - BIF_RET(erts_make_integer(purify_new_fds_inuse(), BIF_P)); -#elif defined(VALGRIND) - /* Not present in valgrind... */ - BIF_RET(make_small(0)); -#endif - } else if (*tp == am_running) { -#if defined(PURIFY) - BIF_RET(purify_is_running() ? am_true : am_false); -#elif defined(VALGRIND) - BIF_RET(RUNNING_ON_VALGRIND ? am_true : am_false); -#endif - } else if (is_list(*tp)) { -#if defined(PURIFY) -# define ERTS_ERROR_CHECKER_PRINTF purify_printf -#elif defined(VALGRIND) -# define ERTS_ERROR_CHECKER_PRINTF VALGRIND_PRINTF -#endif - ErlDrvSizeT buf_size = 8*1024; /* Try with 8KB first */ - char *buf = erts_alloc(ERTS_ALC_T_TMP, buf_size); - ErlDrvSizeT r = erts_iolist_to_buf(*tp, (char*) buf, buf_size - 1); - if (ERTS_IOLIST_TO_BUF_FAILED(r)) { - erts_free(ERTS_ALC_T_TMP, (void *) buf); - if (erts_iolist_size(*tp, &buf_size)) { - goto badarg; - } - buf_size++; - buf = erts_alloc(ERTS_ALC_T_TMP, buf_size); - r = erts_iolist_to_buf(*tp, (char*) buf, buf_size - 1); - ASSERT(r == buf_size - 1); - } - buf[buf_size - 1 - r] = '\0'; - ERTS_ERROR_CHECKER_PRINTF("%s\n", buf); + } +# if defined(VALGRIND) + if (arity == 3 && tp[0] == am_print && is_list(tp[1])) { + char* buf; + if (!iolist_to_tmp_buf(tp[1], &buf)) + goto badarg; + VALGRIND_PRINTF("%s\n", buf); erts_free(ERTS_ALC_T_TMP, (void *) buf); BIF_RET(am_true); #undef ERTS_ERROR_CHECKER_PRINTF @@ -2254,6 +2296,30 @@ info_1_tuple(Process* BIF_P, /* Pointer to current process. */ } else if (*tp == am_running) { BIF_RET(quantify_is_running() ? am_true : am_false); } +# endif +# if defined(ADDRESS_SANITIZER) + if (arity == 3 && ERTS_IS_ATOM_STR("log",tp[0]) && is_list(tp[1])) { + static char *active_log = NULL; + static int active_log_len; + Eterm ret = NIL; + char* buf; + if (!iolist_to_tmp_buf(tp[1], &buf)) + goto badarg; + erts_rwmtx_rwlock(&erts_dist_table_rwmtx); /* random lock abuse */ + __sanitizer_set_report_path(buf); + if (active_log) { + Eterm *hp = HAlloc(BIF_P, 2 * active_log_len); + ret = erts_bld_string_n(&hp, 0, active_log, active_log_len); + erts_free(ERTS_ALC_T_DEBUG, active_log); + } + active_log_len = sys_strlen(buf); + active_log = erts_alloc(ERTS_ALC_T_DEBUG, active_log_len + 1); + sys_memcpy(active_log, buf, active_log_len + 1); + erts_rwmtx_rwunlock(&erts_dist_table_rwmtx); + erts_free(ERTS_ALC_T_TMP, (void *) buf); + BIF_RET(ret); + } +# endif #endif #if defined(__GNUC__) && defined(HAVE_SOLARIS_SPARC_PERFMON) } else if (ERTS_IS_ATOM_STR("ultrasparc_set_pcr", sel)) { @@ -2459,6 +2525,9 @@ BIF_RETTYPE system_info_1(BIF_ALIST_1) #elif defined(VALGRIND) ERTS_DECL_AM(valgrind); BIF_RET(AM_valgrind); +#elif defined(ADDRESS_SANITIZER) + ERTS_DECL_AM(asan); + BIF_RET(AM_asan); #elif defined(GPROF) ERTS_DECL_AM(gprof); BIF_RET(AM_gprof); @@ -4469,6 +4538,28 @@ static void broken_halt_test(Eterm bif_arg_2) erts_exit(ERTS_DUMP_EXIT, "%T", bif_arg_2); } +static void +test_multizero_timeout_in_timeout3(void *vproc) +{ + Process *proc = (Process *) vproc; + ErtsMessage *mp = erts_alloc_message(0, NULL); + ERTS_DECL_AM(multizero_timeout_in_timeout_done); + erts_queue_message(proc, 0, mp, AM_multizero_timeout_in_timeout_done, am_system); + erts_proc_dec_refc(proc); +} + +static void +test_multizero_timeout_in_timeout2(void *vproc) +{ + erts_start_timer_callback(0, test_multizero_timeout_in_timeout3, vproc); +} + +static void +test_multizero_timeout_in_timeout(void *vproc) +{ + erts_start_timer_callback(0, test_multizero_timeout_in_timeout2, vproc); +} + BIF_RETTYPE erts_debug_set_internal_state_2(BIF_ALIST_2) { /* @@ -4850,6 +4941,18 @@ BIF_RETTYPE erts_debug_set_internal_state_2(BIF_ALIST_2) BIF_RET(am_true); } } + else if (ERTS_IS_ATOM_STR("multizero_timeout_in_timeout", BIF_ARG_1)) { + Sint64 timeout; + if (term_to_Sint64(BIF_ARG_2, &timeout)) { + if (timeout < 0) + timeout = 0; + erts_proc_inc_refc(BIF_P); + erts_start_timer_callback((ErtsMonotonicTime) timeout, + test_multizero_timeout_in_timeout, + (void *) BIF_P); + BIF_RET(am_ok); + } + } } BIF_ERROR(BIF_P, BADARG); diff --git a/erts/emulator/beam/erl_bif_persistent.c b/erts/emulator/beam/erl_bif_persistent.c index 91cc03fe57..596604cdec 100644 --- a/erts/emulator/beam/erl_bif_persistent.c +++ b/erts/emulator/beam/erl_bif_persistent.c @@ -439,7 +439,7 @@ BIF_RETTYPE persistent_term_put_2(BIF_ALIST_2) BIF_RETTYPE persistent_term_get_0(BIF_ALIST_0) { - HashTable* hash_table = (HashTable *) erts_atomic_read_nob(&the_hash_table); + HashTable* hash_table; TrapData* trap_data; Eterm res = NIL; Eterm magic_ref; @@ -450,6 +450,8 @@ BIF_RETTYPE persistent_term_get_0(BIF_ALIST_0) ERTS_BIF_YIELD0(&bif_trap_export[BIF_persistent_term_get_0], BIF_P); } + hash_table = (HashTable *) erts_atomic_read_nob(&the_hash_table); + magic_ref = alloc_trap_data(BIF_P); mbp = erts_magic_ref2bin(magic_ref); trap_data = ERTS_MAGIC_BIN_DATA(mbp); @@ -673,7 +675,7 @@ BIF_RETTYPE erts_internal_erase_persistent_terms_0(BIF_ALIST_0) BIF_RETTYPE persistent_term_info_0(BIF_ALIST_0) { - HashTable* hash_table = (HashTable *) erts_atomic_read_nob(&the_hash_table); + HashTable* hash_table; TrapData* trap_data; Eterm res = NIL; Eterm magic_ref; @@ -684,6 +686,8 @@ BIF_RETTYPE persistent_term_info_0(BIF_ALIST_0) ERTS_BIF_YIELD0(&bif_trap_export[BIF_persistent_term_info_0], BIF_P); } + hash_table = (HashTable *) erts_atomic_read_nob(&the_hash_table); + magic_ref = alloc_trap_data(BIF_P); mbp = erts_magic_ref2bin(magic_ref); trap_data = ERTS_MAGIC_BIN_DATA(mbp); diff --git a/erts/emulator/beam/erl_bif_port.c b/erts/emulator/beam/erl_bif_port.c index 78857e9b33..604fc25b6e 100644 --- a/erts/emulator/beam/erl_bif_port.c +++ b/erts/emulator/beam/erl_bif_port.c @@ -60,8 +60,7 @@ BIF_RETTYPE erts_internal_open_port_2(BIF_ALIST_2) Eterm res; char *str; int err_type, err_num; - ErtsLinkData *ldp; - ErtsLink *lnk; + ErtsLink *proc_lnk, *port_lnk; port = open_port(BIF_P, BIF_ARG_1, BIF_ARG_2, &err_type, &err_num); if (!port) { @@ -83,15 +82,15 @@ BIF_RETTYPE erts_internal_open_port_2(BIF_ALIST_2) BIF_RET(res); } - ldp = erts_link_create(ERTS_LNK_TYPE_PORT, BIF_P->common.id, port->common.id); - ASSERT(ldp->a.other.item == port->common.id); - ASSERT(ldp->b.other.item == BIF_P->common.id); + proc_lnk = erts_link_internal_create(ERTS_LNK_TYPE_PORT, port->common.id); + port_lnk = erts_link_internal_create(ERTS_LNK_TYPE_PORT, BIF_P->common.id); /* * This link should not already be present, but can potentially * due to id wrapping... */ - lnk = erts_link_tree_lookup_insert(&ERTS_P_LINKS(BIF_P), &ldp->a); - erts_link_tree_insert(&ERTS_P_LINKS(port), &ldp->b); + if (!!erts_link_tree_lookup_insert(&ERTS_P_LINKS(BIF_P), proc_lnk)) + erts_link_internal_release(proc_lnk); + erts_link_tree_insert(&ERTS_P_LINKS(port), port_lnk); if (port->drv_ptr->flags & ERL_DRV_FLAG_USE_INIT_ACK) { @@ -121,9 +120,6 @@ BIF_RETTYPE erts_internal_open_port_2(BIF_ALIST_2) erts_port_release(port); - if (lnk) - erts_link_release(lnk); - return ret; } diff --git a/erts/emulator/beam/erl_bif_re.c b/erts/emulator/beam/erl_bif_re.c index 568534cab2..0428b4e348 100644 --- a/erts/emulator/beam/erl_bif_re.c +++ b/erts/emulator/beam/erl_bif_re.c @@ -1502,7 +1502,7 @@ re_inspect_2(BIF_ALIST_2) tp = tuple_val(BIF_ARG_1); if (tp[1] != am_re_pattern || is_not_small(tp[2]) || is_not_small(tp[3]) || is_not_small(tp[4]) || - is_not_binary(tp[5])) { + is_not_binary(tp[5]) || binary_size(tp[5]) < 4) { goto error; } if (BIF_ARG_2 != am_namelist) { diff --git a/erts/emulator/beam/erl_bif_trace.c b/erts/emulator/beam/erl_bif_trace.c index 7708e0755c..b688e82e70 100644 --- a/erts/emulator/beam/erl_bif_trace.c +++ b/erts/emulator/beam/erl_bif_trace.c @@ -465,19 +465,25 @@ erts_trace_flags(Eterm List, cpu_timestamp = !0; #endif } else if (is_tuple(item)) { + ERTS_TRACER_CLEAR(&tracer); tracer = erts_term_to_tracer(am_tracer, item); if (tracer == THE_NON_VALUE) goto error; } else goto error; list = CDR(list_val(list)); } - if (is_not_nil(list)) goto error; + if (is_not_nil(list)) { + goto error; + } - if (pMask && mask) *pMask = mask; - if (pTracer && !ERTS_TRACER_IS_NIL(tracer)) *pTracer = tracer; - if (pCpuTimestamp && cpu_timestamp) *pCpuTimestamp = cpu_timestamp; + if (mask) *pMask = mask; + if (!ERTS_TRACER_IS_NIL(tracer)) *pTracer = tracer; + if (cpu_timestamp) *pCpuTimestamp = cpu_timestamp; return !0; + error: + if (tracer != THE_NON_VALUE) + ERTS_TRACER_CLEAR(&tracer); return 0; } @@ -541,6 +547,7 @@ Eterm erts_internal_trace_3(BIF_ALIST_3) } if (!erts_try_seize_code_write_permission(BIF_P)) { + ERTS_TRACER_CLEAR(&tracer); ERTS_BIF_YIELD3(&bif_trap_export[BIF_erts_internal_trace_3], BIF_P, BIF_ARG_1, BIF_ARG_2, BIF_ARG_3); } @@ -1841,10 +1848,13 @@ new_seq_trace_token(Process* p, int ensure_new_heap) make_small(p->seq_trace_lastcnt)); } else if (ensure_new_heap) { + Eterm *mature = p->abandoned_heap ? p->abandoned_heap : p->heap; + Uint mature_size = p->high_water - mature; Eterm* tpl = tuple_val(SEQ_TRACE_TOKEN(p)); ASSERT(arityval(tpl[0]) == 5); if (ErtsInArea(tpl, OLD_HEAP(p), - (OLD_HEND(p) - OLD_HEAP(p))*sizeof(Eterm))) { + (OLD_HEND(p) - OLD_HEAP(p))*sizeof(Eterm)) || + ErtsInArea(tpl, mature, mature_size*sizeof(Eterm))) { hp = HAlloc(p, 6); sys_memcpy(hp, tpl, 6*sizeof(Eterm)); SEQ_TRACE_TOKEN(p) = make_tuple(hp); diff --git a/erts/emulator/beam/erl_binary.h b/erts/emulator/beam/erl_binary.h index f3e3890e94..20344bbdcd 100644 --- a/erts/emulator/beam/erl_binary.h +++ b/erts/emulator/beam/erl_binary.h @@ -369,7 +369,7 @@ erts_free_aligned_binary_bytes(byte* buf) ** These extra bytes where earlier (< R13B04) added by an alignment-bug ** in this code. Do we dare remove this in some major release (R14?) maybe? */ -#if defined(DEBUG) || defined(VALGRIND) +#if defined(DEBUG) || defined(VALGRIND) || defined(ADDRESS_SANITIZER) # define CHICKEN_PAD 0 #else # define CHICKEN_PAD (sizeof(void*) - 1) diff --git a/erts/emulator/beam/erl_gc.c b/erts/emulator/beam/erl_gc.c index 61b4b81668..b74eaccea8 100644 --- a/erts/emulator/beam/erl_gc.c +++ b/erts/emulator/beam/erl_gc.c @@ -697,10 +697,10 @@ garbage_collect(Process* p, ErlHeapFragment *live_hf_end, ErtsMonotonicTime start_time; ErtsSchedulerData *esdp = erts_proc_sched_data(p); erts_aint32_t state; - ERTS_MSACC_PUSH_STATE(); #ifdef USE_VM_PROBES DTRACE_CHARBUF(pidbuf, DTRACE_TERM_BUF_SIZE); #endif + ERTS_MSACC_PUSH_STATE(); ERTS_UNDEF(start_time, 0); ERTS_CHK_MBUF_SZ(p); @@ -1772,15 +1772,6 @@ do_minor(Process *p, ErlHeapFragment *live_hf_end, sys_memcpy(n_heap + new_sz - n, p->stop, n * sizeof(Eterm)); p->stop = n_heap + new_sz - n; -#ifdef USE_VM_PROBES - if (HEAP_SIZE(p) != new_sz && DTRACE_ENABLED(process_heap_grow)) { - DTRACE_CHARBUF(pidbuf, DTRACE_TERM_BUF_SIZE); - - dtrace_proc_str(p, pidbuf); - DTRACE3(process_heap_grow, pidbuf, HEAP_SIZE(p), new_sz); - } -#endif - #ifdef HARDDEBUG disallow_heap_frag_ref_in_heap(p, n_heap, n_htop); #endif @@ -1789,8 +1780,19 @@ do_minor(Process *p, ErlHeapFragment *live_hf_end, HEAP_START(p) = n_heap; HEAP_TOP(p) = n_htop; - HEAP_SIZE(p) = new_sz; HEAP_END(p) = n_heap + new_sz; + +#ifdef USE_VM_PROBES + if (HEAP_SIZE(p) != new_sz && DTRACE_ENABLED(process_heap_grow)) { + DTRACE_CHARBUF(pidbuf, DTRACE_TERM_BUF_SIZE); + Uint old_sz = HEAP_SIZE(p); + + HEAP_SIZE(p) = new_sz; + dtrace_proc_str(p, pidbuf); + DTRACE3(process_heap_grow, pidbuf, old_sz, new_sz); + } +#endif + HEAP_SIZE(p) = new_sz; } /* @@ -1863,15 +1865,6 @@ major_collection(Process* p, ErlHeapFragment *live_hf_end, sys_memcpy(n_heap + new_sz - stk_sz, p->stop, stk_sz * sizeof(Eterm)); p->stop = n_heap + new_sz - stk_sz; -#ifdef USE_VM_PROBES - if (HEAP_SIZE(p) != new_sz && DTRACE_ENABLED(process_heap_grow)) { - DTRACE_CHARBUF(pidbuf, DTRACE_TERM_BUF_SIZE); - - dtrace_proc_str(p, pidbuf); - DTRACE3(process_heap_grow, pidbuf, HEAP_SIZE(p), new_sz); - } -#endif - #ifdef HARDDEBUG disallow_heap_frag_ref_in_heap(p, n_heap, n_htop); #endif @@ -1880,8 +1873,24 @@ major_collection(Process* p, ErlHeapFragment *live_hf_end, HEAP_START(p) = n_heap; HEAP_TOP(p) = n_htop; - HEAP_SIZE(p) = new_sz; HEAP_END(p) = n_heap + new_sz; + +#ifdef USE_VM_PROBES + /* Fire process_heap_grow tracepoint after all heap references have + * been updated. This allows to walk the stack. */ + if (HEAP_SIZE(p) != new_sz && DTRACE_ENABLED(process_heap_grow)) { + DTRACE_CHARBUF(pidbuf, DTRACE_TERM_BUF_SIZE); + Uint old_sz = HEAP_SIZE(p); + + /* Update the heap size before firing tracepoint */ + HEAP_SIZE(p) = new_sz; + + dtrace_proc_str(p, pidbuf); + DTRACE3(process_heap_grow, pidbuf, old_sz, new_sz); + } +#endif + HEAP_SIZE(p) = new_sz; + GEN_GCS(p) = 0; HIGH_WATER(p) = HEAP_TOP(p); diff --git a/erts/emulator/beam/erl_message.c b/erts/emulator/beam/erl_message.c index e57d505805..41574d99ba 100644 --- a/erts/emulator/beam/erl_message.c +++ b/erts/emulator/beam/erl_message.c @@ -1623,6 +1623,15 @@ void erts_factory_undo(ErtsHeapFactory* factory) ERTS_HEAP_FRAG_SIZE(factory->heap_frags_saved->alloc_size)); } } + if (factory->message) { + ASSERT(factory->message->data.attached != ERTS_MSG_COMBINED_HFRAG); + ASSERT(!factory->message->data.heap_frag); + + /* Set the message to NIL in order for it not to be treated as + a distributed message by erts_cleanup_messages */ + factory->message->m[0] = NIL; + erts_cleanup_messages(factory->message); + } } break; diff --git a/erts/emulator/beam/erl_monitor_link.c b/erts/emulator/beam/erl_monitor_link.c index a1c5170386..2145d7b71c 100644 --- a/erts/emulator/beam/erl_monitor_link.c +++ b/erts/emulator/beam/erl_monitor_link.c @@ -1104,8 +1104,8 @@ erts_monitor_size(ErtsMonitor *mon) \* */ #ifdef ERTS_ML_DEBUG -size_t erts_link_a_offset; -size_t erts_link_b_offset; +size_t erts_link_proc_offset; +size_t erts_link_dist_offset; size_t erts_link_key_offset; #endif @@ -1113,8 +1113,8 @@ static ERTS_INLINE void link_init(void) { #ifdef ERTS_ML_DEBUG - erts_link_a_offset = offsetof(ErtsLinkData, a); - erts_link_b_offset = offsetof(ErtsLinkData, b); + erts_link_proc_offset = offsetof(ErtsLinkData, proc); + erts_link_dist_offset = offsetof(ErtsLinkData, dist); erts_link_key_offset = offsetof(ErtsLink, other.item); #endif } @@ -1133,29 +1133,45 @@ erts_link_tree_lookup_insert(ErtsLink **root, ErtsLink *lnk) (ErtsMonLnkNode *) lnk); } +static ErtsMonLnkNode * +create_link_internal(Eterm id, void *vtypep) +{ + ErtsMonLnkNode *lnk = erts_link_internal_create(*((Uint16 *) vtypep), id); + ERTS_ML_ASSERT(ml_cmp_keys(lnk->other.item, id) == 0); + return (ErtsMonLnkNode *) lnk; +} + +ErtsLink *erts_link_internal_tree_lookup_create(ErtsLink **root, int *created, + Uint16 type, Eterm other) +{ + return (ErtsLink *) ml_rbt_lookup_create((ErtsMonLnkNode **) root, + other, create_link_internal, + (void *) &type, + created); +} + typedef struct { Uint16 type; - Eterm a; + Eterm this; } ErtsLinkCreateCtxt; static ErtsMonLnkNode * -create_link(Eterm b, void *vcctxt) +create_link_external(Eterm other, void *vcctxt) { ErtsLinkCreateCtxt *cctxt = vcctxt; - ErtsLinkData *ldp = erts_link_create(cctxt->type, cctxt->a, b); - ERTS_ML_ASSERT(ml_cmp_keys(ldp->a.other.item, b) == 0); - return (ErtsMonLnkNode *) &ldp->a; + ErtsLinkData *ldp = erts_link_external_create(cctxt->type, cctxt->this, other); + return (ErtsMonLnkNode *) &ldp->proc; } -ErtsLink *erts_link_tree_lookup_create(ErtsLink **root, int *created, - Uint16 type, Eterm this, - Eterm other) +ErtsLink *erts_link_external_tree_lookup_create(ErtsLink **root, int *created, + Uint16 type, Eterm this, + Eterm other) { ErtsLinkCreateCtxt cctxt; cctxt.type = type; - cctxt.a = this; + cctxt.this = this; return (ErtsLink *) ml_rbt_lookup_create((ErtsMonLnkNode **) root, - other, create_link, + other, create_link_external, (void *) &cctxt, created); } @@ -1289,23 +1305,17 @@ erts_link_list_foreach_delete_yielding(ErtsLink **list, arg, vyspp, limit); } -ErtsLinkData * -erts_link_create(Uint16 type, Eterm a, Eterm b) +ErtsLink * +erts_link_internal_create(Uint16 type, Eterm id) { - ErtsLinkData *ldp; - + ErtsILink *ilnk; #ifdef ERTS_ML_DEBUG switch (type) { case ERTS_LNK_TYPE_PROC: - ERTS_ML_ASSERT(is_internal_pid(a) && is_internal_pid(a)); + ERTS_ML_ASSERT(is_internal_pid(id)); break; case ERTS_LNK_TYPE_PORT: - ERTS_ML_ASSERT(is_internal_pid(a) || is_internal_pid(b)); - ERTS_ML_ASSERT(is_internal_port(a) || is_internal_port(b)); - break; - case ERTS_LNK_TYPE_DIST_PROC: - ERTS_ML_ASSERT(is_internal_pid(a) || is_internal_pid(b)); - ERTS_ML_ASSERT(is_external_pid(a) || is_external_pid(b)); + ERTS_ML_ASSERT(is_internal_pid(id) || is_internal_port(id)); break; default: ERTS_INTERNAL_ERROR("Invalid link type"); @@ -1313,144 +1323,128 @@ erts_link_create(Uint16 type, Eterm a, Eterm b) } #endif - if (type != ERTS_LNK_TYPE_DIST_PROC) { - ldp = erts_alloc(ERTS_ALC_T_LINK, sizeof(ErtsLinkData)); + ilnk = erts_alloc(ERTS_ALC_T_LINK, sizeof(ErtsILink)); - ldp->a.other.item = b; - ldp->a.flags = (Uint16) 0; + ilnk->link.other.item = id; + ilnk->link.key_offset = (Uint16) offsetof(ErtsLink, other.item); + ilnk->link.offset = (Uint16) 0; + ilnk->link.flags = (Uint16) 0; + ilnk->link.type = type; + ilnk->unlinking = 0; - ldp->b.other.item = a; - ldp->b.flags = (Uint16) 0; - } - else { - ErtsLinkDataExtended *ldep; - Uint size, hsz; - Eterm *hp; - ErlOffHeap oh; + return &ilnk->link; +} - hsz = EXTERNAL_THING_HEAD_SIZE + 1; - if (hsz < ERTS_REF_THING_SIZE - && (is_internal_ordinary_ref(a) - || is_internal_ordinary_ref(b))) { - hsz = ERTS_REF_THING_SIZE; - } -#ifdef DEBUG - if (is_internal_pid(a)) { - ERTS_ML_ASSERT(is_external_pid(b) - || is_internal_ordinary_ref(b)); - ERTS_ML_ASSERT(NC_HEAP_SIZE(b) <= hsz); - } - else { - ERTS_ML_ASSERT(is_internal_pid(b)); - ERTS_ML_ASSERT(is_external_pid(a) - || is_internal_ordinary_ref(a)); - ERTS_ML_ASSERT(NC_HEAP_SIZE(a) <= hsz); - } -#endif +ErtsLinkData * +erts_link_external_create(Uint16 type, Eterm this, Eterm other) +{ + ErtsLinkData *ldp; + ErtsELink *elnk; + Uint size, hsz; + Eterm *hp; + ErlOffHeap oh; - size = sizeof(ErtsLinkDataExtended) - sizeof(Eterm); - size += hsz*sizeof(Eterm); + ERTS_ML_ASSERT(type == ERTS_LNK_TYPE_DIST_PROC); + ERTS_ML_ASSERT(is_internal_pid(this)); + ERTS_ML_ASSERT(is_external_pid(other)); - ldp = erts_alloc(ERTS_ALC_T_LINK_EXT, size); + hsz = EXTERNAL_THING_HEAD_SIZE + 1; - ldp->a.flags = ERTS_ML_FLG_EXTENDED; - ldp->b.flags = ERTS_ML_FLG_EXTENDED; + size = sizeof(ErtsELink) - sizeof(Eterm); + size += hsz*sizeof(Eterm); - ldep = (ErtsLinkDataExtended *) ldp; - hp = &ldep->heap[0]; + ldp = erts_alloc(ERTS_ALC_T_LINK_EXT, size); - ERTS_INIT_OFF_HEAP(&oh); + elnk = (ErtsELink *) ldp; + hp = &elnk->heap[0]; - if (is_internal_pid(a)) { - ldp->a.other.item = STORE_NC(&hp, &oh, b); - ldp->b.other.item = a; - } - else { - ldp->a.other.item = b; - ldp->b.other.item = STORE_NC(&hp, &oh, a); - } + ERTS_INIT_OFF_HEAP(&oh); - ldep->ohhp = oh.first; - ldep->dist = NULL; - } + ldp->proc.other.item = STORE_NC(&hp, &oh, other); + ldp->proc.flags = ERTS_ML_FLG_EXTENDED; + ldp->proc.key_offset = (Uint16) offsetof(ErtsLink, other.item); + ldp->proc.offset = (Uint16) offsetof(ErtsLinkData, proc); + ldp->proc.type = type; - erts_atomic32_init_nob(&ldp->refc, 2); + ldp->dist.other.item = this; + ldp->dist.flags = ERTS_ML_FLG_EXTENDED; + ldp->dist.key_offset = (Uint16) offsetof(ErtsLink, other.item); + ldp->dist.offset = (Uint16) offsetof(ErtsLinkData, dist); + ldp->dist.type = type; - ldp->a.key_offset = (Uint16) offsetof(ErtsLink, other.item); - ldp->a.offset = (Uint16) offsetof(ErtsLinkData, a); - ldp->a.type = type; + elnk->ohhp = oh.first; + elnk->dist = NULL; + elnk->unlinking = 0; - ldp->b.key_offset = (Uint16) offsetof(ErtsLink, other.item); - ldp->b.offset = (Uint16) offsetof(ErtsLinkData, b); - ldp->b.type = type; + erts_atomic32_init_nob(&ldp->refc, 2); return ldp; } /* - * erts_link_destroy__() should only be called from + * erts_link_destroy_elink__() should only be called from * erts_link_release() or erts_link_release_both(). */ void -erts_link_destroy__(ErtsLinkData *ldp) +erts_link_destroy_elink__(ErtsELink *elnk) { - ERTS_ML_ASSERT(erts_atomic32_read_nob(&ldp->refc) == 0); - ERTS_ML_ASSERT(!(ldp->a.flags & ERTS_ML_FLG_IN_TABLE)); - ERTS_ML_ASSERT(!(ldp->b.flags & ERTS_ML_FLG_IN_TABLE)); - ERTS_ML_ASSERT((ldp->a.flags & ERTS_ML_FLGS_SAME) - == (ldp->b.flags & ERTS_ML_FLGS_SAME)); - - if (!(ldp->a.flags & ERTS_ML_FLG_EXTENDED)) - erts_free(ERTS_ALC_T_LINK, ldp); - else { - ErtsLinkDataExtended *ldep = (ErtsLinkDataExtended *) ldp; - ErlOffHeap oh; - if (ldep->ohhp) { - ERTS_INIT_OFF_HEAP(&oh); - oh.first = ldep->ohhp; - erts_cleanup_offheap(&oh); - } - if (ldep->dist) - erts_mon_link_dist_dec_refc(ldep->dist); - erts_free(ERTS_ALC_T_LINK_EXT, ldep); + ErlOffHeap oh; + ERTS_ML_ASSERT(erts_atomic32_read_nob(&elnk->ld.refc) == 0); + ERTS_ML_ASSERT(!(elnk->ld.proc.flags & ERTS_ML_FLG_IN_TABLE)); + ERTS_ML_ASSERT(!(elnk->ld.dist.flags & ERTS_ML_FLG_IN_TABLE)); + ERTS_ML_ASSERT((elnk->ld.proc.flags & ERTS_ML_FLGS_SAME) + == (elnk->ld.dist.flags & ERTS_ML_FLGS_SAME)); + ERTS_ML_ASSERT(elnk->ld.proc.flags & ERTS_ML_FLG_EXTENDED); + ERTS_ML_ASSERT(elnk->ld.dist.flags & ERTS_ML_FLG_EXTENDED); + + if (elnk->ohhp) { + ERTS_INIT_OFF_HEAP(&oh); + oh.first = elnk->ohhp; + erts_cleanup_offheap(&oh); } + if (elnk->dist) + erts_mon_link_dist_dec_refc(elnk->dist); + erts_free(ERTS_ALC_T_LINK_EXT, elnk); } void erts_link_set_dead_dist(ErtsLink *lnk, Eterm nodename) { - ErtsLinkDataExtended *ldep; - ldep = (ErtsLinkDataExtended *) erts_link_to_data(lnk); + ErtsELink *elnk; ERTS_ML_ASSERT(lnk->flags & ERTS_ML_FLG_EXTENDED); ERTS_ML_ASSERT(lnk->type == ERTS_LNK_TYPE_DIST_PROC); - ERTS_ML_ASSERT(!ldep->dist); - ldep->dist = erts_mon_link_dist_create(nodename); - ldep->dist->alive = 0; + elnk = erts_link_to_elink(lnk); + + ERTS_ML_ASSERT(!elnk->dist); + + elnk->dist = erts_mon_link_dist_create(nodename); + elnk->dist->alive = 0; } Uint erts_link_size(ErtsLink *lnk) { Uint size, refc; - ErtsLinkData *ldp = erts_link_to_data(lnk); - if (!(lnk->flags & ERTS_ML_FLG_EXTENDED)) - size = sizeof(ErtsLinkData); + if (!(lnk->flags & ERTS_ML_FLG_EXTENDED)) { + size = sizeof(ErtsLink); + refc = 1; + } else { - ErtsLinkDataExtended *ldep = (ErtsLinkDataExtended *) ldp; + ErtsELink *elnk = erts_link_to_elink(lnk); ASSERT(lnk->type == ERTS_LNK_TYPE_DIST_PROC); - ASSERT(is_external_pid_header(ldep->heap[0])); + ASSERT(is_external_pid_header(elnk->heap[0])); - size = sizeof(ErtsLinkDataExtended); - size += thing_arityval(ldep->heap[0])*sizeof(Eterm); + size = sizeof(ErtsELink); + size += thing_arityval(elnk->heap[0])*sizeof(Eterm); + refc = (Uint) erts_atomic32_read_nob(&elnk->ld.refc); + ASSERT(refc > 0); } - refc = (Uint) erts_atomic32_read_nob(&ldp->refc); - ASSERT(refc > 0); return size / refc; } diff --git a/erts/emulator/beam/erl_monitor_link.h b/erts/emulator/beam/erl_monitor_link.h index d75bc7999b..07f38ac9ca 100644 --- a/erts/emulator/beam/erl_monitor_link.h +++ b/erts/emulator/beam/erl_monitor_link.h @@ -400,6 +400,7 @@ #include "erl_thr_progress.h" #undef ERL_THR_PROGRESS_TSD_TYPE_ONLY +#include "erl_alloc.h" #if defined(DEBUG) || 0 # define ERTS_ML_DEBUG @@ -720,7 +721,7 @@ ErtsMonitor *erts_monitor_tree_lookup(ErtsMonitor *root, Eterm key); * @returns Pointer to a monitor with the * key 'key'. If no monitor with the key * 'key' was found and 'mon' was inserted - * 'mon' is returned. + * 'NULL' is returned. * */ ErtsMonitor *erts_monotor_tree_lookup_insert(ErtsMonitor **root, @@ -1548,18 +1549,26 @@ typedef struct ErtsMonLnkNode__ ErtsLink; typedef int (*ErtsLinkFunc)(ErtsLink *, void *, Sint); +/* Internal Link */ typedef struct { - ErtsLink a; - ErtsLink b; + ErtsLink link; + Uint64 unlinking; +} ErtsILink; + +typedef struct { + ErtsLink proc; + ErtsLink dist; erts_atomic32_t refc; } ErtsLinkData; +/* External Link */ typedef struct { ErtsLinkData ld; struct erl_off_heap_header *ohhp; ErtsMonLnkDist *dist; + Uint64 unlinking; Eterm heap[1]; /* heap start... */ -} ErtsLinkDataExtended; +} ErtsELink; /* * --- Link tree operations --- @@ -1597,14 +1606,14 @@ ErtsLink *erts_link_tree_lookup(ErtsLink *root, Eterm item); * @returns Pointer to a link with the * key 'key'. If no link with the key * 'key' was found and 'lnk' was inserted - * 'lnk' is returned. + * 'NULL' is returned. * */ ErtsLink *erts_link_tree_lookup_insert(ErtsLink **root, ErtsLink *lnk); /** * - * @brief Lookup or create a link in a link tree. + * @brief Lookup or create an external link in a link tree. * * Looks up a link with the key 'other' in the link tree. If it is not * found, creates and insert a link with the key 'other'. @@ -1623,9 +1632,40 @@ ErtsLink *erts_link_tree_lookup_insert(ErtsLink **root, ErtsLink *lnk); * * @param[in] other Id of other entity * + * @returns Pointer to either an already existing + * link in the tree or a newly created + * and inserted link. + * + */ +ErtsLink *erts_link_external_tree_lookup_create(ErtsLink **root, int *created, + Uint16 type, Eterm this, Eterm other); + +/** + * + * @brief Lookup or create an internal link in a link tree. + * + * Looks up a link with the key 'other' in the link tree. If it is not + * found, creates and insert a link with the key 'other'. + * + * @param[in,out] root Pointer to pointer to root of link tree + * + * @param[out] created Pointer to integer. The integer is set to + * a non-zero value if no link with key + * 'other' was found, and a new link + * was created. If a link was found, it + * is set to zero. + * + * @param[in] type Type of link + * + * @param[in] other Id of other entity + * + * @returns Pointer to either an already existing + * link in the tree or a newly created + * and inserted link. + * */ -ErtsLink *erts_link_tree_lookup_create(ErtsLink **root, int *created, - Uint16 type, Eterm this, Eterm other); +ErtsLink *erts_link_internal_tree_lookup_create(ErtsLink **root, int *created, + Uint16 type, Eterm other); /** * @@ -2052,60 +2092,74 @@ int erts_link_list_foreach_delete_yielding(ErtsLink **list, /** * - * @brief Create a link + * @brief Create an external link * - * Can create all types of links + * An external link structure contains two links, one for usage in + * the link tree of the process and one for usage in the dist entry. * - * When the function is called it is assumed that: - * - 'ref' is an internal ordinary reference if type is ERTS_MON_TYPE_PROC, - * ERTS_MON_TYPE_PORT, ERTS_MON_TYPE_TIME_OFFSET, or ERTS_MON_TYPE_RESOURCE - * - 'ref' is NIL if type is ERTS_MON_TYPE_NODE or ERTS_MON_TYPE_NODES - * - 'ref' is and ordinary internal reference or an external reference if - * type is ERTS_MON_TYPE_DIST_PROC - * - 'name' is an atom or NIL if type is ERTS_MON_TYPE_PROC, - * ERTS_MON_TYPE_PORT, or ERTS_MON_TYPE_DIST_PROC - * - 'name is NIL if type is ERTS_MON_TYPE_TIME_OFFSET, ERTS_MON_TYPE_RESOURCE, - * ERTS_MON_TYPE_NODE, or ERTS_MON_TYPE_NODES - * If the above is not true, bad things will happen. * - * @param[in] type ERTS_MON_TYPE_PROC, ERTS_MON_TYPE_PORT, - * ERTS_MON_TYPE_TIME_OFFSET, ERTS_MON_TYPE_DIST_PROC, - * ERTS_MON_TYPE_RESOURCE, ERTS_MON_TYPE_NODE, - * or ERTS_MON_TYPE_NODES + * @param[in] type ERTS_MON_TYPE_DIST_PROC + * + * @param[in] this The process identifier of the local + * process. The link structure in the + * 'dist' field a will have its + * 'other.item' field set to 'this'. + * The 'dist' link structure is to be + * inserted on the distribution entry. * - * @param[in] a The key of entity a. Link structure a will - * have field other.item set to 'b'. + * @param[in] other The process identifier of the remote + * process. The link structure in the + * 'proc' field a will have its + * 'other.item' field set to 'other'. + * The 'proc' link structure is to be + * inserted on the local process. * - * @param[in] b The key of entity b. Link structure b will - * have field other.item set to 'a'. + * @returns A pointer to the link data structure + * containing the link structures. The + * link data structure is in turn part + * of the external link structure + * (ErtsELink). * */ -ErtsLinkData *erts_link_create(Uint16 type, Eterm a, Eterm b); +ErtsLinkData *erts_link_external_create(Uint16 type, Eterm this, Eterm other); /** * - * @brief Get pointer to link data structure + * @brief Create an internal link + * + * @param[in] type ERTS_MON_TYPE_PROC, ERTS_MON_TYPE_PORT, + * + * @param[in] id Id of the entity linked. + * + * @returns A pointer to the link stucture. + */ +ErtsLink *erts_link_internal_create(Uint16 type, Eterm id); + +/** + * + * @brief Get pointer to external link data structure * * @param[in] lnk Pointer to link * - * @returns Pointer to link data structure + * @returns Pointer to external link structure * */ -ERTS_GLB_INLINE ErtsLinkData *erts_link_to_data(ErtsLink *lnk); +ERTS_GLB_INLINE ErtsELink *erts_link_to_elink(ErtsLink *lnk); /** * * @brief Get pointer to the other link structure part of the link * - * @param[in] lnk Pointer to link + * @param[in] lnk Pointer to link structure * - * @param[out] ldpp Pointer to pointer to link data structure, - * if a non-NULL value is passed in the call + * @param[out] elnkpp Pointer to pointer to external link + * data structure, if a non-NULL value + * is passed in the call * - * @returns Pointer to other link + * @returns Pointer to other link structure * */ -ERTS_GLB_INLINE ErtsLink *erts_link_to_other(ErtsLink *lnk, ErtsLinkData **ldpp); +ERTS_GLB_INLINE ErtsLink *erts_link_to_other(ErtsLink *lnk, ErtsELink **elnkpp); /** * @@ -2121,10 +2175,27 @@ ERTS_GLB_INLINE int erts_link_is_in_table(ErtsLink *lnk); /** * + * @brief Release an internal link + * + * When the function is called it is assumed that: + * - 'lnk' link is not part of any list or tree + * - 'lnk' is not referred to by any other structures + * If the above are not true, bad things will happen. + * + * @param[in] lnk Pointer to link + * + */ +ERTS_GLB_INLINE void erts_link_internal_release(ErtsLink *lnk); + +/** + * * @brief Release link * - * When both link halves part of the link have been released the link - * structure will be deallocated. + * Can be used to release a link half of an external + * link as well as an internal link. In the external + * case both link halves part of the external link have + * to been released before the link structure will be + * deallocated. * * When the function is called it is assumed that: * - 'lnk' link is not part of any list or tree @@ -2138,10 +2209,11 @@ ERTS_GLB_INLINE void erts_link_release(ErtsLink *lnk); /** * - * @brief Release both link halves of a link simultaneously + * @brief Release both link halves of an external link + * simultaneously * - * Release both halves of a link simultaneously and deallocate - * the structure. + * Release both halves of an external link simultaneously and + * deallocate the structure. * * When the function is called it is assumed that: * - Neither of the parts of the link are part of any list or tree @@ -2229,39 +2301,43 @@ erts_link_set_dead_dist(ErtsLink *lnk, Eterm nodename); Uint erts_link_size(ErtsLink *lnk); /* internal function... */ -void erts_link_destroy__(ErtsLinkData *ldp); +void erts_link_destroy_elink__(ErtsELink *elnk); /* implementations for globally inlined link functions... */ #if ERTS_GLB_INLINE_INCL_FUNC_DEF #ifdef ERTS_ML_DEBUG -extern size_t erts_link_a_offset; -extern size_t erts_link_b_offset; +extern size_t erts_link_proc_offset; +extern size_t erts_link_dist_offset; extern size_t erts_link_key_offset; #endif -ERTS_GLB_INLINE ErtsLinkData * -erts_link_to_data(ErtsLink *lnk) +ERTS_GLB_INLINE ErtsELink * +erts_link_to_elink(ErtsLink *lnk) { - ErtsLinkData *ldp = erts_ml_node_to_main_struct__((ErtsMonLnkNode *) lnk); + ErtsELink *elnk; + + ERTS_ML_ASSERT(lnk->flags & ERTS_ML_FLG_EXTENDED); + + elnk = erts_ml_node_to_main_struct__((ErtsMonLnkNode *) lnk); #ifdef ERTS_ML_DEBUG - ERTS_ML_ASSERT(erts_link_a_offset == (size_t) ldp->a.offset); - ERTS_ML_ASSERT(erts_link_key_offset == (size_t) ldp->a.key_offset); - ERTS_ML_ASSERT(erts_link_b_offset == (size_t) ldp->b.offset); - ERTS_ML_ASSERT(erts_link_key_offset == (size_t) ldp->b.key_offset); + ERTS_ML_ASSERT(erts_link_proc_offset == (size_t) elnk->ld.proc.offset); + ERTS_ML_ASSERT(erts_link_key_offset == (size_t) elnk->ld.proc.key_offset); + ERTS_ML_ASSERT(erts_link_dist_offset == (size_t) elnk->ld.dist.offset); + ERTS_ML_ASSERT(erts_link_key_offset == (size_t) elnk->ld.dist.key_offset); #endif - return ldp; + return elnk; } ERTS_GLB_INLINE ErtsLink * -erts_link_to_other(ErtsLink *lnk, ErtsLinkData **ldpp) +erts_link_to_other(ErtsLink *lnk, ErtsELink **elnkpp) { - ErtsLinkData *ldp = erts_link_to_data(lnk); - if (ldpp) - *ldpp = ldp; - return lnk == &ldp->a ? &ldp->b : &ldp->a; + ErtsELink *elnk = erts_link_to_elink(lnk); + if (elnkpp) + *elnkpp = elnk; + return lnk == &elnk->ld.proc ? &elnk->ld.dist : &elnk->ld.proc; } ERTS_GLB_INLINE int @@ -2295,23 +2371,38 @@ erts_link_list_last(ErtsLink *list) } ERTS_GLB_INLINE void +erts_link_internal_release(ErtsLink *lnk) +{ + ERTS_ML_ASSERT(lnk->type == ERTS_LNK_TYPE_PROC + || lnk->type == ERTS_LNK_TYPE_PORT); + ERTS_ML_ASSERT(!(lnk->flags & ERTS_ML_FLG_EXTENDED)); + erts_free(ERTS_ALC_T_LINK, lnk); +} + +ERTS_GLB_INLINE void erts_link_release(ErtsLink *lnk) { - ErtsLinkData *ldp = erts_link_to_data(lnk); - ERTS_ML_ASSERT(!(lnk->flags & ERTS_ML_FLG_IN_TABLE)); - ERTS_ML_ASSERT(erts_atomic32_read_nob(&ldp->refc) > 0); - if (erts_atomic32_dec_read_nob(&ldp->refc) == 0) - erts_link_destroy__(ldp); + if (!(lnk->flags & ERTS_ML_FLG_EXTENDED)) + erts_link_internal_release(lnk); + else { + ErtsELink *elnk = erts_link_to_elink(lnk); + ERTS_ML_ASSERT(!(lnk->flags & ERTS_ML_FLG_IN_TABLE)); + ERTS_ML_ASSERT(erts_atomic32_read_nob(&elnk->ld.refc) > 0); + if (erts_atomic32_dec_read_nob(&elnk->ld.refc) == 0) + erts_link_destroy_elink__(elnk); + } } ERTS_GLB_INLINE void erts_link_release_both(ErtsLinkData *ldp) { - ERTS_ML_ASSERT(!(ldp->a.flags & ERTS_ML_FLG_IN_TABLE)); - ERTS_ML_ASSERT(!(ldp->b.flags & ERTS_ML_FLG_IN_TABLE)); + ERTS_ML_ASSERT(!(ldp->proc.flags & ERTS_ML_FLG_IN_TABLE)); + ERTS_ML_ASSERT(!(ldp->dist.flags & ERTS_ML_FLG_IN_TABLE)); ERTS_ML_ASSERT(erts_atomic32_read_nob(&ldp->refc) >= 2); + ERTS_ML_ASSERT(ldp->proc.flags & ERTS_ML_FLG_EXTENDED); + ERTS_ML_ASSERT(ldp->dist.flags & ERTS_ML_FLG_EXTENDED); if (erts_atomic32_add_read_nob(&ldp->refc, (erts_aint32_t) -2) == 0) - erts_link_destroy__(ldp); + erts_link_destroy_elink__((ErtsELink *) ldp); } ERTS_GLB_INLINE ErtsLink * @@ -2342,22 +2433,22 @@ erts_link_tree_key_delete(ErtsLink **root, ErtsLink *lnk) ERTS_GLB_INLINE int erts_link_dist_insert(ErtsLink *lnk, ErtsMonLnkDist *dist) { - ErtsLinkDataExtended *ldep; + ErtsELink *elnk; int insert; ERTS_ML_ASSERT(lnk->flags & ERTS_ML_FLG_EXTENDED); ERTS_ML_ASSERT(lnk->type == ERTS_LNK_TYPE_DIST_PROC); - ldep = (ErtsLinkDataExtended *) erts_link_to_data(lnk); + elnk = erts_link_to_elink(lnk); - ERTS_ML_ASSERT(!ldep->dist); + ERTS_ML_ASSERT(!elnk->dist); ERTS_ML_ASSERT(dist); erts_mtx_lock(&dist->mtx); insert = dist->alive; if (insert) { - ldep->dist = dist; + elnk->dist = dist; erts_mon_link_dist_inc_refc(dist); erts_link_list_insert(&dist->links, lnk); } @@ -2370,15 +2461,15 @@ erts_link_dist_insert(ErtsLink *lnk, ErtsMonLnkDist *dist) ERTS_GLB_INLINE int erts_link_dist_delete(ErtsLink *lnk) { - ErtsLinkDataExtended *ldep; + ErtsELink *elnk; ErtsMonLnkDist *dist; int delete; ERTS_ML_ASSERT(lnk->flags & ERTS_ML_FLG_EXTENDED); ERTS_ML_ASSERT(lnk->type == ERTS_LNK_TYPE_DIST_PROC); - ldep = (ErtsLinkDataExtended *) erts_link_to_data(lnk); - dist = ldep->dist; + elnk = erts_link_to_elink(lnk); + dist = elnk->dist; if (!dist) return -1; diff --git a/erts/emulator/beam/erl_node_tables.c b/erts/emulator/beam/erl_node_tables.c index 929795fcb0..75c6b6abce 100644 --- a/erts/emulator/beam/erl_node_tables.c +++ b/erts/emulator/beam/erl_node_tables.c @@ -1641,18 +1641,21 @@ clear_visited_dist_monitors(DistEntry *dep) static void insert_link_data(ErtsLink *lnk, int type, Eterm id) { - ErtsLinkData *ldp = erts_link_to_data(lnk); - if ((ldp->a.flags & (ERTS_ML_FLG_DBG_VISITED - | ERTS_ML_FLG_EXTENDED)) == ERTS_ML_FLG_EXTENDED) { - ErtsLinkDataExtended *ldep = (ErtsLinkDataExtended *) ldp; - if (ldep->ohhp) { + if ((lnk->flags & (ERTS_ML_FLG_DBG_VISITED + | ERTS_ML_FLG_EXTENDED)) != ERTS_ML_FLG_EXTENDED) { + lnk->flags |= ERTS_ML_FLG_DBG_VISITED; + } + else { + ErtsELink *elnk = erts_link_to_elink(lnk); + if (elnk->ohhp) { ErlOffHeap oh; ERTS_INIT_OFF_HEAP(&oh); - oh.first = ldep->ohhp; + oh.first = elnk->ohhp; insert_offheap(&oh, type, id); } + elnk->ld.proc.flags |= ERTS_ML_FLG_DBG_VISITED; + elnk->ld.dist.flags |= ERTS_ML_FLG_DBG_VISITED; } - ldp->a.flags |= ERTS_ML_FLG_DBG_VISITED; } static int insert_link(ErtsLink *lnk, void *idp, Sint reds) @@ -1664,8 +1667,13 @@ static int insert_link(ErtsLink *lnk, void *idp, Sint reds) static int clear_visited_link(ErtsLink *lnk, void *p, Sint reds) { - ErtsLinkData *ldp = erts_link_to_data(lnk); - ldp->a.flags &= ~ERTS_ML_FLG_DBG_VISITED; + if (!(lnk->flags & ERTS_ML_FLG_EXTENDED)) + lnk->flags &= ~ERTS_ML_FLG_DBG_VISITED; + else { + ErtsELink *elnk = erts_link_to_elink(lnk); + elnk->ld.proc.flags &= ~ERTS_ML_FLG_DBG_VISITED; + elnk->ld.dist.flags &= ~ERTS_ML_FLG_DBG_VISITED; + } return 1; } diff --git a/erts/emulator/beam/erl_port.h b/erts/emulator/beam/erl_port.h index 9983a22f56..917a5714af 100644 --- a/erts/emulator/beam/erl_port.h +++ b/erts/emulator/beam/erl_port.h @@ -368,6 +368,7 @@ Eterm erts_request_io_bytes(Process *c_p); #define ERTS_PORT_REDS_EXIT (CONTEXT_REDS/100) #define ERTS_PORT_REDS_CONNECT (CONTEXT_REDS/200) #define ERTS_PORT_REDS_UNLINK (CONTEXT_REDS/200) +#define ERTS_PORT_REDS_UNLINK_ACK (CONTEXT_REDS/200) #define ERTS_PORT_REDS_LINK (CONTEXT_REDS/200) #define ERTS_PORT_REDS_MONITOR (CONTEXT_REDS/200) #define ERTS_PORT_REDS_DEMONITOR (CONTEXT_REDS/200) @@ -852,7 +853,8 @@ enum { ERTS_P2P_SIG_TYPE_LINK = 8, ERTS_P2P_SIG_TYPE_UNLINK = 9, ERTS_P2P_SIG_TYPE_MONITOR = 10, - ERTS_P2P_SIG_TYPE_DEMONITOR = 11 + ERTS_P2P_SIG_TYPE_DEMONITOR = 11, + ERTS_P2P_SIG_TYPE_UNLINK_ACK = 12 }; #define ERTS_P2P_SIG_TYPE_BITS 4 @@ -914,8 +916,12 @@ struct ErtsProc2PortSigData_ { } link; struct { Eterm port_id; - ErtsLink *lnk; + ErtsSigUnlinkOp *sulnk; } unlink; + struct { + Eterm port_id; + ErtsSigUnlinkOp *sulnk; + } unlink_ack; struct { Eterm port_id; ErtsMonitor *mon; @@ -1004,7 +1010,8 @@ ErtsPortOpResult erts_port_output(Process *, int, Port *, Eterm, Eterm, Eterm *) ErtsPortOpResult erts_port_exit(Process *, int, Port *, Eterm, Eterm, Eterm *); ErtsPortOpResult erts_port_connect(Process *, int, Port *, Eterm, Eterm, Eterm *); ErtsPortOpResult erts_port_link(Process *, Port *, ErtsLink *, Eterm *); -ErtsPortOpResult erts_port_unlink(Process *, Port *, ErtsLink *, Eterm *); +ErtsPortOpResult erts_port_unlink(Process *, Port *, ErtsSigUnlinkOp *, Eterm *); +ErtsPortOpResult erts_port_unlink_ack(Process *, Port *, ErtsSigUnlinkOp *); ErtsPortOpResult erts_port_control(Process *, Port *, unsigned int, Eterm, Eterm *); ErtsPortOpResult erts_port_call(Process *, Port *, unsigned int, Eterm, Eterm *); ErtsPortOpResult erts_port_info(Process *, Port *, Eterm, Eterm *); diff --git a/erts/emulator/beam/erl_proc_sig_queue.c b/erts/emulator/beam/erl_proc_sig_queue.c index 6d383cedcc..7d4820f1db 100644 --- a/erts/emulator/beam/erl_proc_sig_queue.c +++ b/erts/emulator/beam/erl_proc_sig_queue.c @@ -51,7 +51,7 @@ * Note that not all signal are handled using this functionality! */ -#define ERTS_SIG_Q_OP_MAX 14 +#define ERTS_SIG_Q_OP_MAX 15 #define ERTS_SIG_Q_OP_EXIT 0 /* Exit signal due to bif call */ #define ERTS_SIG_Q_OP_EXIT_LINKED 1 /* Exit signal due to link break*/ @@ -67,7 +67,8 @@ #define ERTS_SIG_Q_OP_PROCESS_INFO 11 #define ERTS_SIG_Q_OP_SYNC_SUSPEND 12 #define ERTS_SIG_Q_OP_RPC 13 -#define ERTS_SIG_Q_OP_DIST_SPAWN_REPLY ERTS_SIG_Q_OP_MAX +#define ERTS_SIG_Q_OP_DIST_SPAWN_REPLY 14 +#define ERTS_SIG_Q_OP_UNLINK_ACK ERTS_SIG_Q_OP_MAX #define ERTS_SIG_Q_TYPE_MAX (ERTS_MON_LNK_TYPE_MAX + 5) @@ -87,6 +88,10 @@ #define ERTS_SIG_IS_GEN_EXIT_EXTERNAL(sig) \ (ASSERT(ERTS_SIG_IS_GEN_EXIT(sig)),is_non_value(get_exit_signal_data(sig)->reason)) + +#define ERTS_SIG_LNK_X_FLAG_NORMAL_KILLS (((Uint32) 1) << 0) +#define ERTS_SIG_LNK_X_FLAG_CONNECTION_LOST (((Uint32) 1) << 1) + Process *ERTS_WRITE_UNLIKELY(erts_dirty_process_signal_handler); Process *ERTS_WRITE_UNLIKELY(erts_dirty_process_signal_handler_high); Process *ERTS_WRITE_UNLIKELY(erts_dirty_process_signal_handler_max); @@ -121,7 +126,15 @@ typedef struct { Eterm reason; union { Eterm ref; - int normal_kills; + struct { + Uint32 flags; + /* + * connection_id is only set when the + * ERTS_SIG_LNK_X_FLAG_CONNECTION_LOST + * flag has been set... + */ + Uint32 connection_id; + } link; } u; } ErtsExitSignalData; @@ -132,10 +145,13 @@ typedef struct { typedef struct { ErtsSignalCommon common; + Eterm nodename; + Uint32 connection_id; Eterm local; /* internal pid (immediate) */ Eterm remote; /* external pid (heap for it follow) */ + Uint64 id; Eterm heap[EXTERNAL_THING_HEAD_SIZE + 1]; -} ErtsSigDistLinkOp; +} ErtsSigDistUnlinkOp; typedef struct { Eterm message; @@ -281,39 +297,43 @@ destroy_dist_proc_demonitor(ErtsSigDistProcDemonitor *dmon) erts_free(ERTS_ALC_T_DIST_DEMONITOR, dmon); } -static ERTS_INLINE ErtsSigDistLinkOp * -make_sig_dist_link_op(int op, Eterm local, Eterm remote) +static ERTS_INLINE ErtsSigDistUnlinkOp * +make_sig_dist_unlink_op(int op, Eterm nodename, Uint32 conn_id, + Eterm local, Eterm remote, Uint64 id) { Eterm *hp; ErlOffHeap oh = {0}; - ErtsSigDistLinkOp *sdlnk = erts_alloc(ERTS_ALC_T_SIG_DATA, - sizeof(ErtsSigDistLinkOp)); + ErtsSigDistUnlinkOp *sdulnk = erts_alloc(ERTS_ALC_T_SIG_DATA, + sizeof(ErtsSigDistUnlinkOp)); ASSERT(is_internal_pid(local)); ASSERT(is_external_pid(remote)); - hp = &sdlnk->heap[0]; + hp = &sdulnk->heap[0]; - sdlnk->common.tag = ERTS_PROC_SIG_MAKE_TAG(op, - ERTS_SIG_Q_TYPE_DIST_LINK, - 0); - sdlnk->local = local; - sdlnk->remote = STORE_NC(&hp, &oh, remote); - - ASSERT(&sdlnk->heap[0] < hp); - ASSERT(hp <= &sdlnk->heap[0] + sizeof(sdlnk->heap)/sizeof(sdlnk->heap[0])); - ASSERT(boxed_val(sdlnk->remote) == &sdlnk->heap[0]); + sdulnk->common.tag = ERTS_PROC_SIG_MAKE_TAG(op, + ERTS_SIG_Q_TYPE_DIST_LINK, + 0); + sdulnk->nodename = nodename; + sdulnk->connection_id = conn_id; + sdulnk->local = local; + sdulnk->remote = STORE_NC(&hp, &oh, remote); + sdulnk->id = id; + + ASSERT(&sdulnk->heap[0] < hp); + ASSERT(hp <= &sdulnk->heap[0] + sizeof(sdulnk->heap)/sizeof(sdulnk->heap[0])); + ASSERT(boxed_val(sdulnk->remote) == &sdulnk->heap[0]); - return sdlnk; + return sdulnk; } static ERTS_INLINE void -destroy_sig_dist_link_op(ErtsSigDistLinkOp *sdlnk) +destroy_sig_dist_unlink_op(ErtsSigDistUnlinkOp *sdulnk) { - ASSERT(is_external_pid(sdlnk->remote)); - ASSERT(boxed_val(sdlnk->remote) == &sdlnk->heap[0]); - erts_deref_node_entry(((ExternalThing *) &sdlnk->heap[0])->node, - make_boxed(&sdlnk->heap[0])); - erts_free(ERTS_ALC_T_SIG_DATA, sdlnk); + ASSERT(is_external_pid(sdulnk->remote)); + ASSERT(boxed_val(sdulnk->remote) == &sdulnk->heap[0]); + erts_deref_node_entry(((ExternalThing *) &sdulnk->heap[0])->node, + make_boxed(&sdulnk->heap[0])); + erts_free(ERTS_ALC_T_SIG_DATA, sdulnk); } static ERTS_INLINE ErtsExitSignalData * @@ -408,23 +428,13 @@ sig_enqueue_trace(Process *c_p, ErtsMessage **sigp, int op, ErtsMessage* sig = *sigp; Uint16 type = ERTS_PROC_SIG_TYPE(((ErtsSignal *) sig)->common.tag); Eterm reason, from; + ErtsExitSignalData *xsigd; - if (type == ERTS_SIG_Q_TYPE_GEN_EXIT) { - ErtsExitSignalData *xsigd = get_exit_signal_data(sig); - reason = xsigd->reason; - from = xsigd->from; - } - else { - ErtsLink *lnk = (ErtsLink *) sig, *olnk; - - ASSERT(type == ERTS_LNK_TYPE_PROC - || type == ERTS_LNK_TYPE_PORT - || type == ERTS_LNK_TYPE_DIST_PROC); + ASSERT(type == ERTS_SIG_Q_TYPE_GEN_EXIT); - olnk = erts_link_to_other(lnk, NULL); - reason = lnk->other.item; - from = olnk->other.item; - } + xsigd = get_exit_signal_data(sig); + reason = xsigd->reason; + from = xsigd->from; if (is_pid(from)) { @@ -968,6 +978,12 @@ erts_proc_sig_privqs_len(Process *c_p) return proc_sig_privqs_len(c_p, 0); } +void +erts_proc_sig_destroy_unlink_op(ErtsSigUnlinkOp *sulnk) +{ + erts_free(ERTS_ALC_T_SIG_DATA, sulnk); +} + ErtsDistExternal * erts_proc_sig_get_external(ErtsMessage *msgp) { @@ -996,7 +1012,8 @@ send_gen_exit_signal(Process *c_p, Eterm from_tag, Eterm from, Eterm to, Sint16 op, Eterm reason, ErtsDistExternal *dist_ext, ErlHeapFragment *dist_ext_hfrag, - Eterm ref, Eterm token, int normal_kills) + Eterm ref, Eterm token, int normal_kills, + Uint32 conn_lost, Uint32 conn_id) { ErtsExitSignalData *xsigd; Eterm *hp, *start_hp, s_reason, s_ref, s_message, s_token, s_from; @@ -1134,12 +1151,18 @@ send_gen_exit_signal(Process *c_p, Eterm from_tag, xsigd->reason = s_reason; hfrag->next = dist_ext_hfrag; - if (is_nil(s_ref)) - xsigd->u.normal_kills = normal_kills; - else { + if (is_not_nil(s_ref)) { ASSERT(is_ref(s_ref)); xsigd->u.ref = s_ref; } + else { + xsigd->u.link.flags = 0; + if (normal_kills) + xsigd->u.link.flags |= ERTS_SIG_LNK_X_FLAG_NORMAL_KILLS; + if (conn_lost) + xsigd->u.link.flags |= ERTS_SIG_LNK_X_FLAG_CONNECTION_LOST; + xsigd->u.link.connection_id = conn_id; + } hp += sizeof(ErtsExitSignalData)/sizeof(Eterm); @@ -1282,7 +1305,7 @@ erts_proc_sig_send_exit(Process *c_p, Eterm from, Eterm to, from_tag = dep->sysname; } send_gen_exit_signal(c_p, from_tag, from, to, ERTS_SIG_Q_OP_EXIT, - reason, NULL, NULL, NIL, token, normal_kills); + reason, NULL, NULL, NIL, token, normal_kills, 0, 0); } void @@ -1293,7 +1316,7 @@ erts_proc_sig_send_dist_exit(DistEntry *dep, Eterm reason, Eterm token) { send_gen_exit_signal(NULL, dep->sysname, from, to, ERTS_SIG_Q_OP_EXIT, - reason, dist_ext, hfrag, NIL, token, 0); + reason, dist_ext, hfrag, NIL, token, 0, 0, 0); } @@ -1301,26 +1324,36 @@ void erts_proc_sig_send_link_exit(Process *c_p, Eterm from, ErtsLink *lnk, Eterm reason, Eterm token) { - Eterm to; + Eterm to, from_tag, from_item; + int conn_lost; + Uint32 conn_id; ASSERT(!c_p || c_p->common.id == from); ASSERT(lnk); to = lnk->other.item; - if (is_not_immed(reason) || is_not_nil(token)) { + if (is_value(from)) { ASSERT(is_internal_pid(from) || is_internal_port(from)); - send_gen_exit_signal(c_p, from, from, to, ERTS_SIG_Q_OP_EXIT_LINKED, - reason, NULL, NULL, NIL, token, 0); + from_tag = from_item = from; + conn_id = 0; + conn_lost = 0; } else { - /* Pass signal using old link structure... */ - ErtsSignal *sig = (ErtsSignal *) lnk; - lnk->other.item = reason; /* pass reason via this other.item */ - sig->common.tag = ERTS_PROC_SIG_MAKE_TAG(ERTS_SIG_Q_OP_EXIT_LINKED, - lnk->type, 0); - if (proc_queue_signal(c_p, to, sig, ERTS_SIG_Q_OP_EXIT_LINKED)) - return; /* receiver will destroy lnk structure */ - } - if (lnk) - erts_link_release(lnk); + ErtsLink *olnk; + ErtsELink *elnk; + + ASSERT(reason == am_noconnection); + ASSERT(lnk->flags & ERTS_ML_FLG_EXTENDED); + ASSERT(lnk->type == ERTS_LNK_TYPE_DIST_PROC); + + olnk = erts_link_to_other(lnk, &elnk); + + from_item = olnk->other.item; + from_tag = elnk->dist->nodename; + conn_id = elnk->dist->connection_id; + conn_lost = !0; + } + send_gen_exit_signal(c_p, from_tag, from_item, to, ERTS_SIG_Q_OP_EXIT_LINKED, + reason, NULL, NULL, NIL, token, 0, conn_lost, conn_id); + erts_link_release(lnk); } int @@ -1340,24 +1373,78 @@ erts_proc_sig_send_link(Process *c_p, Eterm to, ErtsLink *lnk) return proc_queue_signal(c_p, to, sig, ERTS_SIG_Q_OP_LINK); } -void -erts_proc_sig_send_unlink(Process *c_p, ErtsLink *lnk) +ErtsSigUnlinkOp * +erts_proc_sig_make_unlink_op(Process *c_p, Eterm from) +{ + Uint64 id; + ErtsSigUnlinkOp *sulnk; + if (c_p) + id = erts_proc_sig_new_unlink_id(c_p); + else { + /* + * *Only* ports are allowed to call without current + * process pointer... + */ + ASSERT(is_internal_port(from)); + id = (Uint64) erts_raw_get_unique_monotonic_integer(); + if (id == 0) + id = (Uint64) erts_raw_get_unique_monotonic_integer(); + } + + ASSERT(id != 0); + + sulnk = erts_alloc(ERTS_ALC_T_SIG_DATA, sizeof(ErtsSigUnlinkOp)); + sulnk->from = from; + sulnk->id = id; + + return sulnk; +} + +Uint64 +erts_proc_sig_send_unlink(Process *c_p, Eterm from, ErtsLink *lnk) { + int res; ErtsSignal *sig; Eterm to; + ErtsSigUnlinkOp *sulnk; + Uint64 id; - ASSERT(lnk); + ASSERT(lnk->type != ERTS_LNK_TYPE_PROC + || lnk->type != ERTS_LNK_TYPE_PORT); + ASSERT(lnk->flags & ERTS_ML_FLG_IN_TABLE); - sig = (ErtsSignal *) lnk; + sulnk = erts_proc_sig_make_unlink_op(c_p, from); + id = sulnk->id; + sig = (ErtsSignal *) sulnk; to = lnk->other.item; + sig->common.tag = ERTS_PROC_SIG_MAKE_TAG(ERTS_SIG_Q_OP_UNLINK, + lnk->type, 0); ASSERT(is_internal_pid(to)); + res = proc_queue_signal(c_p, to, sig, ERTS_SIG_Q_OP_UNLINK); + if (res == 0) { + erts_proc_sig_destroy_unlink_op(sulnk); + return 0; + } + return id; +} - sig->common.tag = ERTS_PROC_SIG_MAKE_TAG(ERTS_SIG_Q_OP_UNLINK, - lnk->type, 0); +void +erts_proc_sig_send_unlink_ack(Process *c_p, Eterm from, ErtsSigUnlinkOp *sulnk) +{ + ErtsSignal *sig = (ErtsSignal *) sulnk; + Eterm to = sulnk->from; + Uint16 type; - if (!proc_queue_signal(c_p, to, sig, ERTS_SIG_Q_OP_UNLINK)) - erts_link_release(lnk); + ASSERT(is_internal_pid(to)); + ASSERT(is_internal_pid(from) || is_internal_port(from)); + + sulnk->from = from; + type = is_internal_pid(from) ? ERTS_LNK_TYPE_PROC : ERTS_LNK_TYPE_PORT; + sig->common.tag = ERTS_PROC_SIG_MAKE_TAG(ERTS_SIG_Q_OP_UNLINK_ACK, + type, 0); + if (!proc_queue_signal(c_p, to, sig, ERTS_SIG_Q_OP_UNLINK_ACK)) + erts_proc_sig_destroy_unlink_op(sulnk); } void @@ -1368,24 +1455,92 @@ erts_proc_sig_send_dist_link_exit(DistEntry *dep, Eterm reason, Eterm token) { send_gen_exit_signal(NULL, dep->sysname, from, to, ERTS_SIG_Q_OP_EXIT_LINKED, - reason, dist_ext, hfrag, NIL, token, 0); + reason, dist_ext, hfrag, NIL, token, 0, 0, 0); } +static void +reply_dist_unlink_ack(Process *c_p, ErtsSigDistUnlinkOp *sdulnk); + void -erts_proc_sig_send_dist_unlink(DistEntry *dep, Eterm from, Eterm to) +erts_proc_sig_send_dist_unlink(DistEntry *dep, Uint32 conn_id, + Eterm from, Eterm to, Uint64 id) { + /* Remote to local */ ErtsSignal *sig; ASSERT(is_internal_pid(to)); ASSERT(is_external_pid(from)); ASSERT(dep == external_pid_dist_entry(from)); - sig = (ErtsSignal *) make_sig_dist_link_op(ERTS_SIG_Q_OP_UNLINK, - to, from); + sig = (ErtsSignal *) make_sig_dist_unlink_op(ERTS_SIG_Q_OP_UNLINK, + dep->sysname, conn_id, + to, from, id); if (!proc_queue_signal(NULL, to, sig, ERTS_SIG_Q_OP_UNLINK)) - destroy_sig_dist_link_op((ErtsSigDistLinkOp *) sig); + reply_dist_unlink_ack(NULL, (ErtsSigDistUnlinkOp *) sig); +} + +void +erts_proc_sig_send_dist_unlink_ack(Process *c_p, DistEntry *dep, + Uint32 conn_id, Eterm from, Eterm to, + Uint64 id) +{ + /* Remote to local */ + ErtsSignal *sig; + + ASSERT(is_internal_pid(to)); + ASSERT(is_external_pid(from)); + ASSERT(dep == external_pid_dist_entry(from)); + + sig = (ErtsSignal *) make_sig_dist_unlink_op(ERTS_SIG_Q_OP_UNLINK_ACK, + dep->sysname, conn_id, + to, from, id); + + if (!proc_queue_signal(c_p, to, sig, ERTS_SIG_Q_OP_UNLINK_ACK)) + destroy_sig_dist_unlink_op((ErtsSigDistUnlinkOp *) sig); +} + +static void +reply_dist_unlink_ack(Process *c_p, ErtsSigDistUnlinkOp *sdulnk) +{ + /* Local to remote */ + ASSERT(is_external_pid(sdulnk->remote)); + + /* + * 'id' is zero if the other side not understand + * unlink-ack signals... + */ + if (sdulnk->id) { + DistEntry *dep = external_pid_dist_entry(sdulnk->remote); + + /* + * Do not set up new a connection; only send unlink ack + * on the same connection which the unlink operation was + * received on... + */ + if (dep != erts_this_dist_entry && sdulnk->nodename == dep->sysname) { + ErtsDSigSendContext ctx; + int code = erts_dsig_prepare(&ctx, dep, c_p, 0, + ERTS_DSP_NO_LOCK, 1, 1, 0); + switch (code) { + case ERTS_DSIG_PREP_CONNECTED: + case ERTS_DSIG_PREP_PENDING: + if (sdulnk->connection_id == ctx.connection_id) { + code = erts_dsig_send_unlink_ack(&ctx, + sdulnk->local, + sdulnk->remote, + sdulnk->id); + ASSERT(code == ERTS_DSIG_SEND_OK); + } + break; + default: + break; + } + } + } + + destroy_sig_dist_unlink_op(sdulnk); } void @@ -1402,7 +1557,7 @@ erts_proc_sig_send_dist_monitor_down(DistEntry *dep, Eterm ref, monitored = from; send_gen_exit_signal(NULL, dep->sysname, monitored, to, ERTS_SIG_Q_OP_MONITOR_DOWN, - reason, dist_ext, hfrag, ref, NIL, 0); + reason, dist_ext, hfrag, ref, NIL, 0, 0, 0); } void @@ -1473,7 +1628,8 @@ erts_proc_sig_send_monitor_down(ErtsMonitor *mon, Eterm reason) } send_gen_exit_signal(NULL, from_tag, monitored, to, ERTS_SIG_Q_OP_MONITOR_DOWN, - reason, NULL, NULL, mdp->ref, NIL, 0); + reason, NULL, NULL, mdp->ref, NIL, + 0, 0, 0); } erts_monitor_release(mon); } @@ -2214,46 +2370,68 @@ handle_exit_signal(Process *c_p, ErtsSigRecvTracing *tracing, { ErtsMessage *conv_msg = NULL; ErtsExitSignalData *xsigd = NULL; - ErtsLinkData *ldp = NULL; /* Avoid erroneous warning... */ - ErtsLink *dlnk = NULL; /* Avoid erroneous warning... */ Eterm tag = ((ErtsSignal *) sig)->common.tag; - Uint16 type = ERTS_PROC_SIG_TYPE(tag); int op = ERTS_PROC_SIG_OP(tag); int destroy = 0; int ignore = 0; int save = 0; int exit = 0; + int linked = 0; int cnt = 1; Eterm reason; Eterm from; - if (type == ERTS_SIG_Q_TYPE_GEN_EXIT) { - xsigd = get_exit_signal_data(sig); - from = xsigd->from; - if (op != ERTS_SIG_Q_OP_EXIT_LINKED) - ignore = 0; + ASSERT(ERTS_PROC_SIG_TYPE(tag) == ERTS_SIG_Q_TYPE_GEN_EXIT); + + xsigd = get_exit_signal_data(sig); + from = xsigd->from; + + if (op == ERTS_SIG_Q_OP_EXIT_LINKED) { + ErtsLink *lnk, *dlnk = NULL; + ErtsELink *elnk = NULL; + lnk = erts_link_tree_lookup(ERTS_P_LINKS(c_p), from); + if (!lnk) + ignore = destroy = !0; /* No longer active */ + else if (lnk->type != ERTS_LNK_TYPE_DIST_PROC) { + if (((ErtsILink *) lnk)->unlinking) + ignore = destroy = !0; /* No longer active */ + else + linked = !0; + } else { - ErtsLink *llnk = erts_link_tree_lookup(ERTS_P_LINKS(c_p), from); - if (!llnk) { - /* Link no longer active; ignore... */ - ignore = !0; - destroy = !0; - } - else { - ignore = 0; - erts_link_tree_delete(&ERTS_P_LINKS(c_p), llnk); - if (llnk->type != ERTS_LNK_TYPE_DIST_PROC) - erts_link_release(llnk); - else { - dlnk = erts_link_to_other(llnk, &ldp); - if (erts_link_dist_delete(dlnk)) - erts_link_release_both(ldp); - else - erts_link_release(llnk); - } + dlnk = erts_link_to_other(lnk, &elnk); + if (elnk->unlinking) + ignore = destroy = !0; /* No longer active */ + else + linked = !0; + if ((xsigd->u.link.flags & ERTS_SIG_LNK_X_FLAG_CONNECTION_LOST) + && xsigd->u.link.connection_id != elnk->dist->connection_id) { + /* + * The exit signal is due to loss of connection. The link + * that triggered this was setup before that connection + * was lost, but was later unlinked. After that, the + * current link was setup using a new connection. That is, + * current link should be left unaffected, and the signal + * should be silently dropped. + */ + linked = 0; + lnk = NULL; + ignore = destroy = !0; } } + if (lnk) { + /* Remove link... */ + erts_link_tree_delete(&ERTS_P_LINKS(c_p), lnk); + if (!elnk) + erts_link_internal_release(lnk); + else if (erts_link_dist_delete(dlnk)) + erts_link_release_both(&elnk->ld); + else + erts_link_release(lnk); + } + } + if (!ignore) { /* This GEN_EXIT was received from another node, decode the exit reason */ if (ERTS_SIG_IS_GEN_EXIT_EXTERNAL(sig)) erts_proc_sig_decode_dist(c_p, ERTS_PROC_LOCK_MAIN, sig, 1); @@ -2265,106 +2443,29 @@ handle_exit_signal(Process *c_p, ErtsSigRecvTracing *tracing, ignore = !0; destroy = !0; } + } - if (!ignore) { + if (!ignore) { - if ((op != ERTS_SIG_Q_OP_EXIT || reason != am_kill) - && (c_p->flags & F_TRAP_EXIT)) { - convert_prepared_sig_to_msg(c_p, sig, - xsigd->message, next_nm_sig); - conv_msg = sig; - } - else if (reason == am_normal && !xsigd->u.normal_kills) { - /* Ignore it... */ - destroy = !0; - ignore = !0; - } - else { - /* Terminate... */ - save = !0; - exit = !0; - if (op == ERTS_SIG_Q_OP_EXIT && reason == am_kill) - reason = am_killed; - } + if ((op != ERTS_SIG_Q_OP_EXIT || reason != am_kill) + && (c_p->flags & F_TRAP_EXIT)) { + convert_prepared_sig_to_msg(c_p, sig, + xsigd->message, next_nm_sig); + conv_msg = sig; } - } - else { /* Link exit */ - ErtsLink *slnk = (ErtsLink *) sig; - ErtsLink *llnk = erts_link_to_other(slnk, &ldp); - - ASSERT(type == ERTS_LNK_TYPE_PROC - || type == ERTS_LNK_TYPE_PORT - || type == ERTS_LNK_TYPE_DIST_PROC); - - from = llnk->other.item; - reason = slnk->other.item; /* reason in other.item ... */ - ASSERT(is_pid(from) || is_internal_port(from)); - ASSERT(is_immed(reason)); - ASSERT(op == ERTS_SIG_Q_OP_EXIT_LINKED); - dlnk = erts_link_tree_key_delete(&ERTS_P_LINKS(c_p), llnk); - if (!dlnk) { - ignore = !0; /* Link no longer active; ignore... */ - ldp = NULL; + else if (reason == am_normal + && !(xsigd->u.link.flags & ERTS_SIG_LNK_X_FLAG_NORMAL_KILLS)) { + /* Ignore it... */ + destroy = !0; + ignore = !0; } else { - Eterm pid; - ErtsMessage *mp; - ErtsProcLocks locks; - Uint hsz; - Eterm *hp; - ErlOffHeap *ohp; - ignore = 0; - if (dlnk == llnk) - dlnk = NULL; - else - ldp = NULL; - - ASSERT(is_immed(reason)); - - if (!(c_p->flags & F_TRAP_EXIT)) { - if (reason == am_normal) - ignore = !0; /* Ignore it... */ - else - exit = !0; /* Terminate... */ - } - else { - - /* - * Create and EXIT message and replace - * the original signal with the message... - */ - - locks = ERTS_PROC_LOCK_MAIN; - - hsz = 4 + NC_HEAP_SIZE(from); - - mp = erts_alloc_message_heap(c_p, &locks, hsz, &hp, &ohp); - - if (locks != ERTS_PROC_LOCK_MAIN) - erts_proc_unlock(c_p, locks & ~ERTS_PROC_LOCK_MAIN); - - pid = STORE_NC(&hp, ohp, from); - - ERL_MESSAGE_TERM(mp) = TUPLE3(hp, am_EXIT, pid, reason); - ERL_MESSAGE_TOKEN(mp) = am_undefined; - if (is_immed(pid)) - ERL_MESSAGE_FROM(mp) = pid; - else { - DistEntry *dep; - ASSERT(is_external_pid(pid)); - dep = external_pid_dist_entry(pid); - ERL_MESSAGE_FROM(mp) = dep->sysname; - } - - /* Replace original signal with the exit message... */ - convert_to_msg(c_p, sig, mp, next_nm_sig); - - cnt += 4; - - conv_msg = mp; - } + /* Terminate... */ + save = !0; + exit = !0; + if (op == ERTS_SIG_Q_OP_EXIT && reason == am_kill) + reason = am_killed; } - destroy = !0; } if (ignore|exit) { @@ -2385,25 +2486,16 @@ handle_exit_signal(Process *c_p, ErtsSigRecvTracing *tracing, if (!exit) { if (conv_msg) erts_proc_notify_new_message(c_p, ERTS_PROC_LOCK_MAIN); - if (op == ERTS_SIG_Q_OP_EXIT_LINKED && tracing->procs) + if (linked && tracing->procs) { + ASSERT(op == ERTS_SIG_Q_OP_EXIT_LINKED); getting_unlinked(c_p, from); + } } if (destroy) { cnt++; - if (type == ERTS_SIG_Q_TYPE_GEN_EXIT) { - sig->next = NULL; - erts_cleanup_messages(sig); - } - else { - if (ldp) - erts_link_release_both(ldp); - else { - if (dlnk) - erts_link_release(dlnk); - erts_link_release((ErtsLink *) sig); - } - } + sig->next = NULL; + erts_cleanup_messages(sig); } *exited = exit; @@ -3366,10 +3458,10 @@ handle_dist_spawn_reply(Process *c_p, ErtsSigRecvTracing *tracing, /* Stale reply; remove link that was setup... */ ErtsLink *lnk = datap->link; if (lnk) { - ErtsLinkData *ldp; - ErtsLink *dlnk = erts_link_to_other(lnk, &ldp); + ErtsELink *elnk; + ErtsLink *dlnk = erts_link_to_other(lnk, &elnk); if (erts_link_dist_delete(dlnk)) - erts_link_release_both(ldp); + erts_link_release_both(&elnk->ld); else erts_link_release(lnk); } @@ -3452,11 +3544,11 @@ handle_dist_spawn_reply(Process *c_p, ErtsSigRecvTracing *tracing, lnk = datap->link; if (lnk) { - ErtsLinkData *ldp; + ErtsELink *elnk; ErtsLink *dlnk; - dlnk = erts_link_to_other(lnk, &ldp); + dlnk = erts_link_to_other(lnk, &elnk); if (erts_link_dist_delete(dlnk)) - erts_link_release_both(ldp); + erts_link_release_both(&elnk->ld); else erts_link_release(lnk); } @@ -3588,10 +3680,10 @@ handle_dist_spawn_reply_exiting(Process *c_p, /* May happen when connection concurrently close... */ ErtsLink *lnk = datap->link; if (lnk) { - ErtsLinkData *ldp; - ErtsLink *dlnk = erts_link_to_other(lnk, &ldp); + ErtsELink *elnk; + ErtsLink *dlnk = erts_link_to_other(lnk, &elnk); if (erts_link_dist_delete(dlnk)) - erts_link_release_both(ldp); + erts_link_release_both(&elnk->ld); else erts_link_release(lnk); } @@ -3954,27 +4046,27 @@ erts_proc_sig_handle_incoming(Process *c_p, erts_aint32_t *statep, } case ERTS_SIG_Q_OP_LINK: { - ErtsLink *rlnk, *lnk = (ErtsLink *) sig; + ErtsLink *lnk, *nlnk = (ErtsLink *) sig; ERTS_PROC_SIG_HDBG_PRIV_CHKQ(c_p, &tracing, next_nm_sig); remove_nm_sig(c_p, sig, next_nm_sig); - rlnk = erts_link_tree_insert_addr_replace(&ERTS_P_LINKS(c_p), - lnk); - if (!rlnk) { + lnk = erts_link_tree_lookup_insert(&ERTS_P_LINKS(c_p), nlnk); + if (!lnk) { if (tracing.procs) - getting_linked(c_p, lnk->other.item); + getting_linked(c_p, nlnk->other.item); } else { - if (rlnk->type != ERTS_LNK_TYPE_DIST_PROC) - erts_link_release(rlnk); + /* Already linked or unlinking... */ + if (nlnk->type != ERTS_LNK_TYPE_DIST_PROC) + erts_link_internal_release(nlnk); else { - ErtsLinkData *ldp; - ErtsLink *dlnk = erts_link_to_other(rlnk, &ldp); + ErtsELink *elnk; + ErtsLink *dlnk = erts_link_to_other(nlnk, &elnk); if (erts_link_dist_delete(dlnk)) - erts_link_release_both(ldp); + erts_link_release_both(&elnk->ld); else - erts_link_release(rlnk); + erts_link_release(nlnk); } } @@ -3984,52 +4076,102 @@ erts_proc_sig_handle_incoming(Process *c_p, erts_aint32_t *statep, case ERTS_SIG_Q_OP_UNLINK: { Uint16 type = ERTS_PROC_SIG_TYPE(tag); - ErtsLinkData *ldp; ErtsLink *llnk; ERTS_PROC_SIG_HDBG_PRIV_CHKQ(c_p, &tracing, next_nm_sig); remove_nm_sig(c_p, sig, next_nm_sig); if (type == ERTS_SIG_Q_TYPE_DIST_LINK) { - ErtsSigDistLinkOp *sdlnk = (ErtsSigDistLinkOp *) sig; - ASSERT(type == ERTS_SIG_Q_TYPE_DIST_LINK); - ASSERT(is_external_pid(sdlnk->remote)); - llnk = erts_link_tree_lookup(ERTS_P_LINKS(c_p), sdlnk->remote); + ErtsSigDistUnlinkOp *sdulnk = (ErtsSigDistUnlinkOp *) sig; + ASSERT(is_external_pid(sdulnk->remote)); + llnk = erts_link_tree_lookup(ERTS_P_LINKS(c_p), sdulnk->remote); if (llnk) { - ErtsLink *dlnk = erts_link_to_other(llnk, &ldp); - erts_link_tree_delete(&ERTS_P_LINKS(c_p), llnk); - if (erts_link_dist_delete(dlnk)) - erts_link_release_both(ldp); - else - erts_link_release(llnk); - cnt += 8; - if (tracing.procs) - getting_unlinked(c_p, sdlnk->remote); + ErtsELink *elnk; + ErtsLink *dlnk = erts_link_to_other(llnk, &elnk); + if (!elnk->unlinking) { + erts_link_tree_delete(&ERTS_P_LINKS(c_p), llnk); + if (erts_link_dist_delete(dlnk)) + erts_link_release_both(&elnk->ld); + else + erts_link_release(llnk); + cnt += 8; + if (tracing.procs) + getting_unlinked(c_p, sdulnk->remote); + } } - destroy_sig_dist_link_op(sdlnk); + reply_dist_unlink_ack(c_p, sdulnk); cnt++; } else { - ErtsLinkData *ldp; - ErtsLink *dlnk, *slnk; - slnk = (ErtsLink *) sig; - llnk = erts_link_to_other(slnk, &ldp); - dlnk = erts_link_tree_key_delete(&ERTS_P_LINKS(c_p), llnk); - if (!dlnk) - erts_link_release(slnk); - else { + ErtsSigUnlinkOp *sulnk = (ErtsSigUnlinkOp *) sig; + llnk = erts_link_tree_lookup(ERTS_P_LINKS(c_p), + sulnk->from); + if (llnk && !((ErtsILink *) llnk)->unlinking) { if (tracing.procs) - getting_unlinked(c_p, llnk->other.item); - if (dlnk == llnk) - erts_link_release_both(ldp); - else { - erts_link_release(slnk); - erts_link_release(dlnk); + getting_unlinked(c_p, sulnk->from); + erts_link_tree_delete(&ERTS_P_LINKS(c_p), llnk); + erts_link_release(llnk); + cnt += 4; + } + if (is_internal_pid(sulnk->from)) + erts_proc_sig_send_unlink_ack(c_p, c_p->common.id, sulnk); + else { + Port *prt; + ASSERT(is_internal_port(sulnk->from)); + prt = erts_port_lookup(sulnk->from, + ERTS_PORT_SFLGS_INVALID_DRIVER_LOOKUP); + if (prt) + erts_port_unlink_ack(c_p, prt, sulnk); + else + erts_proc_sig_destroy_unlink_op(sulnk); + } + } + + ERTS_PROC_SIG_HDBG_PRIV_CHKQ(c_p, &tracing, next_nm_sig); + break; + } + + case ERTS_SIG_Q_OP_UNLINK_ACK: { + Uint16 type = ERTS_PROC_SIG_TYPE(tag); + + ERTS_PROC_SIG_HDBG_PRIV_CHKQ(c_p, &tracing, next_nm_sig); + + remove_nm_sig(c_p, sig, next_nm_sig); + if (type == ERTS_SIG_Q_TYPE_DIST_LINK) { + ErtsSigDistUnlinkOp *sdulnk; + ErtsLink *lnk; + sdulnk = (ErtsSigDistUnlinkOp *) sig; + lnk = erts_link_tree_lookup(ERTS_P_LINKS(c_p), + sdulnk->remote); + if (lnk) { + ErtsELink *elnk = erts_link_to_elink(lnk); + if (elnk->unlinking == sdulnk->id) { + erts_link_tree_delete(&ERTS_P_LINKS(c_p), lnk); + if (erts_link_dist_delete(&elnk->ld.dist)) + erts_link_release_both(&elnk->ld); + else + erts_link_release(lnk); + cnt += 8; } } - cnt += 2; + destroy_sig_dist_unlink_op(sdulnk); } + else { + ErtsSigUnlinkOp *sulnk; + ErtsILink *ilnk; + sulnk = (ErtsSigUnlinkOp *) sig; + ilnk = (ErtsILink *) erts_link_tree_lookup(ERTS_P_LINKS(c_p), + sulnk->from); + + if (ilnk && ilnk->unlinking == sulnk->id) { + erts_link_tree_delete(&ERTS_P_LINKS(c_p), &ilnk->link); + erts_link_internal_release(&ilnk->link); + cnt += 4; + } + erts_proc_sig_destroy_unlink_op(sulnk); + } + ERTS_PROC_SIG_HDBG_PRIV_CHKQ(c_p, &tracing, next_nm_sig); break; } @@ -4398,9 +4540,24 @@ erts_proc_sig_handle_exit(Process *c_p, Sint *redsp, case ERTS_SIG_Q_OP_UNLINK: if (type == ERTS_SIG_Q_TYPE_DIST_LINK) - destroy_sig_dist_link_op((ErtsSigDistLinkOp *) sig); - else - erts_link_release((ErtsLink *) sig); + reply_dist_unlink_ack(c_p, (ErtsSigDistUnlinkOp *) sig); + else if (is_internal_pid(((ErtsSigUnlinkOp *) sig)->from)) + erts_proc_sig_send_unlink_ack(c_p, c_p->common.id, + (ErtsSigUnlinkOp *) sig); + else { + Port *prt; + ASSERT(is_internal_port(((ErtsSigUnlinkOp *) sig)->from)); + prt = erts_port_lookup(((ErtsSigUnlinkOp *) sig)->from, + ERTS_PORT_SFLGS_INVALID_DRIVER_LOOKUP); + if (prt) + erts_port_unlink_ack(c_p, prt, (ErtsSigUnlinkOp *) sig); + else + erts_proc_sig_destroy_unlink_op((ErtsSigUnlinkOp *) sig); + } + break; + + case ERTS_SIG_Q_OP_UNLINK_ACK: + erts_proc_sig_destroy_unlink_op((ErtsSigUnlinkOp *) sig); break; case ERTS_SIG_Q_OP_GROUP_LEADER: { @@ -4515,6 +4672,7 @@ clear_seq_trace_token(ErtsMessage *sig) case ERTS_SIG_Q_OP_DEMONITOR: case ERTS_SIG_Q_OP_LINK: case ERTS_SIG_Q_OP_UNLINK: + case ERTS_SIG_Q_OP_UNLINK_ACK: case ERTS_SIG_Q_OP_TRACE_CHANGE_STATE: case ERTS_SIG_Q_OP_GROUP_LEADER: case ERTS_SIG_Q_OP_IS_ALIVE: @@ -4603,15 +4761,16 @@ erts_proc_sig_signal_size(ErtsSignal *sig) break; case ERTS_SIG_Q_OP_UNLINK: - if (type == ERTS_SIG_Q_TYPE_DIST_LINK) { - size = NC_HEAP_SIZE(((ErtsSigDistLinkOp *) sig)->remote); + case ERTS_SIG_Q_OP_UNLINK_ACK: + if (type != ERTS_SIG_Q_TYPE_DIST_LINK) + size = sizeof(ErtsSigUnlinkOp); + else { + size = NC_HEAP_SIZE(((ErtsSigDistUnlinkOp *) sig)->remote); size--; size *= sizeof(Eterm); - size += sizeof(ErtsSigDistLinkOp); - break; + size += sizeof(ErtsSigDistUnlinkOp); } - /* Fall through... */ - + break; case ERTS_SIG_Q_OP_LINK: size = erts_link_size((ErtsLink *) sig); break; @@ -5264,12 +5423,14 @@ erts_proc_sig_debug_foreach_sig(Process *c_p, case ERTS_SIG_Q_OP_UNLINK: if (type == ERTS_SIG_Q_TYPE_DIST_LINK) { - debug_foreach_sig_fake_oh(((ErtsSigDistLinkOp *) sig)->remote, + debug_foreach_sig_fake_oh(((ErtsSigDistUnlinkOp *) sig)->remote, oh_func, arg); - break; } - /* Fall through... */ - + break; + + case ERTS_SIG_Q_OP_UNLINK_ACK: + break; + case ERTS_SIG_Q_OP_LINK: lnk_func((ErtsLink *) sig, arg, -1); break; diff --git a/erts/emulator/beam/erl_proc_sig_queue.h b/erts/emulator/beam/erl_proc_sig_queue.h index dfd443e3d6..bbff433711 100644 --- a/erts/emulator/beam/erl_proc_sig_queue.h +++ b/erts/emulator/beam/erl_proc_sig_queue.h @@ -30,6 +30,7 @@ * - Persistent monitor message * - Link * - Unlink + * - Unlink Ack * - Group leader * - Is process alive * - Process info request @@ -100,6 +101,12 @@ typedef struct { Eterm tag; } ErtsSignalCommon; +typedef struct { + ErtsSignalCommon common; + Eterm from; + Uint64 id; +} ErtsSigUnlinkOp; + #define ERTS_SIG_HANDLE_REDS_MAX_PREFERED (CONTEXT_REDS/40) #ifdef ERTS_PROC_SIG_HARD_DEBUG @@ -295,19 +302,97 @@ erts_proc_sig_send_link(Process *c_p, Eterm to, ErtsLink *lnk); /** * + * @brief Create a new unlink identifier + * + * The newly created unlink identifier is to be used in an + * unlink operation. + * + * @param[in] c_p Pointer to process struct of + * currently executing process. + * + * @return A new 64-bit unlink identifier + * unique in context of the + * calling process. The identifier + * may be any value but zero. + */ +ERTS_GLB_INLINE Uint64 erts_proc_sig_new_unlink_id(Process *c_p); + +/** + * + * @brief Create an unlink op signal structure + * + * The structure will contain a newly created unlink + * identifier to be used in the operation. + * + * @param[in] c_p Pointer to process struct of + * currently executing process + * ('from' is a process + * identifier), or NULL if not + * called in the context of an + * executing process ('from' is + * a port identifier). + * + * @param[in] from Id (as an erlang term) of + * entity sending the unlink + * signal. + * + * @return A pointer to the unlink op + * structure. + */ +ErtsSigUnlinkOp * +erts_proc_sig_make_unlink_op(Process *c_p, Eterm from); + +/** + * + * @brief Destroy an unlink op signal structure + * + * @param[in] sulnk A pointer to the unlink op + * structure. + */ +void +erts_proc_sig_destroy_unlink_op(ErtsSigUnlinkOp *sulnk); + +/** + * * @brief Send an unlink signal to a process. * * * @param[in] c_p Pointer to process struct of * currently executing process. * + * @param[in] from Id (as an erlang term) of + * entity sending the unlink + * signal. + * * @param[in] lnk Pointer to link structure from * the sending side. It should * contain information about * receiver. */ +Uint64 +erts_proc_sig_send_unlink(Process *c_p, Eterm from, ErtsLink *lnk); + +/** + * + * @brief Send an unlink acknowledgment signal to a process. + * + * + * @param[in] c_p Pointer to process struct of + * currently executing process. + * + * @param[in] from Id (as an erlang term) of + * entity sending the unlink + * signal. + * + * @param[in] sulnk A pointer to the unlink op + * structure. This structure + * was typically received by + * the caller in an unlink + * signal. + */ void -erts_proc_sig_send_unlink(Process *c_p, ErtsLink *lnk); +erts_proc_sig_send_unlink_ack(Process *c_p, Eterm from, + ErtsSigUnlinkOp *sulnk); /** * @@ -343,11 +428,10 @@ erts_proc_sig_send_dist_link_exit(struct dist_entry_ *dep, /** * - * @brief Send an unlink signal to a process. + * @brief Send an unlink signal to a local process. * * This function is used instead of erts_proc_sig_send_unlink() - * when the signal arrives via the distribution and - * therefore no link structure is available. + * when the signal arrives via the distribution. * * @param[in] dep Distribution entry of channel * that the signal arrived on. @@ -356,10 +440,37 @@ erts_proc_sig_send_dist_link_exit(struct dist_entry_ *dep, * * @param[in] to Identifier of receiver. * + * @param[in] id Identifier of unlink operation. */ void -erts_proc_sig_send_dist_unlink(struct dist_entry_ *dep, - Eterm from, Eterm to); +erts_proc_sig_send_dist_unlink(DistEntry *dep, Uint32 conn_id, + Eterm from, Eterm to, Uint64 id); + +/** + * + * @brief Send an unlink acknowledgment signal to a local process. + * + * This function is used instead of erts_proc_sig_send_unlink_ack() + * when the signal arrives via the distribution. + * + * @param[in] c_p Pointer to process struct of + * currently executing process or + * NULL if not called in the context + * of an executing process. + * + * @param[in] dep Distribution entry of channel + * that the signal arrived on. + * + * @param[in] from Identifier of sender. + * + * @param[in] to Identifier of receiver. + * + * @param[in] id Identifier of unlink operation. + */ +void +erts_proc_sig_send_dist_unlink_ack(Process *c_p, DistEntry *dep, + Uint32 conn_id, Eterm from, Eterm to, + Uint64 id); /** * @@ -1069,6 +1180,18 @@ Sint erts_proc_sig_fetch_msgq_len_offs__(Process *proc); #if ERTS_GLB_INLINE_INCL_FUNC_DEF +ERTS_GLB_INLINE Uint64 +erts_proc_sig_new_unlink_id(Process *c_p) +{ + Uint64 id; + ASSERT(c_p); + + id = (Uint64) c_p->uniq++; + if (id == 0) + id = (Uint64) c_p->uniq++; + return id; +} + ERTS_GLB_INLINE Sint erts_proc_sig_fetch(Process *proc) { diff --git a/erts/emulator/beam/erl_process.c b/erts/emulator/beam/erl_process.c index 3c8b1631b4..55d0690d0d 100644 --- a/erts/emulator/beam/erl_process.c +++ b/erts/emulator/beam/erl_process.c @@ -11896,6 +11896,7 @@ erl_create_process(Process* parent, /* Parent of process (default group leader). erts_get_default_proc_tracing(&ERTS_TRACE_FLAGS(p), &ERTS_TRACER(p)); + p->uniq = 1; p->sig_qs.first = NULL; p->sig_qs.last = &p->sig_qs.first; p->sig_qs.cont = NULL; @@ -11939,6 +11940,28 @@ erl_create_process(Process* parent, /* Parent of process (default group leader). p->fp_exception = 0; #endif + if (parent && IS_TRACED(parent)) { + if (ERTS_TRACE_FLAGS(parent) & F_TRACE_SOS) { + ERTS_TRACE_FLAGS(p) |= (ERTS_TRACE_FLAGS(parent) & TRACEE_FLAGS); + erts_tracer_replace(&p->common, ERTS_TRACER(parent)); + } + if (ERTS_TRACE_FLAGS(parent) & F_TRACE_SOS1) { + /* Overrides TRACE_CHILDREN */ + ERTS_TRACE_FLAGS(p) |= (ERTS_TRACE_FLAGS(parent) & TRACEE_FLAGS); + erts_tracer_replace(&p->common, ERTS_TRACER(parent)); + ERTS_TRACE_FLAGS(p) &= ~(F_TRACE_SOS1 | F_TRACE_SOS); + ERTS_TRACE_FLAGS(parent) &= ~(F_TRACE_SOS1 | F_TRACE_SOS); + } + if (so->flags & SPO_LINK && ERTS_TRACE_FLAGS(parent) & (F_TRACE_SOL|F_TRACE_SOL1)) { + ERTS_TRACE_FLAGS(p) |= (ERTS_TRACE_FLAGS(parent)&TRACEE_FLAGS); + erts_tracer_replace(&p->common, ERTS_TRACER(parent)); + if (ERTS_TRACE_FLAGS(parent) & F_TRACE_SOL1) {/*maybe override*/ + ERTS_TRACE_FLAGS(p) &= ~(F_TRACE_SOL1 | F_TRACE_SOL); + ERTS_TRACE_FLAGS(parent) &= ~(F_TRACE_SOL1 | F_TRACE_SOL); + } + } + } + /* seq_trace is handled before regular tracing as the latter may touch the * trace token. */ if (!have_seqtrace(token)) { @@ -12039,39 +12062,17 @@ erl_create_process(Process* parent, /* Parent of process (default group leader). } } - if (parent && IS_TRACED(parent)) { - if (ERTS_TRACE_FLAGS(parent) & F_TRACE_SOS) { - ERTS_TRACE_FLAGS(p) |= (ERTS_TRACE_FLAGS(parent) & TRACEE_FLAGS); - erts_tracer_replace(&p->common, ERTS_TRACER(parent)); - } - if (ERTS_TRACE_FLAGS(parent) & F_TRACE_SOS1) { - /* Overrides TRACE_CHILDREN */ - ERTS_TRACE_FLAGS(p) |= (ERTS_TRACE_FLAGS(parent) & TRACEE_FLAGS); - erts_tracer_replace(&p->common, ERTS_TRACER(parent)); - ERTS_TRACE_FLAGS(p) &= ~(F_TRACE_SOS1 | F_TRACE_SOS); - ERTS_TRACE_FLAGS(parent) &= ~(F_TRACE_SOS1 | F_TRACE_SOS); - } - if (so->flags & SPO_LINK && ERTS_TRACE_FLAGS(parent) & (F_TRACE_SOL|F_TRACE_SOL1)) { - ERTS_TRACE_FLAGS(p) |= (ERTS_TRACE_FLAGS(parent)&TRACEE_FLAGS); - erts_tracer_replace(&p->common, ERTS_TRACER(parent)); - if (ERTS_TRACE_FLAGS(parent) & F_TRACE_SOL1) {/*maybe override*/ - ERTS_TRACE_FLAGS(p) &= ~(F_TRACE_SOL1 | F_TRACE_SOL); - ERTS_TRACE_FLAGS(parent) &= ~(F_TRACE_SOL1 | F_TRACE_SOL); - } - } - if (ARE_TRACE_FLAGS_ON(parent, F_TRACE_PROCS)) { - /* The locks may already be released if seq_trace is enabled as - * well. */ - if ((locks & (ERTS_PROC_LOCK_STATUS|ERTS_PROC_LOCK_TRACE)) - == (ERTS_PROC_LOCK_STATUS|ERTS_PROC_LOCK_TRACE)) { - locks &= ~(ERTS_PROC_LOCK_STATUS|ERTS_PROC_LOCK_TRACE); - erts_proc_unlock(p, ERTS_PROC_LOCK_STATUS|ERTS_PROC_LOCK_TRACE); - erts_proc_unlock(parent, ERTS_PROC_LOCK_STATUS|ERTS_PROC_LOCK_TRACE); - } - trace_proc_spawn(parent, am_spawn, p->common.id, mod, func, args); - if (so->flags & SPO_LINK) - trace_proc(parent, locks, parent, am_link, p->common.id); + if (parent && IS_TRACED_FL(parent, F_TRACE_PROCS)) { + /* The locks may already be released if seq_trace is enabled as well. */ + if ((locks & (ERTS_PROC_LOCK_STATUS|ERTS_PROC_LOCK_TRACE)) + == (ERTS_PROC_LOCK_STATUS|ERTS_PROC_LOCK_TRACE)) { + locks &= ~(ERTS_PROC_LOCK_STATUS|ERTS_PROC_LOCK_TRACE); + erts_proc_unlock(p, ERTS_PROC_LOCK_STATUS|ERTS_PROC_LOCK_TRACE); + erts_proc_unlock(parent, ERTS_PROC_LOCK_STATUS|ERTS_PROC_LOCK_TRACE); } + trace_proc_spawn(parent, am_spawn, p->common.id, mod, func, args); + if (so->flags & SPO_LINK) + trace_proc(parent, locks, parent, am_link, p->common.id); } if (IS_TRACED_FL(p, F_TRACE_PROCS)) { @@ -12097,18 +12098,17 @@ erl_create_process(Process* parent, /* Parent of process (default group leader). if (so->flags & SPO_LINK) { ErtsLink *lnk; - ErtsLinkData *ldp = erts_link_create(ERTS_LNK_TYPE_PROC, - parent->common.id, - p->common.id); - lnk = erts_link_tree_lookup_insert(&ERTS_P_LINKS(parent), &ldp->a); - if (lnk) { + lnk = erts_link_internal_create(ERTS_LNK_TYPE_PROC, p->common.id); + if (!!erts_link_tree_lookup_insert(&ERTS_P_LINKS(parent), lnk)) { /* * This should more or less never happen, but could * potentially happen if pid:s wrap... */ erts_link_release(lnk); } - erts_link_tree_insert(&ERTS_P_LINKS(p), &ldp->b); + lnk = erts_link_internal_create(ERTS_LNK_TYPE_PROC, + parent->common.id); + erts_link_tree_insert(&ERTS_P_LINKS(p), lnk); } /* @@ -12216,11 +12216,11 @@ erl_create_process(Process* parent, /* Parent of process (default group leader). if (so->flags & SPO_LINK) { ErtsLinkData *ldp; - ldp = erts_link_create(ERTS_LNK_TYPE_DIST_PROC, - parent_id, p->common.id); - erts_link_tree_insert(&ERTS_P_LINKS(p), &ldp->b); - if (!erts_link_dist_insert(&ldp->a, so->mld)) { - erts_proc_sig_send_link_exit(NULL, THE_NON_VALUE, &ldp->a, + ldp = erts_link_external_create(ERTS_LNK_TYPE_DIST_PROC, + p->common.id, parent_id); + erts_link_tree_insert(&ERTS_P_LINKS(p), &ldp->proc); + if (!erts_link_dist_insert(&ldp->dist, so->mld)) { + erts_proc_sig_send_link_exit(NULL, THE_NON_VALUE, &ldp->dist, am_noconnection, NIL); } } @@ -12400,6 +12400,7 @@ void erts_init_empty_process(Process *p) ERTS_P_MONITORS(p) = NULL; ERTS_P_LT_MONITORS(p) = NULL; ERTS_P_LINKS(p) = NULL; /* List of links */ + p->uniq = 1; p->sig_qs.first = NULL; p->sig_qs.last = &p->sig_qs.first; p->sig_qs.cont = NULL; @@ -13107,7 +13108,7 @@ erts_proc_exit_handle_dist_link(ErtsLink *lnk, void *vctxt, Sint reds) ErtsMonLnkDist *dist; DistEntry *dep; ErtsLink *dlnk; - ErtsLinkData *ldp = NULL; + ErtsELink *elnk = NULL; ErtsHeapFactory factory; Sint reds_consumed = 0; @@ -13116,8 +13117,8 @@ erts_proc_exit_handle_dist_link(ErtsLink *lnk, void *vctxt, Sint reds) ASSERT(ctxt->dist_state == NIL); ASSERT(!ctxt->yield); - dlnk = erts_link_to_other(lnk, &ldp); - dist = ((ErtsLinkDataExtended *) ldp)->dist; + dlnk = erts_link_to_other(lnk, &elnk); + dist = elnk->dist; ASSERT(is_external_pid(lnk->other.item)); dep = external_pid_dist_entry(lnk->other.item); @@ -13125,7 +13126,7 @@ erts_proc_exit_handle_dist_link(ErtsLink *lnk, void *vctxt, Sint reds) ASSERT(dep != erts_this_dist_entry); if (!erts_link_dist_delete(dlnk)) - ldp = NULL; + elnk = NULL; code = erts_dsig_prepare(&ctx, dep, c_p, ERTS_PROC_LOCK_MAIN, ERTS_DSP_NO_LOCK, 0, 0, 0); @@ -13175,8 +13176,8 @@ erts_proc_exit_handle_dist_link(ErtsLink *lnk, void *vctxt, Sint reds) ASSERT(! "Invalid dsig prep exit monitor result"); break; } - if (ldp) - erts_link_release_both(ldp); + if (elnk) + erts_link_release_both(&elnk->ld); else if (lnk) erts_link_release(lnk); return reds_consumed; @@ -13188,7 +13189,7 @@ erts_proc_exit_handle_link(ErtsLink *lnk, void *vctxt, Sint reds) ErtsProcExitContext *ctxt = (ErtsProcExitContext *) vctxt; Process *c_p = ((ErtsProcExitContext *) vctxt)->c_p; Eterm reason = ((ErtsProcExitContext *) vctxt)->reason; - ErtsLinkData *ldp = NULL; + ErtsELink *elnk = NULL; switch (lnk->type) { case ERTS_LNK_TYPE_PROC: @@ -13220,8 +13221,8 @@ erts_proc_exit_handle_link(ErtsLink *lnk, void *vctxt, Sint reds) int code; if (is_immed(reason)) { - dlnk = erts_link_to_other(lnk, &ldp); - dist = ((ErtsLinkDataExtended *) ldp)->dist; + dlnk = erts_link_to_other(lnk, &elnk); + dist = elnk->dist; ASSERT(is_external_pid(lnk->other.item)); dep = external_pid_dist_entry(lnk->other.item); @@ -13229,7 +13230,7 @@ erts_proc_exit_handle_link(ErtsLink *lnk, void *vctxt, Sint reds) ASSERT(dep != erts_this_dist_entry); if (!erts_link_dist_delete(dlnk)) - ldp = NULL; + elnk = NULL; code = erts_dsig_prepare(&ctx, dep, c_p, 0, ERTS_DSP_NO_LOCK, 1, 1, 0); switch (code) { @@ -13258,8 +13259,8 @@ erts_proc_exit_handle_link(ErtsLink *lnk, void *vctxt, Sint reds) break; } - if (ldp) - erts_link_release_both(ldp); + if (elnk) + erts_link_release_both(&elnk->ld); else if (lnk) erts_link_release(lnk); return 1; diff --git a/erts/emulator/beam/erl_process.h b/erts/emulator/beam/erl_process.h index d89a00714f..9e5ea73869 100644 --- a/erts/emulator/beam/erl_process.h +++ b/erts/emulator/beam/erl_process.h @@ -992,6 +992,7 @@ struct process { Process *next; /* Pointer to next process in run queue */ + Sint64 uniq; /* Used for process unique integer */ ErtsSignalPrivQueues sig_qs; /* Signal queues */ ErtsBifTimers *bif_timers; /* Bif timers aiming at this process */ diff --git a/erts/emulator/beam/erl_ptab.c b/erts/emulator/beam/erl_ptab.c index 38c095fb4a..94ea030f6f 100644 --- a/erts/emulator/beam/erl_ptab.c +++ b/erts/emulator/beam/erl_ptab.c @@ -27,10 +27,10 @@ #ifdef HAVE_CONFIG_H # include "config.h" #endif +#include "global.h" #define ERTS_PTAB_WANT_BIF_IMPL__ #define ERTS_PTAB_WANT_DEBUG_FUNCS__ #include "erl_ptab.h" -#include "global.h" #include "erl_binary.h" typedef struct ErtsPTabListBifData_ ErtsPTabListBifData; diff --git a/erts/emulator/beam/erl_sched_spec_pre_alloc.c b/erts/emulator/beam/erl_sched_spec_pre_alloc.c index 9766e76a83..d24bb727ce 100644 --- a/erts/emulator/beam/erl_sched_spec_pre_alloc.c +++ b/erts/emulator/beam/erl_sched_spec_pre_alloc.c @@ -32,6 +32,7 @@ # include "config.h" #endif +#if !defined(VALGRIND) && !defined(ADDRESS_SANITIZER) #include "erl_process.h" #include "erl_thr_progress.h" @@ -347,3 +348,4 @@ erts_sspa_process_remote_frees(erts_sspa_chunk_header_t *chdr, return res; } +#endif /* !defined(VALGRIND) && !defined(ADDRESS_SANITIZER) */ diff --git a/erts/emulator/beam/external.c b/erts/emulator/beam/external.c index 72edff0c22..1d1c473ab9 100644 --- a/erts/emulator/beam/external.c +++ b/erts/emulator/beam/external.c @@ -5712,7 +5712,66 @@ error: } #define ERTS_TRANSCODE_REDS_FACT 4 +typedef struct { + ErtsHeapFactory factory; + Eterm *hp; +} ErtsTranscodeDecodeState; + +static Eterm +transcode_decode_ctl_msg(ErtsTranscodeDecodeState *state, + SysIOVec *iov, + int end_ix) +{ + Eterm ctl_msg, *hp; + Uint buf_sz; + byte *buf_start, *buf_end; + byte *ptr; + Uint hsz; + + if (end_ix == 3) { + /* The whole control message is in iov[2].iov_base */ + buf_sz = (Uint) iov[2].iov_len; + buf_start = (byte *) iov[2].iov_base; + buf_end = buf_start + buf_sz; + } + else { + /* Control message over multiple buffers... */ + int ix; + buf_sz = 0; + for (ix = 2; ix < end_ix; ix++) + buf_sz += iov[ix].iov_len; + ptr = buf_start = erts_alloc(ERTS_ALC_T_TMP, buf_sz); + buf_end = buf_start + buf_sz; + for (ix = 2; ix < end_ix; ix++) { + sys_memcpy((void *) ptr, + (void *) iov[ix].iov_base, + iov[ix].iov_len); + ptr += iov[ix].iov_len; + } + } + + hsz = decoded_size(buf_start, buf_end, 0, NULL); + state->hp = hp = erts_alloc(ERTS_ALC_T_TMP, hsz*sizeof(Eterm)); + erts_factory_tmp_init(&state->factory, hp, hsz, ERTS_ALC_T_TMP); + + ptr = dec_term(NULL, &state->factory, buf_start, &ctl_msg, NULL, 0); + ASSERT(ptr); (void)ptr; + ASSERT(is_tuple(ctl_msg)); + + if (buf_start != (byte *) iov[2].iov_base) + erts_free(ERTS_ALC_T_TMP, buf_start); + + return ctl_msg; +} + +static void +transcode_decode_state_destroy(ErtsTranscodeDecodeState *state) +{ + erts_factory_close(&state->factory); + erts_free(ERTS_ALC_T_TMP, state->hp); +} +static Sint transcode_dist_obuf(ErtsDistOutputBuf* ob, DistEntry* dep, Uint64 dflags, @@ -5802,46 +5861,15 @@ Sint transcode_dist_obuf(ErtsDistOutputBuf* ob, * this packet to an empty (tick) packet, and inform * spawning process that this is not supported... */ - ErtsHeapFactory factory; - Eterm ctl_msg, ref, pid, token, *tp, *hp; - Uint buf_sz; - byte *buf_start, *buf_end; - byte *ptr; - Uint hsz; + ErtsTranscodeDecodeState tds; + Eterm ctl_msg, ref, pid, token, *tp; int i; hdr += 4; payload_ix = get_int32(hdr); ASSERT(payload_ix >= 3); - if (payload_ix == 3) { - /* The whole control message is in iov[2].iov_base */ - buf_sz = (Uint) iov[2].iov_len; - buf_start = (byte *) iov[2].iov_base; - buf_end = buf_start + buf_sz; - } - else { - /* Control message over multiple buffers... */ - int ix; - buf_sz = 0; - for (ix = 2; ix < payload_ix; ix++) - buf_sz += iov[ix].iov_len; - ptr = buf_start = erts_alloc(ERTS_ALC_T_TMP, buf_sz); - buf_end = buf_start + buf_sz; - for (ix = 2; ix < payload_ix; ix++) { - sys_memcpy((void *) ptr, - (void *) iov[ix].iov_base, - iov[ix].iov_len); - ptr += iov[ix].iov_len; - } - } - - hsz = decoded_size(buf_start, buf_end, 0, NULL); - hp = erts_alloc(ERTS_ALC_T_TMP, hsz*sizeof(Eterm)); - erts_factory_tmp_init(&factory, hp, hsz, ERTS_ALC_T_TMP); - - ptr = dec_term(NULL, &factory, buf_start, &ctl_msg, NULL, 0); - ASSERT(ptr); (void)ptr; + ctl_msg = transcode_decode_ctl_msg(&tds, iov, payload_ix); ASSERT(is_tuple_arity(ctl_msg, 6) || is_tuple_arity(ctl_msg, 8)); @@ -5865,10 +5893,7 @@ Sint transcode_dist_obuf(ErtsDistOutputBuf* ob, NULL, am_notsup, token); - erts_factory_close(&factory); - erts_free(ERTS_ALC_T_TMP, hp); - if (buf_start != (byte *) iov[2].iov_base) - erts_free(ERTS_ALC_T_TMP, buf_start); + transcode_decode_state_destroy(&tds); for (i = 1; i < ob->eiov->vsize; i++) { if (ob->eiov->binv[i]) @@ -5883,6 +5908,119 @@ Sint transcode_dist_obuf(ErtsDistOutputBuf* ob, return 0; return reds; } + + if ((~dflags & DFLAG_UNLINK_ID) + && ep[0] == SMALL_TUPLE_EXT + && ep[1] == 4 + && ep[2] == SMALL_INTEGER_EXT + && (ep[3] == DOP_UNLINK_ID_ACK || ep[3] == DOP_UNLINK_ID)) { + + if (ep[3] == DOP_UNLINK_ID_ACK) { + /* Drop DOP_UNLINK_ID_ACK signal... */ + int i; + for (i = 1; i < ob->eiov->vsize; i++) { + if (ob->eiov->binv[i]) + driver_free_binary(ob->eiov->binv[i]); + } + ob->eiov->vsize = 1; + ob->eiov->size = 0; + } + else { + Eterm ctl_msg, remote, local, *tp; + ErtsTranscodeDecodeState tds; + Uint64 id; + byte *ptr; + ASSERT(ep[3] == DOP_UNLINK_ID); + /* + * Rewrite the DOP_UNLINK_ID signal into a + * DOP_UNLINK signal and send an unlink ack + * to the local sender. + */ + + /* + * decode control message so we get info + * needed for unlink ack signal to send... + */ + ASSERT(get_int32(hdr + 4) == 0); /* No payload */ + ctl_msg = transcode_decode_ctl_msg(&tds, iov, eiov->vsize); + + ASSERT(is_tuple_arity(ctl_msg, 4)); + + tp = tuple_val(ctl_msg); + ASSERT(tp[1] == make_small(DOP_UNLINK_ID)); + + if (!term_to_Uint64(tp[2], &id)) + ERTS_INTERNAL_ERROR("Invalid encoding of DOP_UNLINK_ID signal"); + + local = tp[3]; + remote = tp[4]; + + ASSERT(is_internal_pid(local)); + ASSERT(is_external_pid(remote)); + + /* + * Rewrite buffer to an unlink signal by removing + * second element and change first element to + * DOP_UNLINK. That is, to: {DOP_UNLINK, local, remote} + */ + + ptr = &ep[4]; + switch (*ptr) { + case SMALL_INTEGER_EXT: + ptr += 1; + break; + case INTEGER_EXT: + ptr += 4; + break; + case SMALL_BIG_EXT: + ptr += 1; + ASSERT(*ptr <= 8); + ptr += *ptr + 1; + break; + default: + ERTS_INTERNAL_ERROR("Invalid encoding of DOP_UNLINK_ID signal"); + break; + } + + ASSERT((ptr - ep) <= 16); + ASSERT((ptr - ep) <= iov[2].iov_len); + + *(ptr--) = DOP_UNLINK; + *(ptr--) = SMALL_INTEGER_EXT; + *(ptr--) = 3; + *ptr = SMALL_TUPLE_EXT; + + iov[2].iov_base = ptr; + iov[2].iov_len -= (ptr - ep); + +#ifdef DEBUG + { + ErtsTranscodeDecodeState dbg_tds; + Eterm new_ctl_msg = transcode_decode_ctl_msg(&dbg_tds, + iov, + eiov->vsize); + ASSERT(is_tuple_arity(new_ctl_msg, 3)); + tp = tuple_val(new_ctl_msg); + ASSERT(tp[1] == make_small(DOP_UNLINK)); + ASSERT(tp[2] == local); + ASSERT(eq(tp[3], remote)); + transcode_decode_state_destroy(&dbg_tds); + } +#endif + + /* Send unlink ack to local sender... */ + erts_proc_sig_send_dist_unlink_ack(NULL, dep, + dep->connection_id, + remote, local, id); + + transcode_decode_state_destroy(&tds); + + reds -= 5; + } + if (reds < 0) + return 0; + return reds; + } start_r = r = reds*ERTS_TRANSCODE_REDS_FACT; diff --git a/erts/emulator/beam/io.c b/erts/emulator/beam/io.c index d7fa2f2696..d2c6dffa53 100644 --- a/erts/emulator/beam/io.c +++ b/erts/emulator/beam/io.c @@ -768,7 +768,7 @@ driver_create_port(ErlDrvPort creator_port_ix, /* Creating port */ Port* port; erts_driver_t *driver; erts_mtx_t *driver_lock = NULL; - ErtsLinkData *ldp; + ErtsLink *port_lnk, *proc_lnk; ERTS_CHK_NO_PROC_LOCKS; @@ -814,15 +814,13 @@ driver_create_port(ErlDrvPort creator_port_ix, /* Creating port */ } ERTS_LC_ASSERT(erts_lc_is_port_locked(port)); - ldp = erts_link_create(ERTS_LNK_TYPE_PORT, - port->common.id, pid); - ASSERT(ldp->a.other.item == pid); - ASSERT(ldp->b.other.item == port->common.id); - erts_link_tree_insert(&ERTS_P_LINKS(port), &ldp->a); - - if (!erts_proc_sig_send_link(NULL, pid, &ldp->b)) { - erts_link_tree_delete(&ERTS_P_LINKS(port), &ldp->a); - erts_link_release_both(ldp); + port_lnk = erts_link_internal_create(ERTS_LNK_TYPE_PORT, pid); + erts_link_tree_insert(&ERTS_P_LINKS(port), port_lnk); + proc_lnk = erts_link_internal_create(ERTS_LNK_TYPE_PORT, port->common.id); + if (!erts_proc_sig_send_link(NULL, pid, proc_lnk)) { + erts_link_tree_delete(&ERTS_P_LINKS(port), port_lnk); + erts_link_internal_release(proc_lnk); + erts_link_internal_release(port_lnk); if (driver->handle) { erts_rwmtx_rlock(&erts_driver_list_lock); erts_ddll_decrement_port_count(driver->handle); @@ -1229,14 +1227,9 @@ erts_schedule_port2port_signal(Eterm port_num, ErtsProc2PortSigData *sigdp, int task_flags, ErtsProc2PortSigCallback callback) { - Port *prt = erts_port_lookup_raw(port_num); - - if (!prt) - return -1; - sigdp->caller = ERTS_INVALID_PID; - return erts_port_task_schedule(prt->common.id, + return erts_port_task_schedule(port_num, NULL, ERTS_PORT_TASK_PROC_SIG, sigdp, @@ -2363,16 +2356,17 @@ set_port_connected(int bang_op, if (is_not_internal_pid(connect)) return ERTS_PORT_OP_DROPPED; - lnk = erts_link_tree_lookup_create(&ERTS_P_LINKS(prt), &created, - ERTS_LNK_TYPE_PORT, prt->common.id, - connect); + lnk = erts_link_internal_tree_lookup_create(&ERTS_P_LINKS(prt), + &created, + ERTS_LNK_TYPE_PORT, + connect); if (created) { - ErtsLinkData *ldp; - ErtsLink *olnk = erts_link_to_other(lnk, &ldp); - ASSERT(olnk->other.item == prt->common.id); + ErtsLink *olnk = erts_link_internal_create(ERTS_LNK_TYPE_PORT, + prt->common.id); if (!erts_proc_sig_send_link(NULL, connect, olnk)) { erts_link_tree_delete(&ERTS_P_LINKS(prt), lnk); - erts_link_release_both(ldp); + erts_link_internal_release(lnk); + erts_link_internal_release(olnk); return ERTS_PORT_OP_DROPPED; } if (IS_TRACED_FL(prt, F_TRACE_PORTS)) @@ -2490,24 +2484,27 @@ erts_port_connect(Process *c_p, } static void -port_unlink(Port *prt, ErtsLink *lnk) +port_unlink_failure(Eterm port_id, ErtsSigUnlinkOp *sulnk) { - ErtsLinkData *ldp; - ErtsLink *dlnk, *llnk; + erts_proc_sig_send_unlink_ack(NULL, port_id, sulnk); +} - llnk = erts_link_to_other(lnk, &ldp); - dlnk = erts_link_tree_key_delete(&ERTS_P_LINKS(prt), llnk); - if (!dlnk) - erts_link_release(lnk); +static void +port_unlink(Port *prt, erts_aint32_t state, ErtsSigUnlinkOp *sulnk) +{ + if (state & ERTS_PORT_SFLGS_INVALID_LOOKUP) + port_unlink_failure(prt->common.id, sulnk); else { - if (IS_TRACED_FL(prt, F_TRACE_PORTS)) - trace_port(prt, am_getting_unlinked, llnk->other.item); - if (dlnk == llnk) - erts_link_release_both(ldp); - else { - erts_link_release(lnk); - erts_link_release(dlnk); + ErtsILink *ilnk; + ilnk = (ErtsILink *) erts_link_tree_lookup(ERTS_P_LINKS(prt), + sulnk->from); + if (ilnk && !ilnk->unlinking) { + if (IS_TRACED_FL(prt, F_TRACE_PORTS)) + trace_port(prt, am_getting_unlinked, sulnk->from); + erts_link_tree_delete(&ERTS_P_LINKS(prt), &ilnk->link); + erts_link_internal_release(&ilnk->link); } + erts_proc_sig_send_unlink_ack(NULL, prt->common.id, sulnk); } } @@ -2515,14 +2512,16 @@ static int port_sig_unlink(Port *prt, erts_aint32_t state, int op, ErtsProc2PortSigData *sigdp) { if (op == ERTS_PROC2PORT_SIG_EXEC) - port_unlink(prt, sigdp->u.unlink.lnk); + port_unlink(prt, state, sigdp->u.unlink.sulnk); + else + port_unlink_failure(sigdp->u.unlink.port_id, sigdp->u.unlink.sulnk); if (sigdp->flags & ERTS_P2P_SIG_DATA_FLG_REPLY) port_sched_op_reply(sigdp->caller, sigdp->ref, am_true, prt); return ERTS_PORT_REDS_UNLINK; } ErtsPortOpResult -erts_port_unlink(Process *c_p, Port *prt, ErtsLink *lnk, Eterm *refp) +erts_port_unlink(Process *c_p, Port *prt, ErtsSigUnlinkOp *sulnk, Eterm *refp) { ErtsProc2PortSigData *sigdp; ErtsTryImmDrvCallState try_call_state @@ -2533,13 +2532,16 @@ erts_port_unlink(Process *c_p, Port *prt, ErtsLink *lnk, Eterm *refp) !refp, am_unlink); + ASSERT(is_internal_pid(sulnk->from)); + switch (try_imm_drv_call(&try_call_state)) { case ERTS_TRY_IMM_DRV_CALL_OK: - port_unlink(prt, lnk); + port_unlink(prt, try_call_state.state, sulnk); finalize_imm_drv_call(&try_call_state); BUMP_REDS(c_p, ERTS_PORT_REDS_UNLINK); return ERTS_PORT_OP_DONE; case ERTS_TRY_IMM_DRV_CALL_INVALID_PORT: + port_unlink_failure(prt->common.id, sulnk); return ERTS_PORT_OP_DROPPED; default: /* Schedule call instead... */ @@ -2549,7 +2551,7 @@ erts_port_unlink(Process *c_p, Port *prt, ErtsLink *lnk, Eterm *refp) sigdp = erts_port_task_alloc_p2p_sig_data(); sigdp->flags = ERTS_P2P_SIG_TYPE_UNLINK; sigdp->u.unlink.port_id = prt->common.id; - sigdp->u.unlink.lnk = lnk; + sigdp->u.unlink.sulnk = sulnk; return erts_schedule_proc2port_signal(c_p, prt, @@ -2562,24 +2564,100 @@ erts_port_unlink(Process *c_p, Port *prt, ErtsLink *lnk, Eterm *refp) } static void +port_unlink_ack_failure(Eterm port_id, ErtsSigUnlinkOp *sulnk) +{ + erts_proc_sig_destroy_unlink_op(sulnk); +} + +static void +port_unlink_ack(Port *prt, erts_aint32_t state, ErtsSigUnlinkOp *sulnk) +{ + if (state & ERTS_PORT_SFLGS_INVALID_LOOKUP) + port_unlink_ack_failure(prt->common.id, sulnk); + else { + ErtsILink *ilnk; + ilnk = (ErtsILink *) erts_link_tree_lookup(ERTS_P_LINKS(prt), + sulnk->from); + if (ilnk && ilnk->unlinking == sulnk->id) { + erts_link_tree_delete(&ERTS_P_LINKS(prt), &ilnk->link); + erts_link_internal_release(&ilnk->link); + } + erts_proc_sig_destroy_unlink_op(sulnk); + } +} + +static int +port_sig_unlink_ack(Port *prt, erts_aint32_t state, int op, ErtsProc2PortSigData *sigdp) +{ + if (op == ERTS_PROC2PORT_SIG_EXEC) + port_unlink_ack(prt, state, sigdp->u.unlink_ack.sulnk); + else + port_unlink_ack_failure(sigdp->u.unlink_ack.port_id, sigdp->u.unlink_ack.sulnk); + return ERTS_PORT_REDS_UNLINK_ACK; +} + +ErtsPortOpResult +erts_port_unlink_ack(Process *c_p, Port *prt, ErtsSigUnlinkOp *sulnk) +{ + ErtsProc2PortSigData *sigdp; + ErtsTryImmDrvCallState try_call_state + = ERTS_INIT_TRY_IMM_DRV_CALL_STATE(c_p, + prt, + ERTS_PORT_SFLGS_DEAD, + 0, + !0, + am_unlink); + + ASSERT(c_p); + sulnk->from = c_p->common.id; + + switch (try_imm_drv_call(&try_call_state)) { + case ERTS_TRY_IMM_DRV_CALL_OK: + port_unlink_ack(prt, try_call_state.state, sulnk); + finalize_imm_drv_call(&try_call_state); + BUMP_REDS(c_p, ERTS_PORT_REDS_UNLINK_ACK); + return ERTS_PORT_OP_DONE; + case ERTS_TRY_IMM_DRV_CALL_INVALID_PORT: + port_unlink_ack_failure(prt->common.id, sulnk); + return ERTS_PORT_OP_DROPPED; + default: + /* Schedule call instead... */ + break; + } + + sigdp = erts_port_task_alloc_p2p_sig_data(); + sigdp->flags = ERTS_P2P_SIG_TYPE_UNLINK_ACK; + sigdp->u.unlink_ack.port_id = prt->common.id; + sigdp->u.unlink_ack.sulnk = sulnk; + + return erts_schedule_proc2port_signal(c_p, + prt, + c_p->common.id, + NULL, + sigdp, + 0, + NULL, + port_sig_unlink_ack); +} + +static void port_link_failure(Eterm port_id, ErtsLink *lnk) { erts_proc_sig_send_link_exit(NULL, port_id, lnk, am_noproc, NIL); } static void -port_link(Port *prt, erts_aint32_t state, ErtsLink *lnk) +port_link(Port *prt, erts_aint32_t state, ErtsLink *nlnk) { if (state & ERTS_PORT_SFLGS_INVALID_LOOKUP) - port_link_failure(prt->common.id, lnk); + port_link_failure(prt->common.id, nlnk); else { - ErtsLink *rlnk; - rlnk = erts_link_tree_insert_addr_replace(&ERTS_P_LINKS(prt), - lnk); - if (rlnk) - erts_link_release(rlnk); + ErtsLink *lnk; + lnk = erts_link_tree_lookup_insert(&ERTS_P_LINKS(prt), nlnk); + if (lnk) + erts_link_release(nlnk); else if (IS_TRACED_FL(prt, F_TRACE_PORTS)) - trace_port(prt, am_getting_linked, lnk->other.item); + trace_port(prt, am_getting_linked, nlnk->other.item); } } @@ -2780,19 +2858,37 @@ erts_port_demonitor(Process *c_p, Port *prt, ErtsMonitor *mon) port_sig_demonitor); } + +/* Unlink (an internal process) from a port */ static void -init_ack_send_reply(Port *port, Eterm resp) +unlink_proc(Port *prt, Eterm pid) { + ErtsILink *ilnk; - if (!is_internal_port(resp)) { - Eterm proc = port->async_open_port->to; - ErtsLink *lnk = erts_link_tree_lookup(ERTS_P_LINKS(port), - proc); - if (lnk) { - erts_link_tree_delete(&ERTS_P_LINKS(port), lnk); - erts_proc_sig_send_unlink(NULL, lnk); + ASSERT(prt); + ASSERT(is_internal_pid(pid)); + + ilnk = (ErtsILink *) erts_link_tree_lookup(ERTS_P_LINKS(prt), + pid); + if (ilnk && !ilnk->unlinking) { + Uint64 id = erts_proc_sig_send_unlink(NULL, + prt->common.id, + &ilnk->link); + if (id != 0) + ilnk->unlinking = id; + else { + erts_link_tree_delete(&ERTS_P_LINKS(prt), &ilnk->link); + erts_link_internal_release(&ilnk->link); } } +} + +static void +init_ack_send_reply(Port *port, Eterm resp) +{ + + if (!is_internal_port(resp)) + unlink_proc(port, port->async_open_port->to); port_sched_op_reply(port->async_open_port->to, port->async_open_port->ref, resp, @@ -7079,7 +7175,6 @@ driver_failure_term(ErlDrvPort ix, Eterm term, int eof) int driver_exit(ErlDrvPort ix, int err) { Port* prt = erts_drvport2port(ix); - ErtsLink *lnk; Eterm connected; ERTS_CHK_NO_PROC_LOCKS; @@ -7088,11 +7183,7 @@ int driver_exit(ErlDrvPort ix, int err) return -1; connected = ERTS_PORT_GET_CONNECTED(prt); - lnk = erts_link_tree_lookup(ERTS_P_LINKS(prt), connected); - if (lnk) { - erts_link_tree_delete(&ERTS_P_LINKS(prt), lnk); - erts_proc_sig_send_unlink(NULL, lnk); - } + unlink_proc(prt, connected); if (err == 0) return driver_failure_term(ix, am_normal, 0); diff --git a/erts/emulator/beam/msg_instrs.tab b/erts/emulator/beam/msg_instrs.tab index 48ab87eed7..3d350e65cb 100644 --- a/erts/emulator/beam/msg_instrs.tab +++ b/erts/emulator/beam/msg_instrs.tab @@ -106,8 +106,10 @@ i_loop_rec(Dest) { c_p->arity = 0; c_p->current = NULL; c_p->fcalls = FCALLS; + ERTS_UNREQ_PROC_MAIN_LOCK(c_p); FCALLS -= erts_proc_sig_receive_helper(c_p, FCALLS, neg_o_reds, &msgp, &get_out); + ERTS_REQ_PROC_MAIN_LOCK(c_p); SWAPIN; if (ERTS_UNLIKELY(msgp == NULL)) { if (get_out) { diff --git a/erts/emulator/beam/time.c b/erts/emulator/beam/time.c index d054a81cd7..05f932ad84 100644 --- a/erts/emulator/beam/time.c +++ b/erts/emulator/beam/time.c @@ -871,6 +871,8 @@ erts_bump_timers(ErtsTimerWheel *tiw, ErtsMonotonicTime curr_time) } if (tiw->pos >= bump_to) { + if (tiw->at_once.nto) + continue; ERTS_MSACC_POP_STATE_M_X(); break; } diff --git a/erts/emulator/drivers/common/inet_drv.c b/erts/emulator/drivers/common/inet_drv.c index 00708f1478..49db7077de 100644 --- a/erts/emulator/drivers/common/inet_drv.c +++ b/erts/emulator/drivers/common/inet_drv.c @@ -82,6 +82,32 @@ /* All platforms fail on malloc errors. */ #define FATAL_MALLOC +/* The linux kernel sctp include files have an alignment bug + that causes warnings of this type to appear: + + drivers/common/inet_drv.c:3196:47: error: taking address of packed member of 'struct sctp_paddr_change' may result in an unaligned pointer value [-Werror=address-of-packed-member] + 3196 | i = load_inet_get_address(spec, i, desc, &sptr->spc_aaddr); + + So we need to suppress those, without disabling all warning + diagnostics of that type. + + See https://lore.kernel.org/patchwork/patch/1108122/ for the + patch that fixes this bug. In a few years we should be able to + remove this suppression. */ +#ifdef HAVE_GCC_DIAG_IGNORE_WADDRESS_OF_PACKED_MEMBER +#define PUSH_SUPPRESS_ADDRESS_OF_PACKED_MEMBER() \ + _Pragma("GCC diagnostic push") \ + _Pragma("GCC diagnostic ignored \"-Waddress-of-packed-member\"") \ + do { } while(0) +#define POP_SUPPRESS_ADDRESS_OF_PACKED_MEMBER() \ + _Pragma("GCC diagnostic pop") \ + do { } while(0) +#else +#define PUSH_SUPPRESS_ADDRESS_OF_PACKED_MEMBER() \ + do { } while(0) +#define POP_SUPPRESS_ADDRESS_OF_PACKED_MEMBER() \ + do { } while(0) +#endif #include "erl_driver.h" @@ -601,15 +627,6 @@ static int my_strncasecmp(const char *s1, const char *s2, size_t n) #include "packet_parser.h" -#define get_int24(s) ((((unsigned char*) (s))[0] << 16) | \ - (((unsigned char*) (s))[1] << 8) | \ - (((unsigned char*) (s))[2])) - -#define get_little_int32(s) ((((unsigned char*) (s))[3] << 24) | \ - (((unsigned char*) (s))[2] << 16) | \ - (((unsigned char*) (s))[1] << 8) | \ - (((unsigned char*) (s))[0])) - #if defined(HAVE_SYS_UN_H) || defined(SO_BINDTODEVICE) /* strnlen doesn't exist everywhere */ @@ -3193,7 +3210,9 @@ static int sctp_parse_async_event ASSERT(sptr->spc_length <= sz); /* No buffer overrun */ i = LOAD_ATOM (spec, i, am_sctp_paddr_change); + PUSH_SUPPRESS_ADDRESS_OF_PACKED_MEMBER(); i = load_inet_get_address(spec, i, desc, &sptr->spc_aaddr); + POP_SUPPRESS_ADDRESS_OF_PACKED_MEMBER(); switch (sptr->spc_state) { @@ -8150,7 +8169,9 @@ static int load_paddrinfo (ErlDrvTermData * spec, int i, { i = LOAD_ATOM (spec, i, am_sctp_paddrinfo); i = LOAD_ASSOC_ID (spec, i, pai->spinfo_assoc_id); + PUSH_SUPPRESS_ADDRESS_OF_PACKED_MEMBER(); i = load_inet_get_address(spec, i, desc, &pai->spinfo_address); + POP_SUPPRESS_ADDRESS_OF_PACKED_MEMBER(); switch(pai->spinfo_state) { case SCTP_ACTIVE: @@ -8670,7 +8691,9 @@ static ErlDrvSSizeT sctp_fill_opts(inet_descriptor* desc, ASSERT(0); } i = LOAD_ASSOC_ID (spec, i, sp.sspp_assoc_id); + PUSH_SUPPRESS_ADDRESS_OF_PACKED_MEMBER(); i = load_inet_get_address(spec, i, desc, &sp.sspp_addr); + POP_SUPPRESS_ADDRESS_OF_PACKED_MEMBER(); i = LOAD_TUPLE (spec, i, 3); i = LOAD_TUPLE (spec, i, 2); break; @@ -8730,7 +8753,9 @@ static ErlDrvSSizeT sctp_fill_opts(inet_descriptor* desc, i = LOAD_ATOM (spec, i, am_sctp_peer_addr_params); i = LOAD_ATOM (spec, i, am_sctp_paddrparams); i = LOAD_ASSOC_ID (spec, i, ap.spp_assoc_id); + PUSH_SUPPRESS_ADDRESS_OF_PACKED_MEMBER(); i = load_inet_get_address(spec, i, desc, &ap.spp_address); + POP_SUPPRESS_ADDRESS_OF_PACKED_MEMBER(); i = LOAD_INT (spec, i, ap.spp_hbinterval); i = LOAD_INT (spec, i, ap.spp_pathmaxrxt); @@ -10045,6 +10070,15 @@ static void tcp_inet_stop(ErlDrvData e) tcp_close_check(desc); tcp_clear_input(desc); +#ifdef HAVE_SENDFILE + if(desc->tcp_add_flags & TCP_ADDF_SENDFILE) { + desc->tcp_add_flags &= ~TCP_ADDF_SENDFILE; + close(desc->sendfile.dup_file_fd); + DEBUGF(("tcp_inet_stop(%p): SENDFILE dup closed %d\r\n", + desc->inet.port, desc->sendfile.dup_file_fd)); + } +#endif + DEBUGF(("tcp_inet_stop(%p) }\r\n", desc->inet.port)); inet_stop(INETP(desc)); } @@ -10060,12 +10094,6 @@ static void tcp_inet_stop(ErlDrvData e) * will be freed through tcp_inet_stop later on. */ static void tcp_desc_close(tcp_descriptor* desc) { -#ifdef HAVE_SENDFILE - if(desc->tcp_add_flags & TCP_ADDF_SENDFILE) { - desc->tcp_add_flags &= ~TCP_ADDF_SENDFILE; - close(desc->sendfile.dup_file_fd); - } -#endif tcp_clear_input(desc); tcp_clear_output(desc); @@ -10440,6 +10468,13 @@ static ErlDrvSSizeT tcp_inet_ctl(ErlDrvData e, unsigned int cmd, return ctl_error(EINVAL, rbuf, rsize); } else if (!IS_CONNECTED(INETP(desc))) { return ctl_error(ENOTCONN, rbuf, rsize); + } else if (desc->tcp_add_flags & TCP_ADDF_SENDFILE) { + /* This should not happen as prim_inet.erl makes + sure that only the controlling process can + use the sendfile operation. But we add this + check here anyways just in case that prim_inet + is changed... */ + return ctl_error(EINVAL, rbuf, rsize); } sys_memcpy(&raw_file_fd, buf, sizeof(raw_file_fd)); @@ -10447,6 +10482,9 @@ static ErlDrvSSizeT tcp_inet_ctl(ErlDrvData e, unsigned int cmd, desc->sendfile.dup_file_fd = dup(raw_file_fd); + DEBUGF(("tcp_inet_ctl(%p): SENDFILE dup %d\r\n", + desc->inet.port, desc->sendfile.dup_file_fd)); + if(desc->sendfile.dup_file_fd == -1) { return ctl_error(errno, rbuf, rsize); } @@ -10499,6 +10537,11 @@ static void tcp_inet_send_timeout(ErlDrvData e, ErlDrvTermData dummy) if (desc->send_timeout_close) { tcp_desc_close(desc); } + /* Q: Why not keep port busy as send queue may still be full (ERL-1390)? + * + * A: If kept busy, a following send call would hang without a timeout + * as it would get suspended in erlang:port_command waiting on busy port. + */ } /* @@ -10625,11 +10668,9 @@ static void tcp_inet_flush(ErlDrvData e) #ifdef HAVE_SENDFILE /* The old file driver aborted when it was stopped during sendfile, so - * we'll clear the flag and discard all output. */ + * we'll clear the flag and discard all output. It is the job of + * tcp_inet_stop to close the extra sendfile fd. */ if(desc->tcp_add_flags & TCP_ADDF_SENDFILE) { - desc->tcp_add_flags &= ~TCP_ADDF_SENDFILE; - close(desc->sendfile.dup_file_fd); - discard_output = 1; } #endif @@ -11790,6 +11831,9 @@ static int tcp_sendfile_completed(tcp_descriptor* desc) { desc->tcp_add_flags &= ~TCP_ADDF_SENDFILE; close(desc->sendfile.dup_file_fd); + DEBUGF(("tcp_sendfile_completed(%p): SENDFILE dup closed %d\r\n", + desc->inet.port, desc->sendfile.dup_file_fd)); + /* While we flushed the output queue prior to sending the file, we've * deferred clearing busy status until now as there's no point in doing so * while we still have a file to send. diff --git a/erts/emulator/internal_doc/NewLinking.tla b/erts/emulator/internal_doc/NewLinking.tla new file mode 100644 index 0000000000..832bbaf037 --- /dev/null +++ b/erts/emulator/internal_doc/NewLinking.tla @@ -0,0 +1,194 @@ +\* +\* %CopyrightBegin% +\* +\* Copyright Ericsson AB 2021. All Rights Reserved. +\* +\* Licensed under the Apache License, Version 2.0 (the "License"); +\* you may not use this file except in compliance with the License. +\* You may obtain a copy of the License at +\* +\* http://www.apache.org/licenses/LICENSE-2.0 +\* +\* Unless required by applicable law or agreed to in writing, software +\* distributed under the License is distributed on an "AS IS" BASIS, +\* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +\* See the License for the specific language governing permissions and +\* limitations under the License. +\* +\* %CopyrightEnd% +\* + +(* + --- Model of the new link protocol introduced in Erlang/OTP 23.3 --- + + The protocol is documented in the ERTS User's Guide -> + Distribution Protocol -> Protocol Between Connected Nodes -> + Link Protocol -> New Link Protocol + + This model only models a link between two processes. This since a link + between one pair of processes is completely independent of links between + other pairs of processes. This model also assumes that the connection + between the processes does not fail. In the real world a connection can + of course fail. This is however taken care of by clearing the link + information on both ends when a connection fails, and tracking which + instantiation of a connection between the nodes signals arrive on and + ignoring signals from old instantiations of connections. That is, + connection loss is trivially taken care of since we just start over + again from scratch if the connection is lost. + + The documentation of the protocol talks about "process local information + about links". This information is stored in the process state record below. + The 'other' field contains the identifier of the other process. The 'type' + field acts as "active flag". The link is active when 'type' equals + "linked" and not active when 'type' equals "unlinked". If the + 'wait_unlink_ack' field contains a value larger than or equal to zero + it is the "unlink id" of an unlink request we have issued and are waiting + for to get acknowledged. If the 'wait_unlink_ack' field contains -1 we + are not waiting for an acknowledgment. When 'type' equals "unlinked" and + 'wait_unlink_ack' equals -1, we would in the documented protocol have + removed the "process local information about the link". In this model we, + however, keep the state, but in this state instead of removing it. + Messages are tagged with a message number in order to model the signal + order of Erlang. The message number of the unlink signal is also used as + "unlink id". + + The model has been checked with the following parameters: + + Declared constants: + PROCS <- {"a", "b"} + + Temporal formula: + FairSpec + + Deadlock: + disabled + + Invariants: + TypeOK + ValidState + + State Constraint: + /\ \E proc \in PROCS : procState[proc].next_send =< 15 + /\ Cardinality(msgs) =< 10 + + That is, we have checked all states where processes send up to 15 signals + with at most 10 outstanding signals. + + Deadlock checking has been disabled since we intentionally stop when + we have no outstanding signals (in 'Next') in order to avoid checking + signal sequences equivalent to sequences we already have checked. + +*) +------------------------------ MODULE NewLinking ------------------------------ +EXTENDS Integers, TLC, FiniteSets +CONSTANTS PROCS \* PROCS should be a set of exactly two process names + +VARIABLES + procState, \* Set of process states; procState[proc] is state of proc + msgs \* Set of messages sent + +vars == <<procState, msgs>> + +\* Set of possible process states... +procStateRec == + [self : PROCS, + other: PROCS, + type : {"linked", "unlinked"}, + wait_unlink_ack : Nat \cup {-1}, + next_send : Nat, + next_recv : Nat] + +\* Set of possible messages... +Messages == + [type : {"link", "unlink", "unlink_ack"}, from : PROCS, to : PROCS, msg_no : Nat, ack : Nat \cup {-1}] + +TypeOK == + /\ procState \in [PROCS -> procStateRec] + /\ msgs \subseteq Messages + +Init == + /\ msgs = {} + /\ procState = [p \in PROCS |-> [self |-> p, + other |-> CHOOSE p2 \in PROCS : p2 /= p, + type |-> "unlinked", + next_send |-> 0, + next_recv |-> 0, + wait_unlink_ack |-> -1]] + +MkMsg(self, mtype, accnr) == + [type |-> mtype, + from |-> procState[self].self, + to |-> procState[self].other, + msg_no |-> procState[self].next_send, + ack |-> accnr] + +Link(self) == + /\ procState[self].type = "unlinked" + /\ msgs' = msgs \cup {MkMsg(self, "link", -1)} + /\ procState' = [procState EXCEPT ![self].type = "linked", + ![self].next_send = @ + 1, + ![self].wait_unlink_ack = -1] + +Unlink(self) == + /\ procState[self].type = "linked" + /\ msgs' = msgs \cup {MkMsg(self, "unlink", -1)} + /\ procState' = [procState EXCEPT ![self].type = "unlinked", + ![self].next_send = @ + 1, + ![self].wait_unlink_ack = procState[self].next_send] + +RecvLink(self, msg) == + LET type == IF procState[self].wait_unlink_ack /= -1 + THEN "unlinked" + ELSE "linked" + IN /\ msgs' = msgs \ {msg} + /\ procState' = [procState EXCEPT ![self].type = type, + ![self].next_recv = @ + 1] + +RecvUnlink(self, msg) == + /\ msgs' = (msgs \ {msg}) \cup {MkMsg(self, + "unlink_ack", + procState[self].next_recv)} + /\ procState' = [procState EXCEPT ![self].type = "unlinked", + ![self].next_recv = @ + 1, + ![self].next_send = @ + 1] + +RecvUnlinkAck(self, msg) == + LET wack == IF procState[self].wait_unlink_ack = msg.ack + THEN -1 + ELSE procState[self].wait_unlink_ack + IN /\ msgs' = msgs \ {msg} + /\ procState' = [procState EXCEPT ![self].next_recv = @ + 1, + ![self].wait_unlink_ack = wack] + +Recv(self) == + /\ \E m \in msgs : /\ m.to = self + /\ m.msg_no = procState[self].next_recv + /\ LET msg == CHOOSE m \in msgs : /\ m.to = self + /\ m.msg_no = procState[self].next_recv + IN CASE msg.type = "link" -> RecvLink(self, msg) + [] msg.type = "unlink" -> RecvUnlink(self, msg) + [] msg.type = "unlink_ack" -> RecvUnlinkAck(self, msg) + +(* + If we have no outstanding messages; both processes should + have the same view about whether they are linked or not... +*) +ValidState == + IF msgs /= {} + THEN TRUE + ELSE \A p \in PROCS : \A p2 \in PROCS : procState[p].type = procState[p2].type + +Next == + /\ (msgs /= {} \/ \A p \in PROCS : procState[p].next_send = 0) + /\ \E p \in PROCS : \/ Recv(p) + \/ Link(p) + \/ Unlink(p) + +Spec == Init /\ [][Next]_vars + +FairSpec == Spec /\ WF_vars(Next) + +============================================================================= +\* Modification History +\* Last modified Mon Jan 25 11:26:06 CET 2021 by rickard.green +\* Created Wed Jan 20 13:11:46 CET 2021 by rickard.green diff --git a/erts/emulator/nifs/common/zlib_nif.c b/erts/emulator/nifs/common/zlib_nif.c index b709ed5a6f..2710c586c6 100644 --- a/erts/emulator/nifs/common/zlib_nif.c +++ b/erts/emulator/nifs/common/zlib_nif.c @@ -370,6 +370,7 @@ static int zlib_flush_queue(int (*codec)(z_stream*, int), ErlNifEnv *env, d->s.next_in = input_vec[vec_idx].iov_base; d->s.avail_in = block_size; + /* We don't flush until reaching the end of our input. */ res = codec(&d->s, Z_NO_FLUSH); ASSERT(d->s.avail_in == 0 || d->s.avail_out == 0 || res != Z_OK); @@ -395,7 +396,12 @@ static int zlib_flush_queue(int (*codec)(z_stream*, int), ErlNifEnv *env, res = Z_BUF_ERROR; } - if(res == Z_OK && flush != Z_NO_FLUSH && (*bytes_remaining == 0)) { + if(res == Z_OK && (*bytes_remaining == 0) && d->s.avail_out > 0) { + /* We've reached the end of our input and need to flush the zlib state. + * + * Note that we do this even when the flush parameter is Z_NO_FLUSH as + * we may have filled our output buffer on the previous call. It will + * nop when there's nothing left to flush. */ d->s.next_in = NULL; d->s.avail_in = 0; diff --git a/erts/emulator/sys/common/erl_mmap.c b/erts/emulator/sys/common/erl_mmap.c index b0d9fc0776..78c20ea98d 100644 --- a/erts/emulator/sys/common/erl_mmap.c +++ b/erts/emulator/sys/common/erl_mmap.c @@ -2130,13 +2130,18 @@ void erts_mmap_init(ErtsMemMapper* mm, ErtsMMapInit *init) { static int is_first_call = 1; - int virtual_map = 0; char *start = NULL, *end = NULL; UWord pagesize; + int virtual_map = 0; + + (void)virtual_map; + #if defined(__WIN32__) - SYSTEM_INFO sysinfo; - GetSystemInfo(&sysinfo); - pagesize = (UWord) sysinfo.dwPageSize; + { + SYSTEM_INFO sysinfo; + GetSystemInfo(&sysinfo); + pagesize = (UWord) sysinfo.dwPageSize; + } #elif defined(_SC_PAGESIZE) pagesize = (UWord) sysconf(_SC_PAGESIZE); #elif defined(HAVE_GETPAGESIZE) diff --git a/erts/emulator/sys/common/erl_mmap.h b/erts/emulator/sys/common/erl_mmap.h index 1a6de44dfd..7a3fdd0aa9 100644 --- a/erts/emulator/sys/common/erl_mmap.h +++ b/erts/emulator/sys/common/erl_mmap.h @@ -49,7 +49,8 @@ * See the following message on how MAP_NORESERVE was treated on FreeBSD: * <http://lists.llvm.org/pipermail/cfe-commits/Week-of-Mon-20150202/122958.html> */ -# if defined(MAP_FIXED) && (defined(MAP_NORESERVE) || defined(__FreeBSD__)) +# if (defined(MAP_FIXED) && (defined(MAP_NORESERVE) || defined(__FreeBSD__)) \ + && !defined(ADDRESS_SANITIZER)) # define ERTS_HAVE_OS_PHYSICAL_MEMORY_RESERVATION 1 # endif #endif @@ -207,20 +208,16 @@ ERTS_GLB_INLINE void erts_mem_discard(void *p, UWord size); #include <sys/mman.h> ERTS_GLB_INLINE void erts_mem_discard(void *ptr, UWord size) { + /* Note that we don't fall back to MADV_DONTNEED since it promises that + * the given region will be zeroed on access, which turned out to be + * too much of a performance hit. */ #ifdef MADV_FREE - /* This is preferred as it doesn't necessarily free the pages right - * away, which is a bit faster than MADV_DONTNEED. */ madvise(ptr, size, MADV_FREE); #else - madvise(ptr, size, MADV_DONTNEED); + (void)ptr; + (void)size; #endif } -#elif defined(HAVE_SYS_MMAN_H) && defined(HAVE_POSIX_MADVISE) && !(defined(__sun) || defined(__sun__)) - #include <sys/mman.h> - - ERTS_GLB_INLINE void erts_mem_discard(void *ptr, UWord size) { - posix_madvise(ptr, size, POSIX_MADV_DONTNEED); - } #elif defined(_WIN32) #include <winbase.h> diff --git a/erts/emulator/sys/unix/sys.c b/erts/emulator/sys/unix/sys.c index beaa466f81..46a035214b 100644 --- a/erts/emulator/sys/unix/sys.c +++ b/erts/emulator/sys/unix/sys.c @@ -49,6 +49,10 @@ #include <sys/ioctl.h> #endif +#ifdef ADDRESS_SANITIZER +# include <sanitizer/asan_interface.h> +#endif + #define ERTS_WANT_BREAK_HANDLING #define WANT_NONBLOCKING /* must define this to pull in defs from sys.h */ #include "sys.h" @@ -386,6 +390,9 @@ void erts_sys_sigsegv_handler(int signo) { */ int erts_sys_is_area_readable(char *start, char *stop) { +#ifdef ADDRESS_SANITIZER + return __asan_region_is_poisoned(start, stop-start) == NULL; +#else int fds[2]; if (!pipe(fds)) { /* We let write try to figure out if the pointers are readable */ @@ -400,7 +407,7 @@ erts_sys_is_area_readable(char *start, char *stop) { return 1; } return 0; - +#endif } static ERTS_INLINE int diff --git a/erts/emulator/test/alloc_SUITE.erl b/erts/emulator/test/alloc_SUITE.erl index 51406c6934..97cd90d72c 100644 --- a/erts/emulator/test/alloc_SUITE.erl +++ b/erts/emulator/test/alloc_SUITE.erl @@ -19,8 +19,8 @@ -module(alloc_SUITE). -author('rickard.green@uab.ericsson.se'). --export([all/0, suite/0, init_per_testcase/2, end_per_testcase/2]). - +-export([all/0, suite/0, init_per_testcase/2, end_per_testcase/2, + init_per_suite/1, end_per_suite/1]). -export([basic/1, coalesce/1, threads/1, @@ -47,6 +47,18 @@ all() -> bucket_mask, rbtree, mseg_clear_cache, erts_mmap, cpool, migration, cpool_opt]. +init_per_suite(Config) -> + case test_server:memory_checker() of + MC when MC =:= valgrind; MC =:= asan -> + %% No point testing own allocators under valgrind or asan. + {skip, "Memory checker " ++ atom_to_list(MC)}; + none -> + Config + end. + +end_per_suite(_Config) -> + ok. + init_per_testcase(Case, Config) when is_list(Config) -> [{testcase, Case},{debug,false}|Config]. diff --git a/erts/emulator/test/async_ports_SUITE_data/cport.c b/erts/emulator/test/async_ports_SUITE_data/cport.c index 033aff382a..621b109444 100644 --- a/erts/emulator/test/async_ports_SUITE_data/cport.c +++ b/erts/emulator/test/async_ports_SUITE_data/cport.c @@ -10,27 +10,19 @@ #endif typedef unsigned char byte; +int write_exact(byte *buf, int len); -int read_cmd(byte *buf) -{ - int len; - if (read_exact(buf, 4) != 4) - return(-1); +int read_exact(byte *buf, int len); - len = (buf[0] << 24) | (buf[1] << 16) | (buf[2] << 8) | buf[3]; - return read_exact(buf, len); -} - -int write_cmd(byte *buf, int len) +int write_exact(byte *buf, int len) { - byte li[4]; - li[0] = (len >> 24) & 0xff; - li[1] = (len >> 16) & 0xff; - li[2] = (len >> 8) & 0xff; - li[3] = len & 0xff; - write_exact(&li, 4); - - return write_exact(buf, len); + int i, wrote = 0; + do { + if ((i = write(1, buf+wrote, len-wrote)) < 0) + return (i); + wrote += i; + } while (wrote<len); + return len; } int read_exact(byte *buf, int len) @@ -46,15 +38,26 @@ int read_exact(byte *buf, int len) return len; } -int write_exact(byte *buf, int len) +int write_cmd(byte *buf, int len) { - int i, wrote = 0; - do { - if ((i = write(1, buf+wrote, len-wrote)) < 0) - return (i); - wrote += i; - } while (wrote<len); - return len; + byte li[4]; + li[0] = (len >> 24) & 0xff; + li[1] = (len >> 16) & 0xff; + li[2] = (len >> 8) & 0xff; + li[3] = len & 0xff; + write_exact(li, 4); + + return write_exact(buf, len); +} + +int read_cmd(byte *buf) +{ + int len; + if (read_exact(buf, 4) != 4) + return(-1); + + len = (buf[0] << 24) | (buf[1] << 16) | (buf[2] << 8) | buf[3]; + return read_exact(buf, len); } byte static_buf[31457280]; // 30 mb diff --git a/erts/emulator/test/bif_SUITE.erl b/erts/emulator/test/bif_SUITE.erl index a5f2e70d71..924c24b2fb 100644 --- a/erts/emulator/test/bif_SUITE.erl +++ b/erts/emulator/test/bif_SUITE.erl @@ -790,7 +790,12 @@ erlang_halt(Config) when is_list(Config) -> % This test triggers a segfault when dumping a crash dump % to make sure that we can handle it properly. + + %% Prevent address sanitizer from catching SEGV in slave node + AsanOpts = add_asan_opt("handle_segv=0"), {ok,N4} = slave:start(H, halt_node4), + reset_asan_opts(AsanOpts), + CrashDump = filename:join(proplists:get_value(priv_dir,Config), "segfault_erl_crash.dump"), true = rpc:call(N4, os, putenv, ["ERL_CRASH_DUMP",CrashDump]), @@ -808,6 +813,25 @@ erlang_halt(Config) when is_list(Config) -> ok end. +add_asan_opt(Opt) -> + case test_server:is_asan() of + true -> + case os:getenv("ASAN_OPTIONS") of + false -> + os:putenv("ASAN_OPTIONS", Opt), + undefined; + AO -> + os:putenv("ASAN_OPTIONS", AO ++ [$: | Opt]), + AO + end; + _ -> + false + end. + +reset_asan_opts(false) -> ok; +reset_asan_opts(undefined) -> os:unsetenv("ASAN_OPTIONS"); +reset_asan_opts(AO) -> os:putenv("ASAN_OPTIONS", AO). + wait_until_stable_size(_File,-10) -> {error,enoent}; wait_until_stable_size(File,PrevSz) -> diff --git a/erts/emulator/test/distribution_SUITE.erl b/erts/emulator/test/distribution_SUITE.erl index 4982d979ec..128c9b07e0 100644 --- a/erts/emulator/test/distribution_SUITE.erl +++ b/erts/emulator/test/distribution_SUITE.erl @@ -1750,14 +1750,14 @@ start_monitor(Offender,P) -> just_stay_alive -> ok end end), - Ref = receive + Res = receive {Q,ref,R} -> - R + {Q, R} after 5000 -> error end, - io:format("Ref is ~p~n",[Ref]), - ok. + io:format("Res is ~p~n",[Res]), + Res. start_link(Offender,P) -> Parent = self(), Q = spawn(Offender, @@ -1769,14 +1769,14 @@ start_link(Offender,P) -> just_stay_alive -> ok end end), - Ref = receive + Res = receive {Q,ref,R} -> R after 5000 -> error end, - io:format("Ref is ~p~n",[Ref]), - ok. + io:format("Res is ~p~n",[Res]), + Res. %% Test dist messages with valid structure (binary to term ok) but malformed control content bad_dist_structure(Config) when is_list(Config) -> @@ -1924,20 +1924,38 @@ bad_dist_fragments(Config) when is_list(Config) -> [{hdr, 3, binary:part(Msg, 10,byte_size(Msg)-10)}, close]), - start_monitor(Offender,P), - ExitVictim = spawn(Victim, fun() -> receive ok -> ok end end), - send_bad_fragments(Offender, Victim, P,{?DOP_PAYLOAD_EXIT,P,ExitVictim},2, + ExitVictim = spawn(Victim, fun() -> + receive + {link, Proc} -> + link(Proc), + Parent ! {self(), linked} + end, + receive ok -> ok end + end), + OP1 = start_link(Offender,ExitVictim), + ExitVictim ! {link, OP1}, + receive {ExitVictim, linked} -> ok end, + send_bad_fragments(Offender, Victim, ExitVictim,{?DOP_PAYLOAD_EXIT,OP1,ExitVictim},0, [{hdr, 1, [131]}]), - start_monitor(Offender,P), Exit2Victim = spawn(Victim, fun() -> receive ok -> ok end end), - send_bad_fragments(Offender, Victim, P,{?DOP_PAYLOAD_EXIT2,P,Exit2Victim},2, + {OP2, _} = start_monitor(Offender,Exit2Victim), + send_bad_fragments(Offender, Victim, Exit2Victim,{?DOP_PAYLOAD_EXIT2,OP2,Exit2Victim},0, [{hdr, 1, [132]}]), - start_monitor(Offender,P), - DownVictim = spawn(Victim, fun() -> receive ok -> ok end end), - DownRef = erlang:monitor(process, DownVictim), - send_bad_fragments(Offender, Victim, P,{?DOP_PAYLOAD_MONITOR_P_EXIT,P,DownVictim,DownRef},2, + DownVictim = spawn(Victim, fun() -> + receive + {monitor, Proc} -> + DR = erlang:monitor(process, Proc), + Parent ! {self(), DR} + end, + Parent ! {self, DR}, + receive ok -> ok end + end), + {OP3, _} = start_monitor(Offender,DownVictim), + DownVictim ! {monitor, OP3}, + DownRef = receive {DownVictim, DR} -> DR end, + send_bad_fragments(Offender, Victim, DownVictim,{?DOP_PAYLOAD_MONITOR_P_EXIT,OP3,DownVictim,DownRef},0, [{hdr, 1, [133]}]), P ! two, diff --git a/erts/emulator/test/erl_drv_thread_SUITE_data/rwlock.c b/erts/emulator/test/erl_drv_thread_SUITE_data/rwlock.c index 98d0162b55..9c1b242007 100644 --- a/erts/emulator/test/erl_drv_thread_SUITE_data/rwlock.c +++ b/erts/emulator/test/erl_drv_thread_SUITE_data/rwlock.c @@ -18,6 +18,7 @@ */ #include "testcase_driver.h" +#include <stdio.h> #ifdef __WIN32__ #include <windows.h> @@ -25,6 +26,7 @@ #include <unistd.h> #endif #include <errno.h> +#include <stdio.h> #define NO_OF_THREADS 17 diff --git a/erts/emulator/test/erl_link_SUITE.erl b/erts/emulator/test/erl_link_SUITE.erl index ed444f2599..7e8a6b2d2c 100644 --- a/erts/emulator/test/erl_link_SUITE.erl +++ b/erts/emulator/test/erl_link_SUITE.erl @@ -45,7 +45,11 @@ otp_5772_dist_link/1, otp_5772_monitor/1, otp_5772_dist_monitor/1, - otp_7946/1]). + otp_7946/1, + otp_17127_local_link_with_simultaneous_link_unlink/1, + otp_17127_dist_link_with_simultaneous_link_unlink/1, + otp_17127_local_random/1, + otp_17127_dist_random/1]). -export([init_per_testcase/2, end_per_testcase/2]). @@ -55,6 +59,7 @@ -record(erl_link, {type, % process | port | dist_process pid = [], + state, % linked | unlinking id}). % This is to be kept in sync with erl_bif_info.c (make_monitor_list) @@ -74,7 +79,10 @@ all() -> [links, dist_links, monitor_nodes, process_monitors, dist_process_monitors, busy_dist_port_monitor, busy_dist_port_link, otp_5772_link, otp_5772_dist_link, - otp_5772_monitor, otp_5772_dist_monitor, otp_7946]. + otp_5772_monitor, otp_5772_dist_monitor, otp_7946, + otp_17127_local_link_with_simultaneous_link_unlink, + otp_17127_dist_link_with_simultaneous_link_unlink, + otp_17127_local_random, otp_17127_dist_random]. init_per_testcase(Func, Config) when is_atom(Func), is_list(Config) -> case catch erts_debug:get_internal_state(available_internal_state) of @@ -109,6 +117,8 @@ dist_links(Config) when is_list(Config) -> TP4 = spawn(?MODULE, test_proc, []), TP5 = spawn(?MODULE, test_proc, []), TP6 = spawn(Node, ?MODULE, test_proc, []), + io:format("TP4=~p~nTP5=~p~nTP6=~p~n", [TP4, TP5, TP6]), + true = tp_call(TP6, fun() -> link(TP4) end), check_link(TP4, TP6), true = tp_call(TP5, @@ -490,6 +500,119 @@ otp_7946(Config) when is_list(Config) -> Reason = noconnection end. +otp_17127_local_link_with_simultaneous_link_unlink(Config) when is_list(Config) -> + otp_17127_link_with_simultaneous_link_unlink_test(node(), node()). + +otp_17127_dist_link_with_simultaneous_link_unlink(Config) when is_list(Config) -> + [NodeName] = get_names(1, otp_17127), + {ok, Node} = start_node(NodeName), + Res = otp_17127_link_with_simultaneous_link_unlink_test(node(), Node), + stop_node(Node), + Res. + +otp_17127_link_with_simultaneous_link_unlink_test(NodeA, NodeB) -> + FunA = fun (Other) -> + link(Other) + end, + FunB = fun (Other) -> + link(Other), + unlink(Other) + end, + otp_17127_test(NodeA, FunA, NodeB, FunB). + +otp_17127_local_random(Config) when is_list(Config) -> + otp_17127_random_test(node(), node(), 100). + +otp_17127_dist_random(Config) when is_list(Config) -> + [NodeName] = get_names(1, otp_17127), + {ok, Node} = start_node(NodeName), + Res = otp_17127_random_test(node(), Node, 20), + stop_node(Node), + Res. + +otp_17127_random_test(_NodeA, _NodeB, 0) -> + ok; +otp_17127_random_test(NodeA, NodeB, N) -> + Fun = fun (Other) -> + rand_proc(Other, rand:uniform(500)) + end, + otp_17127_test(NodeA, Fun, NodeB, Fun), + otp_17127_random_test(NodeA, NodeB, N-1). + +rand_proc(_Other, 0) -> + ok; +rand_proc(Other, N) -> + case rand:uniform(3) of + 1 -> link(Other); + 2 -> unlink(Other); + 3 -> erlang:yield() + end, + rand_proc(Other, N-1). + +otp_17127_test(NodeA, FunA, NodeB, FunB) -> + process_flag(priority, high), + {SchedA, SchedB} = case NodeA == NodeB of + false -> + {[], []}; + true -> + NS = erlang:system_info(schedulers_online), + process_flag(scheduler, 1), + {[{scheduler, (1 rem NS) + 1}], + [{scheduler, (2 rem NS) + 1}]} + end, + A = spawn_opt(NodeA, + fun () -> + receive + {go, Tester, Other, Later} -> + busy_wait_until(Later), + FunA(Other), + receive ping -> Other ! pong end, + receive pling -> ok end, + Tester ! {self(), done} + end, + receive after infinity -> ok end + end, SchedA), + B = spawn_opt(NodeB, + fun () -> + receive + {go, Tester, Other, Later} -> + busy_wait_until(Later), + FunB(Other), + Other ! ping, + receive pong -> Other ! pling end, + Tester ! {self(), done} + end, + receive after infinity -> ok end + end, SchedB), + io:format("A = ~p~nB = ~p~n", [A, B]), + Later = case NodeA == NodeB of + true -> + GoTime = (erlang:monotonic_time() + + erlang:convert_time_unit(100, millisecond, native)), + fun () -> + erlang:monotonic_time() >= GoTime + end; + false -> + GoTime = (os:system_time(nanosecond) + + erlang:convert_time_unit(500, millisecond, nanosecond)), + fun () -> + os:system_time(nanosecond) >= GoTime + end + end, + erlang:yield(), + A ! {go, self(), B, Later}, + B ! {go, self(), A, Later}, + receive {A, done} -> ok end, + receive {B, done} -> ok end, + try + check_consistent_link_state(A, B), + true = rpc:call(node(A), erlang, is_process_alive, [A]), + true = rpc:call(node(B), erlang, is_process_alive, [B]) + after + exit(A, kill), + exit(B, kill) + end. + %% %% -- Internal utils -------------------------------------------------------- %% @@ -668,6 +791,13 @@ wait_until(Fun) -> end end. +busy_wait_until(Fun) -> + case Fun() of + true -> ok; + _ -> + busy_wait_until(Fun) + end. + forever(Fun) -> Fun(), forever(Fun). @@ -782,9 +912,13 @@ find_erl_monitor(Pid, Item) -> find_erl_link(Obj, Type, Item) when is_pid(Item); is_port(Item) -> LinkList = get_link_list(Obj), io:format("~p LinkList: ~p~n", [Obj, LinkList]), - lists:foldl(fun (#erl_link{type = T, pid = I} = EL, + lists:foldl(fun (#erl_link{type = T, pid = I, state = linked} = EL, Acc) when T == Type, I == Item -> [EL|Acc]; + ({erl_link, T, P, I}, + Acc) when T == Type, P == Item -> + %% Old emulator without state (linked if record exists)... + [#erl_link{type = T, pid = P, id = I, state = linked}|Acc]; (_, Acc) -> Acc end, @@ -794,9 +928,13 @@ find_erl_link(Obj, Type, Id) when is_integer(Id) -> %% Find by Id LinkList = get_link_list(Obj), io:format("~p LinkList: ~p~n", [Obj, LinkList]), - lists:foldl(fun (#erl_link{type = T, id = I} = EL, + lists:foldl(fun (#erl_link{type = T, id = I, state = linked} = EL, Acc) when T == Type, I == Id -> [EL|Acc]; + ({erl_link, T, P, I}, + Acc) when T == Type, I == Id -> + %% Old emulator without state (linked if record exists)... + [#erl_link{type = T, pid = P, id = I, state = linked}|Acc]; (_, Acc) -> Acc end, @@ -816,14 +954,25 @@ get_link_type(A, B) when is_pid(A), dist_process end. +check_consistent_link_state(A, B) -> + %% Both processes should agree on whether + %% they are linked or not... + LinkType = get_link_type(A, B), + case find_erl_link(A, LinkType, B) of + [] -> + check_unlink(A, B), + io:format("~p and ~p are not linked~n", [A, B]); + _ -> + check_link(A, B), + io:format("~p and ~p are linked~n", [A, B]) + end. + check_link(A, B) when node(A) == node(B) -> LinkType = get_link_type(A, B), [#erl_link{type = LinkType, - pid = B, - id = Id}] = find_erl_link(A, LinkType, B), + pid = B}] = find_erl_link(A, LinkType, B), [#erl_link{type = LinkType, - pid = A, - id = Id}] = find_erl_link(B, LinkType, A), + pid = A}] = find_erl_link(B, LinkType, A), [] = find_erl_link({node(A), node(B)}, LinkType, A), diff --git a/erts/emulator/test/erts_debug_SUITE.erl b/erts/emulator/test/erts_debug_SUITE.erl index 32efd6cf84..01473874e8 100644 --- a/erts/emulator/test/erts_debug_SUITE.erl +++ b/erts/emulator/test/erts_debug_SUITE.erl @@ -229,7 +229,10 @@ alloc_blocks_size(Config) when is_list(Config) -> ok = rpc:call(Node, ?MODULE, do_alloc_blocks_size, []), true = test_server:stop_node(Node) end, - F("+Meamax"), + case test_server:is_asan() of + false -> F("+Meamax"); + true -> skip + end, F("+Meamin"), F(""), ok. diff --git a/erts/emulator/test/hash_SUITE.erl b/erts/emulator/test/hash_SUITE.erl index c4a700d1a7..86b4460b38 100644 --- a/erts/emulator/test/hash_SUITE.erl +++ b/erts/emulator/test/hash_SUITE.erl @@ -640,13 +640,18 @@ test_phash2_plus_bin_helper2(Bin, TransformerFun, ExtraBytes, ExtraBits, Expecte end. run_when_enough_resources(Fun) -> - case {total_memory(), erlang:system_info(wordsize)} of - {Mem, 8} when is_integer(Mem) andalso Mem >= 31 -> + Bits = 8 * erlang:system_info({wordsize,external}), + Mem = total_memory(), + Build = erlang:system_info(build_type), + + if Bits =:= 64, is_integer(Mem), Mem >= 31, + Build =/= valgrind, Build =/= asan -> Fun(); - {Mem, WordSize} -> + + true -> {skipped, - io_lib:format("Not enough resources (System Memory >= ~p, Word Size = ~p)", - [Mem, WordSize])} + io_lib:format("Not enough resources (System Memory = ~p, Bits = ~p, Build = ~p)", + [Mem, Bits, Build])} end. %% Total memory in GB diff --git a/erts/emulator/test/os_signal_SUITE.erl b/erts/emulator/test/os_signal_SUITE.erl index 6bafb0e18c..7bd8985dc7 100644 --- a/erts/emulator/test/os_signal_SUITE.erl +++ b/erts/emulator/test/os_signal_SUITE.erl @@ -275,6 +275,15 @@ t_sigalrm(_Config) -> ok. t_sigchld_fork(_Config) -> + case test_server:is_asan() of + true -> + %% Avoid false leak reports from forked process + {skip, "Address sanitizer"}; + false -> + sigchld_fork() + end. + +sigchld_fork() -> Pid1 = setup_service(), ok = os:set_signal(sigchld, handle), {ok,OsPid} = os_signal_SUITE:fork(), diff --git a/erts/emulator/test/persistent_term_SUITE.erl b/erts/emulator/test/persistent_term_SUITE.erl index f4511eb483..a988864c2d 100644 --- a/erts/emulator/test/persistent_term_SUITE.erl +++ b/erts/emulator/test/persistent_term_SUITE.erl @@ -22,8 +22,10 @@ -include_lib("common_test/include/ct.hrl"). -export([all/0,suite/0,init_per_suite/1,end_per_suite/1, + init_per_testcase/2, end_per_testcase/2, basic/1,purging/1,sharing/1,get_trapping/1, destruction/1, + get_all_race/1, info/1,info_trapping/1,killed_while_trapping/1, off_heap_values/1,keys/1,collisions/1, init_restart/1, put_erase_trapping/1, @@ -40,6 +42,7 @@ suite() -> all() -> [basic,purging,sharing,get_trapping,info,info_trapping, destruction, + get_all_race, killed_while_trapping,off_heap_values,keys,collisions, init_restart, put_erase_trapping, killed_while_trapping_put, killed_while_trapping_erase]. @@ -56,6 +59,15 @@ end_per_suite(Config) -> erts_debug:set_internal_state(available_internal_state, false), Config. +init_per_testcase(_, Config) -> + Config. + +end_per_testcase(_, _Config) -> + ok; +end_per_testcase(get_all_race, _Config) -> + get_all_race_cleanup(), + ok. + basic(_Config) -> Chk = chk(), N = 777, @@ -781,3 +793,31 @@ repeat(_Fun, 0) -> repeat(Fun, N) -> Fun(), repeat(Fun, N-1). + + +%% OTP-17298 +get_all_race(_Config) -> + N = 20 * erlang:system_info(schedulers_online), + persistent_term:put(get_all_race, N), + SPs = [spawn_link(fun() -> gar_setter(Seq) end) || Seq <- lists:seq(1, N)], + GPs = [spawn_link(fun gar_getter/0) || _ <- lists:seq(1, N)], + receive after 2000 -> ok end, + [begin unlink(Pid), exit(Pid,kill) end || Pid <- (SPs ++ GPs)], + ok. + +get_all_race_cleanup() -> + N = persistent_term:get(get_all_race, 0), + _ = persistent_term:erase(get_all_race), + [_ = persistent_term:erase(Seq) || Seq <- lists:seq(1, N)], + ok. + +gar_getter() -> + erts_debug:set_internal_state(reds_left, 1), + _ = persistent_term:get(), + gar_getter(). + +gar_setter(Key) -> + erts_debug:set_internal_state(reds_left, 1), + persistent_term:erase(Key), + persistent_term:put(Key, {complex, term}), + gar_setter(Key). diff --git a/erts/emulator/test/process_SUITE.erl b/erts/emulator/test/process_SUITE.erl index 2e9a427240..5619807791 100644 --- a/erts/emulator/test/process_SUITE.erl +++ b/erts/emulator/test/process_SUITE.erl @@ -1613,10 +1613,13 @@ process_flag_badarg(Config) when is_list(Config) -> chk_badarg(fun () -> process_flag(priority, 4711) end), chk_badarg(fun () -> process_flag(save_calls, hmmm) end), - P= spawn_link(fun () -> receive die -> ok end end), + {P,Mref} = spawn_monitor(fun () -> receive "in vain" -> no end end), chk_badarg(fun () -> process_flag(P, save_calls, hmmm) end), chk_badarg(fun () -> process_flag(gurka, save_calls, hmmm) end), - P ! die, + exit(P, die), + chk_badarg(fun () -> process_flag(P, save_calls, 0) end), + {'DOWN', Mref, process, P, die} = receive M -> M end, + chk_badarg(fun () -> process_flag(P, save_calls, 0) end), ok. -include_lib("stdlib/include/ms_transform.hrl"). diff --git a/erts/emulator/test/timer_bif_SUITE.erl b/erts/emulator/test/timer_bif_SUITE.erl index 2382dc992e..68048b9771 100644 --- a/erts/emulator/test/timer_bif_SUITE.erl +++ b/erts/emulator/test/timer_bif_SUITE.erl @@ -31,7 +31,8 @@ same_time_yielding_with_cancel/1, same_time_yielding_with_cancel_other/1, % same_time_yielding_with_cancel_other_accessor/1, auto_cancel_yielding/1, - suspended_scheduler_timeout/1]). + suspended_scheduler_timeout/1, + multizero_timeout_in_timeout/1]). -include_lib("common_test/include/ct.hrl"). @@ -70,7 +71,8 @@ all() -> same_time_yielding_with_cancel_other, % same_time_yielding_with_cancel_other_accessor, auto_cancel_yielding, - suspended_scheduler_timeout]. + suspended_scheduler_timeout, + multizero_timeout_in_timeout]. %% Basic start_timer/3 functionality @@ -657,6 +659,26 @@ suspended_scheduler_timeout(Config) when is_list(Config) -> end, ok. +multizero_timeout_in_timeout(Config) when is_list(Config) -> + Timeout = 500, + MaxTimeoutDiff = 1000, + + %% We want to operate on the same timer wheel all the time... + process_flag(scheduler, erlang:system_info(schedulers_online)), + + erlang:send_after(5*(Timeout+MaxTimeoutDiff), self(), pling), + erlang:yield(), + Start = erlang:monotonic_time(), + erts_debug:set_internal_state(multizero_timeout_in_timeout, Timeout), + receive multizero_timeout_in_timeout_done -> ok end, + End = erlang:monotonic_time(), + Time = erlang:convert_time_unit(End-Start, native, millisecond), + io:format("Time=~p~n", [Time]), + true = Time < Timeout + MaxTimeoutDiff, + ok. + + + process_is_cleaned_up(P) when is_pid(P) -> undefined == erts_debug:get_internal_state({process_status, P}). diff --git a/erts/emulator/test/trace_SUITE.erl b/erts/emulator/test/trace_SUITE.erl index 025f5282fb..951502cb61 100644 --- a/erts/emulator/test/trace_SUITE.erl +++ b/erts/emulator/test/trace_SUITE.erl @@ -1684,6 +1684,18 @@ bad_flag(Config) when is_list(Config) -> {'EXIT', {badarg, _}} = (catch erlang:trace(new, true, [not_a_valid_flag])), + + %% Leaks of {tracer,_} in OTP 23.2 + Pid = spawn(fun() -> receive die -> ok end end), + 1 = erlang:trace(Pid, true, [{tracer, self()}, + {tracer, self()}]), + Pid ! die, + {'EXIT', {badarg, _}} = + (catch erlang:trace(new, true, [{tracer, self()} + | improper])), + {'EXIT', {badarg, _}} = + (catch erlang:trace(new, true, [{tracer, self()}, + not_a_valid_flag])), ok. %% Test erlang:trace_delivered/1 diff --git a/erts/emulator/test/trace_call_time_SUITE.erl b/erts/emulator/test/trace_call_time_SUITE.erl index 00235bffa5..d4595e6647 100644 --- a/erts/emulator/test/trace_call_time_SUITE.erl +++ b/erts/emulator/test/trace_call_time_SUITE.erl @@ -65,6 +65,7 @@ -export([all/0, suite/0, init_per_testcase/2, end_per_testcase/2, not_run/1]). -export([basic/1, on_and_off/1, info/1, + apply_bif_bug/1, abb_worker/1, disable_ongoing/1, pause_and_restart/1, scheduling/1, called_function/1, combo/1, bif/1, nif/1]). @@ -91,6 +92,7 @@ all() -> false -> [basic, on_and_off, info, pause_and_restart, scheduling, disable_ongoing, + apply_bif_bug, combo, bif, nif, called_function, dead_tracer, return_stop, catch_crash] end. @@ -827,3 +829,26 @@ loop() -> Pid ! {self(), answer, erlang:apply(M, F, A)}, loop() end. + +%% OTP-17290, GH-4635 +apply_bif_bug(_Config) -> + Pid = spawn(?MODULE, abb_worker, [self()]), + erlang:trace(Pid, true, [call]), + erlang:trace_pattern({?MODULE,abb_foo,'_'}, true, [call_time]), + erlang:trace_pattern({erlang,display,1}, true, [call_time]), + Pid ! {call, erlang, display, ["Hej"]}, + receive + done -> ok + end, + erlang:trace_pattern({'_','_','_'}, false, [call_time]). + +abb_worker(Papa) -> + receive + {call, M, F, Args} -> + abb_foo(M, F, Args), + Papa ! done + end. + + +abb_foo(M,F,Args) -> + apply(M,F,Args). diff --git a/erts/etc/unix/cerl.src b/erts/etc/unix/cerl.src index 539425b954..3889308a67 100644 --- a/erts/etc/unix/cerl.src +++ b/erts/etc/unix/cerl.src @@ -43,6 +43,7 @@ # -purecov Run emulator compiled for purecov # -gcov Run emulator compiled for gcov # -valgrind Run emulator compiled for valgrind +# -asan Run emulator compiled for address-sanitizer # -lcnt Run emulator compiled for lock counting # -icount Run emulator compiled for instruction counting # -rr Run emulator under "rr record" @@ -75,8 +76,8 @@ GDB= GDBBP= GDBARGS= TYPE= -debug= run_valgrind=no +run_asan=no run_rr=no skip_erlexec=no @@ -220,6 +221,12 @@ while [ $# -gt 0 ]; do run_valgrind=yes skip_erlexec=yes ;; + "-asan") + shift + cargs="$cargs -asan" + run_asan=yes + TYPE=.asan + ;; "-emu_type") shift cargs="$cargs -$1" @@ -271,6 +278,28 @@ if [ $skip_erlexec = yes ]; then set -- $beam_args IFS="$SAVE_IFS" fi +if [ $run_asan = yes ]; then + # Leak sanitizer options + if [ "x${LSAN_OPTIONS#*suppressions=}" = "x$LSAN_OPTIONS" ]; then + export LSAN_OPTIONS + if [ "x$ERL_TOP" != "x" ]; then + LSAN_OPTIONS="$LSAN_OPTIONS:suppressions=$ERL_TOP/erts/emulator/asan/suppress" + else + echo "No leak-sanitizer suppression file found in \$LSAN_OPTIONS" + echo "and \$ERL_TOP not set." + fi + fi + # Address sanitizer options + export ASAN_OPTIONS + if [ "x$ASAN_LOG_DIR" != "x" ]; then + if [ "x${ASAN_OPTIONS#*log_path=}" = "x$ASAN_OPTIONS" ]; then + ASAN_OPTIONS="$ASAN_OPTIONS:log_path=$ASAN_LOG_DIR/$EMU_NAME-$ASAN_LOGFILE_PREFIX-0" + fi + fi + if [ "x${ASAN_OPTIONS#*halt_on_error=}" = "x$ASAN_OPTIONS" ]; then + ASAN_OPTIONS="$ASAN_OPTIONS:halt_on_error=false" + fi +fi if [ "x$GDB" = "x" ]; then if [ $run_valgrind = yes ]; then valversion=`valgrind --version` diff --git a/erts/etc/win32/erl.c b/erts/etc/win32/erl.c index e960fb1238..ae58cf04be 100644 --- a/erts/etc/win32/erl.c +++ b/erts/etc/win32/erl.c @@ -74,7 +74,8 @@ int wmain(int argc, wchar_t **argv) wslpathlen = wcslen(wslpath); } } - pathlen = (wcslen(path) + wslpathlen + wcslen(erlexec_dir) + 2); + /* Add size for path delimiters and eos */ + pathlen = (wcslen(path) + wslpathlen + wcslen(erlexec_dir) + 3); npath = (wchar_t *) malloc(pathlen*sizeof(wchar_t)); if(wslpathlen > 0) { swprintf(npath,pathlen,L"%s;%s;%s",erlexec_dir,path,wslpath); diff --git a/erts/include/internal/ethread_header_config.h.in b/erts/include/internal/ethread_header_config.h.in index 6309f10439..300068b952 100644 --- a/erts/include/internal/ethread_header_config.h.in +++ b/erts/include/internal/ethread_header_config.h.in @@ -76,6 +76,12 @@ /* Define if x86/x86_64 out of order instructions should be synchronized */ #undef ETHR_X86_OUT_OF_ORDER +/* Define if you have the powerpc lwsync instruction */ +#undef ETHR_PPC_HAVE_LWSYNC + +/* Define if you do not have the powerpc lwsync instruction */ +#undef ETHR_PPC_HAVE_NO_LWSYNC + /* Define if only run in Sparc TSO mode */ #undef ETHR_SPARC_TSO @@ -86,10 +92,20 @@ #undef ETHR_SPARC_RMO /* Define as a boolean indicating whether you have a gcc compatible compiler - capable of generating the ARM DMB instruction, and are compiling for an ARM - processor with ARM DMB instruction support, or not */ + capable of generating the ARM 'dmb sy' instruction, and are compiling for + an ARM processor with ARM DMB instruction support, or not */ #undef ETHR_HAVE_GCC_ASM_ARM_DMB_INSTRUCTION +/* Define as a boolean indicating whether you have a gcc compatible compiler + capable of generating the ARM 'dmb ld' instruction, and are compiling for + an ARM processor with ARM DMB instruction support, or not */ +#undef ETHR_HAVE_GCC_ASM_ARM_DMB_LD_INSTRUCTION + +/* Define as a boolean indicating whether you have a gcc compatible compiler + capable of generating the ARM 'dmb st' instruction, and are compiling for + an ARM processor with ARM DMB instruction support, or not */ +#undef ETHR_HAVE_GCC_ASM_ARM_DMB_ST_INSTRUCTION + /* Define as a bitmask corresponding to the word sizes that __sync_synchronize() can handle on your system */ #undef ETHR_HAVE___sync_synchronize diff --git a/erts/include/internal/gcc/ethr_membar.h b/erts/include/internal/gcc/ethr_membar.h index 643b243683..aeef8115a3 100644 --- a/erts/include/internal/gcc/ethr_membar.h +++ b/erts/include/internal/gcc/ethr_membar.h @@ -149,14 +149,51 @@ ethr_full_fence__(void) __asm__ __volatile__("dmb sy" : : : "memory"); } +#if ETHR_HAVE_GCC_ASM_ARM_DMB_ST_INSTRUCTION static __inline__ __attribute__((__always_inline__)) void ethr_store_fence__(void) { + /* StoreStore */ __asm__ __volatile__("dmb st" : : : "memory"); } +#endif + +#if ETHR_HAVE_GCC_ASM_ARM_DMB_LD_INSTRUCTION +static __inline__ __attribute__((__always_inline__)) void +ethr_load_fence__(void) +{ + /* LoadLoad and LoadStore */ + __asm__ __volatile__("dmb ld" : : : "memory"); +} +#endif -#define ETHR_MEMBAR(B) \ - ETHR_CHOOSE_EXPR((B) == ETHR_StoreStore, ethr_store_fence__(), ethr_full_fence__()) +#if ETHR_HAVE_GCC_ASM_ARM_DMB_ST_INSTRUCTION && ETHR_HAVE_GCC_ASM_ARM_DMB_LD_INSTRUCTION +/* sy, st & ld */ +#define ETHR_MEMBAR(B) \ + ETHR_CHOOSE_EXPR((B) == ETHR_StoreStore, \ + ethr_store_fence__(), \ + ETHR_CHOOSE_EXPR((B) & (ETHR_StoreStore \ + | ETHR_StoreLoad), \ + ethr_full_fence__(), \ + ethr_load_fence__())) +#elif ETHR_HAVE_GCC_ASM_ARM_DMB_ST_INSTRUCTION +/* sy & st */ +#define ETHR_MEMBAR(B) \ + ETHR_CHOOSE_EXPR((B) == ETHR_StoreStore, \ + ethr_store_fence__(), \ + ethr_full_fence__()) +#elif ETHR_HAVE_GCC_ASM_ARM_DMB_LD_INSTRUCTION +/* sy & ld */ +#define ETHR_MEMBAR(B) \ + ETHR_CHOOSE_EXPR((B) & (ETHR_StoreStore \ + | ETHR_StoreLoad), \ + ethr_full_fence__(), \ + ethr_load_fence__()) +#else +/* sy */ +#define ETHR_MEMBAR(B) \ + ethr_full_fence__() +#endif #elif ETHR_HAVE___sync_synchronize @@ -205,9 +242,13 @@ ethr_full_fence__(void) /* * Define ETHR_READ_DEPEND_MEMORY_BARRIER for all architechtures * not known to order data dependent loads + * + * This is a bit too conservative, but better safe than sorry... + * Add more archs as needed... */ -#if !defined(__ia64__) && !defined(__arm__) +#if !defined(__ia64__) && !defined(__arm__) && !defined(__arm64__) \ + && !defined(__aarch32__) && !defined(__aarch64__) # define ETHR_READ_DEPEND_MEMORY_BARRIER ETHR_MEMBAR(ETHR_LoadLoad) #endif diff --git a/erts/include/internal/gcc/ethread.h b/erts/include/internal/gcc/ethread.h index 12b41f8704..5584648614 100644 --- a/erts/include/internal/gcc/ethread.h +++ b/erts/include/internal/gcc/ethread.h @@ -44,6 +44,10 @@ #undef ETHR_GCC_RELB_VERSIONS__ #undef ETHR_GCC_RELB_MOD_VERSIONS__ #undef ETHR_GCC_MB_MOD_VERSIONS__ +#undef ETHR_TRUST_GCC_ATOMIC_BUILTINS_MEMORY_BARRIERS__ + +#define ETHR_TRUST_GCC_ATOMIC_BUILTINS_MEMORY_BARRIERS__ \ + ETHR_TRUST_GCC_ATOMIC_BUILTINS_MEMORY_BARRIERS /* * True GNU GCCs before version 4.8 do not emit a memory barrier @@ -52,18 +56,46 @@ */ #undef ETHR___atomic_load_ACQUIRE_barrier_bug #if ETHR_GCC_COMPILER != ETHR_GCC_COMPILER_TRUE + +#if ETHR_GCC_COMPILER == ETHR_GCC_COMPILER_CLANG \ + && defined(__apple_build_version__) \ + && __clang_major__ >= 12 +/* Apples clang verified not to have this bug */ +# define ETHR___atomic_load_ACQUIRE_barrier_bug 0 +/* Also trust builtin barriers */ +# undef ETHR_TRUST_GCC_ATOMIC_BUILTINS_MEMORY_BARRIERS__ +# define ETHR_TRUST_GCC_ATOMIC_BUILTINS_MEMORY_BARRIERS__ 1 +# else /* - * A gcc compatible compiler. We have no information + * Another gcc compatible compiler. We have no information * about the existence of this bug, but we assume * that it is not impossible that it could have * been "inherited". Therefore, until we are certain * that the bug does not exist, we assume that it * does. */ -# define ETHR___atomic_load_ACQUIRE_barrier_bug ETHR_GCC_VERSIONS_MASK__ +# define ETHR___atomic_load_ACQUIRE_barrier_bug ETHR_GCC_VERSIONS_MASK__ +# endif + #elif !ETHR_AT_LEAST_GCC_VSN__(4, 8, 0) /* True gcc of version < 4.8, i.e., bug exist... */ # define ETHR___atomic_load_ACQUIRE_barrier_bug ETHR_GCC_VERSIONS_MASK__ +#elif ETHR_AT_LEAST_GCC_VSN__(8, 3, 0) \ + && (defined(__arm64__) || defined(__aarch64__) || defined(__arm__)) \ + && ETHR_SIZEOF_PTR == 8 +/* Verified not to have this bug */ +# define ETHR___atomic_load_ACQUIRE_barrier_bug 0 +/* Also trust builtin barriers */ +# undef ETHR_TRUST_GCC_ATOMIC_BUILTINS_MEMORY_BARRIERS__ +# define ETHR_TRUST_GCC_ATOMIC_BUILTINS_MEMORY_BARRIERS__ 1 +#elif ETHR_AT_LEAST_GCC_VSN__(9, 3, 0) \ + && (defined(__powerpc__) || defined(__ppc__) || defined(__powerpc64__)) \ + && ETHR_SIZEOF_PTR == 8 +/* Verified not to have this bug */ +# define ETHR___atomic_load_ACQUIRE_barrier_bug 0 +/* Also trust builtin barriers */ +# undef ETHR_TRUST_GCC_ATOMIC_BUILTINS_MEMORY_BARRIERS__ +# define ETHR_TRUST_GCC_ATOMIC_BUILTINS_MEMORY_BARRIERS__ 1 #else /* True gcc of version >= 4.8 */ /* * Sizes less than or equal to word size have been fixed, @@ -87,7 +119,7 @@ #define ETHR_GCC_RELAXED_VERSIONS__ ETHR_GCC_VERSIONS_MASK__ #define ETHR_GCC_RELAXED_MOD_VERSIONS__ ETHR_GCC_VERSIONS_MASK__ -#if ETHR_TRUST_GCC_ATOMIC_BUILTINS_MEMORY_BARRIERS +#if ETHR_TRUST_GCC_ATOMIC_BUILTINS_MEMORY_BARRIERS__ # define ETHR_GCC_ACQB_VERSIONS__ ETHR_GCC_VERSIONS_MASK__ # define ETHR_GCC_ACQB_MOD_VERSIONS__ ETHR_GCC_VERSIONS_MASK__ # define ETHR_GCC_RELB_VERSIONS__ ETHR_GCC_VERSIONS_MASK__ diff --git a/erts/lib_src/Makefile.in b/erts/lib_src/Makefile.in index bb43d51d97..8fddd3479e 100644 --- a/erts/lib_src/Makefile.in +++ b/erts/lib_src/Makefile.in @@ -81,6 +81,11 @@ CFLAGS=@DEBUG_CFLAGS@ -DVALGRIND TYPE_SUFFIX=.valgrind PRE_LD= else +ifeq ($(TYPE),asan) +CFLAGS=@DEBUG_CFLAGS@ +TYPE_SUFFIX=.asan +PRE_LD= +else ifeq ($(TYPE),gprof) CFLAGS += -DGPROF -pg TYPE_SUFFIX=.gprof @@ -117,6 +122,7 @@ endif endif endif endif +endif OPSYS=@OPSYS@ sol2CFLAGS= diff --git a/erts/lib_src/common/erl_misc_utils.c b/erts/lib_src/common/erl_misc_utils.c index 17506b87ef..b35d53be7d 100644 --- a/erts/lib_src/common/erl_misc_utils.c +++ b/erts/lib_src/common/erl_misc_utils.c @@ -30,6 +30,7 @@ #include "erl_misc_utils.h" #if !defined(__WIN32__) /* UNIX */ +# include <stdarg.h> # include <stdio.h> # include <sys/types.h> # include <sys/param.h> @@ -1095,15 +1096,21 @@ get_cgroup_v1_base_dir(const char *controller) { return NULL; } -static const char* -get_cgroup_path(const char *controller) { +enum cgroup_version_t { + ERTS_CGROUP_NONE, + ERTS_CGROUP_V1, + ERTS_CGROUP_V2 +}; + +static enum cgroup_version_t +get_cgroup_path(const char *controller, const char **path) { char line_buf[10 << 10]; FILE *var_file; var_file = fopen("/proc/self/mountinfo", "r"); if (var_file == NULL) { - return NULL; + return ERTS_CGROUP_NONE; } while (fgets(line_buf, sizeof(line_buf), var_file)) { @@ -1138,7 +1145,9 @@ get_cgroup_path(const char *controller) { if (csv_contains(controllers, controller, ' ')) { free((void*)cgc_path); fclose(var_file); - return strdup(mount_path); + + *path = strdup(mount_path); + return ERTS_CGROUP_V2; } } free((void*)cgc_path); @@ -1147,42 +1156,51 @@ get_cgroup_path(const char *controller) { const char *base_dir = get_cgroup_v1_base_dir(controller); if (base_dir) { - const char *result; - if (strcmp(root_path, base_dir)) { - result = str_combine(mount_path, base_dir); + *path = str_combine(mount_path, base_dir); } else { - result = strdup(mount_path); + *path = strdup(mount_path); } free((void*)base_dir); fclose(var_file); - return result; + + return ERTS_CGROUP_V1; } } } } fclose(var_file); - return NULL; + + return ERTS_CGROUP_NONE; } -static int read_cgroup_var(const char *group_path, const char *var_name, - ssize_t *out) { +static int read_cgroup_interface(const char *group_path, const char *if_name, + int arg_count, const char *format, ...) { const char *var_path; int res; - var_path = str_combine(group_path, var_name); + var_path = str_combine(group_path, if_name); res = 0; if (var_path) { - FILE *var_file = fopen(var_path, "r"); + FILE *var_file; + + var_file = fopen(var_path, "r"); free((void*)var_path); if (var_file) { - if (fscanf(var_file, "%zi", out) == 1) { + va_list va_args; + + va_start(va_args, format); + + if (vfscanf(var_file, format, va_args) == arg_count) { res = 1; } + + va_end(va_args); + fclose(var_file); } } @@ -1197,35 +1215,44 @@ static int read_cgroup_var(const char *group_path, const char *var_name, static int read_cpu_quota(int limit) { - const char *cgroup_path = get_cgroup_path("cpu"); + ssize_t cfs_period_us, cfs_quota_us; + const char *cgroup_path; + int succeeded; - if (cgroup_path) { - ssize_t cfs_period_us, cfs_quota_us; - int succeeded; + switch (get_cgroup_path("cpu", &cgroup_path)) { + case ERTS_CGROUP_V1: + succeeded = read_cgroup_interface(cgroup_path, "/cpu.cfs_quota_us", + 1, "%zi", &cfs_quota_us) && + read_cgroup_interface(cgroup_path, "/cpu.cfs_period_us", + 1, "%zi", &cfs_period_us); - cfs_period_us = -1; - cfs_quota_us = -1; - - succeeded = - read_cgroup_var(cgroup_path, "/cpu.cfs_quota_us", &cfs_quota_us) && - read_cgroup_var(cgroup_path, "/cpu.cfs_period_us", &cfs_period_us); + free((void*)cgroup_path); + break; + case ERTS_CGROUP_V2: + succeeded = read_cgroup_interface(cgroup_path, "/cpu.max", + 2, "%zi %zi", &cfs_quota_us, &cfs_period_us); free((void*)cgroup_path); + break; + default: + succeeded = 0; + break; + } - if (succeeded) { - if (cfs_period_us > 0 && cfs_quota_us > 0) { - size_t quota = cfs_quota_us / cfs_period_us; + if (succeeded) { + if (cfs_period_us > 0 && cfs_quota_us > 0) { + size_t quota = cfs_quota_us / cfs_period_us; - if (quota == 0) { - quota = 1; - } - if (quota > 0 && quota <= (size_t)limit) { - return quota; - } + if (quota == 0) { + quota = 1; } - return limit; + if (quota > 0 && quota <= (size_t)limit) { + return quota; + } } + + return limit; } return 0; diff --git a/erts/vsn.mk b/erts/vsn.mk index cc7958578f..ef6d146974 100644 --- a/erts/vsn.mk +++ b/erts/vsn.mk @@ -18,7 +18,7 @@ # %CopyrightEnd% # -VSN = 11.1.8 +VSN = 11.2 # Port number 4365 in 4.2 # Port number 4366 in 4.3 diff --git a/lib/common_test/doc/src/ct.xml b/lib/common_test/doc/src/ct.xml index c1f638b580..366d856e08 100644 --- a/lib/common_test/doc/src/ct.xml +++ b/lib/common_test/doc/src/ct.xml @@ -141,7 +141,7 @@ <desc><marker id="add_config-2"/> <p>Loads configuration variables using the specified callback module and configuration string. The callback module is to be either loaded or - present in the code part. Loaded configuration variables can later + present in the code path. Loaded configuration variables can later be removed using function <seemfa marker="#remove_config/2"><c>ct:remove_config/2</c></seemfa>. </p> @@ -166,7 +166,7 @@ <seemfa marker="#break/2"><c>ct:break/2</c></seemfa> is to be called instead.</p> <p>A cancelled timetrap is not automatically reactivated after the - break, but must be started exlicitly with + break, but must be started explicitly with <seemfa marker="#timetrap/1"><c>ct:timetrap/1</c></seemfa>.</p> <p>In order for the break/continue functionality to work, <c>Common Test</c> must release the shell process controlling <c>stdin</c>. @@ -1090,7 +1090,7 @@ <v>Reason = term()</v> </type> <desc><marker id="remove_config-2"/> - <p>Removes configuration variables (together wih their aliases) + <p>Removes configuration variables (together with their aliases) that were loaded with specified callback module and configuration string.</p> </desc> @@ -1435,7 +1435,7 @@ </type> <desc><marker id="step-4"/> <p>Steps through a test case with the debugger. If option - <c>config</c> has been specifed, breakpoints are also set on + <c>config</c> has been specified, breakpoints are also set on the configuration functions in <c>Suite</c>.</p> <p>See also <seemfa marker="#run/3"><c>ct:run/3</c></seemfa>.</p> diff --git a/lib/common_test/doc/src/notes.xml b/lib/common_test/doc/src/notes.xml index c4f5c7bb11..1ba1765347 100644 --- a/lib/common_test/doc/src/notes.xml +++ b/lib/common_test/doc/src/notes.xml @@ -33,6 +33,21 @@ <file>notes.xml</file> </header> +<section><title>Common_Test 1.20</title> + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + Various address sanitizer support.</p> + <p> + Own Id: OTP-16959 Aux Id: PR-2965 </p> + </item> + </list> + </section> + +</section> + <section><title>Common_Test 1.19.1</title> <section><title>Improvements and New Features</title> diff --git a/lib/common_test/src/test_server.erl b/lib/common_test/src/test_server.erl index 20c229dde4..b3cc2af145 100644 --- a/lib/common_test/src/test_server.erl +++ b/lib/common_test/src/test_server.erl @@ -21,7 +21,7 @@ -define(DEFAULT_TIMETRAP_SECS, 60). %%% TEST_SERVER_CTRL INTERFACE %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% --export([run_test_case_apply/1,init_target_info/0,init_valgrind/0]). +-export([run_test_case_apply/1,init_target_info/0,init_memory_checker/0]). -export([cover_compile/1,cover_analyse/2]). %%% TEST_SERVER_SUP INTERFACE %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -48,10 +48,8 @@ -export([is_cover/0,is_debug/0,is_commercial/0]). -export([break/1,break/2,break/3,continue/0,continue/1]). +-export([memory_checker/0, is_valgrind/0, is_asan/0]). -%%% DEBUGGER INTERFACE %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% --export([valgrind_new_leaks/0, valgrind_format/2, - is_valgrind/0]). %%% PRIVATE EXPORTED %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -export([]). @@ -60,6 +58,7 @@ -include("test_server_internal.hrl"). -include_lib("kernel/include/file.hrl"). + init_target_info() -> [$.|Emu] = code:objfile_extension(), {_, OTPRel} = init:script_id(), @@ -73,8 +72,8 @@ init_target_info() -> username=test_server_sup:get_username(), cookie=atom_to_list(erlang:get_cookie())}. -init_valgrind() -> - valgrind_new_leaks(). +init_memory_checker() -> + check_memory_leaks(). %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -367,19 +366,50 @@ stick_all_sticky(Node,Sticky) -> %% cover. run_test_case_apply({CaseNum,Mod,Func,Args,Name,RunInit,TimetrapData}) -> - case is_valgrind() of - false -> - ok; - true -> - valgrind_format("Test case #~w ~w:~w/1", [CaseNum, Mod, Func]), - os:putenv("VALGRIND_LOGFILE_INFIX",atom_to_list(Mod)++"."++ - atom_to_list(Func)++"-") - end, + MC = case {Func, memory_checker()} of + {init_per_suite, _} -> none; % skip init/end_per_suite/group + {init_per_group, _} -> none; % as CaseNum is always 0 + {end_per_group, _} -> none; + {end_per_suite, _} -> none; + {_, valgrind} -> + valgrind_format("Test case #~w ~w:~w/1", [CaseNum, Mod, Func]), + os:putenv("VALGRIND_LOGFILE_INFIX",atom_to_list(Mod)++"."++ + atom_to_list(Func)++"-"), + valgrind; + {_, asan} -> + %% Address sanitizer does not support printf in log file + %% but it lets us change the log file on the fly. So we use + %% that to give each test case its own log file. + case asan_take_logpath() of + false -> false; + {LogPath, OtherOpts} -> + LogDir = filename:dirname(LogPath), + LogFile = filename:basename(LogPath), + [Exe, App | _ ] = string:lexemes(LogFile, "-"), + NewLogFile = io_lib:format("~s-~s-tc-~4..0w-~w-~w", + [Exe,App,CaseNum, Mod, Func]), + NewLogPath = filename:join(LogDir, NewLogFile), + + %% Do leak check and then change asan log file + %% for this running beam executable. + erlang:system_info({memory_checker, check_leaks}), + _PrevLog = erlang:system_info({memory_checker, log, NewLogPath}), + + %% Set log file name for subnodes + %% that may be created by this test case + NewOpts = asan_make_opts(["log_path="++NewLogPath++".subnode" + | OtherOpts]), + os:putenv("ASAN_OPTIONS", NewOpts) + end, + asan; + {_, none} -> + node + end, ProcBef = erlang:system_info(process_count), Result = run_test_case_apply(Mod, Func, Args, Name, RunInit, TimetrapData), ProcAft = erlang:system_info(process_count), - valgrind_new_leaks(), + check_memory_leaks(MC), DetFail = get(test_server_detected_fail), {Result,DetFail,ProcBef,ProcAft}. @@ -2053,7 +2083,8 @@ timetrap_scale_factor() -> { 3, fun() -> has_superfluous_schedulers() end}, { 6, fun() -> is_debug() end}, {10, fun() -> is_cover() end}, - {10, fun() -> is_valgrind() end} + {10, fun() -> is_valgrind() end}, + {2, fun() -> is_asan() end} ]). timetrap_scale_factor(Scales) -> @@ -2962,22 +2993,36 @@ is_commercial() -> %% %% Returns true if valgrind is running, else false is_valgrind() -> - case catch erlang:system_info({valgrind, running}) of - {'EXIT', _} -> false; - Res -> Res + memory_checker() =:= valgrind. + +%% Returns true if address-sanitizer is running, else false +is_asan() -> + memory_checker() =:= asan. + +%% Returns the error checker running (valgrind | asan | none). +memory_checker() -> + case catch erlang:system_info({memory_checker, running}) of + {'EXIT', _} -> none; + EC -> EC end. + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% DEBUGGER INTERFACE %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -%% valgrind_new_leaks() -> ok +%% check_memory_leaks() -> ok %% -%% Checks for new memory leaks if Valgrind is active. -valgrind_new_leaks() -> - catch erlang:system_info({valgrind, memory}), +%% Checks for memory leaks if Valgrind or Address-sanitizer is active. +check_memory_leaks() -> + check_memory_leaks(memory_checker()). + +check_memory_leaks(valgrind) -> + catch erlang:system_info({memory_checker, check_leaks}), + ok; +check_memory_leaks(_) -> ok. %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -2987,9 +3032,31 @@ valgrind_new_leaks() -> %% %% Outputs the formatted string to Valgrind's logfile,if Valgrind is active. valgrind_format(Format, Args) -> - (catch erlang:system_info({valgrind, io_lib:format(Format, Args)})), + (catch erlang:system_info({memory_checker, print, io_lib:format(Format, Args)})), ok. +asan_take_logpath() -> + case os:getenv("ASAN_OPTIONS") of + false -> false; + S -> + Opts = string:lexemes(S, ":"), + asan_take_logpath_loop(Opts, []) + end. + +asan_take_logpath_loop(["log_path="++LogPath | T], Acc) -> + {LogPath, T ++ Acc}; +asan_take_logpath_loop([Opt | T], Acc) -> + asan_take_logpath_loop(T, [Opt | Acc]); +asan_take_logpath_loop([], _) -> + false. + +asan_make_opts([A|T]) -> + asan_make_opts(T, A). + +asan_make_opts([], Acc) -> + Acc; +asan_make_opts([A|T], Acc) -> + asan_make_opts(T, A ++ [$: | Acc]). %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% diff --git a/lib/common_test/src/test_server_ctrl.erl b/lib/common_test/src/test_server_ctrl.erl index 995594dd59..dbd5537206 100644 --- a/lib/common_test/src/test_server_ctrl.erl +++ b/lib/common_test/src/test_server_ctrl.erl @@ -2195,7 +2195,7 @@ do_add_end_per_suite_and_skip(LastMod, LastRef, Mod, FwMod) -> %% Runs the specified tests, then displays/logs the summary. run_test_cases(TestSpec, Config, TimetrapData) -> - test_server:init_valgrind(), + test_server:init_memory_checker(), case lists:member(no_src, get(test_server_logopts)) of true -> ok; diff --git a/lib/common_test/src/test_server_node.erl b/lib/common_test/src/test_server_node.erl index 8deea353d7..edfb1fbd92 100644 --- a/lib/common_test/src/test_server_node.erl +++ b/lib/common_test/src/test_server_node.erl @@ -645,7 +645,38 @@ find_release(latest) -> find_release(previous) -> "kaka"; find_release(Rel) -> - find_release(os:type(), Rel). + case find_release(os:type(), Rel) of + none -> + find_release_path(Rel); + Else -> + Else + end. + +find_release_path(Rel) -> + Paths = string:lexemes(os:getenv("PATH"), ":"), + find_release_path(Paths, Rel). +find_release_path([Path|T], Rel) -> + case os:find_executable("erl", Path) of + false -> + find_release_path(T, Rel); + ErlExec -> + Pattern = filename:join([Path,"..","releases","*","OTP_VERSION"]), + case filelib:wildcard(Pattern) of + [VersionFile] -> + {ok, VsnBin} = file:read_file(VersionFile), + [MajorVsn|_] = string:lexemes(VsnBin, "."), + case unicode:characters_to_list(MajorVsn) of + Rel -> + ErlExec; + _Else -> + find_release_path(T, Rel) + end; + _Else -> + find_release_path(T, Rel) + end + end; +find_release_path([], _) -> + none. find_release({unix,sunos}, Rel) -> case os:cmd("uname -p") of diff --git a/lib/common_test/test_server/ts_run.erl b/lib/common_test/test_server/ts_run.erl index 7e12b9652c..ce454dce9c 100644 --- a/lib/common_test/test_server/ts_run.erl +++ b/lib/common_test/test_server/ts_run.erl @@ -197,17 +197,25 @@ make_command(Vars, Spec, State) -> {ok,Cwd} = file:get_cwd(), TestDir = State#state.test_dir, TestPath = filename:nativename(TestDir), - Erl = case os:getenv("TS_RUN_VALGRIND") of + Erl = case os:getenv("TS_RUN_EMU") of false -> ct:get_progname(); - _ -> + "valgrind" -> case State#state.file of Dir when is_list(Dir) -> os:putenv("VALGRIND_LOGFILE_PREFIX", Dir++"-"); _ -> ok end, - "cerl -valgrind" + "cerl -valgrind"; + "asan" -> + case State#state.file of + App when is_list(App) -> + os:putenv("ASAN_LOGFILE_PREFIX", App); + _ -> + ok + end, + "cerl -asan" end, Naming = case ts_lib:var(longnames, Vars) of @@ -261,9 +269,10 @@ run_batch(Vars, _Spec, State) -> ts_lib:progress(Vars, 1, "Command: ~ts~n", [Command]), io:format(user, "Command: ~ts~n",[Command]), Port = open_port({spawn, Command}, [stream, in, eof, exit_status]), - Timeout = 30000 * case os:getenv("TS_RUN_VALGRIND") of + Timeout = 30000 * case os:getenv("TS_RUN_EMU") of false -> 1; - _ -> 100 + "valgrind" -> 100; + "asan" -> 2 end, tricky_print_data(Port, Timeout). diff --git a/lib/common_test/vsn.mk b/lib/common_test/vsn.mk index 1994d75518..d1e8e2e8ba 100644 --- a/lib/common_test/vsn.mk +++ b/lib/common_test/vsn.mk @@ -1 +1 @@ -COMMON_TEST_VSN = 1.19.1 +COMMON_TEST_VSN = 1.20 diff --git a/lib/compiler/doc/src/notes.xml b/lib/compiler/doc/src/notes.xml index b9978401a5..0d8fafb523 100644 --- a/lib/compiler/doc/src/notes.xml +++ b/lib/compiler/doc/src/notes.xml @@ -32,6 +32,28 @@ <p>This document describes the changes made to the Compiler application.</p> +<section><title>Compiler 7.6.7</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p>Fixed a bug in the type optimization pass that could + yield incorrect values or cause the wrong clauses to be + executed.</p> + <p> + Own Id: OTP-17073</p> + </item> + <item> + <p>Fixed a bug in the validator that could cause it to + reject valid code.</p> + <p> + Own Id: OTP-17126 Aux Id: ERL-1471 </p> + </item> + </list> + </section> + +</section> + <section><title>Compiler 7.6.6</title> <section><title>Fixed Bugs and Malfunctions</title> @@ -287,6 +309,22 @@ </section> +<section><title>Compiler 7.5.4.3</title> + + <section><title>Improvements and New Features</title> + <list> + <item> + <p>Fixed a bug in the type optimization pass that could + yield incorrect values or cause the wrong clauses to be + executed.</p> + <p> + Own Id: OTP-17073</p> + </item> + </list> + </section> + +</section> + <section><title>Compiler 7.5.4.2</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/compiler/src/beam_validator.erl b/lib/compiler/src/beam_validator.erl index b5295147e5..055debe620 100644 --- a/lib/compiler/src/beam_validator.erl +++ b/lib/compiler/src/beam_validator.erl @@ -191,8 +191,6 @@ validate_0([{function, Name, Arity, Entry, Code} | Fs], Module, Level, Ft) -> h=0, %%Available heap size for floats. hf=0, - %% Floating point state. - fls=undefined, %% List of hot catch/try tags ct=[], %% Previous instruction was setelement/3. @@ -303,8 +301,9 @@ init_function_args(-1, Vst) -> init_function_args(X, Vst) -> init_function_args(X - 1, create_term(any, argument, [], {x,X}, Vst)). -kill_heap_allocation(St) -> - St#st{h=0,hf=0}. +kill_heap_allocation(#vst{current=St0}=Vst) -> + St = St0#st{h=0,hf=0}, + Vst#vst{current=St}. validate_branches(MFA, Vst) -> #vst{ branched=Targets0, labels=Labels0 } = Vst, @@ -393,7 +392,6 @@ vi({fmove,Src,{fr,_}=Dst}, Vst) -> set_freg(Dst, Vst); vi({fmove,{fr,_}=Src,Dst}, Vst0) -> assert_freg_set(Src, Vst0), - assert_fls(checked, Vst0), Vst = eat_heap_float(Vst0), create_term(#t_float{}, fmove, [], Dst, Vst); vi({kill,Reg}, Vst) -> @@ -680,9 +678,8 @@ vi({gc_bif,Op,{f,Fail},Live,Ss,Dst}, Vst0) -> %% Heap allocations and X registers are killed regardless of whether we %% fail or not, as we may fail after GC. - #vst{current=St0} = Vst0, - St = kill_heap_allocation(St0), - Vst = prune_x_regs(Live, Vst0#vst{current=St}), + Vst1 = kill_heap_allocation(Vst0), + Vst = prune_x_regs(Live, Vst1), validate_bif(gc_bif, Op, Fail, Ss, Dst, Vst0, Vst); @@ -728,13 +725,10 @@ vi({wait,{f,Lbl}}, Vst) -> vi({wait_timeout,{f,Lbl},Src}, Vst0) -> assert_no_exception(Lbl), - %% Note that the receive marker is not cleared since we may re-enter the - %% loop while waiting. If we time out we'll be transferred to a timeout - %% instruction that clears the marker. assert_term(Src, Vst0), verify_y_init(Vst0), - Vst = branch(Lbl, prune_x_regs(0, Vst0)), + Vst = branch(Lbl, schedule_out(0, Vst0)), branch(?EXCEPTION_LABEL, Vst); %% @@ -771,14 +765,15 @@ vi({try_end,Reg}, #vst{current=#st{ct=[Tag|_]}}=Vst) -> vi({try_case,Reg}, #vst{current=#st{ct=[Tag|_]}}=Vst0) -> case get_tag_type(Reg, Vst0) of {trytag,_Fail}=Tag -> - %% Kill the catch tag, all x registers, and the receive marker. + %% Kill the catch tag and all other state (as if we've been + %% scheduled out with no live registers). Only previously allocated + %% Y registers are alive at this point. Vst1 = kill_catch_tag(Reg, Vst0), - Vst2 = prune_x_regs(0, Vst1), - Vst3 = set_receive_marker(none, Vst2), + Vst2 = schedule_out(0, Vst1), %% Class:Error:Stacktrace - Vst4 = create_term(#t_atom{}, try_case, [], {x,0}, Vst3), - Vst = create_term(any, try_case, [], {x,1}, Vst4), + Vst3 = create_term(#t_atom{}, try_case, [], {x,0}, Vst2), + Vst = create_term(any, try_case, [], {x,1}, Vst3), create_term(any, try_case, [], {x,2}, Vst); Type -> error({wrong_tag_type,Type}) @@ -932,26 +927,13 @@ vi({fconv,Src,{fr,_}=Dst}, Vst) -> assert_term(Src, Vst), branch(?EXCEPTION_LABEL, Vst, - fun(FailVst) -> - %% This is a hack to supress assert_float_checked/1 in - %% fork_state/2, since this instruction is legal even when - %% the state is unchecked. - set_fls(checked, FailVst) - end, fun(SuccVst0) -> SuccVst = update_type(fun meet/2, number, Src, SuccVst0), set_freg(Dst, SuccVst) end); vi(fclearerror, Vst) -> - case get_fls(Vst) of - undefined -> ok; - checked -> ok; - Fls -> error({bad_floating_point_state,Fls}) - end, - set_fls(cleared, Vst); -vi({fcheckerror,_}, Vst0) -> - assert_fls(cleared, Vst0), - Vst = set_fls(checked, Vst0), + Vst; +vi({fcheckerror, _}, Vst) -> branch(?EXCEPTION_LABEL, Vst); %% @@ -1125,8 +1107,6 @@ validate_var_info([], _Reg, Vst) -> %% The stackframe must have a known size and be initialized. %% Does not return to the instruction following the call. validate_tail_call(Deallocate, Func, Live, #vst{current=#st{numy=NumY}}=Vst0) -> - assert_float_checked(Vst0), - verify_y_init(Vst0), verify_live(Live, Vst0), verify_call_args(Func, Live, Vst0), @@ -1153,18 +1133,15 @@ validate_tail_call(Deallocate, Func, Live, #vst{current=#st{numy=NumY}}=Vst0) -> %% The instruction will return to the instruction following the call. validate_body_call(Func, Live, #vst{current=#st{numy=NumY}}=Vst) when is_integer(NumY)-> - assert_float_checked(Vst), - verify_y_init(Vst), verify_live(Live, Vst), verify_call_args(Func, Live, Vst), - SuccFun = fun(#vst{current=St0}=SuccVst0) -> + SuccFun = fun(SuccVst0) -> {RetType, _, _} = call_types(Func, Live, SuccVst0), true = RetType =/= none, %Assertion. - St = St0#st{f=init_fregs()}, - SuccVst = prune_x_regs(0, SuccVst0#vst{current=St}), + SuccVst = schedule_out(0, SuccVst0), create_term(RetType, call, [], {x,0}, SuccVst) end, @@ -1180,13 +1157,6 @@ validate_body_call(Func, Live, validate_body_call(_, _, #vst{current=#st{numy=NumY}}) -> error({allocated, NumY}). -assert_float_checked(Vst) -> - case get_fls(Vst) of - undefined -> ok; - checked -> ok; - Fls -> error({unsafe_instruction,{float_error_state,Fls}}) - end. - init_try_catch_branch(Kind, Dst, Fail, Vst0) -> assert_no_exception(Fail), @@ -1336,7 +1306,6 @@ verify_return(#vst{current=#st{recv_marker=Mark}}) when Mark =/= none -> %% the message. error({return_with_receive_marker,Mark}); verify_return(Vst) -> - assert_float_checked(Vst), verify_no_ct(Vst), kill_state(Vst). @@ -1349,7 +1318,6 @@ verify_return(Vst) -> %% validate_bif(Kind, Op, Fail, Ss, Dst, OrigVst, Vst) -> - assert_float_checked(Vst), case {will_bif_succeed(Op, Ss, Vst), Fail} of {yes, _} -> %% This BIF cannot fail (neither throw nor branch), make sure it's @@ -1734,22 +1702,29 @@ test_heap(Heap, Live, Vst0) -> heap_alloc(Heap, Vst). heap_alloc(Heap, #vst{current=St0}=Vst) -> - St1 = kill_heap_allocation(St0), - St = heap_alloc_1(Heap, St1), + {HeapWords, Floats} = heap_alloc_1(Heap), + + St = St0#st{h=HeapWords,hf=Floats}, + Vst#vst{current=St}. -heap_alloc_1({alloc,Alloc}, St) -> - heap_alloc_2(Alloc, St); -heap_alloc_1(HeapWords, St) when is_integer(HeapWords) -> - St#st{h=HeapWords}. +heap_alloc_1({alloc, Alloc}) -> + heap_alloc_2(Alloc, 0, 0); +heap_alloc_1(HeapWords) when is_integer(HeapWords) -> + {HeapWords, 0}. -heap_alloc_2([{words,HeapWords}|T], St0) -> - St = St0#st{h=HeapWords}, - heap_alloc_2(T, St); -heap_alloc_2([{floats,Floats}|T], St0) -> - St = St0#st{hf=Floats}, - heap_alloc_2(T, St); -heap_alloc_2([], St) -> St. +heap_alloc_2([{words, HeapWords} | T], 0, Floats) -> + heap_alloc_2(T, HeapWords, Floats); +heap_alloc_2([{floats, Floats} | T], HeapWords, 0) -> + heap_alloc_2(T, HeapWords, Floats); +heap_alloc_2([], HeapWords, Floats) -> + {HeapWords, Floats}. + +schedule_out(Live, Vst0) when is_integer(Live) -> + Vst1 = prune_x_regs(Live, Vst0), + Vst2 = kill_heap_allocation(Vst1), + Vst = kill_fregs(Vst2), + set_receive_marker(none, Vst). prune_x_regs(Live, #vst{current=St0}=Vst) when is_integer(Live) -> #st{fragile=Fragile0,xs=Xs0} = St0, @@ -1788,22 +1763,10 @@ assert_arities(_) -> error(bad_tuple_arity_list). %%% -%%% Floating point checking. -%%% -%%% Possible values for the fls field (=floating point error state). -%%% -%%% undefined - Undefined (initial state). No float operations allowed. -%%% -%%% cleared - fclearerror/0 has been executed. Float operations -%%% are allowed (such as fadd). -%%% -%%% checked - fcheckerror/1 has been executed. It is allowed to -%%% move values out of floating point registers. -%%% -%%% The following instructions may be executed in any state: +%%% Floating point helpers. %%% -%%% fconv Src {fr,_} -%%% fmove Src {fr,_} %% Move INTO floating point register. +%%% fconv Src {fr,_} +%%% fmove Src {fr,_} %% Move known float INTO floating point register. %%% is_float_arith_bif(fadd, [_, _]) -> true; @@ -1813,25 +1776,16 @@ is_float_arith_bif(fnegate, [_]) -> true; is_float_arith_bif(fsub, [_, _]) -> true; is_float_arith_bif(_, _) -> false. -validate_float_arith_bif(Ss, Dst, Vst0) -> - _ = [assert_freg_set(S, Vst0) || S <- Ss], - assert_fls(cleared, Vst0), - Vst = set_fls(cleared, Vst0), +validate_float_arith_bif(Ss, Dst, Vst) -> + _ = [assert_freg_set(S, Vst) || S <- Ss], set_freg(Dst, Vst). -assert_fls(Fls, Vst) -> - case get_fls(Vst) of - Fls -> ok; - OtherFls -> error({bad_floating_point_state,OtherFls}) - end. - -set_fls(Fls, #vst{current=#st{}=St}=Vst) when is_atom(Fls) -> - Vst#vst{current=St#st{fls=Fls}}. - -get_fls(#vst{current=#st{fls=Fls}}) when is_atom(Fls) -> Fls. - init_fregs() -> 0. +kill_fregs(#vst{current=St0}=Vst) -> + St = St0#st{f=init_fregs()}, + Vst#vst{current=St}. + set_freg({fr,Fr}=Freg, #vst{current=#st{f=Fregs0}=St}=Vst) -> check_limit(Freg), Bit = 1 bsl Fr, @@ -2276,7 +2230,7 @@ new_value(Type, Op, Ss, #vst{current=#st{vs=Vs0}=St,ref_ctr=Counter}=Vst) -> {Ref, Vst#vst{current=St#st{vs=Vs},ref_ctr=Counter+1}}. kill_catch_tag(Reg, #vst{current=#st{ct=[Tag|Tags]}=St}=Vst0) -> - Vst = Vst0#vst{current=St#st{ct=Tags,fls=undefined}}, + Vst = Vst0#vst{current=St#st{ct=Tags}}, Tag = get_tag_type(Reg, Vst), %Assertion. kill_tag(Reg, Vst). @@ -2535,10 +2489,6 @@ branch(Fail, Vst) -> fork_state(?EXCEPTION_LABEL, Vst0) -> #vst{current=#st{ct=CatchTags,numy=NumY}} = Vst0, - %% Floating-point exceptions must be checked before any other kind of - %% exception can be raised. - assert_float_checked(Vst0), - %% The stack will be scanned looking for a catch tag, so all Y registers %% must be initialized. verify_y_init(Vst0), diff --git a/lib/compiler/test/beam_validator_SUITE.erl b/lib/compiler/test/beam_validator_SUITE.erl index 6f6391d3ba..38be30b165 100644 --- a/lib/compiler/test/beam_validator_SUITE.erl +++ b/lib/compiler/test/beam_validator_SUITE.erl @@ -28,7 +28,7 @@ dead_code/1, overwrite_catchtag/1,overwrite_trytag/1,accessing_tags/1,bad_catch_try/1, cons_guard/1, - freg_range/1,freg_uninit/1,freg_state/1, + freg_range/1,freg_uninit/1, bad_bin_match/1,bad_dsetel/1, state_after_fault_in_catch/1,no_exception_in_catch/1, undef_label/1,illegal_instruction/1,failing_gc_guard_bif/1, @@ -63,7 +63,7 @@ groups() -> unsafe_catch,dead_code, overwrite_catchtag,overwrite_trytag,accessing_tags, bad_catch_try,cons_guard,freg_range,freg_uninit, - freg_state,bad_bin_match,bad_dsetel, + bad_bin_match,bad_dsetel, state_after_fault_in_catch,no_exception_in_catch, undef_label,illegal_instruction,failing_gc_guard_bif, map_field_lists,cover_bin_opt,val_dsetel, @@ -290,28 +290,6 @@ freg_uninit(Config) when is_list(Config) -> {uninitialized_reg,{fr,0}}}}] = Errors, ok. -freg_state(Config) when is_list(Config) -> - Errors = do_val(freg_state, Config), - [{{t,sum_1,2}, - {{bif,fmul,{f,0},[{fr,0},{fr,1}],{fr,0}}, - 6, - {bad_floating_point_state,undefined}}}, - {{t,sum_2,2}, - {{fmove,{fr,0},{x,0}}, - 8, - {bad_floating_point_state,cleared}}}, - {{t,sum_3,2}, - {{bif,'-',{f,0},[{x,1},{x,0}],{x,1}}, - 8, - {unsafe_instruction,{float_error_state,cleared}}}}, - {{t,sum_4,2}, - {{fcheckerror,{f,0}}, - 4, - {bad_floating_point_state,undefined}}}, - {{t,sum_5,2}, - {fclearerror,5,{bad_floating_point_state,cleared}}}] = Errors, - ok. - bad_bin_match(Config) when is_list(Config) -> [{{t,t,1},{return,5,{match_context,{x,0}}}}] = do_val(bad_bin_match, Config), diff --git a/lib/compiler/test/beam_validator_SUITE_data/freg_state.S b/lib/compiler/test/beam_validator_SUITE_data/freg_state.S deleted file mode 100644 index 7466763482..0000000000 --- a/lib/compiler/test/beam_validator_SUITE_data/freg_state.S +++ /dev/null @@ -1,59 +0,0 @@ -{module, freg_state}. %% version = 0 - -{exports, [{sum_1,2},{sum_2,2},{sum_3,2},{sum_4,2},{sum_5,2}]}. - -{attributes, []}. - - -{function, sum_1, 2, 2}. - {label,1}. - {func_info,{atom,t},{atom,sum_1},2}. - {label,2}. - {fconv,{x,0},{fr,0}}. - {fconv,{x,1},{fr,1}}. - {bif,fmul,{f,0},[{fr,0},{fr,1}],{fr,0}}. - {'%live',1}. - return. - -{function, sum_2, 2, 4}. - {label,3}. - {func_info,{atom,t},{atom,sum_2},2}. - {label,4}. - {fconv,{x,0},{fr,0}}. - {fconv,{x,1},{fr,1}}. - fclearerror. - {bif,fmul,{f,0},[{fr,0},{fr,1}],{fr,0}}. - {fmove,{fr,0},{x,0}}. - {'%live',1}. - return. - -{function, sum_3, 2, 6}. - {label,5}. - {func_info,{atom,t},{atom,sum_3},2}. - {label,6}. - {fconv,{x,0},{fr,0}}. - {fconv,{x,1},{fr,1}}. - fclearerror. - {bif,fmul,{f,0},[{fr,0},{fr,1}],{fr,0}}. - {bif,'-',{f,0},[{x,1},{x,0}],{x,1}}. - {fcheckerror,{f,0}}. - {fmove,{fr,0},{x,0}}. - {'%live',1}. - return. - -{function, sum_4, 2, 8}. - {label,6}. - {func_info,{atom,t},{atom,sum_4},2}. - {label,8}. - {fcheckerror,{f,0}}. - {fmove,{fr,0},{x,0}}. - {'%live',1}. - return. - -{function, sum_5, 2, 10}. - {label,9}. - {func_info,{atom,t},{atom,sum_5},2}. - {label,10}. - fclearerror. - fclearerror. - return. diff --git a/lib/compiler/test/float_SUITE.erl b/lib/compiler/test/float_SUITE.erl index 586dfe8102..bf154eeb62 100644 --- a/lib/compiler/test/float_SUITE.erl +++ b/lib/compiler/test/float_SUITE.erl @@ -22,7 +22,7 @@ init_per_group/2,end_per_group/2, pending/1,bif_calls/1,math_functions/1,mixed_float_and_int/1, subtract_number_type/1,float_followed_by_guard/1, - fconv_line_numbers/1]). + fconv_line_numbers/1,exception_signals/1]). -include_lib("common_test/include/ct.hrl"). @@ -31,7 +31,8 @@ suite() -> [{ct_hooks,[ts_install_cth]}]. all() -> [pending, bif_calls, math_functions, mixed_float_and_int, subtract_number_type, - float_followed_by_guard,fconv_line_numbers]. + float_followed_by_guard,fconv_line_numbers, + exception_signals]. groups() -> []. @@ -220,5 +221,22 @@ fconv_line_numbers_1(A) -> false end, Stacktrace). +%% ERL-1471: compiler generated invalid 'fclearerror' / 'fcheckerror' +%% sequences. +exception_signals(Config) when is_list(Config) -> + 2.0 = exception_signals_1(id(25), id(true), []), + 2.0 = exception_signals_1(id(25), id(false), []), + 2.0 = exception_signals_1(id(25.0), id(true), []), + 2.0 = exception_signals_1(id(25.0), id(false), []), + ok. + +exception_signals_1(Width, Value, _Opts) -> + Height = Width / 25.0, + _Middle = case Value of + true -> Width / 2.0; + false -> 0 + end, + _More = Height + 1. + id(I) -> I. diff --git a/lib/compiler/vsn.mk b/lib/compiler/vsn.mk index 421b5ac2a1..c628ece88c 100644 --- a/lib/compiler/vsn.mk +++ b/lib/compiler/vsn.mk @@ -1 +1 @@ -COMPILER_VSN = 7.6.6 +COMPILER_VSN = 7.6.7 diff --git a/lib/crypto/c_src/Makefile.in b/lib/crypto/c_src/Makefile.in index ecccc33d8d..0821bd8d00 100644 --- a/lib/crypto/c_src/Makefile.in +++ b/lib/crypto/c_src/Makefile.in @@ -59,11 +59,17 @@ TYPEMARKER = .gprof TYPE_EXTRA_CFLAGS = -DGPROF -pg TYPE_FLAGS = $(CFLAGS) $(TYPE_EXTRA_CFLAGS) else +ifeq ($(TYPE),asan) +TYPEMARKER = .asan +TYPE_FLAGS = $(CFLAGS) -fsanitize=address -fsanitize-recover=address -fno-omit-frame-pointer -DADDRESS_SANITIZER +LDFLAGS += -fsanitize=address +else TYPEMARKER = TYPE_FLAGS = $(CFLAGS) endif endif endif +endif # ---------------------------------------------------- # Release directory specification @@ -159,7 +165,7 @@ ALL_STATIC_CFLAGS = @DED_STATIC_CFLAGS@ $(TYPE_EXTRA_CFLAGS) $(CONFIGURE_ARGS) $ _create_dirs := $(shell mkdir -p $(OBJDIR) $(LIBDIR)) -debug opt valgrind: $(NIF_LIB) $(CALLBACK_LIB) $(TEST_ENGINE_LIB) +debug opt valgrind asan: $(NIF_LIB) $(CALLBACK_LIB) $(TEST_ENGINE_LIB) static_lib: $(NIF_ARCHIVE) diff --git a/lib/crypto/c_src/crypto.c b/lib/crypto/c_src/crypto.c index b88413d873..4559a15b07 100644 --- a/lib/crypto/c_src/crypto.c +++ b/lib/crypto/c_src/crypto.c @@ -163,8 +163,8 @@ static int initialize(ErlNifEnv* env, ERL_NIF_TERM load_info) const ERL_NIF_TERM* tpl_array; int vernum; ErlNifBinary lib_bin; - char lib_buf[1000]; #ifdef HAVE_DYNAMIC_CRYPTO_LIB + char lib_buf[1000]; void *handle; #endif diff --git a/lib/crypto/c_src/crypto_callback.c b/lib/crypto/c_src/crypto_callback.c index 0244952a65..53b4bbf1e0 100644 --- a/lib/crypto/c_src/crypto_callback.c +++ b/lib/crypto/c_src/crypto_callback.c @@ -106,9 +106,8 @@ static void crypto_free(void* ptr CCB_FILE_LINE_ARGS) #ifdef OPENSSL_THREADS /* vvvvvvvvvvvvvvv OPENSSL_THREADS vvvvvvvvvvvvvvvv */ -#if OPENSSL_VERSION_NUMBER < 0x10100000 +#if OPENSSL_VERSION_NUMBER < PACKED_OPENSSL_VERSION_PLAIN(1,1,0) static ErlNifRWLock** lock_vec = NULL; /* Static locks used by openssl */ -#endif #include <openssl/crypto.h> @@ -132,8 +131,6 @@ static INLINE void locking(int mode, ErlNifRWLock* lock) } } -#if OPENSSL_VERSION_NUMBER < PACKED_OPENSSL_VERSION_PLAIN(1,1,0) - /* TODO: there should be an enif_atomic32_add_return() */ typedef int (*add_lock_function_t)(int *var, int incr, int type, const char *file, int line); @@ -192,21 +189,18 @@ DLLEXPORT struct crypto_callbacks* get_crypto_callbacks(int nlocks) &crypto_realloc, &crypto_free, -#if OPENSSL_VERSION_NUMBER < 0x10100000 -#ifdef OPENSSL_THREADS +#if defined OPENSSL_THREADS && OPENSSL_VERSION_NUMBER < PACKED_OPENSSL_VERSION_PLAIN(1,1,0) NULL, /* add_lock_function, filled in below */ &locking_function, &id_function, &dyn_create_function, &dyn_lock_function, &dyn_destroy_function -#endif /* OPENSSL_THREADS */ -#endif +#endif /* OPENSSL_THREADS && PACKED_OPENSSL_VERSION_PLAIN(1,1,0) */ }; if (!is_initialized) { -#if OPENSSL_VERSION_NUMBER < 0x10100000 -#ifdef OPENSSL_THREADS +#if defined OPENSSL_THREADS && OPENSSL_VERSION_NUMBER < PACKED_OPENSSL_VERSION_PLAIN(1,1,0) the_struct.add_lock_function = get_add_lock_function(); if (nlocks > 0) { int i; @@ -223,18 +217,15 @@ DLLEXPORT struct crypto_callbacks* get_crypto_callbacks(int nlocks) goto err; } } -#endif -#endif +#endif /* OPENSSL_THREADS && PACKED_OPENSSL_VERSION_PLAIN(1,1,0) */ is_initialized = 1; } return &the_struct; -#if OPENSSL_VERSION_NUMBER < 0x10100000 -#ifdef OPENSSL_THREADS +#if defined OPENSSL_THREADS && OPENSSL_VERSION_NUMBER < PACKED_OPENSSL_VERSION_PLAIN(1,1,0) err: return NULL; #endif -#endif } #ifdef HAVE_DYNAMIC_CRYPTO_LIB diff --git a/lib/crypto/c_src/engine.c b/lib/crypto/c_src/engine.c index b2fdcfaa9f..82ba50e4bd 100644 --- a/lib/crypto/c_src/engine.c +++ b/lib/crypto/c_src/engine.c @@ -23,6 +23,7 @@ #ifdef HAS_ENGINE_SUPPORT struct engine_ctx { ENGINE *engine; + int is_functional; char *id; }; @@ -44,6 +45,12 @@ static void engine_ctx_dtor(ErlNifEnv* env, struct engine_ctx* ctx) { enif_free(ctx->id); } else PRINTF_ERR0(" empty ctx->id=NULL"); + + if (ctx->engine) { + if (ctx->is_functional) + ENGINE_finish(ctx->engine); + ENGINE_free(ctx->engine); + } } int get_engine_and_key_id(ErlNifEnv *env, ERL_NIF_TERM key, char ** id, ENGINE **e) @@ -144,6 +151,7 @@ ERL_NIF_TERM engine_by_id_nif(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[ if ((ctx = enif_alloc_resource(engine_ctx_rtype, sizeof(struct engine_ctx))) == NULL) goto err; ctx->engine = engine; + ctx->is_functional = 0; ctx->id = engine_id; /* ctx now owns engine_id */ engine_id = NULL; @@ -181,7 +189,7 @@ ERL_NIF_TERM engine_init_nif(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[] if (!ENGINE_init(ctx->engine)) return ERROR_Atom(env, "engine_init_failed"); - + ctx->is_functional = 1; return atom_ok; bad_arg: @@ -200,11 +208,13 @@ ERL_NIF_TERM engine_free_nif(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[] // Get Engine ASSERT(argc == 1); - if (!enif_get_resource(env, argv[0], engine_ctx_rtype, (void**)&ctx)) + if (!enif_get_resource(env, argv[0], engine_ctx_rtype, (void**)&ctx) + || ctx->is_functional) goto bad_arg; if (!ENGINE_free(ctx->engine)) goto err; + ctx->engine = NULL; return atom_ok; bad_arg: @@ -223,11 +233,13 @@ ERL_NIF_TERM engine_finish_nif(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv // Get Engine ASSERT(argc == 1); - if (!enif_get_resource(env, argv[0], engine_ctx_rtype, (void**)&ctx)) + if (!enif_get_resource(env, argv[0], engine_ctx_rtype, (void**)&ctx) + || !ctx->is_functional) goto bad_arg; if (!ENGINE_finish(ctx->engine)) goto err; + ctx->is_functional = 0; return atom_ok; bad_arg: @@ -265,7 +277,8 @@ ERL_NIF_TERM engine_ctrl_cmd_strings_nif(ErlNifEnv* env, int argc, const ERL_NIF // Get Engine ASSERT(argc == 3); - if (!enif_get_resource(env, argv[0], engine_ctx_rtype, (void**)&ctx)) + if (!enif_get_resource(env, argv[0], engine_ctx_rtype, (void**)&ctx) + || !ctx->engine) goto bad_arg; PRINTF_ERR1("Engine Id: %s\r\n", ENGINE_get_id(ctx->engine)); @@ -333,7 +346,8 @@ ERL_NIF_TERM engine_add_nif(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv[]) // Get Engine ASSERT(argc == 1); - if (!enif_get_resource(env, argv[0], engine_ctx_rtype, (void**)&ctx)) + if (!enif_get_resource(env, argv[0], engine_ctx_rtype, (void**)&ctx) + || !ctx->engine) goto bad_arg; if (!ENGINE_add(ctx->engine)) @@ -360,7 +374,8 @@ ERL_NIF_TERM engine_remove_nif(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv // Get Engine ASSERT(argc == 1); - if (!enif_get_resource(env, argv[0], engine_ctx_rtype, (void**)&ctx)) + if (!enif_get_resource(env, argv[0], engine_ctx_rtype, (void**)&ctx) + || !ctx->engine) goto bad_arg; if (!ENGINE_remove(ctx->engine)) @@ -387,7 +402,8 @@ ERL_NIF_TERM engine_register_nif(ErlNifEnv* env, int argc, const ERL_NIF_TERM ar // Get Engine ASSERT(argc == 2); - if (!enif_get_resource(env, argv[0], engine_ctx_rtype, (void**)&ctx)) + if (!enif_get_resource(env, argv[0], engine_ctx_rtype, (void**)&ctx) + || !ctx->engine) goto bad_arg; if (!enif_get_uint(env, argv[1], &method)) goto bad_arg; @@ -492,7 +508,8 @@ ERL_NIF_TERM engine_unregister_nif(ErlNifEnv* env, int argc, const ERL_NIF_TERM // Get Engine ASSERT(argc == 2); - if (!enif_get_resource(env, argv[0], engine_ctx_rtype, (void**)&ctx)) + if (!enif_get_resource(env, argv[0], engine_ctx_rtype, (void**)&ctx) + || !ctx->engine) goto bad_arg; if (!enif_get_uint(env, argv[1], &method)) goto bad_arg; @@ -592,6 +609,7 @@ ERL_NIF_TERM engine_get_first_nif(ErlNifEnv* env, int argc, const ERL_NIF_TERM a if ((ctx = enif_alloc_resource(engine_ctx_rtype, sizeof(struct engine_ctx))) == NULL) goto err; + ctx->is_functional = 0; ctx->engine = engine; ctx->id = NULL; @@ -623,10 +641,14 @@ ERL_NIF_TERM engine_get_next_nif(ErlNifEnv* env, int argc, const ERL_NIF_TERM ar // Get Engine ASSERT(argc == 1); - if (!enif_get_resource(env, argv[0], engine_ctx_rtype, (void**)&ctx)) + if (!enif_get_resource(env, argv[0], engine_ctx_rtype, (void**)&ctx) + || !ctx->engine) goto bad_arg; - if ((engine = ENGINE_get_next(ctx->engine)) == NULL) { + engine = ENGINE_get_next(ctx->engine); + ctx->engine = NULL; + + if (engine == NULL) { if (!enif_alloc_binary(0, &engine_bin)) goto err; engine_bin.size = 0; @@ -636,6 +658,7 @@ ERL_NIF_TERM engine_get_next_nif(ErlNifEnv* env, int argc, const ERL_NIF_TERM ar if ((next_ctx = enif_alloc_resource(engine_ctx_rtype, sizeof(struct engine_ctx))) == NULL) goto err; next_ctx->engine = engine; + next_ctx->is_functional = 0; next_ctx->id = NULL; result = enif_make_resource(env, next_ctx); @@ -667,7 +690,8 @@ ERL_NIF_TERM engine_get_id_nif(ErlNifEnv* env, int argc, const ERL_NIF_TERM argv // Get Engine ASSERT(argc == 1); - if (!enif_get_resource(env, argv[0], engine_ctx_rtype, (void**)&ctx)) + if (!enif_get_resource(env, argv[0], engine_ctx_rtype, (void**)&ctx) + || !ctx->engine) goto bad_arg; if ((engine_id = ENGINE_get_id(ctx->engine)) == NULL) { @@ -705,7 +729,8 @@ ERL_NIF_TERM engine_get_name_nif(ErlNifEnv* env, int argc, const ERL_NIF_TERM ar // Get Engine ASSERT(argc == 1); - if (!enif_get_resource(env, argv[0], engine_ctx_rtype, (void**)&ctx)) + if (!enif_get_resource(env, argv[0], engine_ctx_rtype, (void**)&ctx) + || !ctx->engine) goto bad_arg; if ((engine_name = ENGINE_get_name(ctx->engine)) == NULL) { diff --git a/lib/crypto/c_src/info.c b/lib/crypto/c_src/info.c index 573039203c..1d7e744995 100644 --- a/lib/crypto/c_src/info.c +++ b/lib/crypto/c_src/info.c @@ -26,6 +26,8 @@ char *crypto_callback_name = "crypto_callback.debug"; # elif defined(VALGRIND) char *crypto_callback_name = "crypto_callback.valgrind"; +# elif defined(ADDRESS_SANITIZER) +char *crypto_callback_name = "crypto_callback.asan"; # else char *crypto_callback_name = "crypto_callback"; # endif diff --git a/lib/crypto/c_src/openssl_config.h b/lib/crypto/c_src/openssl_config.h index cf63bd6051..1a11ab12fe 100644 --- a/lib/crypto/c_src/openssl_config.h +++ b/lib/crypto/c_src/openssl_config.h @@ -120,7 +120,7 @@ #endif #if defined(HAS_EVP_PKEY_CTX) \ - && OPENSSL_VERSION_NUMBER < PACKED_OPENSSL_VERSION_PLAIN(1,0,2) + && OPENSSL_VERSION_NUMBER < PACKED_OPENSSL_VERSION_PLAIN(1,1,0) /* EVP is slow on antique crypto libs. * DISABLE_EVP_* is 0 or 1 from the configure script */ diff --git a/lib/crypto/c_src/otp_test_engine.c b/lib/crypto/c_src/otp_test_engine.c index 6416925b05..f5fff85b14 100644 --- a/lib/crypto/c_src/otp_test_engine.c +++ b/lib/crypto/c_src/otp_test_engine.c @@ -375,8 +375,6 @@ int pem_passwd_cb_fun(char *buf, int size, int rwflag, void *password) return 0; } -#endif - #if defined(FAKE_RSA_IMPL) /* RSA sign. This returns a fixed string so the test case can test that it was called instead of the cryptolib default RSA sign */ @@ -454,3 +452,5 @@ static int test_rsa_free(RSA *rsa) } #endif /* if defined(FAKE_RSA_IMPL) */ + +#endif /* if defined(HAVE_EC) */ diff --git a/lib/crypto/doc/src/Makefile b/lib/crypto/doc/src/Makefile index f48a79e8d1..b4926d6d7c 100644 --- a/lib/crypto/doc/src/Makefile +++ b/lib/crypto/doc/src/Makefile @@ -48,4 +48,4 @@ TOP_SPECS_FILE = specs.xml include $(ERL_TOP)/make/doc.mk -valgrind: +valgrind asan: diff --git a/lib/crypto/doc/src/notes.xml b/lib/crypto/doc/src/notes.xml index d2b65cd275..304e263f88 100644 --- a/lib/crypto/doc/src/notes.xml +++ b/lib/crypto/doc/src/notes.xml @@ -31,6 +31,54 @@ </header> <p>This document describes the changes made to the Crypto application.</p> +<section><title>Crypto 4.9</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Fix minor memory leaks in crypto ENGINE and robustify the + code.</p> + <p> + Own Id: OTP-17212</p> + </item> + <item> + <p> + The otp_test_engine no longer fails if NO_EC* is set in + the OpenSSL configuration.</p> + <p> + Own Id: OTP-17256 Aux Id: PR-4580, GH-4573 </p> + </item> + </list> + </section> + + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + Various address sanitizer support.</p> + <p> + Own Id: OTP-16959 Aux Id: PR-2965 </p> + </item> + <item> + <p> + EVP is now disabled for OpenSSL cryptolib versions up to + and including 1.0.2</p> + <p> + Own Id: OTP-17116 Aux Id: PR-2972 </p> + </item> + <item> + <p> + Warning for unused C function removed</p> + <p> + Own Id: OTP-17145 Aux Id: OTP-17105, PR-2872 </p> + </item> + </list> + </section> + +</section> + <section><title>Crypto 4.8.3</title> <section><title>Fixed Bugs and Malfunctions</title> @@ -242,6 +290,21 @@ </section> +<section><title>Crypto 4.6.5.2</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Adding missing flag in BN-calls in SRP.</p> + <p> + Own Id: OTP-17107</p> + </item> + </list> + </section> + +</section> + <section><title>Crypto 4.6.5.1</title> <section><title>Improvements and New Features</title> @@ -554,6 +617,21 @@ </section> +<section><title>Crypto 4.4.2.3</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Adding missing flag in BN-calls in SRP.</p> + <p> + Own Id: OTP-17107</p> + </item> + </list> + </section> + +</section> + <section><title>Crypto 4.4.2.2</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/crypto/src/Makefile b/lib/crypto/src/Makefile index 1753ba4f36..c3f1c859e5 100644 --- a/lib/crypto/src/Makefile +++ b/lib/crypto/src/Makefile @@ -61,7 +61,7 @@ ERL_COMPILE_FLAGS += -DCRYPTO_VSN=\"$(VSN)\" -Werror -I../include # Targets # ---------------------------------------------------- -debug opt valgrind: $(TARGET_FILES) $(APP_TARGET) $(APPUP_TARGET) +debug opt valgrind asan: $(TARGET_FILES) $(APP_TARGET) $(APPUP_TARGET) clean: rm -f $(TARGET_FILES) $(APP_TARGET) $(APPUP_TARGET) diff --git a/lib/crypto/src/crypto.erl b/lib/crypto/src/crypto.erl index d178ecc28e..b38d13d844 100644 --- a/lib/crypto/src/crypto.erl +++ b/lib/crypto/src/crypto.erl @@ -2985,10 +2985,10 @@ get_test_engine() -> end. check_otp_test_engine(LibDir) -> - case filelib:wildcard("otp_test_engine*", LibDir) of - [] -> + case choose_otp_test_engine(LibDir) of + false -> {error, notexist}; - [LibName|_] -> % In case of Valgrind there could be more than one + LibName -> LibPath = filename:join(LibDir,LibName), case filelib:is_file(LibPath) of true -> @@ -2999,3 +2999,20 @@ check_otp_test_engine(LibDir) -> end. +choose_otp_test_engine(LibDir) -> + LibNames = filelib:wildcard("otp_test_engine.*", LibDir), + Type = atom_to_list(erlang:system_info(build_type)), + choose_otp_test_engine(LibNames, Type, false). + +choose_otp_test_engine([LibName | T], Type, Acc) -> + case string:lexemes(LibName, ".") of + [_, Type, _SO] -> + LibName; %% Choose typed if exists (valgrind,asan) + [_, _SO] -> + %% Fallback on typeless (opt) + choose_otp_test_engine(T, Type, LibName); + _ -> + choose_otp_test_engine(T, Type, Acc) + end; +choose_otp_test_engine([], _, Acc) -> + Acc. diff --git a/lib/crypto/vsn.mk b/lib/crypto/vsn.mk index 0fb42b3993..1aaf84ba37 100644 --- a/lib/crypto/vsn.mk +++ b/lib/crypto/vsn.mk @@ -1 +1 @@ -CRYPTO_VSN = 4.8.3 +CRYPTO_VSN = 4.9 diff --git a/lib/dialyzer/doc/src/notes.xml b/lib/dialyzer/doc/src/notes.xml index 7667ab9459..eac56be316 100644 --- a/lib/dialyzer/doc/src/notes.xml +++ b/lib/dialyzer/doc/src/notes.xml @@ -32,6 +32,21 @@ <p>This document describes the changes made to the Dialyzer application.</p> +<section><title>Dialyzer 4.3.1</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Correct handling of PLTs in the GUI.</p> + <p> + Own Id: OTP-17091</p> + </item> + </list> + </section> + +</section> + <section><title>Dialyzer 4.3</title> <section><title>Improvements and New Features</title> diff --git a/lib/dialyzer/src/dialyzer_gui_wx.erl b/lib/dialyzer/src/dialyzer_gui_wx.erl index f47d90b91f..dd0d1b8979 100644 --- a/lib/dialyzer/src/dialyzer_gui_wx.erl +++ b/lib/dialyzer/src/dialyzer_gui_wx.erl @@ -498,10 +498,10 @@ gui_loop(#gui_state{backend_pid = BackendPid, doc_plt = DocPlt, end, ExplanationPid = spawn_link(Fun), gui_loop(State#gui_state{expl_pid = ExplanationPid}); - {BackendPid, done, NewPlt, NewDocPlt} -> + {BackendPid, done, _NewPlt, NewDocPlt} -> message(State, "Analysis done"), - dialyzer_plt:delete(NewPlt), config_gui_stop(State), + dialyzer_plt:delete(State#gui_state.doc_plt), gui_loop(State#gui_state{doc_plt = NewDocPlt}); {'EXIT', BackendPid, {error, Reason}} -> free_editor(State, ?DIALYZER_ERROR_TITLE, Reason), diff --git a/lib/dialyzer/vsn.mk b/lib/dialyzer/vsn.mk index 3e7cebb1c9..803d121b53 100644 --- a/lib/dialyzer/vsn.mk +++ b/lib/dialyzer/vsn.mk @@ -1 +1 @@ -DIALYZER_VSN = 4.3 +DIALYZER_VSN = 4.3.1 diff --git a/lib/eldap/doc/src/notes.xml b/lib/eldap/doc/src/notes.xml index 946db5b93d..667c0075cd 100644 --- a/lib/eldap/doc/src/notes.xml +++ b/lib/eldap/doc/src/notes.xml @@ -31,6 +31,21 @@ </header> <p>This document describes the changes made to the Eldap application.</p> +<section><title>Eldap 1.2.9</title> + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + Add ability to specify size limit on ldap requests</p> + <p> + Own Id: OTP-17166 Aux Id: PR-2904 </p> + </item> + </list> + </section> + +</section> + <section><title>Eldap 1.2.8</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/eldap/include/eldap.hrl b/lib/eldap/include/eldap.hrl index 7c12cd4f2b..b670de871f 100644 --- a/lib/eldap/include/eldap.hrl +++ b/lib/eldap/include/eldap.hrl @@ -7,6 +7,7 @@ -record(eldap_search, { base = [], % Baseobject filter = [], % Search conditions + size_limit = 0, % Setting default size limit to 0 makes it unlimited scope=wholeSubtree, % Search scope deref=derefAlways, % Dereference attributes = [], % Attributes to be returned diff --git a/lib/eldap/src/eldap.erl b/lib/eldap/src/eldap.erl index 9b7e254dfe..c647083024 100644 --- a/lib/eldap/src/eldap.erl +++ b/lib/eldap/src/eldap.erl @@ -326,6 +326,8 @@ parse_search_args([{base, Base}|T],A) -> parse_search_args(T,A#eldap_search{base = Base}); parse_search_args([{filter, Filter}|T],A) -> parse_search_args(T,A#eldap_search{filter = Filter}); +parse_search_args([{size_limit, SizeLimit}|T],A) when is_integer(SizeLimit) -> + parse_search_args(T,A#eldap_search{size_limit = SizeLimit}); parse_search_args([{scope, Scope}|T],A) -> parse_search_args(T,A#eldap_search{scope = Scope}); parse_search_args([{deref, Deref}|T],A) -> @@ -749,7 +751,7 @@ do_search_0(Data, A, Controls) -> Req = #'SearchRequest'{baseObject = A#eldap_search.base, scope = v_scope(A#eldap_search.scope), derefAliases = v_deref(A#eldap_search.deref), - sizeLimit = 0, % no size limit + sizeLimit = v_size_limit(A#eldap_search.size_limit), timeLimit = v_timeout(A#eldap_search.timeout), typesOnly = v_bool(A#eldap_search.types_only), filter = v_filter(A#eldap_search.filter), @@ -777,6 +779,9 @@ collect_search_responses(Data, S, ID, {ok,Msg}, Acc, Ref) success -> log2(Data, "search reply = searchResDone ~n", []), {ok,Acc,Ref,Data}; + sizeLimitExceeded -> + log2(Data, "[TRUNCATED] search reply = searchResDone ~n", []), + {ok,Acc,Ref,Data}; referral -> {{ok, {referral,R#'LDAPResult'.referral}}, Data}; Reason -> @@ -1088,6 +1093,9 @@ v_bool(true) -> true; v_bool(false) -> false; v_bool(_Bool) -> throw({error,concat(["not Boolean: ",_Bool])}). +v_size_limit(I) when is_integer(I), I>=0 -> I; +v_size_limit(_I) -> throw({error,concat(["size_limit not positive integer: ",_I])}). + v_timeout(I) when is_integer(I), I>=0 -> I; v_timeout(_I) -> throw({error,concat(["timeout not positive integer: ",_I])}). diff --git a/lib/eldap/test/eldap_basic_SUITE.erl b/lib/eldap/test/eldap_basic_SUITE.erl index a337fe1c21..683e1d5393 100644 --- a/lib/eldap/test/eldap_basic_SUITE.erl +++ b/lib/eldap/test/eldap_basic_SUITE.erl @@ -49,9 +49,12 @@ search_filter_and/1, search_filter_and_not/1, search_filter_equalityMatch/1, + search_filter_equalityMatch_objectClass_exists/1, search_filter_final/1, search_filter_initial/1, search_filter_or/1, + search_filter_or_sizelimit_ok/1, + search_filter_or_sizelimit_exceeded/1, search_filter_substring_any/1, search_non_existant/1, search_referral/1, @@ -118,6 +121,7 @@ groups() -> more_add, add_referral, search_filter_equalityMatch, + search_filter_equalityMatch_objectClass_exists, search_filter_substring_any, search_filter_initial, search_filter_final, @@ -126,6 +130,8 @@ groups() -> search_filter_and_not, search_two_hits, search_referral, + search_filter_or_sizelimit_ok, + search_filter_or_sizelimit_exceeded, modify, modify_referral, delete, @@ -569,6 +575,17 @@ search_filter_equalityMatch(Config) -> scope=eldap:singleLevel()}). %%%---------------------------------------------------------------- +search_filter_equalityMatch_objectClass_exists(Config) -> + BasePath = proplists:get_value(eldap_path, Config), + ExpectedDN = "cn=Jonas Jonsson," ++ BasePath, + {ok, #eldap_search_result{entries=[#eldap_entry{object_name=ExpectedDN}]}} = + eldap:search(proplists:get_value(handle, Config), + #eldap_search{base = BasePath, + filter = eldap:'and'([eldap:equalityMatch("sn", "Jonsson"), + eldap:present("objectclass")]), + scope=eldap:singleLevel()}). + +%%%---------------------------------------------------------------- search_filter_substring_any(Config) -> BasePath = proplists:get_value(eldap_path, Config), ExpectedDN = "cn=Jonas Jonsson," ++ BasePath, @@ -627,6 +644,39 @@ search_filter_or(Config) -> ExpectedDNs = lists:sort([DN || #eldap_entry{object_name=DN} <- Es]). %%%---------------------------------------------------------------- +search_filter_or_sizelimit_ok(Config) -> + H = proplists:get_value(handle, Config), + BasePath = proplists:get_value(eldap_path, Config), + ExpectedDNs = lists:sort(["cn=Foo Bar," ++ BasePath, + "ou=Team," ++ BasePath]), + {ok, #eldap_search_result{entries=Es}} = + eldap:search(H, + #eldap_search{base = BasePath, + filter = eldap:'or'([eldap:substrings("sn", [{any, "a"}]), + eldap:equalityMatch("ou","Team")]), + size_limit = 2, + scope=eldap:singleLevel()}), + ExpectedDNs = lists:sort([DN || #eldap_entry{object_name=DN} <- Es]). + +%%%---------------------------------------------------------------- +search_filter_or_sizelimit_exceeded(Config) -> + H = proplists:get_value(handle, Config), + BasePath = proplists:get_value(eldap_path, Config), + %% The quesry without the {size_limit,1} option would return two answers: + ExpectedDNs = ["cn=Foo Bar," ++ BasePath, + "ou=Team," ++ BasePath], + %% Expect exact one of the two answers, but we don't know which: + {ok, #eldap_search_result{entries=[E]}} = + eldap:search(H, + #eldap_search{base = BasePath, + filter = eldap:'or'([eldap:substrings("sn", [{any, "a"}]), + eldap:equalityMatch("ou","Team")]), + size_limit = 1, + scope=eldap:singleLevel()}), + #eldap_entry{object_name=DN} = E, + true = lists:member(DN, ExpectedDNs). + +%%%---------------------------------------------------------------- search_filter_and_not(Config) -> H = proplists:get_value(handle, Config), BasePath = proplists:get_value(eldap_path, Config), diff --git a/lib/eldap/vsn.mk b/lib/eldap/vsn.mk index 8f969e6945..f0b9745f08 100644 --- a/lib/eldap/vsn.mk +++ b/lib/eldap/vsn.mk @@ -1 +1 @@ -ELDAP_VSN = 1.2.8 +ELDAP_VSN = 1.2.9 diff --git a/lib/erl_interface/doc/src/notes.xml b/lib/erl_interface/doc/src/notes.xml index 7b904b9ff5..fde35537ba 100644 --- a/lib/erl_interface/doc/src/notes.xml +++ b/lib/erl_interface/doc/src/notes.xml @@ -453,6 +453,39 @@ </section> +<section><title>Erl_Interface 3.11.3.1</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Fix link error "multiple definition of + `ei_default_socket_callbacks'" for gcc version 10 or when + built with gcc option -fno-common. Error exists since + OTP-21.3.</p> + <p> + Own Id: OTP-16412 Aux Id: PR-2503 </p> + </item> + </list> + </section> + + + <section><title>Known Bugs and Problems</title> + <list> + <item> + <p> + The <c>ei</c> API for decoding/encoding terms is not + fully 64-bit compatible since terms that have a + representation on the external term format larger than 2 + GB cannot be handled.</p> + <p> + Own Id: OTP-16607 Aux Id: OTP-16608 </p> + </item> + </list> + </section> + +</section> + <section><title>Erl_Interface 3.11.3</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/inets/doc/src/httpd.xml b/lib/inets/doc/src/httpd.xml index 7d82463d12..21d9a87304 100644 --- a/lib/inets/doc/src/httpd.xml +++ b/lib/inets/doc/src/httpd.xml @@ -137,7 +137,7 @@ be more that one server that has the same bind_address and port. If this property is not explicitly set, it is assumed that the <seeerl marker="#prop_bind_address"><c>bind_address</c></seeerl> and - <seeerl marker="#prop_port"><c>port</c></seeerl>uniquely identifies the HTTP server. + <seeerl marker="#prop_port"><c>port</c></seeerl> uniquely identifies the HTTP server. </p> </item> @@ -256,9 +256,9 @@ <tag><marker id="max_client_body_chunk"></marker>{max_client_body_chunk, integer()}</tag> <item> - <p>Enforces chunking of a HTTP PUT or POST body data to be deliverd + <p>Enforces chunking of a HTTP PUT or POST body data to be delivered to the mod_esi callback. Note this is not supported for mod_cgi. - Default is no limit e.i the whole body is deliverd as one entity, which could + Default is no limit e.i the whole body is delivered as one entity, which could be very memory consuming. <seeerl marker="mod_esi">mod_esi(3)</seeerl>. </p> </item> @@ -275,7 +275,7 @@ 1590. File suffixes are mapped to MIME types before file delivery. The mapping between file suffixes and MIME types can be specified as an Apache-like file or directly in the property list. Such - a file can look like the follwoing:</p> + a file can look like the following:</p> <pre> # MIME type Extension text/html html htm @@ -863,7 +863,7 @@ Transport: TLS <tag><marker id="prop_block_time"></marker>{block_time, integer()}</tag> <item> <p>Specifies the number of minutes a user is blocked. After - this timehas passed, the user automatically regains access. + this time has passed, the user automatically regains access. Default is <c>60</c>.</p> </item> @@ -1110,7 +1110,7 @@ Transport: TLS <p>If <c>Body</c> is returned and equal to <c>{Fun,Arg}</c>, the web server tries <c>apply/2</c> on <c>Fun</c> with <c>Arg</c> as argument. The web server expects that the fun either - returns a list <c>(Body)</c> that is an HTTP repsonse, or the + returns a list <c>(Body)</c> that is an HTTP response, or the atom <c>sent</c> if the HTTP response is sent back to the client. If <c>close</c> is returned from the fun, something has gone wrong and the server signals this to the client by diff --git a/lib/jinterface/doc/src/notes.xml b/lib/jinterface/doc/src/notes.xml index fca0b46878..8603433fbd 100644 --- a/lib/jinterface/doc/src/notes.xml +++ b/lib/jinterface/doc/src/notes.xml @@ -31,6 +31,31 @@ </header> <p>This document describes the changes made to the Jinterface application.</p> +<section><title>Jinterface 1.11.1</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + A <seeguide + marker="erts:erl_dist_protocol#new_link_protocol">new + link protocol</seeguide> has been introduced which + prevents links from ending up in an inconsistent state + where one participant considers itself linked while the + other doesn't. This bug has always existed in the + distributed case, but has since OTP 21 also existed in + the node local case since the distributed link protocol + then was adopted also for node local links. The bug + could, however, only trigger if both participants + operated on the link simultaneously.</p> + <p> + Own Id: OTP-17127</p> + </item> + </list> + </section> + +</section> + <section><title>Jinterface 1.11</title> <section><title>Improvements and New Features</title> diff --git a/lib/jinterface/java_src/com/ericsson/otp/erlang/AbstractConnection.java b/lib/jinterface/java_src/com/ericsson/otp/erlang/AbstractConnection.java index 26f6ffcd97..fb7c6869b7 100644 --- a/lib/jinterface/java_src/com/ericsson/otp/erlang/AbstractConnection.java +++ b/lib/jinterface/java_src/com/ericsson/otp/erlang/AbstractConnection.java @@ -76,6 +76,8 @@ public abstract class AbstractConnection extends Thread { protected static final int exitTTTag = 13; protected static final int regSendTTTag = 16; protected static final int exit2TTTag = 18; + protected static final int unlinkIdTag = 35; + protected static final int unlinkIdAckTag = 36; // MD5 challenge messsage tags protected static final int ChallengeReply = 'r'; @@ -355,10 +357,8 @@ public abstract class AbstractConnection extends Thread { // link to pid /** - * Create a link between the local node and the specified process on the - * remote node. If the link is still active when the remote process - * terminates, an exit signal will be sent to this connection. Use - * {@link #sendUnlink unlink()} to remove the link. + * + * Send link signal to remote process. * * @param dest * the Erlang PID of the remote process. @@ -393,9 +393,8 @@ public abstract class AbstractConnection extends Thread { } /** - * Remove a link between the local node and the specified process on the - * remote node. This method deactivates links created with {@link #sendLink - * link()}. + * + * Send unlink signal to remote process. * * @param dest * the Erlang PID of the remote process. @@ -404,7 +403,8 @@ public abstract class AbstractConnection extends Thread { * if the connection is not active or a communication error * occurs. */ - protected void sendUnlink(final OtpErlangPid from, final OtpErlangPid dest) + protected void sendUnlink(final OtpErlangPid from, final OtpErlangPid dest, + long unlink_id) throws IOException { if (!connected) { throw new IOException("Not connected"); @@ -417,11 +417,29 @@ public abstract class AbstractConnection extends Thread { header.write1(passThrough); header.write1(version); - // header - header.write_tuple_head(3); - header.write_long(unlinkTag); - header.write_any(from); - header.write_any(dest); + if ((peer.flags & AbstractNode.dFlagUnlinkId) != 0) { + // header + header.write_tuple_head(4); + header.write_long(unlinkIdTag); + header.write_long(unlink_id); + header.write_any(from); + header.write_any(dest); + } + else { + /* + * A node that isn't capable of talking the new link protocol. + * + * Send an old unlink op, and send ourselves an unlink-ack. We may + * end up in an inconsistent state as we could before the new link + * protocol was introduced... + */ + // header + header.write_tuple_head(3); + header.write_long(unlinkTag); + header.write_any(from); + header.write_any(dest); + deliver(new OtpMsg(unlinkIdAckTag, dest, from, unlink_id)); + } // fix up length in preamble header.poke4BE(0, header.size() - 4); @@ -429,6 +447,45 @@ public abstract class AbstractConnection extends Thread { do_send(header); } + /** + * Send unlink acknowledgment signal to remote process. + * + * @param dest + * the Erlang PID of the remote process. + * + * @exception java.io.IOException + * if the connection is not active or a communication error + * occurs. + */ + protected void sendUnlinkAck(final OtpErlangPid from, final OtpErlangPid dest, + long unlink_id) + throws IOException { + if (!connected) { + throw new IOException("Not connected"); + } + if ((peer.flags & AbstractNode.dFlagUnlinkId) != 0) { + @SuppressWarnings("resource") + final OtpOutputStream header = new OtpOutputStream(headerLen); + + // preamble: 4 byte length + "passthrough" tag + header.write4BE(0); // reserve space for length + header.write1(passThrough); + header.write1(version); + + // header + header.write_tuple_head(4); + header.write_long(unlinkIdAckTag); + header.write_long(unlink_id); + header.write_any(from); + header.write_any(dest); + // fix up length in preamble + header.poke4BE(0, header.size() - 4); + + do_send(header); + } + + } + /* used internally when "processes" terminate */ protected void sendExit(final OtpErlangPid from, final OtpErlangPid dest, final OtpErlangObject reason) throws IOException { @@ -685,7 +742,21 @@ public abstract class AbstractConnection extends Thread { from = (OtpErlangPid) head.elementAt(1); to = (OtpErlangPid) head.elementAt(2); - deliver(new OtpMsg(tag, from, to)); + deliver(new OtpMsg(tag, from, to, 0)); + break; + + case unlinkIdTag: // { UNLINK_ID, UnlinkId, FromPid, ToPid} + case unlinkIdAckTag: // { UNLINK_ID_Ack, UnlinkId, FromPid, ToPid} + if (traceLevel >= ctrlThreshold) { + System.out.println("<- " + headerType(head) + " " + + head); + } + + long unlink_id = ((OtpErlangLong) head.elementAt(1)).longValue(); + from = (OtpErlangPid) head.elementAt(2); + to = (OtpErlangPid) head.elementAt(3); + + deliver(new OtpMsg(tag, from, to, unlink_id)); break; // absolutely no idea what to do with these, so we ignore @@ -877,6 +948,12 @@ public abstract class AbstractConnection extends Thread { case unlinkTag: return "UNLINK"; + case unlinkIdTag: + return "UNLINK_ID"; + + case unlinkIdAckTag: + return "UNLINK_ID_ACK"; + case regSendTag: return "REG_SEND"; diff --git a/lib/jinterface/java_src/com/ericsson/otp/erlang/AbstractNode.java b/lib/jinterface/java_src/com/ericsson/otp/erlang/AbstractNode.java index fa6db9a046..09add55819 100644 --- a/lib/jinterface/java_src/com/ericsson/otp/erlang/AbstractNode.java +++ b/lib/jinterface/java_src/com/ericsson/otp/erlang/AbstractNode.java @@ -94,6 +94,7 @@ public class AbstractNode implements OtpTransportFactory { static final int dFlagMapTag = 0x20000; static final int dFlagBigCreation = 0x40000; static final int dFlagHandshake23 = 0x1000000; + static final int dFlagUnlinkId = 0x2000000; int ntype = NTYPE_R6; int proto = 0; // tcp/ip @@ -105,7 +106,8 @@ public class AbstractNode implements OtpTransportFactory { | dflagNewFunTags | dFlagUtf8Atoms | dFlagMapTag | dFlagExportPtrTag | dFlagBigCreation - | dFlagHandshake23; + | dFlagHandshake23 + | dFlagUnlinkId; /* initialize hostname and default cookie */ static { diff --git a/lib/jinterface/java_src/com/ericsson/otp/erlang/Link.java b/lib/jinterface/java_src/com/ericsson/otp/erlang/Link.java index 18aa825759..78890d1cde 100644 --- a/lib/jinterface/java_src/com/ericsson/otp/erlang/Link.java +++ b/lib/jinterface/java_src/com/ericsson/otp/erlang/Link.java @@ -23,11 +23,14 @@ package com.ericsson.otp.erlang; class Link { private final OtpErlangPid local; private final OtpErlangPid remote; + private long unlinking = 0; private int hashCodeValue = 0; public Link(final OtpErlangPid local, final OtpErlangPid remote) { this.local = local; this.remote = remote; + this.unlinking = 0; + } public OtpErlangPid local() { @@ -47,6 +50,14 @@ class Link { || local.equals(aremote) && remote.equals(alocal); } + public long getUnlinking() { + return this.unlinking; + } + + public void setUnlinking(long unlink_id) { + this.unlinking = unlink_id; + } + @Override public int hashCode() { if (hashCodeValue == 0) { diff --git a/lib/jinterface/java_src/com/ericsson/otp/erlang/Links.java b/lib/jinterface/java_src/com/ericsson/otp/erlang/Links.java index 5f1bd40e76..032c7480af 100644 --- a/lib/jinterface/java_src/com/ericsson/otp/erlang/Links.java +++ b/lib/jinterface/java_src/com/ericsson/otp/erlang/Links.java @@ -23,6 +23,7 @@ package com.ericsson.otp.erlang; class Links { Link[] links; int count; + int active; Links() { this(10); @@ -31,68 +32,132 @@ class Links { Links(final int initialSize) { links = new Link[initialSize]; count = 0; + active = 0; } - synchronized void addLink(final OtpErlangPid local, - final OtpErlangPid remote) { - if (find(local, remote) == -1) { + // Try add link and return if it was added or not... + // If already existing it will not be added again. + // If force is true it is added even if it is + // currently unlinking; otherwise not. + synchronized boolean addLink(final OtpErlangPid local, + final OtpErlangPid remote, + final boolean force) { + int i = find(local, remote); + if (i != -1) { + if (links[i].getUnlinking() != 0 && force) { + links[i].setUnlinking(0); + active++; + return true; + } + return false; + } + else { if (count >= links.length) { final Link[] tmp = new Link[count * 2]; System.arraycopy(links, 0, tmp, 0, count); links = tmp; } links[count++] = new Link(local, remote); + active++; + return true; } } - synchronized void removeLink(final OtpErlangPid local, - final OtpErlangPid remote) { + // Try remove link and return whether it was active or not... + synchronized boolean removeLink(final OtpErlangPid local, + final OtpErlangPid remote) { int i; if ((i = find(local, remote)) != -1) { + long unlinking = links[i].getUnlinking(); count--; links[i] = links[count]; links[count] = null; + if (unlinking == 0) { + active--; + return true; + } } + return false; } - synchronized boolean exists(final OtpErlangPid local, - final OtpErlangPid remote) { - return find(local, remote) != -1; + // Try remove active link and return whether it was removed or not... + synchronized boolean removeActiveLink(final OtpErlangPid local, + final OtpErlangPid remote) { + int i; + + if ((i = find(local, remote)) != -1) { + long unlinking = links[i].getUnlinking(); + if (unlinking != 0) + return false; + count--; + active--; + links[i] = links[count]; + links[count] = null; + return true; + } + return false; } - synchronized int find(final OtpErlangPid local, final OtpErlangPid remote) { - for (int i = 0; i < count; i++) { - if (links[i].equals(local, remote)) { - return i; - } + // Remove link if unlink_id match and return whether it was removed or not... + synchronized boolean removeUnlinkingLink(final OtpErlangPid local, + final OtpErlangPid remote, + final long unlink_id) { + int i; + + if (unlink_id == 0) { + return false; } - return -1; + + if ((i = find(local, remote)) != -1) { + long unlinking = links[i].getUnlinking(); + if (unlinking != unlink_id) + return false; + count--; + links[i] = links[count]; + links[count] = null; + return true; + } + return false; } - int count() { - return count; + synchronized boolean setUnlinking(final OtpErlangPid local, + final OtpErlangPid remote, + final long unlink_id) { + int i; + + if (unlink_id == 0) { + return false; + } + + if ((i = find(local, remote)) != -1) { + if (links[i].getUnlinking() == 0) { + links[i].setUnlinking(unlink_id); + active--; + return true; + } + } + return false; } - /* all local pids get notified about broken connection */ - synchronized OtpErlangPid[] localPids() { - OtpErlangPid[] ret = null; - if (count != 0) { - ret = new OtpErlangPid[count]; - for (int i = 0; i < count; i++) { - ret[i] = links[i].local(); + synchronized int find(final OtpErlangPid local, final OtpErlangPid remote) { + for (int i = 0; i < count; i++) { + if (links[i].equals(local, remote)) { + return i; } } - return ret; + return -1; } - /* all remote pids get notified about failed pid */ synchronized OtpErlangPid[] remotePids() { OtpErlangPid[] ret = null; - if (count != 0) { - ret = new OtpErlangPid[count]; - for (int i = 0; i < count; i++) { - ret[i] = links[i].remote(); + if (active != 0) { + int a = 0; + ret = new OtpErlangPid[active]; + for (int i = 0; a < active; i++) { + if (links[i].getUnlinking() == 0) { + ret[a++] = links[i].remote(); + } } } return ret; @@ -112,13 +177,4 @@ class Links { return ret; } - /* returns a copy of the link table */ - synchronized Link[] links() { - Link[] ret = null; - if (count != 0) { - ret = new Link[count]; - System.arraycopy(links, 0, ret, 0, count); - } - return ret; - } } diff --git a/lib/jinterface/java_src/com/ericsson/otp/erlang/OtpConnection.java b/lib/jinterface/java_src/com/ericsson/otp/erlang/OtpConnection.java index eb3eaa1f15..0cfb1bb39d 100644 --- a/lib/jinterface/java_src/com/ericsson/otp/erlang/OtpConnection.java +++ b/lib/jinterface/java_src/com/ericsson/otp/erlang/OtpConnection.java @@ -49,6 +49,8 @@ import java.io.IOException; public class OtpConnection extends AbstractConnection { protected OtpSelf self; protected GenericQueue queue; // messages get delivered here + protected Links links; + private long unlink_id; /* * Accept an incoming connection from a remote node. Used by {@link @@ -96,7 +98,58 @@ public class OtpConnection extends AbstractConnection { @Override public void deliver(final OtpMsg msg) { - queue.put(msg); + switch (msg.type()) { + case OtpMsg.exitTag: + case OtpMsg.linkTag: + case OtpMsg.unlinkTag: + case AbstractConnection.unlinkIdTag: + case AbstractConnection.unlinkIdAckTag: + handle_link_operation(msg); + break; + default: + queue.put(msg); + break; + } + } + + private synchronized void handle_link_operation(final OtpMsg m) { + final OtpErlangPid remote = m.getSenderPid(); + switch (m.type()) { + case OtpMsg.linkTag: + // only queue up link-message if link was added... + if (links.addLink(self.pid(), remote, false)) { + queue.put(m); + } + break; + + case OtpMsg.unlinkTag: + case AbstractConnection.unlinkIdTag: { + final long unlink_id = m.getUnlinkId(); + // only queue up unlink-message if link was removed... + if (links.removeActiveLink(self.pid(), remote)) { + // Use old unlinkTag without unlink id for + // backwards compatibility... + queue.put(new OtpMsg(OtpMsg.unlinkTag, self.pid(), + remote)); + } + try { + super.sendUnlinkAck(self.pid(), remote, unlink_id); + } catch (final Exception e) { + } + break; + } + + case AbstractConnection.unlinkIdAckTag: + links.removeUnlinkingLink(self.pid(), remote, m.getUnlinkId()); + break; + + case OtpMsg.exitTag: + // only queue up exit-message if link was removed... + if (links.removeActiveLink(self.pid(), remote)) { + queue.put(m); + } + break; + } } /** @@ -544,7 +597,14 @@ public class OtpConnection extends AbstractConnection { * occurs. */ public void link(final OtpErlangPid dest) throws IOException { - super.sendLink(self.pid(), dest); + if (links.addLink(self.pid(), dest, true)) { + try { + super.sendLink(self.pid(), dest); + } catch (final IOException e) { + links.removeLink(self.pid(), dest); // restore... + throw e; + } + } } /** @@ -560,7 +620,17 @@ public class OtpConnection extends AbstractConnection { * occurs. */ public void unlink(final OtpErlangPid dest) throws IOException { - super.sendUnlink(self.pid(), dest); + long unlink_id = this.unlink_id++; + if (unlink_id == 0) + unlink_id = this.unlink_id++; + if (links.setUnlinking(self.pid(), dest, unlink_id)) { + try { + super.sendUnlink(self.pid(), dest, unlink_id); + } catch (final IOException e) { + links.addLink(self.pid(), dest, true); // restore... + throw e; + } + } } /** diff --git a/lib/jinterface/java_src/com/ericsson/otp/erlang/OtpCookedConnection.java b/lib/jinterface/java_src/com/ericsson/otp/erlang/OtpCookedConnection.java index 011709beab..70ecc5b695 100644 --- a/lib/jinterface/java_src/com/ericsson/otp/erlang/OtpCookedConnection.java +++ b/lib/jinterface/java_src/com/ericsson/otp/erlang/OtpCookedConnection.java @@ -122,9 +122,7 @@ public class OtpCookedConnection extends AbstractConnection { switch (msg.type()) { case OtpMsg.linkTag: - if (delivered) { - links.addLink(msg.getRecipientPid(), msg.getSenderPid()); - } else { + if (!delivered) { try { // no such pid - send exit to sender super.sendExit(msg.getRecipientPid(), msg.getSenderPid(), @@ -133,13 +131,7 @@ public class OtpCookedConnection extends AbstractConnection { } } break; - - case OtpMsg.unlinkTag: - case OtpMsg.exitTag: - links.removeLink(msg.getRecipientPid(), msg.getSenderPid()); - break; - - case OtpMsg.exit2Tag: + default: break; } @@ -200,30 +192,39 @@ public class OtpCookedConnection extends AbstractConnection { } } - /* - * snoop for outgoing links and update own table - */ - synchronized void link(final OtpErlangPid from, final OtpErlangPid to) - throws OtpErlangExit { + void link(final OtpErlangPid from, final OtpErlangPid to) + throws OtpErlangExit { try { super.sendLink(from, to); - links.addLink(from, to); } catch (final IOException e) { throw new OtpErlangExit("noproc", to); } } - /* - * snoop for outgoing unlinks and update own table - */ - synchronized void unlink(final OtpErlangPid from, final OtpErlangPid to) { - links.removeLink(from, to); + void unlink(final OtpErlangPid from, final OtpErlangPid to, final long unlink_id) { + try { + super.sendUnlink(from, to , unlink_id); + } catch (final IOException e) { + } + } + + void unlink_ack(final OtpErlangPid from, final OtpErlangPid to, final long unlink_id) { try { - super.sendUnlink(from, to); + super.sendUnlinkAck(from, to , unlink_id); } catch (final IOException e) { } } + synchronized void node_link(OtpErlangPid local, OtpErlangPid remote, boolean add) + { + if (add) { + links.addLink(local, remote, true); + } + else { + links.removeLink(local, remote); + } + } + /* * When the connection fails - send exit to all local pids with links * through this connection diff --git a/lib/jinterface/java_src/com/ericsson/otp/erlang/OtpMbox.java b/lib/jinterface/java_src/com/ericsson/otp/erlang/OtpMbox.java index 29a8bc1540..3d46d21d60 100644 --- a/lib/jinterface/java_src/com/ericsson/otp/erlang/OtpMbox.java +++ b/lib/jinterface/java_src/com/ericsson/otp/erlang/OtpMbox.java @@ -84,6 +84,7 @@ public class OtpMbox { GenericQueue queue; String name; Links links; + private long unlink_id; // package constructor: called by OtpNode:createMbox(name) // to create a named mbox @@ -91,6 +92,7 @@ public class OtpMbox { this.self = self; this.home = home; this.name = name; + this.unlink_id = 1; queue = new GenericQueue(); links = new Links(10); } @@ -516,7 +518,10 @@ public class OtpMbox { * or could not be reached. * */ - public void link(final OtpErlangPid to) throws OtpErlangExit { + public synchronized void link(final OtpErlangPid to) throws OtpErlangExit { + if (!links.addLink(self, to, true)) + return; /* Already linked... */ + try { final String node = to.node(); if (node.equals(home.node())) { @@ -526,17 +531,18 @@ public class OtpMbox { } else { final OtpCookedConnection conn = home.getConnection(node); if (conn != null) { - conn.link(self, to); + conn.link(self, to); // may throw 'noproc' + conn.node_link(self, to, true); } else { throw new OtpErlangExit("noproc", to); } } } catch (final OtpErlangExit e) { + links.removeLink(self, to); throw e; } catch (final Exception e) { } - links.addLink(self, to); } /** @@ -552,25 +558,41 @@ public class OtpMbox { * from. * */ - public void unlink(final OtpErlangPid to) { - links.removeLink(self, to); - - try { - final String node = to.node(); - if (node.equals(home.node())) { - home.deliver(new OtpMsg(OtpMsg.unlinkTag, self, to)); - } else { - final OtpCookedConnection conn = home.getConnection(node); - if (conn != null) { - conn.unlink(self, to); + public synchronized void unlink(final OtpErlangPid to) { + long unlink_id = this.unlink_id++; + if (unlink_id == 0) + unlink_id = this.unlink_id++; + if (links.setUnlinking(self, to, unlink_id)) { + try { + final String node = to.node(); + if (node.equals(home.node())) { + home.deliver(new OtpMsg(OtpMsg.unlinkTag, self, to)); + } else { + final OtpCookedConnection conn = home.getConnection(node); + if (conn != null) { + conn.unlink(self, to, unlink_id); + } } + } catch (final Exception e) { } - } catch (final Exception e) { } } /** * <p> + * Get information about all processes and/or mail boxes currently + * linked to this mail box. + * </p> + * + * @return an array of all pids currently linked to this mail box. + * + */ + public synchronized OtpErlangPid[] linked() { + return links.remotePids(); + } + + /** + * <p> * Create a connection to a remote node. * </p> * @@ -688,40 +710,92 @@ public class OtpMbox { * called by OtpNode to deliver message to this mailbox. * * About exit and exit2: both cause exception to be raised upon receive(). - * However exit (not 2) causes any link to be removed as well, while exit2 - * leaves any links intact. + * However exit (not 2) only has an effect if there exist a link. */ void deliver(final OtpMsg m) { switch (m.type()) { + case OtpMsg.exitTag: case OtpMsg.linkTag: - links.addLink(self, m.getSenderPid()); + case OtpMsg.unlinkTag: + case AbstractConnection.unlinkIdTag: + case AbstractConnection.unlinkIdAckTag: + handle_link_operation(m); + break; + default: + queue.put(m); + break; + } + } + + private synchronized void handle_link_operation(final OtpMsg m) { + final OtpErlangPid remote = m.getSenderPid(); + final String node = remote.node(); + final boolean is_local = node.equals(home.node()); + final OtpCookedConnection conn = is_local ? null : home.getConnection(node); + + switch (m.type()) { + case OtpMsg.linkTag: + if (links.addLink(self, remote, false)) { + if (!is_local) { + if (conn != null) + conn.node_link(self, remote, true); + else { + links.removeLink(self, remote); + queue.put(new OtpMsg(OtpMsg.exitTag, remote, self, + new OtpErlangAtom("noconnection"))); + } + } + } break; case OtpMsg.unlinkTag: - links.removeLink(self, m.getSenderPid()); + case AbstractConnection.unlinkIdTag: { + final long unlink_id = m.getUnlinkId(); + final boolean removed = links.removeActiveLink(self, remote); + try { + if (is_local) { + home.deliver(new OtpMsg(AbstractConnection.unlinkIdAckTag, + self, remote, unlink_id)); + } else if (conn != null) { + if (removed) + conn.node_link(self, remote, false); + conn.unlink_ack(self, remote, unlink_id); + } + } catch (final Exception e) { + } break; + } - case OtpMsg.exitTag: - links.removeLink(self, m.getSenderPid()); - queue.put(m); + case AbstractConnection.unlinkIdAckTag: + links.removeUnlinkingLink(self, m.getSenderPid(), m.getUnlinkId()); break; - case OtpMsg.exit2Tag: - default: - queue.put(m); + case OtpMsg.exitTag: + if (links.removeActiveLink(self, m.getSenderPid())) { + queue.put(m); + } break; } } // used to break all known links to this mbox - void breakLinks(final OtpErlangObject reason) { + synchronized void breakLinks(final OtpErlangObject reason) { final Link[] l = links.clearLinks(); if (l != null) { final int len = l.length; for (int i = 0; i < len; i++) { - exit(1, l[i].remote(), reason); + if (l[i].getUnlinking() == 0) { + OtpErlangPid remote = l[i].remote(); + final String node = remote.node(); + if (!node.equals(home.node())) { + final OtpCookedConnection conn = home.getConnection(node); + if (conn != null) + conn.node_link(self, remote, false); + } + exit(1, remote, reason); + } } } } diff --git a/lib/jinterface/java_src/com/ericsson/otp/erlang/OtpMsg.java b/lib/jinterface/java_src/com/ericsson/otp/erlang/OtpMsg.java index 5bbcf2ab9e..0f883d5deb 100644 --- a/lib/jinterface/java_src/com/ericsson/otp/erlang/OtpMsg.java +++ b/lib/jinterface/java_src/com/ericsson/otp/erlang/OtpMsg.java @@ -68,6 +68,7 @@ public class OtpMsg { protected OtpErlangPid from; protected OtpErlangPid to; protected String toName; + protected long unlink_id; // send has receiver pid but no sender information OtpMsg(final OtpErlangPid to, final OtpInputStream paybuf) { @@ -77,6 +78,7 @@ public class OtpMsg { toName = null; this.paybuf = paybuf; payload = null; + this.unlink_id = 0; } // send has receiver pid but no sender information @@ -87,6 +89,7 @@ public class OtpMsg { toName = null; paybuf = null; this.payload = payload; + this.unlink_id = 0; } // send_reg has sender pid and receiver name @@ -98,6 +101,7 @@ public class OtpMsg { to = null; this.paybuf = paybuf; payload = null; + this.unlink_id = 0; } // send_reg has sender pid and receiver name @@ -109,6 +113,7 @@ public class OtpMsg { to = null; paybuf = null; this.payload = payload; + this.unlink_id = 0; } // exit (etc) has from, to, reason @@ -117,8 +122,10 @@ public class OtpMsg { this.tag = tag; this.from = from; this.to = to; + this.unlink_id = 0; paybuf = null; payload = reason; + this.unlink_id = 0; } // special case when reason is an atom (i.e. most of the time) @@ -129,19 +136,52 @@ public class OtpMsg { this.to = to; paybuf = null; payload = new OtpErlangAtom(reason); + this.unlink_id = 0; } - // other message types (link, unlink) + // other message types (link and old unlink) OtpMsg(final int tag, final OtpErlangPid from, final OtpErlangPid to) { // convert TT-tags to equiv non-TT versions - int atag = tag; - if (tag > 10) { - atag -= 10; - } + this.tag = drop_tt_tag(tag); + this.from = from; + this.to = to; + this.unlink_id = 0; + } - this.tag = atag; + // unlink + OtpMsg(final int tag, final OtpErlangPid from, final OtpErlangPid to, + final long unlink_id) { + // convert TT-tags to equiv non-TT versions + this.tag = drop_tt_tag(tag); this.from = from; this.to = to; + this.unlink_id = unlink_id; + } + + private int drop_tt_tag(final int tag) { + switch (tag) { + case AbstractConnection.sendTTTag: + return OtpMsg.sendTag; + case AbstractConnection.exitTTTag: + return OtpMsg.exitTag; + case AbstractConnection.regSendTTTag: + return OtpMsg.regSendTag; + case AbstractConnection.exit2TTTag: + return OtpMsg.exit2Tag; + default: + return tag; + } + } + + /** + * Get unlink identifier of an unlink or unlink acknowledgment + * message. For package internal use only. + * + * @return the serialized Erlang term contained in this message. + * + */ + long getUnlinkId() { + return this.unlink_id; } /** diff --git a/lib/jinterface/test/jinterface_SUITE.erl b/lib/jinterface/test/jinterface_SUITE.erl index 468981a557..1602f4a8b2 100644 --- a/lib/jinterface/test/jinterface_SUITE.erl +++ b/lib/jinterface/test/jinterface_SUITE.erl @@ -31,7 +31,12 @@ java_internal_send_receive_different_nodes/1, java_internal_send_receive_self/1, java_link_and_exit/1, erl_link_and_exit/1, - erl_link_java_exit/1, java_link_erl_exit/1, + erl_link_unlink_link_and_exit/1, + erl_link_java_unlink_link_and_exit/1, + simultaneous_erl_link_java_link_unlink/1, + simultaneous_erl_link_unlink_java_link/1, + erl_link_java_exit/1, + java_link_erl_exit/1, internal_link_linking_exits/1, internal_link_linked_exits/1, internal_unlink_linking_exits/1, internal_unlink_linked_exits/1, normal_exit/1, kill_mbox/1,kill_erl_proc_from_java/1, @@ -77,6 +82,10 @@ -define(kill_mbox_from_erlang,12). -define(erl_exit_with_reason_any_term,13). -define(java_exit_with_reason_any_term,14). +-define(erl_link_unlink_link_and_exit, 15). +-define(erl_link_java_unlink_link_and_exit, 16). +-define(simultaneous_erl_link_java_link_unlink, 17). +-define(simultaneous_erl_link_unlink_java_link, 18). %% Test cases in NodeStatusHandler.java @@ -148,6 +157,10 @@ link_unlink() -> %% Implemented in MboxLinkUnlink.java java_link_and_exit, erl_link_and_exit, + erl_link_unlink_link_and_exit, + erl_link_java_unlink_link_and_exit, + simultaneous_erl_link_java_link_unlink, + simultaneous_erl_link_unlink_java_link, erl_link_java_exit, java_link_erl_exit, internal_link_linking_exits, @@ -393,6 +406,77 @@ erl_link_and_exit(Config) when is_list(Config) -> end, erl_java_link(LinkFun,erl_link_and_exit,Config). +erl_link_unlink_link_and_exit(Config) when is_list(Config) -> + LinkFun = fun(Mbox) -> + link(Mbox), + unlink(Mbox), + link(Mbox), + Mbox ! {?erl_link_unlink_link_and_exit,self(),?link_test_reason}, + receive ok -> ok end, + exit(?link_test_reason) + end, + erl_java_link(LinkFun,erl_link_unlink_link_and_exit,Config). + +erl_link_java_unlink_link_and_exit(Config) when is_list(Config) -> + LinkFun = fun(Mbox) -> + link(Mbox), + Mbox ! {?erl_link_java_unlink_link_and_exit,self(),?link_test_reason}, + receive ok -> ok end, + exit(?link_test_reason) + end, + erl_java_link(LinkFun,erl_link_java_unlink_link_and_exit,Config). + +simultaneous_erl_link_java_link_unlink(Config) when is_list(Config) -> + LinkFun = fun(Mbox) -> + GoTime = os:system_time(millisecond) + 500, + Mbox ! {?simultaneous_erl_link_java_link_unlink, self(), GoTime}, + spin_wait_until(fun () -> + os:system_time(millisecond) >= GoTime + end), + link(Mbox), + receive check_link -> ok end, + %% We now know the unlink should have reached us and we + %% should have sent the unlink ack... + Mbox ! check_link, + {links, Links} = process_info(self(), links), + Expect = case lists:member(Mbox, Links) of + true -> linked; + false -> not_linked + end, + io:format("Expect = ~p~n", [Expect]), + receive + MboxResult -> + MboxResult = Expect + end, + exit(?link_test_reason) + end, + erl_java_link(LinkFun,simultaneous_erl_link_java_link_unlink,Config). + +simultaneous_erl_link_unlink_java_link(Config) when is_list(Config) -> + LinkFun = fun(Mbox) -> + GoTime = os:system_time(millisecond) + 500, + Mbox ! {?simultaneous_erl_link_unlink_java_link, self(), GoTime}, + spin_wait_until(fun () -> + os:system_time(millisecond) >= GoTime + end), + link(Mbox), + unlink(Mbox), + Mbox ! check_link, + receive check_link -> ok end, + {links, Links} = process_info(self(), links), + Expect = case lists:member(Mbox, Links) of + true -> linked; + false -> not_linked + end, + io:format("Expect = ~p~n", [Expect]), + receive + MboxResult -> + MboxResult = Expect + end, + exit(?link_test_reason) + end, + erl_java_link(LinkFun,simultaneous_erl_link_unlink_java_link,Config). + %%%----------------------------------------------------------------- erl_link_java_exit(doc) -> ["MboxLinkUnlink.java: " @@ -856,3 +940,11 @@ erl_status_server([{Tag,NodeName,Up}|Rest],_) -> end; erl_status_server([],From) -> From ! done. + +spin_wait_until(Fun) -> + case Fun() of + true -> ok; + _ -> spin_wait_until(Fun) + end. + + diff --git a/lib/jinterface/test/jinterface_SUITE_data/MboxLinkUnlink.java b/lib/jinterface/test/jinterface_SUITE_data/MboxLinkUnlink.java index 8cb4aa2ed6..a0018eee9c 100644 --- a/lib/jinterface/test/jinterface_SUITE_data/MboxLinkUnlink.java +++ b/lib/jinterface/test/jinterface_SUITE_data/MboxLinkUnlink.java @@ -50,6 +50,10 @@ class MboxLinkUnlink { private static final int kill_mbox_from_erlang = 12; private static final int erl_exit_with_reason_any_term = 13; private static final int java_exit_with_reason_any_term = 14; + private static final int erl_link_unlink_link_and_exit = 15; + private static final int erl_link_java_unlink_link_and_exit = 16; + private static final int simultaneous_erl_link_java_link_unlink = 17; + private static final int simultaneous_erl_link_unlink_java_link = 18; private static boolean dbg = true; @@ -94,15 +98,74 @@ class MboxLinkUnlink { break; case erl_exit_with_reason_any_term: case erl_link_and_exit: + case erl_link_unlink_link_and_exit: dbg("Java got \"erl_link_and_exit\" or " + - "\"erl_exit_with_reason_any_term\""); + "\"erl_exit_with_reason_any_term\" or " + + "\"erl_link_unlink_link_and_exit\""); + if (!is_linked(mbox, (OtpErlangPid)tuple.elementAt(1))) + System.exit(17); mbox.send((OtpErlangPid)tuple.elementAt(1), new OtpErlangAtom("ok")); waiting = true; expected = tuple.elementAt(2); mbox.receive(1000); System.exit(2); - break; + break; + case erl_link_java_unlink_link_and_exit: + dbg("Java got \"erl_link_java_unlink_link_and_exit\""); + mbox.unlink((OtpErlangPid)tuple.elementAt(1)); + mbox.unlink((OtpErlangPid)tuple.elementAt(1)); + mbox.link((OtpErlangPid)tuple.elementAt(1)); + if (!is_linked(mbox, (OtpErlangPid)tuple.elementAt(1))) + System.exit(16); + mbox.send((OtpErlangPid)tuple.elementAt(1), + new OtpErlangAtom("ok")); + waiting = true; + expected = tuple.elementAt(2); + mbox.receive(1000); + System.exit(2); + break; + case simultaneous_erl_link_java_link_unlink: { + dbg("Java got \"simultaneous_erl_link_java_link_unlink\""); + OtpErlangPid remote = (OtpErlangPid) tuple.elementAt(1); + long go_time = ((OtpErlangLong) tuple.elementAt(2)).longValue(); + spin_wait_until(go_time); + mbox.link(remote); + mbox.unlink(remote); + OtpErlangAtom check_link = new OtpErlangAtom("check_link"); + mbox.send(remote, check_link); + OtpErlangObject chk_msg = mbox.receive(2000); + if (chk_msg == null) + System.exit(14); + else if (!((OtpErlangAtom) chk_msg).equals(check_link)) + System.exit(15); + mbox.send(remote, new OtpErlangAtom(is_linked(mbox, remote) + ? "linked" + : "not_linked")); + mbox.close(); + break; + } + case simultaneous_erl_link_unlink_java_link: { + dbg("Java got \"simultaneous_erl_link_unlink_java_link\""); + OtpErlangPid remote = (OtpErlangPid) tuple.elementAt(1); + long go_time = ((OtpErlangLong) tuple.elementAt(2)).longValue(); + spin_wait_until(go_time); + mbox.link(remote); + OtpErlangAtom check_link = new OtpErlangAtom("check_link"); + OtpErlangObject chk_msg = mbox.receive(2000); + if (chk_msg == null) + System.exit(14); + else if (!((OtpErlangAtom) chk_msg).equals(check_link)) + System.exit(15); + // We now know the unlink should have reached us and we + // should have sent the unlink ack... + mbox.send(remote, check_link); + mbox.send(remote, new OtpErlangAtom(is_linked(mbox, remote) + ? "linked" + : "not_linked")); + mbox.close(); + break; + } case erl_link_java_exit: dbg("Java got \"erl_link_java_exit\""); mbox.exit(tuple.elementAt(2)); @@ -213,4 +276,23 @@ class MboxLinkUnlink { if (dbg) System.out.println(str); } + private static void spin_wait_until(final long ms) { + long time; + do { + time = System.currentTimeMillis(); + } while (time < ms); + } + + private static boolean is_linked(OtpMbox mbox, OtpErlangPid pid) { + final OtpErlangPid[] linked = mbox.linked(); + if (linked != null) { + final int len = linked.length; + for (int i = 0; i < len; i++) { + if (pid.equals(linked[i])) + return true; + } + } + return false; + } + } diff --git a/lib/jinterface/vsn.mk b/lib/jinterface/vsn.mk index f253cf7431..5209c63af3 100644 --- a/lib/jinterface/vsn.mk +++ b/lib/jinterface/vsn.mk @@ -1 +1 @@ -JINTERFACE_VSN = 1.11 +JINTERFACE_VSN = 1.11.1 diff --git a/lib/kernel/doc/src/erl_epmd.xml b/lib/kernel/doc/src/erl_epmd.xml index 03aa949516..f6fe3c0a9e 100644 --- a/lib/kernel/doc/src/erl_epmd.xml +++ b/lib/kernel/doc/src/erl_epmd.xml @@ -56,8 +56,10 @@ <desc> <p>Registers the node with <c>epmd</c> and tells epmd what port will be used for the current node. It returns a creation number. This number is - incremented on each register to help with identifying if a node is - reconnecting to epmd.</p> + incremented on each register to help differentiate a new node instance + connecting to epmd with the same name.</p> + <p>After the node has successfully registered with epmd it will automatically + attempt reconnect to the daemon if the connection is broken.</p> </desc> </func> diff --git a/lib/kernel/doc/src/erpc.xml b/lib/kernel/doc/src/erpc.xml index 64032a7f94..8b15c38a56 100644 --- a/lib/kernel/doc/src/erpc.xml +++ b/lib/kernel/doc/src/erpc.xml @@ -347,7 +347,7 @@ <desc> <p> The same as calling - <seemfa marker="#call/5"><c>erpc:multicall(<anno>Nodes</anno>,erlang,apply,[<anno>Fun</anno>,[]],<anno>Timeout</anno>)</c></seemfa>. + <seemfa marker="#multicall/5"><c>erpc:multicall(<anno>Nodes</anno>,erlang,apply,[<anno>Fun</anno>,[]],<anno>Timeout</anno>)</c></seemfa>. May raise all the same exceptions as <c>erpc:multicall/5</c> plus an <c>{erpc, badarg}</c> <c>error</c> exception if <c><anno>Fun</anno></c> is not a fun of @@ -471,7 +471,7 @@ my_multicall(Nodes, Module, Function, Args) -> <desc> <p> The same as calling - <seemfa marker="#cast/4"><c>erpc:multicast(<anno>Nodes</anno>,erlang,apply,[<anno>Fun</anno>,[]])</c></seemfa>. + <seemfa marker="#multicast/4"><c>erpc:multicast(<anno>Nodes</anno>,erlang,apply,[<anno>Fun</anno>,[]])</c></seemfa>. </p> <p><c>erpc:multicast/2</c> fails with an <c>{erpc, badarg}</c> <c>error</c> exception if:</p> @@ -586,7 +586,7 @@ my_call(Node, Module, Function, Args, Timeout) -> <desc> <p> The same as calling - <seemfa marker="#call/5"><c>erpc:send_request(<anno>Node</anno>,erlang,apply,[<anno>Fun</anno>,[]])</c></seemfa>. + <seemfa marker="#send_request/4"><c>erpc:send_request(<anno>Node</anno>,erlang,apply,[<anno>Fun</anno>,[]])</c></seemfa>. </p> <p><c>erpc:send_request/2</c> fails with an <c>{erpc, badarg}</c> <c>error</c> exception if:</p> diff --git a/lib/kernel/doc/src/kernel_app.xml b/lib/kernel/doc/src/kernel_app.xml index bbe4526f0c..d4f707951d 100644 --- a/lib/kernel/doc/src/kernel_app.xml +++ b/lib/kernel/doc/src/kernel_app.xml @@ -460,7 +460,7 @@ MaxT = TickTime + TickTime / 4</code> </item> <tag><c>shell_history_file_bytes = integer()</c></tag> <item> - <p>how many bytes the shell should remember. By default, the + <p>How many bytes the shell should remember. By default, the value is set to 512kb, and the minimal value is 50kb.</p> </item> <tag><c>shell_history_path = string()</c></tag> diff --git a/lib/kernel/doc/src/logger.xml b/lib/kernel/doc/src/logger.xml index d4876e5e07..801a2c1193 100644 --- a/lib/kernel/doc/src/logger.xml +++ b/lib/kernel/doc/src/logger.xml @@ -156,7 +156,7 @@ logger:error("error happened because: ~p", [Reason]). % Without macro the field named <c>config</c>. See the <seeerl marker="logger_std_h"><c>logger_std_h(3)</c></seeerl> and <seeerl marker="logger_disk_log_h"><c>logger_disk_log_h(3)</c></seeerl> - manual pages for information about the specifc configuration + manual pages for information about the specific configuration for these handlers.</p> <p>See the <seetype marker="logger_formatter#config"> <c>logger_formatter(3)</c></seetype> manual page for @@ -913,7 +913,7 @@ start(_, []) -> <fsummary>Unset the log level for all modules in the specified application.</fsummary> <desc> <p>Unset the log level for all the modules of the specified application.</p> - <p>This function is a convinience function that calls + <p>This function is a utility function that calls <seemfa marker="#unset_module_level/1">logger:unset_module_level/2</seemfa> for each module associated with an application.</p> </desc> @@ -1160,7 +1160,7 @@ logger:set_proxy_config(maps:merge(Old, Config)). </type> <desc> <p>This callback function is optional.</p> - <p>The function is called on a temporary process when an new + <p>The function is called on a temporary process when a new handler is about to be added. The purpose is to verify the configuration and initiate all resources needed by the handler.</p> @@ -1199,7 +1199,7 @@ logger:set_proxy_config(maps:merge(Old, Config)). <c>set_handler_config/2,3</c></seemfa>, and <c>update</c> if it originates from <seemfa marker="#update_handler_config/2"> <c>update_handler_config/2,3</c></seemfa>. The handler can - use this parameteter to decide how to update the value of + use this parameter to decide how to update the value of the <c>config</c> field, that is, the handler specific configuration data. Typically, if <c>SetOrUpdate</c> equals <c>set</c>, values that are not specified must be diff --git a/lib/kernel/doc/src/notes.xml b/lib/kernel/doc/src/notes.xml index 4f84830044..a735a86df9 100644 --- a/lib/kernel/doc/src/notes.xml +++ b/lib/kernel/doc/src/notes.xml @@ -31,6 +31,60 @@ </header> <p>This document describes the changes made to the Kernel application.</p> +<section><title>Kernel 7.3</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + The range check for compression pointers in DNS encoding + was faulty, which caused incorrect label compression + encoding for very large DNS messages; larger than about + 16 kBytes, such as AXFR responses. This more than 11 year + old bug has now been corrected.</p> + <p> + Own Id: OTP-13641 Aux Id: PR-2959 </p> + </item> + <item> + <p> + Fix of internal links in the <c>erpc</c> documentation.</p> + <p> + Own Id: OTP-17202 Aux Id: PR-4516 </p> + </item> + <item> + <p> + Fix bug where complex seq_trace tokens (that is lists, + tuples, maps etc) could becomes corrupted by the GC. The + bug was introduced in OTP-21.</p> + <p> + Own Id: OTP-17209 Aux Id: PR-3039 </p> + </item> + <item> + <p>When running Xref in the <c>modules</c> mode, the + Debugger application would show up as a depency for the + Kernel applications.</p> + <p> + Own Id: OTP-17223 Aux Id: GH-4546, PR-4554 </p> + </item> + </list> + </section> + + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + <seeerl marker="erl_epmd"><c>erl_epmd</c></seeerl> (the + epmd client) will now try to reconnect to the local EPMD + if the connection is broken.</p> + <p> + Own Id: OTP-17178 Aux Id: PR-3003 </p> + </item> + </list> + </section> + +</section> + <section><title>Kernel 7.2.1</title> <section><title>Fixed Bugs and Malfunctions</title> @@ -616,6 +670,22 @@ </section> +<section><title>Kernel 6.5.2.2</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p>When running Xref in the <c>modules</c> mode, the + Debugger application would show up as a depency for the + Kernel applications.</p> + <p> + Own Id: OTP-17223 Aux Id: GH-4546, PR-4554 </p> + </item> + </list> + </section> + +</section> + <section><title>Kernel 6.5.2.1</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/kernel/doc/src/os.xml b/lib/kernel/doc/src/os.xml index 666aca988f..fc172b4306 100644 --- a/lib/kernel/doc/src/os.xml +++ b/lib/kernel/doc/src/os.xml @@ -37,27 +37,11 @@ use, these functions can be of help in enabling a program to run on most platforms.</p> - <note> - <p> - File operations used to accept filenames containing - null characters (integer value zero). This caused - the name to be truncated and in some cases arguments - to primitive operations to be mixed up. Filenames - containing null characters inside the filename - are now <em>rejected</em> and will cause primitive - file operations to fail. - </p> - <p> - Also environment variable operations used to accept - names and values of environment variables containing - null characters (integer value zero). This caused - operations to silently produce erroneous results. - Environment variable names and values containing - null characters inside the name or value are now - <em>rejected</em> and will cause environment variable - operations to fail. - </p> - </note> + <note> + <p>The functions in this module will raise a <c>badarg</c> exception + if their arguments contain invalid characters according to the + description in the "Data Types" section.</p> + </note> </description> <datatypes> @@ -67,11 +51,9 @@ <p>A string containing valid characters on the specific OS for environment variable names using <seemfa marker="file#native_name_encoding/0"><c>file:native_name_encoding()</c></seemfa> - encoding. Note that specifically null characters (integer - value zero) and <c>$=</c> characters are not allowed. - However, note that not all invalid characters necessarily - will cause the primitiv operations to fail, but may instead - produce invalid results. + encoding. Null characters (integer value zero) are not allowed. On Unix, + <c>=</c> characters are not allowed. On Windows, a <c>=</c> character is only + allowed as the very first character in the string. </p> </desc> </datatype> @@ -81,10 +63,7 @@ <p>A string containing valid characters on the specific OS for environment variable values using <seemfa marker="file#native_name_encoding/0"><c>file:native_name_encoding()</c></seemfa> - encoding. Note that specifically null characters (integer - value zero) are not allowed. However, note that not all - invalid characters necessarily will cause the primitiv - operations to fail, but may instead produce invalid results. + encoding. Null characters (integer value zero) are not allowed. </p> </desc> </datatype> @@ -96,7 +75,7 @@ set, a strings containing valid characters on the specific OS for environment variable names and values using <seemfa marker="file#native_name_encoding/0"><c>file:native_name_encoding()</c></seemfa> - encoding. The first <c>$=</c> characters appearing in + encoding. The first <c>=</c> characters appearing in the string separates environment variable name (on the left) from environment variable value (on the right). </p> @@ -105,14 +84,11 @@ <datatype> <name name="os_command"/> <desc> - <p>All characters needs to be valid characters on the - specific OS using - <seemfa marker="file#native_name_encoding/0"><c>file:native_name_encoding()</c></seemfa> - encoding. Note that specifically null characters (integer - value zero) are not allowed. However, note that not all - invalid characters not necessarily will cause - <seemfa marker="#cmd/1"><c>os:cmd/1</c></seemfa> - to fail, but may instead produce invalid results. + <p>All characters needs to be valid characters on the specific + OS using <seemfa + marker="file#native_name_encoding/0"><c>file:native_name_encoding()</c></seemfa> + encoding. Null characters (integer value zero) are not + allowed. </p> </desc> </datatype> @@ -123,7 +99,7 @@ <taglist> <tag><c>max_size</c></tag> <item> - <p>The maximum size of the data returned by the <c>os:cmd</c> call. + <p>The maximum size of the data returned by the <c>os:cmd/2</c> call. See the <seemfa marker="#cmd/2"><c>os:cmd/2</c></seemfa> documentation for more details.</p> </item> @@ -141,11 +117,6 @@ <p>Executes <c><anno>Command</anno></c> in a command shell of the target OS, captures the standard output of the command, and returns this result as a string.</p> - <warning><p>Previous implementation used to allow all characters - as long as they were integer values greater than or equal to zero. - This sometimes lead to unwanted results since null characters - (integer value zero) often are interpreted as string termination. The - current implementation rejects these.</p></warning> <p><em>Examples:</em></p> <code type="none"> LsOut = os:cmd("ls"), % on unix platform @@ -264,15 +235,6 @@ DirOut = os:cmd("dir"), % on Win32 platform</code> <p>On Unix platforms, the environment is set using UTF-8 encoding if Unicode filename translation is in effect. On Windows, the environment is set using wide character interfaces.</p> - <note> - <p> - <c><anno>VarName</anno></c> is not allowed to contain - an <c>$=</c> character. Previous implementations used - to just let the <c>$=</c> character through which - silently caused erroneous results. Current implementation - will instead throw a <c>badarg</c> exception. - </p> - </note> </desc> </func> diff --git a/lib/kernel/src/erl_epmd.erl b/lib/kernel/src/erl_epmd.erl index 7cc84b2475..96806ae3e7 100644 --- a/lib/kernel/src/erl_epmd.erl +++ b/lib/kernel/src/erl_epmd.erl @@ -53,13 +53,15 @@ -import(lists, [reverse/1]). --record(state, {socket, port_no = -1, name = ""}). +-record(state, {socket, port_no = -1, name = "", family}). -type state() :: #state{}. -include("inet_int.hrl"). -include("erl_epmd.hrl"). -include_lib("kernel/include/inet.hrl"). +-define(RECONNECT_TIME, 2000). + %%%---------------------------------------------------------------------- %%% API %%%---------------------------------------------------------------------- @@ -228,7 +230,8 @@ handle_call({register, Name, PortNo, Family}, _From, State) -> {alive, Socket, Creation} -> S = State#state{socket = Socket, port_no = PortNo, - name = Name}, + name = Name, + family = Family}, {reply, {ok, Creation}, S}; Error -> case init:get_argument(erl_epmd_port) of @@ -263,7 +266,17 @@ handle_cast(_, State) -> -spec handle_info(term(), state()) -> {'noreply', state()}. handle_info({tcp_closed, Socket}, State) when State#state.socket =:= Socket -> + erlang:send_after(?RECONNECT_TIME, self(), reconnect), {noreply, State#state{socket = -1}}; +handle_info(reconnect, State) when State#state.socket =:= -1 -> + case do_register_node(State#state.name, State#state.port_no, State#state.family) of + {alive, Socket, _Creation} -> + %% ignore the received creation + {noreply, State#state{socket = Socket}}; + _Error -> + erlang:send_after(?RECONNECT_TIME, self(), reconnect), + {noreply, State} + end; handle_info(_, State) -> {noreply, State}. diff --git a/lib/kernel/src/error_handler.erl b/lib/kernel/src/error_handler.erl index a89ef83261..a1ea460a45 100644 --- a/lib/kernel/src/error_handler.erl +++ b/lib/kernel/src/error_handler.erl @@ -24,6 +24,10 @@ %% "error_handler: add no_native compiler directive" -compile(no_native). +%% See the comment before the int/0 function for an explanation +%% why this option is needed. +-compile(no_module_opt). + %% Callbacks called from the run-time system. -export([undefined_function/3,undefined_lambda/3,breakpoint/3]). @@ -84,7 +88,10 @@ raise_undef_exception(Module, Func, Args) -> crash({Module,Func,Args,[]}). %% Used to make the call to the 'int' module a "weak" one, to avoid -%% building strong components in xref or dialyzer. +%% making Kernel a visible dependency to Debugger in xref. (To ensure +%% that the call in breakpoint/3 is kept as an apply to an unknown +%% module, this module must be compiled with the 'no_module_opt' +%% option to turn off inter-function type analysis.) int() -> int. diff --git a/lib/kernel/src/inet_dns.erl b/lib/kernel/src/inet_dns.erl index e03f124fe6..18103771dc 100644 --- a/lib/kernel/src/inet_dns.erl +++ b/lib/kernel/src/inet_dns.erl @@ -697,7 +697,7 @@ encode_labels(Bin, Comp0, Pos, [L|Ls]=Labels) when 1 =< byte_size(L), byte_size(L) =< 63 -> case gb_trees:lookup(Labels, Comp0) of none -> - Comp = if Pos < (3 bsl 14) -> + Comp = if Pos < (1 bsl 14) -> %% Just in case - compression %% pointers cannot reach further gb_trees:insert(Labels, Pos, Comp0); diff --git a/lib/kernel/src/kernel.appup.src b/lib/kernel/src/kernel.appup.src index a8490ecefa..796bec68b3 100644 --- a/lib/kernel/src/kernel.appup.src +++ b/lib/kernel/src/kernel.appup.src @@ -52,7 +52,8 @@ {<<"^7\\.1$">>,[restart_new_emulator]}, {<<"^7\\.1\\.0(?:\\.[0-9]+)+$">>,[restart_new_emulator]}, {<<"^7\\.2$">>,[restart_new_emulator]}, - {<<"^7\\.2\\.0(?:\\.[0-9]+)+$">>,[restart_new_emulator]}], + {<<"^7\\.2\\.0(?:\\.[0-9]+)+$">>,[restart_new_emulator]}, + {<<"^7\\.2\\.1(?:\\.[0-9]+)*$">>,[restart_new_emulator]}], [{<<"^6\\.0$">>,[restart_new_emulator]}, {<<"^6\\.0\\.0(?:\\.[0-9]+)+$">>,[restart_new_emulator]}, {<<"^6\\.0\\.1(?:\\.[0-9]+)*$">>,[restart_new_emulator]}, @@ -77,4 +78,5 @@ {<<"^7\\.1$">>,[restart_new_emulator]}, {<<"^7\\.1\\.0(?:\\.[0-9]+)+$">>,[restart_new_emulator]}, {<<"^7\\.2$">>,[restart_new_emulator]}, - {<<"^7\\.2\\.0(?:\\.[0-9]+)+$">>,[restart_new_emulator]}]}. + {<<"^7\\.2\\.0(?:\\.[0-9]+)+$">>,[restart_new_emulator]}, + {<<"^7\\.2\\.1(?:\\.[0-9]+)*$">>,[restart_new_emulator]}]}. diff --git a/lib/kernel/test/erl_distribution_SUITE.erl b/lib/kernel/test/erl_distribution_SUITE.erl index 67faa4911c..1c7b067375 100644 --- a/lib/kernel/test/erl_distribution_SUITE.erl +++ b/lib/kernel/test/erl_distribution_SUITE.erl @@ -30,6 +30,7 @@ nodenames/1, hostnames/1, illegal_nodenames/1, hidden_node/1, dyn_node_name/1, + epmd_reconnect/1, setopts/1, table_waste/1, net_setuptime/1, inet_dist_options_options/1, @@ -54,6 +55,7 @@ tick_serv_test/2, tick_serv_test1/1, run_remote_test/1, dyn_node_name_do/2, + epmd_reconnect_do/2, setopts_do/2, keep_conn/1, time_ping/1]). @@ -64,6 +66,8 @@ -export([pinger/1]). -define(DUMMY_NODE,dummy@test01). +-define(ALT_EPMD_PORT, "12321"). +-define(ALT_EPMD_CMD, "epmd -port "++?ALT_EPMD_PORT). %%----------------------------------------------------------------- %% The distribution is mainly tested in the big old test_suite. @@ -82,6 +86,7 @@ all() -> tick, tick_change, nodenames, hostnames, illegal_nodenames, connect_node, dyn_node_name, + epmd_reconnect, hidden_node, setopts, table_waste, net_setuptime, inet_dist_options_options, {group, monitor_nodes}, @@ -117,9 +122,15 @@ init_per_testcase(TC, Config) when TC == hostnames; file:make_dir("hostnames_nodedir"), file:write_file("hostnames_nodedir/ignore_core_files",""), Config; +init_per_testcase(epmd_reconnect, Config) -> + [] = os:cmd(?ALT_EPMD_CMD++" -relaxed_command_check -daemon"), + Config; init_per_testcase(Func, Config) when is_atom(Func), is_list(Config) -> Config. +end_per_testcase(epmd_reconnect, _Config) -> + os:cmd(?ALT_EPMD_CMD++" -kill"), + ok; end_per_testcase(_Func, _Config) -> ok. @@ -427,6 +438,83 @@ tick_cli_test1(Node) -> end end. +epmd_reconnect(Config) when is_list(Config) -> + NodeNames = [N1,N2,N3] = get_nodenames(3, ?FUNCTION_NAME), + Nodes = [atom_to_list(full_node_name(NN)) || NN <- NodeNames], + + DCfg = "-epmd_port "++?ALT_EPMD_PORT, + + {_N1F,Port1} = start_node_unconnected(DCfg, N1, ?MODULE, run_remote_test, + ["epmd_reconnect_do", atom_to_list(node()), "1" | Nodes]), + {_N2F,Port2} = start_node_unconnected(DCfg, N2, ?MODULE, run_remote_test, + ["epmd_reconnect_do", atom_to_list(node()), "2" | Nodes]), + {_N3F,Port3} = start_node_unconnected(DCfg, N3, ?MODULE, run_remote_test, + ["epmd_reconnect_do", atom_to_list(node()), "3" | Nodes]), + Ports = [Port1, Port2, Port3], + + ok = reap_ports(Ports), + + ok. + +reap_ports([]) -> + ok; +reap_ports(Ports) -> + case (receive M -> M end) of + {Port, Message} -> + case lists:member(Port, Ports) andalso Message of + {data,String} -> + io:format("~p: ~s\n", [Port, String]), + reap_ports(Ports); + {exit_status,0} -> + reap_ports(Ports -- [Port]) + end + end. + +epmd_reconnect_do(_Node, ["1", Node1, Node2, Node3]) -> + Names = [Name || Name <- [hd(string:tokens(Node, "@")) || Node <- [Node1, Node2, Node3]]], + %% wait until all nodes are registered + ok = wait_for_names(Names), + "Killed" ++_ = os:cmd(?ALT_EPMD_CMD++" -kill"), + open_port({spawn, ?ALT_EPMD_CMD}, []), + %% check that all nodes reregister with epmd + ok = wait_for_names(Names), + lists:foreach(fun(Node) -> + ANode = list_to_atom(Node), + pong = net_adm:ping(ANode), + {epmd_reconnect_do, ANode} ! {stop, Node1, Node} + end, [Node2, Node3]), + ok; +epmd_reconnect_do(_Node, ["2", Node1, Node2, _Node3]) -> + register(epmd_reconnect_do, self()), + receive {stop, Node1, Node2} -> + ok + after 7000 -> + exit(timeout) + end; +epmd_reconnect_do(_Node, ["3", Node1, _Node2, Node3]) -> + register(epmd_reconnect_do, self()), + receive {stop, Node1, Node3} -> + ok + after 7000 -> + exit(timeout) + end. + +wait_for_names(Names) -> + %% wait for up to 3 seconds (the current retry timer in erl_epmd is 2s) + wait_for_names(lists:sort(Names), 30, 100). + +wait_for_names(Names, N, Wait) when N > 0 -> + try + {ok, Info} = erl_epmd:names(), + Names = lists:sort([Name || {Name, _Port} <- Info]), + ok + catch + error:{badmatch, _} -> + timer:sleep(Wait), + wait_for_names(Names, N-1, Wait) + end. + + dyn_node_name(Config) when is_list(Config) -> %%run_dist_configs(fun dyn_node_name/2, Config). dyn_node_name("", Config). diff --git a/lib/kernel/test/gen_tcp_api_SUITE.erl b/lib/kernel/test/gen_tcp_api_SUITE.erl index 900196d26f..9e92919bf0 100644 --- a/lib/kernel/test/gen_tcp_api_SUITE.erl +++ b/lib/kernel/test/gen_tcp_api_SUITE.erl @@ -326,12 +326,15 @@ t_accept_timeout(Config) when is_list(Config) -> %% Test that gen_tcp:connect/4 (with timeout) works. t_connect_timeout(Config) when is_list(Config) -> + ?TC_TRY(t_connect_timeout, fun() -> do_connect_timeout(Config) end). + +do_connect_timeout(Config)-> %%BadAddr = {134,138,177,16}, %%TcpPort = 80, {ok, BadAddr} = unused_ip(), TcpPort = 45638, ok = ?P("Connecting to ~p, port ~p", [BadAddr, TcpPort]), - connect_timeout({gen_tcp,connect,[BadAddr,TcpPort,?INET_BACKEND_OPTS(Config),200]}, 0.2, 5.0). + connect_timeout({gen_tcp,connect, [BadAddr,TcpPort, ?INET_BACKEND_OPTS(Config),200]}, 0.2, 5.0). %% Test that gen_tcp:connect/3 handles non-existings hosts, and other %% invalid things. @@ -376,17 +379,29 @@ t_recv_eof(Config) when is_list(Config) -> %% Test using message delimiter $X. t_recv_delim(Config) when is_list(Config) -> + ?TC_TRY(t_recv_delim, fun() -> do_recv_delim(Config) end). + +do_recv_delim(Config) -> + ?P("init"), {ok, L} = gen_tcp:listen(0, ?INET_BACKEND_OPTS(Config)), {ok, Port} = inet:port(L), Opts = ?INET_BACKEND_OPTS(Config) ++ [{active,false}, {packet,line}, {line_delimiter,$X}], {ok, Client} = gen_tcp:connect(localhost, Port, Opts), {ok, A} = gen_tcp:accept(L), + ?P("send the data"), ok = gen_tcp:send(A, "abcXefgX"), - {ok, "abcX"} = gen_tcp:recv(Client, 0, 200), - {ok, "efgX"} = gen_tcp:recv(Client, 0, 200), + %% Why do we need a timeout? + %% Sure, normally there would be no delay, + %% but this testcase has nothing to do with timeouts? + ?P("read the first chunk"), + {ok, "abcX"} = gen_tcp:recv(Client, 0), %, 200), + ?P("read the first chunk"), + {ok, "efgX"} = gen_tcp:recv(Client, 0), %, 200), + ?P("cleanup"), ok = gen_tcp:close(Client), ok = gen_tcp:close(A), + ?P("done"), ok. %%% gen_tcp:shutdown/2 @@ -414,36 +429,60 @@ t_shutdown_both(Config) when is_list(Config) -> ok. t_shutdown_error(Config) when is_list(Config) -> + ?TC_TRY(t_shutdown_error, fun() -> do_shutdown_error(Config) end). + +do_shutdown_error(Config) -> + ?P("create listen socket"), {ok, L} = gen_tcp:listen(0, ?INET_BACKEND_OPTS(Config)), + ?P("shutdown socket (with How = read_write)"), {error, enotconn} = gen_tcp:shutdown(L, read_write), + ?P("close socket"), ok = gen_tcp:close(L), + ?P("shutdown socket again (with How = read_write)"), {error, closed} = gen_tcp:shutdown(L, read_write), + ?P("done"), ok. t_shutdown_async(Config) when is_list(Config) -> + ?TC_TRY(t_shutdown_async, fun() -> do_shutdown_async(Config) end). + +do_shutdown_async(Config) -> {OS, _} = os:type(), + ?P("create listen socket"), {ok, L} = gen_tcp:listen(0, ?INET_BACKEND_OPTS(Config) ++ [{sndbuf, 4096}]), {ok, Port} = inet:port(L), + ?P("connect"), {ok, Client} = gen_tcp:connect(localhost, Port, ?INET_BACKEND_OPTS(Config) ++ [{recbuf, 4096}, {active, false}]), + ?P("accept connection"), {ok, S} = gen_tcp:accept(L), + ?P("create payload"), PayloadSize = 1024 * 1024, Payload = lists:duplicate(PayloadSize, $.), + ?P("send payload"), ok = gen_tcp:send(S, Payload), + ?P("verify queue size"), case erlang:port_info(S, queue_size) of {queue_size, N} when N > 0 -> ok; {queue_size, 0} when OS =:= win32 -> ok; {queue_size, 0} = T -> ct:fail({unexpected, T}) end, + ?P("shutdown(write) accepted socket"), ok = gen_tcp:shutdown(S, write), + ?P("recv from connected socket"), {ok, Buf} = gen_tcp:recv(Client, PayloadSize), + ?P("recv(0) from connected socket (expect closed)"), {error, closed} = gen_tcp:recv(Client, 0), + ?P("verify recv data"), case length(Buf) of - PayloadSize -> ok; - Sz -> ct:fail({payload_size, + PayloadSize -> ?P("done"), ok; + Sz -> ?P("ERROR: " + "~n extected: ~p" + "~n received: ~p", [PayloadSize, Sz]), + ct:fail({payload_size, {expected, PayloadSize}, {received, Sz}}) end. @@ -505,26 +544,32 @@ do_t_fdconnect(Config) -> Path = proplists:get_value(data_dir, Config), Lib = "gen_tcp_api_SUITE", ?P("try load util nif lib"), - case erlang:load_nif(filename:join(Path,Lib), []) of + case erlang:load_nif(filename:join(Path, Lib), []) of ok -> ok; + {error, {reload, ReasonStr}} -> + ?P("already loaded: " + "~n ~s", [ReasonStr]), + ok; {error, Reason} -> ?P("UNEXPECTED - failed loading util nif lib: " "~n ~p", [Reason]), ?SKIPT("failed loading util nif lib") end, ?P("try create listen socket"), - L = case gen_tcp:listen(0, ?INET_BACKEND_OPTS(Config) ++ [{active, false}]) of + L = case gen_tcp:listen(0, + ?INET_BACKEND_OPTS(Config) ++ [{active, false}]) of {ok, LSock} -> LSock; {error, eaddrnotavail = LReason} -> ?SKIPT(listen_failed_str(LReason)) end, {ok, Port} = inet:port(L), - ?P("try create file descriptor (fd)"), + ?P("try create file descriptor"), FD = gen_tcp_api_SUITE:getsockfd(), - ?P("try connect to using file descriptor (~w)", [FD]), - Client = case gen_tcp:connect(localhost, Port, ?INET_BACKEND_OPTS(Config) ++ + ?P("try connect using file descriptor (~w)", [FD]), + Client = case gen_tcp:connect(localhost, Port, + ?INET_BACKEND_OPTS(Config) ++ [{fd, FD}, {port, 20002}, {active, false}]) of @@ -698,46 +743,91 @@ t_local_basic(Config) -> t_local_unbound(Config) -> + ?TC_TRY(t_local_unbound, fun() -> do_local_unbound(Config) end). + +do_local_unbound(Config) -> + ?P("create local (server) filename"), SFile = local_filename(server), SAddr = {local,bin_filename(SFile)}, _ = file:delete(SFile), %% InetBackendOpts = ?INET_BACKEND_OPTS(Config), + ?P("create listen socket with ifaddr ~p", [SAddr]), L = ok(gen_tcp:listen(0, InetBackendOpts ++ [{ifaddr,SAddr},{active,false}])), + ?P("listen socket created: ~p" + "~n => try connect", [L]), C = ok(gen_tcp:connect(SAddr, 0, InetBackendOpts ++ [{active,false}])), + ?P("connected: ~p" + "~n => try accept", [C]), S = ok(gen_tcp:accept(L)), + ?P("accepted: ~p" + "~n => sockname", [S]), SAddr = ok(inet:sockname(L)), - {error,enotconn} = inet:peername(L), + ?P("sockname: ~p" + "~n => peername (expect enotconn)", [SAddr]), + {error, enotconn} = inet:peername(L), + ?P("try local handshake"), local_handshake(S, SAddr, C, {local,<<>>}), + ?P("close listen socket"), ok = gen_tcp:close(L), + ?P("close accepted socket"), ok = gen_tcp:close(S), + ?P("close connected socket"), ok = gen_tcp:close(C), + ?P("delete (local) file"), ok = file:delete(SFile), + ?P("done"), ok. t_local_fdopen(Config) -> + ?TC_TRY(t_local_fdopen, fun() -> do_local_fdopen(Config) end). + +do_local_fdopen(Config) -> + ?P("create local (server) filename"), SFile = local_filename(server), SAddr = {local,bin_filename(SFile)}, _ = file:delete(SFile), %% InetBackendOpts = ?INET_BACKEND_OPTS(Config), + ?P("create listen socket with ifaddr ~p", [SAddr]), L = ok(gen_tcp:listen(0, InetBackendOpts ++ [{ifaddr,SAddr},{active,false}])), + ?P("listen socket created: ~p" + "~n => try connect", [L]), C0 = ok(gen_tcp:connect(SAddr, 0, InetBackendOpts ++ [{active,false}])), + ?P("connected: ~p" + "~n => get fd", [C0]), Fd = ok(prim_inet:getfd(C0)), + ?P("FD: ~p" + "~n => ignore fd", [Fd]), ok = prim_inet:ignorefd(C0, true), + ?P("ignored fd:" + "~n => try fdopen (local)"), C = ok(gen_tcp:fdopen(Fd, [local])), + ?P("fd open: ~p" + "~n => try accept", [C]), S = ok(gen_tcp:accept(L)), + ?P("accepted: ~p" + "~n => sockname", [S]), SAddr = ok(inet:sockname(L)), + ?P("sockname: ~p" + "~n => peername (expect enotconn)", [SAddr]), {error,enotconn} = inet:peername(L), + ?P("try local handshake"), local_handshake(S, SAddr, C, {local,<<>>}), + ?P("close listen socket"), ok = gen_tcp:close(L), + ?P("close accepted socket"), ok = gen_tcp:close(S), + ?P("close connected socket (final)"), ok = gen_tcp:close(C), + ?P("close connected socket (pre)"), ok = gen_tcp:close(C0), + ?P("delete (local) file"), ok = file:delete(SFile), + ?P("done"), ok. t_local_fdopen_listen(Config) -> @@ -829,27 +919,48 @@ t_local_fdopen_connect_unbound(Config) -> ok. t_local_abstract(Config) -> + ?TC_TRY(t_local_abstract, fun() -> do_local_abstract(Config) end). + +do_local_abstract(Config) -> + ?P("only run on linux"), case os:type() of - {unix,linux} -> + {unix, linux} -> AbstAddr = {local,<<>>}, InetBackendOpts = ?INET_BACKEND_OPTS(Config), + ?P("create listen socket"), L = ok(gen_tcp:listen( 0, InetBackendOpts ++ [{ifaddr,AbstAddr},{active,false}])), - {local,_} = SAddr = ok(inet:sockname(L)), + ?P("listen socket created: ~p" + "~n => sockname", [L]), + {local, _} = SAddr = ok(inet:sockname(L)), + ?P("(listen socket) sockname verified" + "~n => try connect"), C = ok(gen_tcp:connect( SAddr, 0, InetBackendOpts ++ [{ifaddr,AbstAddr},{active,false}])), + ?P("connected: ~p" + "~n => sockname", [C]), {local,_} = CAddr = ok(inet:sockname(C)), + ?P("(connected socket) sockname verified" + "~n => try accept"), S = ok(gen_tcp:accept(L)), + ?P("accepted: ~p" + "~n => peername (expect enotconn)", [S]), {error,enotconn} = inet:peername(L), + ?P("try local handshake"), local_handshake(S, SAddr, C, CAddr), + ?P("close listen socket"), ok = gen_tcp:close(L), + ?P("close accepted socket"), ok = gen_tcp:close(S), + ?P("close connected socket"), ok = gen_tcp:close(C), + ?P("done"), ok; _ -> + ?P("skip (unless linux)"), {skip,"AF_LOCAL Abstract Addresses only supported on Linux"} end. @@ -868,19 +979,36 @@ local_handshake(S, SAddr, C, CAddr) -> ok. t_accept_inet6_tclass(Config) when is_list(Config) -> + ?TC_TRY(t_accept_inet6_tclass, fun() -> do_accept_inet6_tclass(Config) end). + +do_accept_inet6_tclass(Config) -> TClassOpt = {tclass,8#56 bsl 2}, % Expedited forwarding Loopback = {0,0,0,0,0,0,0,1}, + ?P("create listen socket with tclass: ~p", [TClassOpt]), case gen_tcp:listen(0, ?INET_BACKEND_OPTS(Config) ++ [inet6, {ip, Loopback}, TClassOpt]) of - {ok,L} -> + {ok, L} -> + ?P("listen socket created: " + "~n ~p", [L]), LPort = ok(inet:port(L)), + ?P("try to connect to port ~p", [LPort]), Sa = ok(gen_tcp:connect(Loopback, LPort, ?INET_BACKEND_OPTS(Config))), + ?P("connected: ~p" + "~n => accept connection", [Sa]), Sb = ok(gen_tcp:accept(L)), + ?P("accepted: ~p" + "~n => getopts (tclass)", [Sb]), [TClassOpt] = ok(inet:getopts(Sb, [tclass])), + ?P("tclass verified => close accepted socket"), ok = gen_tcp:close(Sb), + ?P("close connected socket"), ok = gen_tcp:close(Sa), + ?P("close listen socket"), ok = gen_tcp:close(L), + ?P("done"), ok; - {error,_} -> + {error, _Reason} -> + ?P("ERROR: Failed create listen socket" + "~n ~p", [_Reason]), {skip,"IPv6 TCLASS not supported"} end. @@ -940,6 +1068,8 @@ connect_timeout({M,F,A}, Lower, Upper) -> {skip, "Not tested -- got error " ++ atom_to_list(E)}; {error, enetunreach = E} -> {skip, "Not tested -- got error " ++ atom_to_list(E)}; + {error, ehostunreach = E} -> + {skip, "Not tested -- got error " ++ atom_to_list(E)}; {ok, Socket} -> % What the... Pinfo = erlang:port_info(Socket), Db = inet_db:lookup_socket(Socket), @@ -971,18 +1101,20 @@ unused_ip() -> %% This is not supported on all platforms (yet), so... try net:getifaddrs() of {ok, IfAddrs} -> - io:format("we = ~p," - "unused_ip = ~p" - " ~p" - "~n", [Hent, IP, IfAddrs]); + ?P("~n we = ~p" + "~n unused_ip = ~p" + "~n ~p", [Hent, IP, IfAddrs]); {error, _} -> - io:format("we = ~p, unused_ip = ~p~n", [Hent, IP]) + ?P("~n we: ~p" + "~n unused_ip: ~p", [Hent, IP]) catch _:_:_ -> - io:format("we = ~p, unused_ip = ~p~n", [Hent, IP]) + ?P("~n we: ~p" + "~n unused_ip: ~p", [Hent, IP]) end; true -> - io:format("we = ~p, unused_ip = ~p~n", [Hent, IP]) + ?P("~n we: ~p" + "~n unused_ip: ~p", [Hent, IP]) end, IP. diff --git a/lib/kernel/test/gen_tcp_misc_SUITE.erl b/lib/kernel/test/gen_tcp_misc_SUITE.erl index e0713002ed..2c2725ad30 100644 --- a/lib/kernel/test/gen_tcp_misc_SUITE.erl +++ b/lib/kernel/test/gen_tcp_misc_SUITE.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 1998-2020. All Rights Reserved. +%% Copyright Ericsson AB 1998-2021. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. @@ -1573,42 +1573,63 @@ econnreset_after_async_send_active_once(Config) when is_list(Config) -> do_econnreset_after_async_send_active_once(Config) -> {OS, _} = os:type(), + ?P("create listen socket with active = false"), {ok, L} = ?LISTEN(Config, 0, [{active, false}, {recbuf, 4096}]), {ok, Port} = inet:port(L), + ?P("create connect socket (~w)", [Port]), Client = case ?CONNECT(Config, localhost, Port, - [{active, false}, - {sndbuf, 4096}, - {show_econnreset, true}]) of + [{active, false}, + {sndbuf, 4096}, + {show_econnreset, true}]) of {ok, CSock} -> CSock; {error, eaddrnotavail = Reason} -> ?SKIPT(connect_failed_str(Reason)) end, + ?P("create accept socket"), {ok,S} = gen_tcp:accept(L), + ?P("close listen socket"), ok = gen_tcp:close(L), + ?P("create payload"), Payload = lists:duplicate(1024 * 1024, $.), + ?P("[connect] send payload"), ok = gen_tcp:send(Client, Payload), + ?P("[connect] verify socket queue size"), case erlang:port_info(Client, queue_size) of {queue_size, N} when N > 0 -> ok; {queue_size, 0} when OS =:= win32 -> ok; {queue_size, 0} = T -> ct:fail(T) end, + ?P("[accept] send something"), ok = gen_tcp:send(S, "Whatever"), + ?P("sleep some"), ok = ct:sleep(20), + ?P("[accept] set socket option linger: {true, 0}"), ok = inet:setopts(S, [{linger, {true, 0}}]), + ?P("[accept] close socket"), ok = gen_tcp:close(S), + ?P("sleep some"), ok = ct:sleep(20), + ?P("receive 'unexpected message'"), ok = receive Msg -> {unexpected_msg, Msg} after 0 -> ok end, + ?P("[connect] set socket option active: once"), ok = inet:setopts(Client, [{active, once}]), + ?P("[connect] expect econreset"), receive {tcp_error, Client, econnreset} -> + ?P("[connect] received econreset -> expect socket close message"), receive {tcp_closed, Client} -> + ?P("[connect] received socket close message - done"), ok; Other -> + ?P("[connect] received unexpected message: " + "~n ~p", [Other]), ct:fail({unexpected1, Other}) end; Other -> + ?P("[connect] received unexpected message: " + "~n ~p", [Other]), ct:fail({unexpected2, Other}) end. @@ -1735,7 +1756,7 @@ do_linger_zero(Config) -> {ok, Port} = inet:port(L), ?P("connect (create client socket)"), Client = case ?CONNECT(Config, localhost, Port, - [{active, false}, {sndbuf, 4096}]) of + [{active, false}, {sndbuf, 4096}]) of {ok, CSock} -> CSock; {error, eaddrnotavail = Reason} -> @@ -1745,13 +1766,16 @@ do_linger_zero(Config) -> {ok, S} = gen_tcp:accept(L), ?P("close listen socket"), ok = gen_tcp:close(L), + ?P("create payload"), PayloadSize = 1024 * 1024, Payload = lists:duplicate(PayloadSize, $.), + ?P("ensure empty queue"), lz_ensure_non_empty_queue(Client, Payload, OS), ?P("linger: {true, 0}"), ok = inet:setopts(Client, [{linger, {true, 0}}]), ?P("close client socket"), ok = gen_tcp:close(Client), + ?P("sleep some"), ok = ct:sleep(1), ?P("verify client socket (port) not connected"), @@ -1766,7 +1790,7 @@ do_linger_zero(Config) -> ok. %% THIS DOES NOT WORK FOR 'SOCKET' -lz_ensure_non_empty_queue(Sock, Payload, OS) -> +lz_ensure_non_empty_queue(Sock, Payload, OS) when is_port(Sock) -> lz_ensure_non_empty_queue(Sock, Payload, OS, 1). -define(LZ_MAX_SENDS, 3). @@ -2486,21 +2510,34 @@ do_partial_recv_and_close_4(Config) -> test_prio_put_get(Config) -> Tos = 3 bsl 5, + ?P("test_prio_put_get -> create listen socket"), {ok,L1} = ?LISTEN(Config, 0, [{active,false}]), + ?P("test_prio_put_get -> set opts prio (= 3)"), ok = inet:setopts(L1,[{priority,3}]), + ?P("test_prio_put_get -> set opts tos (= ~p)", [Tos]), ok = inet:setopts(L1,[{tos,Tos}]), + ?P("test_prio_put_get -> verify opts prio and tos"), {ok,[{priority,3},{tos,Tos}]} = inet:getopts(L1,[priority,tos]), + ?P("test_prio_put_get -> set opts prio (= 3)"), ok = inet:setopts(L1,[{priority,3}]), % Dont destroy each other + ?P("test_prio_put_get -> verify opts prio and tos"), {ok,[{priority,3},{tos,Tos}]} = inet:getopts(L1,[priority,tos]), + ?P("test_prio_put_get -> set opts reuseaddr (= true)"), ok = inet:setopts(L1,[{reuseaddr,true}]), % Dont let others destroy + ?P("test_prio_put_get -> verify opts prio and tos"), {ok,[{priority,3},{tos,Tos}]} = inet:getopts(L1,[priority,tos]), + ?P("test_prio_put_get -> close listen socket"), gen_tcp:close(L1), + ?P("test_prio_put_get -> done"), ok. test_prio_accept(Config) -> - {ok,Sock}=?LISTEN(Config, 0,[binary,{packet,0},{active,false}, - {reuseaddr,true},{priority,4}]), - {ok,Port} = inet:port(Sock), + ?P("test_prio_accept -> create listen socket"), + {ok, Sock} = ?LISTEN(Config, 0, [binary,{packet,0},{active,false}, + {reuseaddr,true},{priority,4}]), + ?P("test_prio_accept -> get port number of listen socket"), + {ok, Port} = inet:port(Sock), + ?P("test_prio_accept -> connect to port ~p", [Port]), Sock2 = case ?CONNECT(Config, "localhost",Port,[binary,{packet,0}, {active,false}, {reuseaddr,true}, @@ -2510,22 +2547,33 @@ test_prio_accept(Config) -> {error, eaddrnotavail = Reason} -> ?SKIPT(connect_failed_str(Reason)) end, - {ok,Sock3}=gen_tcp:accept(Sock), - {ok,[{priority,4}]} = inet:getopts(Sock,[priority]), - {ok,[{priority,4}]} = inet:getopts(Sock2,[priority]), - {ok,[{priority,4}]} = inet:getopts(Sock3,[priority]), + ?P("test_prio_accept -> connected => accept connection"), + {ok, Sock3} = gen_tcp:accept(Sock), + ?P("test_prio_accept -> accepted => getopts prio for listen socket"), + {ok, [{priority,4}]} = inet:getopts(Sock, [priority]), + ?P("test_prio_accept -> getopts prio for connected socket"), + {ok, [{priority,4}]} = inet:getopts(Sock2, [priority]), + ?P("test_prio_accept -> getopts prio for accepted socket"), + {ok, [{priority,4}]} = inet:getopts(Sock3, [priority]), + ?P("test_prio_accept -> close listen socket"), gen_tcp:close(Sock), + ?P("test_prio_accept -> close connected socket"), gen_tcp:close(Sock2), + ?P("test_prio_accept -> close accepted socket"), gen_tcp:close(Sock3), + ?P("test_prio_accept -> done"), ok. test_prio_accept2(Config) -> Tos1 = 4 bsl 5, Tos2 = 3 bsl 5, - {ok,Sock}=?LISTEN(Config, 0,[binary,{packet,0},{active,false}, - {reuseaddr,true},{priority,4}, - {tos,Tos1}]), - {ok,Port} = inet:port(Sock), + ?P("test_prio_accept2 -> create listen socket"), + {ok, Sock} = ?LISTEN(Config, 0,[binary,{packet,0},{active,false}, + {reuseaddr,true},{priority,4}, + {tos,Tos1}]), + ?P("test_prio_accept2 -> get port number of listen socket"), + {ok, Port} = inet:port(Sock), + ?P("test_prio_accept2 -> connect to port ~p", [Port]), Sock2 = case ?CONNECT(Config, "localhost",Port,[binary,{packet,0}, {active,false}, {reuseaddr,true}, @@ -2536,22 +2584,33 @@ test_prio_accept2(Config) -> {error, eaddrnotavail = Reason} -> ?SKIPT(connect_failed_str(Reason)) end, - {ok,Sock3}=gen_tcp:accept(Sock), + ?P("test_prio_accept2 -> connected => accept connection"), + {ok, Sock3} = gen_tcp:accept(Sock), + ?P("test_prio_accept2 -> accepted => getopts prio and tos for listen socket"), {ok,[{priority,4},{tos,Tos1}]} = inet:getopts(Sock,[priority,tos]), + ?P("test_prio_accept2 -> getopts prio and tos for connected socket"), {ok,[{priority,4},{tos,Tos2}]} = inet:getopts(Sock2,[priority,tos]), + ?P("test_prio_accept2 -> getopts prio and tos for accepted socket"), {ok,[{priority,4},{tos,Tos1}]} = inet:getopts(Sock3,[priority,tos]), + ?P("test_prio_accept2 -> close listen socket"), gen_tcp:close(Sock), + ?P("test_prio_accept2 -> close connected socket"), gen_tcp:close(Sock2), + ?P("test_prio_accept2 -> close accepted socket"), gen_tcp:close(Sock3), + ?P("test_prio_accept2 -> done"), ok. test_prio_accept3(Config) -> Tos1 = 4 bsl 5, Tos2 = 3 bsl 5, - {ok,Sock}=?LISTEN(Config, 0,[binary,{packet,0},{active,false}, - {reuseaddr,true}, - {tos,Tos1}]), + ?P("test_prio_accept3 -> create listen socket"), + {ok, Sock} = ?LISTEN(Config, 0,[binary,{packet,0},{active,false}, + {reuseaddr,true}, + {tos,Tos1}]), + ?P("test_prio_accept3 -> get port number of listen socket"), {ok,Port} = inet:port(Sock), + ?P("test_prio_accept3 -> connect to port ~p", [Port]), Sock2 = case ?CONNECT(Config, "localhost",Port,[binary,{packet,0}, {active,false}, {reuseaddr,true}, @@ -2561,19 +2620,29 @@ test_prio_accept3(Config) -> {error, eaddrnotavail = Reason} -> ?SKIPT(connect_failed_str(Reason)) end, - {ok,Sock3}=gen_tcp:accept(Sock), - {ok,[{priority,0},{tos,Tos1}]} = inet:getopts(Sock,[priority,tos]), - {ok,[{priority,0},{tos,Tos2}]} = inet:getopts(Sock2,[priority,tos]), - {ok,[{priority,0},{tos,Tos1}]} = inet:getopts(Sock3,[priority,tos]), + ?P("test_prio_accept3 -> connected => accept connection"), + {ok, Sock3} = gen_tcp:accept(Sock), + ?P("test_prio_accept3 -> " + "accepted => getopts prio and tos for listen socket"), + {ok, [{priority,0},{tos,Tos1}]} = inet:getopts(Sock, [priority,tos]), + ?P("test_prio_accept3 -> getopts prio and tos for connected socket"), + {ok, [{priority,0},{tos,Tos2}]} = inet:getopts(Sock2, [priority,tos]), + ?P("test_prio_accept3 -> getopts prio and tos for accepted socket"), + {ok, [{priority,0},{tos,Tos1}]} = inet:getopts(Sock3, [priority,tos]), + ?P("test_prio_accept3 -> close listen socket"), gen_tcp:close(Sock), + ?P("test_prio_accept3 -> close connected socket"), gen_tcp:close(Sock2), + ?P("test_prio_accept3 -> close accepted socket"), gen_tcp:close(Sock3), + ?P("test_prio_accept3 -> done"), ok. test_prio_accept_async(Config) -> Tos1 = 4 bsl 5, Tos2 = 3 bsl 5, Ref = make_ref(), + ?P("test_prio_accept_async -> create prio server"), spawn(?MODULE, priority_server, [Config, {self(),Ref}]), Port = receive {Ref,P} -> P @@ -2582,6 +2651,7 @@ test_prio_accept_async(Config) -> receive after 3000 -> ok end, + ?P("test_prio_accept_async -> connect to port ~p", [Port]), Sock2 = case ?CONNECT(Config, "localhost",Port,[binary,{packet,0}, {active,false}, {reuseaddr,true}, @@ -2592,6 +2662,8 @@ test_prio_accept_async(Config) -> {error, eaddrnotavail = Reason} -> ?SKIPT(connect_failed_str(Reason)) end, + ?P("test_prio_accept_async -> " + "connected => await prio and tos for listen socket"), receive {Ref,{ok,[{priority,4},{tos,Tos1}]}} -> ok; @@ -2599,6 +2671,7 @@ test_prio_accept_async(Config) -> ct:fail({missmatch,Error}) after 5000 -> ct:fail({error,"helper process timeout"}) end, + ?P("test_prio_accept_async -> await prio and tos for accepted socket"), receive {Ref,{ok,[{priority,4},{tos,Tos1}]}} -> ok; @@ -2607,8 +2680,11 @@ test_prio_accept_async(Config) -> after 5000 -> ct:fail({error,"helper process timeout"}) end, - {ok,[{priority,4},{tos,Tos2}]} = inet:getopts(Sock2,[priority,tos]), + ?P("test_prio_accept_async -> getopts prio and tos for connected socket"), + {ok,[{priority,4},{tos,Tos2}]} = inet:getopts(Sock2, [priority,tos]), + ?P("test_prio_accept_async -> close connected socket"), catch gen_tcp:close(Sock2), + ?P("test_prio_accept_async -> done"), ok. priority_server(Config, {Parent,Ref}) -> @@ -2624,32 +2700,40 @@ priority_server(Config, {Parent,Ref}) -> ok. test_prio_fail(Config) -> + ?P("test_prio_fail -> create listen socket"), {ok,L} = ?LISTEN(Config, 0, [{active,false}]), + ?P("test_prio_fail -> try set (and fail) opts prio (= 1000)"), {error,_} = inet:setopts(L,[{priority,1000}]), + ?P("test_prio_fail -> close listen socket"), gen_tcp:close(L), + ?P("test_prio_fail -> done"), ok. test_prio_udp() -> Tos = 3 bsl 5, + ?P("test_prio_udp -> create UDP socket (open)"), {ok,S} = gen_udp:open(0,[{active,false},binary,{tos, Tos}, {priority,3}]), + ?P("test_prio_udp -> getopts prio and tos"), {ok,[{priority,3},{tos,Tos}]} = inet:getopts(S,[priority,tos]), + ?P("test_prio_fail -> close socket"), gen_udp:close(S), + ?P("test_prio_fail -> done"), ok. %% Tests the so_priority and ip_tos options on sockets when applicable. so_priority(Config) when is_list(Config) -> - try do_so_priority(Config) - catch - throw:{skip, _} = SKIP -> - SKIP - end. + ?TC_TRY(so_priority, fun() -> do_so_priority(Config) end). do_so_priority(Config) -> + ?P("create listen socket"), {ok,L} = ?LISTEN(Config, 0, [{active,false}]), + ?P("set opts on listen socket: prio to 1"), ok = inet:setopts(L,[{priority,1}]), + ?P("verify prio"), case inet:getopts(L,[priority]) of {ok,[{priority,1}]} -> + ?P("close listen socket"), gen_tcp:close(L), test_prio_put_get(Config), test_prio_accept(Config), @@ -2658,12 +2742,15 @@ do_so_priority(Config) -> test_prio_accept_async(Config), test_prio_fail(Config), test_prio_udp(), + ?P("done"), ok; - _ -> + _X -> case os:type() of {unix,linux} -> case os:version() of {X,Y,_} when (X > 2) or ((X =:= 2) and (Y >= 4)) -> + ?P("so prio should work on this version: " + "~n ~p", [_X]), ct:fail({error, "so_priority should work on this " "OS, but does not"}); @@ -2742,8 +2829,9 @@ recvtclass(Config) -> %% platforms - change {unix,_} to false? %% pktoptions is not supported for IPv4 -recvtos_ok({unix,openbsd}, OSVer) -> not semver_lt(OSVer, {6,8,0}); -recvtos_ok({unix,darwin}, OSVer) -> not semver_lt(OSVer, {19,4,0}); +recvtos_ok({unix,netbsd}, _OSVer) -> false; +recvtos_ok({unix,openbsd}, _OSVer) -> false; % not semver_lt(OSVer, {6,9,0}); +recvtos_ok({unix,darwin}, OSVer) -> false; % not semver_lt(OSVer, {19,6,0}); %% Using the option returns einval, so it is not implemented. recvtos_ok({unix,freebsd}, OSVer) -> not semver_lt(OSVer, {12,2,0}); recvtos_ok({unix,sunos}, OSVer) -> not semver_lt(OSVer, {5,12,0}); @@ -2754,8 +2842,9 @@ recvtos_ok({unix,_}, _) -> true; recvtos_ok(_, _) -> false. %% pktoptions is not supported for IPv4 -recvttl_ok({unix,openbsd}, OSVer) -> not semver_lt(OSVer, {6,8,0}); -recvttl_ok({unix,darwin}, OSVer) -> not semver_lt(OSVer, {19,4,0}); +recvttl_ok({unix,netbsd}, _OSVer) -> false; +recvttl_ok({unix,openbsd}, _OSVer) -> false; % not semver_lt(OSVer, {6,9,0}); +recvttl_ok({unix,darwin}, OSVer) -> false; % not semver_lt(OSVer, {19,6,0}); %% Using the option returns einval, so it is not implemented. recvttl_ok({unix,freebsd}, OSVer) -> not semver_lt(OSVer, {12,2,0}); recvttl_ok({unix,sunos}, OSVer) -> not semver_lt(OSVer, {5,12,0}); @@ -2766,8 +2855,9 @@ recvttl_ok({unix,_}, _) -> true; recvttl_ok(_, _) -> false. %% pktoptions is not supported for IPv6 -recvtclass_ok({unix,openbsd}, OSVer) -> not semver_lt(OSVer, {6,8,0}); -recvtclass_ok({unix,darwin}, OSVer) -> not semver_lt(OSVer, {19,4,0}); +recvtclass_ok({unix,netbsd}, _OSVer) -> false; +recvtclass_ok({unix,openbsd}, _OSVer) -> false; % not semver_lt(OSVer, {6,9,0}); +recvtclass_ok({unix,darwin}, OSVer) -> false; % not semver_lt(OSVer, {19,6,0}); recvtclass_ok({unix,sunos}, OSVer) -> not semver_lt(OSVer, {5,12,0}); %% Using the option returns einval, so it is not implemented. recvtclass_ok({unix,freebsd}, OSVer) -> not semver_lt(OSVer, {12,2,0}); @@ -3004,6 +3094,8 @@ collect_accepts(N,Tmo) -> ?SKIPT(accept_failed_str(Reason)); {accepted,P,Msg} -> + ?P("received accepted from ~p: " + "~n ~p", [P, Msg]), NextN = if N =:= infinity -> N; true -> N - 1 end, [{P,Msg}] ++ collect_accepts(NextN, Tmo - (millis()-A)) @@ -3029,6 +3121,8 @@ collect_connects(Tmo) -> ?SKIPT(connect_failed_str(Reason)); {connected,P,Msg} -> + ?P("received connected from ~p: " + "~n ~p", [P, Msg]), [{P,Msg}] ++ collect_connects(Tmo-(millis() - A)) after Tmo -> @@ -3320,6 +3414,7 @@ accept_timeouts_mixed(Config) when is_list(Config) -> end. do_accept_timeouts_mixed(Config) -> + ?P("create listen socket"), LS = case ?LISTEN(Config, 0,[]) of {ok, LSocket} -> LSocket; @@ -3328,30 +3423,46 @@ do_accept_timeouts_mixed(Config) -> end, Parent = self(), {ok,PortNo}=inet:port(LS), + ?P("create acceptor process 1 (with timeout 1000)"), P1 = spawn(mktmofun(1000,Parent,LS)), + ?P("await ~p accepting", [P1]), wait_until_accepting(P1,500), + ?P("create acceptor process 2 (with timeout 2000)"), P2 = spawn(mktmofun(2000,Parent,LS)), + ?P("await ~p accepting", [P2]), wait_until_accepting(P2,500), + ?P("create acceptor process 3 (with timeout 3000)"), P3 = spawn(mktmofun(3000,Parent,LS)), + ?P("await ~p accepting", [P3]), wait_until_accepting(P3,500), + ?P("create acceptor process 4 (with timeout 4000)"), P4 = spawn(mktmofun(4000,Parent,LS)), + ?P("await ~p accepting", [P4]), wait_until_accepting(P4,500), + ?P("expect accept from 1 (~p) with timeout", [P1]), ok = ?EXPECT_ACCEPTS([{P1,{error,timeout}}],infinity,1500), + ?P("connect"), case ?CONNECT(Config, "localhost", PortNo, []) of {ok, _} -> ok; {error, eaddrnotavail = Reason1} -> ?SKIPT(connect_failed_str(Reason1)) end, + ?P("expect accept from 2 (~p) with success", [P2]), ok = ?EXPECT_ACCEPTS([{P2,{ok,Port0}}] when is_port(Port0),infinity,100), + ?P("expect accept from 3 (~p) with timeout", [P3]), ok = ?EXPECT_ACCEPTS([{P3,{error,timeout}}],infinity,2000), + ?P("connect"), case ?CONNECT(Config, "localhost", PortNo, []) of {error, eaddrnotavail = Reason2} -> ?SKIPT(connect_failed_str(Reason2)); _ -> ok end, - ok = ?EXPECT_ACCEPTS([{P4,{ok,Port1}}] when is_port(Port1),infinity,100). + ?P("expect accept from 4 (~p) with success", [P4]), + ok = ?EXPECT_ACCEPTS([{P4,{ok,Port1}}] when is_port(Port1),infinity,100), + ?P("done"), + ok. %% Check that single acceptor behaves as expected when killed. killing_acceptor(Config) when is_list(Config) -> @@ -3362,25 +3473,35 @@ killing_acceptor(Config) when is_list(Config) -> end. do_killing_acceptor(Config) -> + ?P("create listen socket"), LS = case ?LISTEN(Config, 0,[]) of {ok, LSocket} -> LSocket; {error, eaddrnotavail = Reason} -> ?SKIPT(listen_failed_str(Reason)) end, + ?P("create acceptor process"), Pid = spawn( fun() -> erlang:display({accepted,self(),gen_tcp:accept(LS)}) end), + ?P("sleep some"), receive after 100 -> ok end, + ?P("get status for listen socket"), {ok,L1} = prim_inet:getstatus(LS), + ?P("verify listen socket accepting"), true = lists:member(accepting, L1), + ?P("kill acceptor"), exit(Pid,kill), + ?P("sleep some"), receive after 100 -> ok end, + ?P("get status for listen socket"), {ok,L2} = prim_inet:getstatus(LS), + ?P("verify listen socket *not* accepting"), false = lists:member(accepting, L2), + ?P("done"), ok. %% Check that multi acceptors behaves as expected when killed. @@ -3426,6 +3547,7 @@ killing_multi_acceptors2(Config) when is_list(Config) -> end. do_killing_multi_acceptors2(Config) -> + ?P("create listen socket"), LS = case ?LISTEN(Config, 0,[]) of {ok, LSocket} -> LSocket; @@ -3433,46 +3555,67 @@ do_killing_multi_acceptors2(Config) -> ?SKIPT(listen_failed_str(Reason)) end, Parent = self(), - {ok,PortNo}=inet:port(LS), + ?P("get port number for listen socket"), + {ok, PortNo} = inet:port(LS), F = fun() -> Parent ! {accepted,self(),gen_tcp:accept(LS)} end, F2 = mktmofun(1000,Parent,LS), + ?P("create acceptor process 1"), Pid = spawn(F), + ?P("create acceptor process 2"), Pid2 = spawn(F), + ?P("wait some"), receive after 100 -> ok end, - {ok,L1} = prim_inet:getstatus(LS), + ?P("get status for listen socket"), + {ok, L1} = prim_inet:getstatus(LS), + ?P("verify listen socket *is* accepting"), true = lists:member(accepting, L1), + ?P("kill acceptor process 1"), exit(Pid,kill), + ?P("wait some"), receive after 100 -> ok end, - {ok,L2} = prim_inet:getstatus(LS), + ?P("get status for listen socket"), + {ok, L2} = prim_inet:getstatus(LS), + ?P("verify listen socket *is* accepting"), true = lists:member(accepting, L2), + ?P("kill acceptor process 1"), exit(Pid2,kill), + ?P("wait some"), receive after 100 -> ok end, - {ok,L3} = prim_inet:getstatus(LS), + ?P("get status for listen socket"), + {ok, L3} = prim_inet:getstatus(LS), + ?P("verify listen socket is *not* accepting"), false = lists:member(accepting, L3), + ?P("create acceptor process 3"), Pid3 = spawn(F2), + ?P("wait some"), receive after 100 -> ok end, + ?P("get status for listen socket"), {ok,L4} = prim_inet:getstatus(LS), + ?P("verify listen socket *is* accepting"), true = lists:member(accepting, L4), - ?CONNECT(Config, "localhost",PortNo,[]), + ?P("connect to port ~p", [PortNo]), + ?CONNECT(Config, "localhost", PortNo,[]), + ?P("accepts"), ok = ?EXPECT_ACCEPTS([{Pid3,{ok,Port}}] when is_port(Port),1,100), - {ok,L5} = prim_inet:getstatus(LS), + ?P("get status for listen socket"), + {ok, L5} = prim_inet:getstatus(LS), + ?P("verify listen socket *is* accepting"), false = lists:member(accepting, L5), + ?P("done"), ok. %% Checks that multi-accept works when more than one accept can be %% done at once (wb test of inet_driver). several_accepts_in_one_go(Config) when is_list(Config) -> - try do_several_accepts_in_one_go(Config) - catch - throw:{skip, _} = SKIP -> - SKIP - end. + ?TC_TRY(several_accepts_in_one_go, + fun() -> do_several_accepts_in_one_go(Config) end). do_several_accepts_in_one_go(Config) -> + ?P("create listen socket"), LS = case ?LISTEN(Config, 0,[]) of {ok, LSock} -> LSock; @@ -3480,15 +3623,25 @@ do_several_accepts_in_one_go(Config) -> ?SKIPT(listen_failed_str(Reason)) end, Parent = self(), - {ok,PortNo}=inet:port(LS), - F1 = fun() -> Parent ! {accepted,self(),gen_tcp:accept(LS)} end, - F2 = fun() -> Parent ! {connected,self(),?CONNECT(Config, "localhost",PortNo,[])} end, + {ok, PortNo} = inet:port(LS), + F1 = fun() -> ?P("acceptor starting"), + Parent ! {accepted,self(),gen_tcp:accept(LS)} + end, + F2 = fun() -> ?P("connector starting"), + Parent ! {connected,self(),?CONNECT(Config, "localhost",PortNo,[])} + end, Ns = lists:seq(1,8), + ?P("start acceptors"), _ = [spawn(F1) || _ <- Ns], + ?P("await accept timeouts"), ok = ?EXPECT_ACCEPTS([],1,500), % wait for tmo + ?P("start connectors"), _ = [spawn(F2) || _ <- Ns], + ?P("await accepts"), ok = ?EXPECT_ACCEPTS([{_,{ok,_}},{_,{ok,_}},{_,{ok,_}},{_,{ok,_}},{_,{ok,_}},{_,{ok,_}},{_,{ok,_}},{_,{ok,_}}],8,15000), + ?P("await connects"), ok = ?EXPECT_CONNECTS([{_,{ok,_}},{_,{ok,_}},{_,{ok,_}},{_,{ok,_}},{_,{ok,_}},{_,{ok,_}},{_,{ok,_}},{_,{ok,_}}],1000), + ?P("done"), ok. flush(Msgs) -> @@ -4992,46 +5145,71 @@ port_failed_str(Reason) -> %% 30-second test for gen_tcp in {active, N} mode, ensuring it does not get stuck. %% Verifies that erl_check_io properly handles extra EPOLLIN signals. bidirectional_traffic(Config) when is_list(Config) -> + ?TC_TRY(bidirectional_traffic, + fun() -> do_bidirectional_traffic(Config) end). + +do_bidirectional_traffic(_Config) -> + ?P("begin"), Workers = erlang:system_info(schedulers_online) * 2, + ?P("Use ~w workers", [Workers]), Payload = crypto:strong_rand_bytes(32), + ?P("create listen socket"), {ok, LSock} = gen_tcp:listen(0, [binary, {packet, 0}, {active, false}, {reuseaddr, true}]), %% get all sockets to know failing ends {ok, Port} = inet:port(LSock), + ?P("listen socket port number ~w", [Port]), Control = self(), + ?P("create ~w receivers", [Workers]), Receivers = [spawn_link(fun () -> exchange(LSock, Port, Payload, Control) end) || _ <- lists:seq(1, Workers)], + ?P("await the result"), Result = receive {timeout, Socket, Total} -> + ?P("timeout msg for ~p: ~w", [Socket, Total]), {fail, {timeout, Socket, Total}}; {error, Socket, Reason} -> + ?P("error msg for ~p: ~p", [Socket, Reason]), {fail, {error, Socket, Reason}} after 30000 -> - %% if it does not fail in 30 seconds, it most likely works - ok + %% if it does not fail in 30 seconds, it most likely works + ?P("timeout => success?"), + ok end, + ?P("terminate receivers"), [begin unlink(Rec), exit(Rec, kill) end || Rec <- Receivers], + ?P("done"), Result. exchange(LSock, Port, Payload, Control) -> %% spin up client _ClntRcv = spawn( fun () -> - {ok, Client} = gen_tcp:connect("localhost", Port, [binary, {packet, 0}, {active, ?ACTIVE_N}]), - send_recv_loop(Client, Payload, Control) + ?P("connect"), + {ok, Client} = + gen_tcp:connect("localhost", + Port, + [binary, {packet, 0}, {active, ?ACTIVE_N}]), + ?P("connected: ~p", [Client]), + send_recv_loop(Client, Payload, Control) end), + ?P("accept"), {ok, Socket} = gen_tcp:accept(LSock), + ?P("accepted: ~p", [Socket]), %% sending process send_recv_loop(Socket, Payload, Control). send_recv_loop(Socket, Payload, Control) -> %% {active, N} must be set to active > 12 to trigger the issue %% {active, 30} seems to trigger it quite often & reliably + ?P("set (initial) active: ~p", [?ACTIVE_N]), inet:setopts(Socket, [{active, ?ACTIVE_N}]), + ?P("spawn sender"), _Snd = spawn_link( fun Sender() -> _ = gen_tcp:send(Socket, Payload), Sender() end), + ?P("begin recv"), recv(Socket, 0, Control). recv(Socket, Total, Control) -> @@ -5042,10 +5220,32 @@ recv(Socket, Total, Control) -> inet:setopts(Socket, [{active, ?ACTIVE_N}]), recv(Socket, Total, Control); {tcp_closed, Socket} -> + ?P("[recv] closed when total received: ~w", [Total]), exit(terminate); - Other-> + Other -> + ?P("[recv] received unexpected when total received: ~w" + "~n ~p" + "~n Socket: ~p" + "~n Port stat: ~p" + "~n Port status: ~p" + "~n Port Info: ~p", + [Total, Other, + Socket, + (catch inet:getstat(Socket)), + (catch prim_inet:getstatus(Socket)), + (catch erlang:port_info(Socket))]), Control ! {error, Socket, Other} after 2000 -> - %% no data received in 2 seconds, test failed - Control ! {timeout, Socket, Total} + %% no data received in 2 seconds, test failed + ?P("[recv] received nothing when total received: ~w" + "~n Socket: ~p" + "~n Port stat: ~p" + "~n Port status: ~p" + "~n Port Info: ~p", + [Total, + Socket, + (catch inet:getstat(Socket)), + (catch prim_inet:getstatus(Socket)), + (catch erlang:port_info(Socket))]), + Control ! {timeout, Socket, Total} end. diff --git a/lib/kernel/test/gen_udp_SUITE.erl b/lib/kernel/test/gen_udp_SUITE.erl index a71237e11f..a52f70933e 100644 --- a/lib/kernel/test/gen_udp_SUITE.erl +++ b/lib/kernel/test/gen_udp_SUITE.erl @@ -752,7 +752,9 @@ do_sendtclass() -> %% Using the option returns einval, so it is not implemented. recvtos_ok({unix,darwin}, OSVer) -> not semver_lt(OSVer, {17,6,0}); %% Using the option returns einval, so it is not implemented. -recvtos_ok({unix,openbsd}, OSVer) -> not semver_lt(OSVer, {6,8,0}); +recvtos_ok({unix,openbsd}, _OSVer) -> false; % not semver_lt(OSVer, {6,9,0}); +%% Using the option returns einval, so it is not implemented. +recvtos_ok({unix,netbsd}, _OSVer) -> false; %% Using the option returns einval, so it is not implemented. recvtos_ok({unix,sunos}, OSVer) -> not semver_lt(OSVer, {5,12,0}); %% @@ -781,7 +783,8 @@ recvtclass_ok(_, _) -> false. %% Using the option returns einval, so it is not implemented. sendtos_ok({unix,darwin}, OSVer) -> not semver_lt(OSVer, {19,0,0}); -sendtos_ok({unix,openbsd}, OSVer) -> not semver_lt(OSVer, {6,8,0}); +sendtos_ok({unix,netbsd}, _OSVer) -> false; +sendtos_ok({unix,openbsd}, _OSVer) -> false; % not semver_lt(OSVer, {6,9,0}); sendtos_ok({unix,sunos}, OSVer) -> not semver_lt(OSVer, {5,12,0}); sendtos_ok({unix,linux}, OSVer) -> not semver_lt(OSVer, {4,0,0}); sendtos_ok({unix,freebsd}, OSVer) -> not semver_lt(OSVer, {12,2,0}); @@ -790,13 +793,13 @@ sendtos_ok({unix,_}, _) -> true; sendtos_ok(_, _) -> false. %% Using the option returns einval, so it is not implemented. -sendttl_ok({unix,darwin}, OSVer) -> not semver_lt(OSVer, {19,4,0}); +sendttl_ok({unix,darwin}, OSVer) -> false; % not semver_lt(OSVer, {19,6,0}); sendttl_ok({unix,linux}, OSVer) -> not semver_lt(OSVer, {4,0,0}); %% Using the option returns enoprotoopt, so it is not implemented. sendttl_ok({unix,freebsd}, OSVer) -> not semver_lt(OSVer, {12,2,0}); %% Option has no effect sendttl_ok({unix,sunos}, OSVer) -> not semver_lt(OSVer, {5,12,0}); -sendttl_ok({unix,openbsd}, OSVer) -> not semver_lt(OSVer, {6,8,0}); +sendttl_ok({unix,openbsd}, _OSVer) -> false; % not semver_lt(OSVer, {6,9,0}); %% sendttl_ok({unix,_}, _) -> true; sendttl_ok(_, _) -> false. diff --git a/lib/kernel/test/inet_res_SUITE.erl b/lib/kernel/test/inet_res_SUITE.erl index 91ff883466..54686c326a 100644 --- a/lib/kernel/test/inet_res_SUITE.erl +++ b/lib/kernel/test/inet_res_SUITE.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2009-2020. All Rights Reserved. +%% Copyright Ericsson AB 2009-2021. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. @@ -33,7 +33,8 @@ ]). -export([basic/1, resolve/1, edns0/1, txt_record/1, files_monitor/1, last_ms_answer/1, intermediate_error/1, - servfail_retry_timeout_default/1, servfail_retry_timeout_1000/1 + servfail_retry_timeout_default/1, servfail_retry_timeout_1000/1, + label_compression_limit/1 ]). -export([ gethostbyaddr/0, gethostbyaddr/1, @@ -71,7 +72,9 @@ suite() -> all() -> [basic, resolve, edns0, txt_record, files_monitor, last_ms_answer, - intermediate_error, servfail_retry_timeout_default, servfail_retry_timeout_1000, + intermediate_error, + servfail_retry_timeout_default, servfail_retry_timeout_1000, + label_compression_limit, gethostbyaddr, gethostbyaddr_v6, gethostbyname, gethostbyname_v6, getaddr, getaddr_v6, ipv4_to_ipv6, host_and_addr]. @@ -974,6 +977,82 @@ servfail_retry_timeout_1000(Config) when is_list(Config) -> %% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%% Test that label encoding compression limits at 14 bits pointer size + +label_compression_limit(Config) when is_list(Config) -> + FirstSz = 8, + Count = 512, + Sz = 20, + %% We create a DNS message with an answer list containing + %% 1+512+1 RR:s. The first label is 8 chars that with message + %% and RR overhead places the second label on offset 32. + %% All other labels are 20 chars that with RR overhead + %% places them on offsets of N * 32. + %% + %% The labels are: "ZZZZZZZZ", then; "AAAAAAAAAAAAAAAAAAAA", + %% "AAAAAAAAAAAAAAAAAAAB", incrementing, so no one is + %% equal and can not be compressed, until the last one + %% that refers to the second to last one, so it could be compressed. + %% + %% However, the second to last label lands on offset 512 * 32 = 16384 + %% which is out of reach for compression since compression uses + %% a 14 bit reference from the start of the message. + %% + %% The last label can only be compressed when we instead + %% generate a message with one less char in the first label, + %% placing the second to last label on offset 16383. + %% + %% So, MsgShort can use compression for the last RR + %% by referring to the second to last RR, but MsgLong can not. + %% + %% Disclaimer: + %% All offsets and overheads are deduced + %% through trial and observation + %% + [D | Domains] = gen_domains(Count, lists:duplicate(Sz, $A), []), + LastD = "Y." ++ D, + DomainsShort = + [lists:duplicate(FirstSz-1, $Z) | + lists:reverse(Domains, [D, LastD])], + DomainsLong = + [lists:duplicate(FirstSz, $Z) | + lists:reverse(Domains, [D, LastD])], + MsgShort = gen_msg(DomainsShort), + MsgLong = gen_msg(DomainsLong), + DataShort = inet_dns:encode(MsgShort), + DataShortSz = byte_size(DataShort), + ?P("DataShort[~w]:~n ~p~n", [DataShortSz, DataShort]), + DataLong = inet_dns:encode(MsgLong), + DataLongSz = byte_size(DataLong), + ?P("DataLong[~w]:~n ~p~n", [DataLongSz, DataLong]), + %% When we increase the first RR size by 1, the compressed + %% label that occupied a 2 bytes reference instead becomes + %% a label with 1 byte size and a final empty label size 1 + 0 = DataLongSz - (DataShortSz+1 - 2 + 1+Sz+1), + ok. + +gen_msg(Domains) -> + inet_dns:make_msg( + [{header, inet_dns:make_header()}, + {anlist, gen_rrs(Domains)}]). + +gen_rrs(Domains) -> + [inet_dns:make_rr([{class,in},{type,a},{domain,D}]) || + D <- Domains]. + +gen_domains(0, _Domain, Acc) -> + Acc; +gen_domains(N, Domain, Acc) -> + gen_domains( + N - 1, incr_domain(Domain), [lists:reverse(Domain) | Acc]). + +incr_domain([$Z | Domain]) -> + [$A | incr_domain(Domain)]; +incr_domain([Char | Domain]) -> + [Char+1 | Domain]. + + +%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Compatibility tests. Call the inet_SUITE tests, but with %% lookup = [file,dns] instead of [native] diff --git a/lib/kernel/test/interactive_shell_SUITE.erl b/lib/kernel/test/interactive_shell_SUITE.erl index f058b53497..3993f99ed0 100644 --- a/lib/kernel/test/interactive_shell_SUITE.erl +++ b/lib/kernel/test/interactive_shell_SUITE.erl @@ -766,7 +766,7 @@ start_runerl_node(RunErl,Erl,Tempdir,Nodename,Args) -> start_runerl_command(RunErl, Tempdir, Cmd) -> FullCmd = "\""++RunErl++"\" "++Tempdir++"/ "++Tempdir++" \""++Cmd++"\"", - ct:pal("~s",[FullCmd]), + ct:pal("~ts",[FullCmd]), os:cmd(FullCmd). start_toerl_server(ToErl,Tempdir) -> diff --git a/lib/kernel/test/kernel_test_lib.erl b/lib/kernel/test/kernel_test_lib.erl index a0d62023f2..c4de916b1a 100644 --- a/lib/kernel/test/kernel_test_lib.erl +++ b/lib/kernel/test/kernel_test_lib.erl @@ -1383,7 +1383,7 @@ linux_info_lookup_collect(Key1, [Key2, Value|Rest], Values) -> linux_info_lookup_collect(_, _, Values) -> lists:reverse(Values). -maybe_skip(HostInfo) -> +maybe_skip(_HostInfo) -> %% We have some crap machines that causes random test case failures %% for no obvious reason. So, attempt to identify those without actually @@ -1442,14 +1442,19 @@ maybe_skip(HostInfo) -> true end, SkipWindowsOnVirtual = + %% fun() -> + %% SysMan = win_sys_info_lookup(system_manufacturer, HostInfo), + %% case string:to_lower(SysMan) of + %% "vmware" ++ _ -> + %% true; + %% _ -> + %% false + %% end + %% end, fun() -> - SysMan = win_sys_info_lookup(system_manufacturer, HostInfo), - case string:to_lower(SysMan) of - "vmware" ++ _ -> - true; - _ -> - false - end + %% The host has been replaced and the VM has been reinstalled + %% so for now we give it a chance... + false end, COND = [{unix, [{linux, LinuxVersionVerify}, {darwin, DarwinVersionVerify}]}, diff --git a/lib/kernel/test/os_SUITE_data/my_echo.c b/lib/kernel/test/os_SUITE_data/my_echo.c index 712c828bb5..439f812af1 100644 --- a/lib/kernel/test/os_SUITE_data/my_echo.c +++ b/lib/kernel/test/os_SUITE_data/my_echo.c @@ -1,3 +1,4 @@ +#include <stdio.h> #ifdef __WIN32__ #include <windows.h> @@ -25,8 +26,6 @@ int wmain(int argc, wchar_t **argv) } #else -#include <stdio.h> - int main(int argc, char** argv) { diff --git a/lib/kernel/test/sendfile_SUITE.erl b/lib/kernel/test/sendfile_SUITE.erl index 36e6ca555f..a2a19caf35 100644 --- a/lib/kernel/test/sendfile_SUITE.erl +++ b/lib/kernel/test/sendfile_SUITE.erl @@ -23,7 +23,7 @@ -include_lib("common_test/include/ct.hrl"). -include_lib("kernel/include/file.hrl"). --export([all/0, init_per_suite/1, end_per_suite/1, init_per_testcase/2]). +-export([all/0, init_per_suite/1, end_per_suite/1, init_per_testcase/2, end_per_testcase/2]). -export([sendfile_server/2, sendfile_do_recv/2, init/1, handle_event/2]). @@ -107,7 +107,27 @@ init_per_testcase(TC,Config) when TC == t_sendfile_recvduring; {skip,"Not supported"} end; init_per_testcase(_TC,Config) -> - Config. + case read_fd_info() of + {ok, NumFDs, FDDetails} -> + [{fds,NumFDs},{details,FDDetails}|Config]; + {error,_Reason} -> + Config + end. + +end_per_testcase(_TC,Config) -> + case proplists:get_value(fds, Config) of + undefined -> + ok; + NumOldFDs -> + case read_fd_info() of + {ok, NumFDs, FDDetails} when NumFDs =/= NumOldFDs -> + ct:log("FDs: ~n~ts~nOldFDs: ~n~ts~n", + [FDDetails,proplists:get_value(details,Config)]), + {fail,"Too many (or too few) fds open"}; + _ -> + ok + end + end. t_sendfile_small(Config) when is_list(Config) -> Filename = proplists:get_value(small_file, Config), @@ -171,7 +191,8 @@ t_sendfile_big_size(Config) -> {ok, #file_info{size = Size}} = file:read_file_info(Filename), {ok,D} = file:open(Filename,[read|FileOpts]), - {ok, Size} = file:sendfile(D, Sock,0,Size,SendfileOpts), + {ok,Size} = file:sendfile(D,Sock,0,Size,SendfileOpts), + ok = file:close(D), Size end, @@ -507,6 +528,18 @@ sendfile(Filename,Sock,Opts) -> Res end. +%% This function returns the number of open fds on a system +%% and also a string representing more detailed information +%% for debugging. +%% It only supports linux for now. +read_fd_info() -> + ProcFd = "/proc/" ++ os:getpid() ++ "/fd", + case file:list_dir(ProcFd) of + {ok, FDs} -> + {ok, length(FDs), os:cmd("ls -l " ++ ProcFd)}; + Error -> + Error + end. %% Error handler diff --git a/lib/kernel/test/seq_trace_SUITE.erl b/lib/kernel/test/seq_trace_SUITE.erl index f8efd1ffea..4a00b8d3d0 100644 --- a/lib/kernel/test/seq_trace_SUITE.erl +++ b/lib/kernel/test/seq_trace_SUITE.erl @@ -26,7 +26,7 @@ init_per_group/2,end_per_group/2, init_per_testcase/2,end_per_testcase/2]). -export([token_set_get/1, tracer_set_get/1, print/1, - old_heap_token/1, + old_heap_token/1,mature_heap_token/1, send/1, distributed_send/1, recv/1, distributed_recv/1, trace_exit/1, distributed_exit/1, call/1, port/1, port_clean_token/1, @@ -54,7 +54,7 @@ suite() -> all() -> [token_set_get, tracer_set_get, print, send, send_literal, distributed_send, recv, distributed_recv, trace_exit, - old_heap_token, + old_heap_token, mature_heap_token, distributed_exit, call, port, match_set_seq_token, port_clean_token, gc_seq_token, label_capability_mismatch, @@ -538,18 +538,24 @@ call(Config) when is_list(Config) -> %% The token should follow spawn, just like it follows messages. inherit_on_spawn(Config) when is_list(Config) -> - lists:foreach(fun (Test) -> - inherit_on_spawn_test(Test) - end, - [spawn, spawn_link, spawn_monitor, - spawn_opt, spawn_request]), + lists:foreach( + fun (Test) -> + lists:foreach( + fun (TraceFlags) -> + inherit_on_spawn_test(Test, TraceFlags) + end, + combinations(spawn_trace_flags())) + end, + [spawn, spawn_link, spawn_monitor, + spawn_opt, spawn_request]), ok. -inherit_on_spawn_test(Spawn) -> - io:format("Testing ~p()~n", [Spawn]), +inherit_on_spawn_test(Spawn, TraceFlags) -> + io:format("Testing ~p() with ~p trace flags~n", [Spawn, TraceFlags]), seq_trace:reset_trace(), start_tracer(), + start_spawn_tracer(TraceFlags), Ref = make_ref(), seq_trace:set_token(label,Ref), @@ -580,6 +586,7 @@ inherit_on_spawn_test(Spawn) -> receive {gurka,Ref} -> ok end, seq_trace:reset_trace(), + erlang:trace(self(),false,[procs|TraceFlags]), Sequence = lists:keysort(3, stop_tracer(6)), io:format("Sequence: ~p~n", [Sequence]), @@ -644,9 +651,35 @@ inherit_on_spawn_test(Spawn) -> GurkaMsg}, _} = RGurkaMsg, + + Links = not(spawn =:= spawn orelse Spawn =:= spawn_monitor), + SoL = lists:member(set_on_link,TraceFlags) orelse + lists:member(set_on_first_link,TraceFlags), + SoS = lists:member(set_on_spawn,TraceFlags) orelse + lists:member(set_on_first_sapwn,TraceFlags), + + NoTraceMessages = + if + SoS andalso Links -> + 4; + SoS andalso not Links -> + 2; + SoL andalso Links -> + 4; + SoL andalso not Links-> + 1; + Links andalso not SoL andalso not SoS -> + 2; + not Links andalso not SoL andalso not SoS -> + 1 + end, + + TraceMessages = stop_spawn_tracer(NoTraceMessages), + unlink(Other), exit(Other, kill), + ok. inherit_on_dist_spawn(Config) when is_list(Config) -> @@ -971,6 +1004,24 @@ old_heap_token(Config) when is_list(Config) -> {label,NewLabel} = seq_trace:get_token(label), ok. +%% Verify changing label on existing token when it resides on mature heap. +%% Bug caused faulty ref from old to new heap. +mature_heap_token(Config) when is_list(Config) -> + + seq_trace:set_token(label, 1), + erlang:garbage_collect(self(), [{type, minor}]), + %% Now token should be on mature heap + %% Set a new non-literal label which should reside on new-heap. + NewLabel = {self(), "new label"}, + seq_trace:set_token(label, NewLabel), + + %% If bug, we now have a ref from mature to new heap. If we now GC + %% twice the token will refer to deallocated memory. + erlang:garbage_collect(self(), [{type, minor}]), + erlang:garbage_collect(self(), [{type, minor}]), + {label,NewLabel} = seq_trace:get_token(label), + ok. + match_set_seq_token(doc) -> ["Tests that match spec function set_seq_token does not " @@ -1366,7 +1417,6 @@ start_tracer(Node) -> unlink(Pid), Pid end. - set_token_flags([]) -> ok; @@ -1379,6 +1429,51 @@ set_token_flags([Flag|Flags]) -> seq_trace:set_token(Flag, true), set_token_flags(Flags). +start_spawn_tracer(TraceFlags) -> + + %% Disable old trace flags + erlang:trace(self(), false, spawn_trace_flags()), + + Me = self(), + Ref = make_ref(), + Pid = spawn_link( + fun () -> + register(spawn_tracer, self()), + Me ! Ref, + (fun F(Data) -> + receive + {get, N, StopRef, Pid} when N =< length(Data) -> + Pid ! {lists:reverse(Data), StopRef}; + M when element(1,M) =:= trace -> + F([M|Data]) + end + end)([]) + end), + receive + Ref -> + erlang:trace(self(),true,[{tracer,Pid}, procs | TraceFlags]) + end. + +stop_spawn_tracer(N) -> + Ref = make_ref(), + spawn_tracer ! {get, N, Ref, self()}, + receive + {Data, Ref} -> + Data + end. + +spawn_trace_flags() -> + [set_on_spawn, set_on_link, set_on_spawn, + set_on_first_link, set_on_first_spawn]. + +combinations(Flags) -> + %% Do a bit of sofs magic to create a list of lists with + %% all the combinations of all the flags above + Set = sofs:from_term(Flags), + Product = sofs:product(list_to_tuple(lists:duplicate(length(Flags),Set))), + Combinations = [lists:usort(tuple_to_list(T)) || T <- sofs:to_external(Product)], + [[] | lists:usort(Combinations)]. + check_ts(no_timestamp, Ts) -> try no_timestamp = Ts diff --git a/lib/kernel/test/zlib_SUITE.erl b/lib/kernel/test/zlib_SUITE.erl index 52ae1b3ae6..215c91ef76 100644 --- a/lib/kernel/test/zlib_SUITE.erl +++ b/lib/kernel/test/zlib_SUITE.erl @@ -395,6 +395,7 @@ api_inflateReset(Config) when is_list(Config) -> api_inflate2(Config) when is_list(Config) -> Data = [<<1,2,2,3,3,3,4,4,4,4>>], Compressed = zlib:compress(Data), + Z1 = zlib:open(), ?m(ok, zlib:inflateInit(Z1)), ?m([], zlib:inflate(Z1, <<>>)), @@ -408,7 +409,20 @@ api_inflate2(Config) when is_list(Config) -> ?m(ok, zlib:inflateEnd(Z1)), ?m(ok, zlib:inflateInit(Z1)), ?m(?EXIT(data_error), zlib:inflate(Z1, <<2,1,2,1,2>>)), - ?m(ok, zlib:close(Z1)). + ?m(ok, zlib:close(Z1)), + + %% OTP-17299: we failed to fully flush the zlib state if we ran out of + %% input and filled the internal output buffer at the same time. + EdgeCaseData = <<"gurka", 0:16384/integer-unit:8>>, + EdgeCaseZipped = zlib:zip(EdgeCaseData), + Z2 = zlib:open(), + ?m(ok, zlib:inflateInit(Z2, -15)), + Unzipped = iolist_to_binary(zlib:inflate(Z2, EdgeCaseZipped)), + ?m(EdgeCaseData, Unzipped), + ?m(ok, zlib:inflateEnd(Z2)), + ?m(ok, zlib:close(Z2)), + + ok. %% Test inflate/3; same as inflate/2 but with the default options inverted. api_inflate3(Config) when is_list(Config) -> diff --git a/lib/kernel/vsn.mk b/lib/kernel/vsn.mk index 4a226539a3..c04299ae88 100644 --- a/lib/kernel/vsn.mk +++ b/lib/kernel/vsn.mk @@ -1 +1 @@ -KERNEL_VSN = 7.2.1 +KERNEL_VSN = 7.3 diff --git a/lib/megaco/test/megaco_config_SUITE.erl b/lib/megaco/test/megaco_config_SUITE.erl index 4d34e09d0d..557406f8c4 100644 --- a/lib/megaco/test/megaco_config_SUITE.erl +++ b/lib/megaco/test/megaco_config_SUITE.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2000-2019. All Rights Reserved. +%% Copyright Ericsson AB 2000-2021. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. @@ -600,7 +600,7 @@ transaction_id_counter_mg(Config) when is_list(Config) -> %% Await the counter worker procs termination i("await the counter working procs completion"), - await_completion_counter_working_procs(Pids), + ok = await_completion_counter_working_procs(Pids), %% Verify result i("verify counter result"), @@ -656,15 +656,52 @@ start_counter_working_procs([Pid | Pids]) -> Pid ! start, start_counter_working_procs(Pids). -await_completion_counter_working_procs([]) -> - ok; await_completion_counter_working_procs(Pids) -> + await_completion_counter_working_procs(Pids, [], []). + +await_completion_counter_working_procs([], _OKs, [] = _ERRs) -> + ok; +await_completion_counter_working_procs([], _OKs, ERRs) -> + {error, ERRs}; +await_completion_counter_working_procs(Pids, OKs, ERRs) -> receive {'EXIT', Pid, normal} -> + %% i("counter working process completion[~w, ~w, ~w] -> " + %% "Expected exit from counter process: " + %% "~n Pid: ~p", + %% [length(Pids), length(OKs), length(ERRs), Pid]), + Pids2 = lists:delete(Pid, Pids), + await_completion_counter_working_procs(Pids2, [Pid | OKs], ERRs); + {'EXIT', Pid, Reason} -> + e("counter working process completion[~w, ~w, ~w] -> " + "Unexpected exit from counter process: " + "~n Pid: ~p" + "~n Reason: ~p", + [length(Pids), length(OKs), length(ERRs), Pid, Reason]), Pids2 = lists:delete(Pid, Pids), - await_completion_counter_working_procs(Pids2); - _Any -> + await_completion_counter_working_procs(Pids2, OKs, [Pid | ERRs]); + + Any -> + e("counter working process completion[~w, ~w, ~w] -> " + "Unexpected message: " + "~n ~p", [length(Pids), length(OKs), length(ERRs), Any]), await_completion_counter_working_procs(Pids) + + after 10000 -> + %% If nothing has happened for this long, something is wrong: + %% Check system events + case megaco_test_global_sys_monitor:events() of + [] -> + i("counter working process completion[~w, ~w, ~w] -> " + "idle", [length(Pids), length(OKs), length(ERRs)]), + await_completion_counter_working_procs(Pids); + SysEvs -> + e("counter working process completion[~w, ~w, ~w] -> " + "system event(s): " + "~n ~p", + [length(Pids), length(OKs), length(ERRs), SysEvs]), + ?SKIP("TC idle with system events") + end end. @@ -677,119 +714,123 @@ transaction_id_counter_mgc(doc) -> "transaction counter handling of the application " "in with several connections (MGC). "]; transaction_id_counter_mgc(Config) when is_list(Config) -> - put(verbosity, ?TEST_VERBOSITY), - put(sname, "TEST"), - put(tc, transaction_id_counter_mgc), - process_flag(trap_exit, true), - - i("starting"), - - {ok, _ConfigPid} = megaco_config:start_link(), - - %% Basic user data - UserMid = {deviceName, "mgc"}, - UserConfig = [ - {min_trans_id, 1} - ], - - %% Basic connection data - RemoteMids = - [ - {deviceName, "mg01"}, - {deviceName, "mg02"}, - {deviceName, "mg03"}, - {deviceName, "mg04"}, - {deviceName, "mg05"}, - {deviceName, "mg06"}, - {deviceName, "mg07"}, - {deviceName, "mg08"}, - {deviceName, "mg09"}, - {deviceName, "mg10"} - ], - RecvHandles = - [ - #megaco_receive_handle{local_mid = UserMid, - encoding_mod = ?MODULE, - encoding_config = [], - send_mod = ?MODULE}, - #megaco_receive_handle{local_mid = UserMid, - encoding_mod = ?MODULE, - encoding_config = [], - send_mod = ?MODULE}, - #megaco_receive_handle{local_mid = UserMid, - encoding_mod = ?MODULE, - encoding_config = [], - send_mod = ?MODULE}, - #megaco_receive_handle{local_mid = UserMid, - encoding_mod = ?MODULE, - encoding_config = [], - send_mod = ?MODULE}, - #megaco_receive_handle{local_mid = UserMid, - encoding_mod = ?MODULE, - encoding_config = [], - send_mod = ?MODULE}, - #megaco_receive_handle{local_mid = UserMid, - encoding_mod = ?MODULE, - encoding_config = [], - send_mod = ?MODULE}, - #megaco_receive_handle{local_mid = UserMid, - encoding_mod = ?MODULE, - encoding_config = [], - send_mod = ?MODULE}, - #megaco_receive_handle{local_mid = UserMid, - encoding_mod = ?MODULE, - encoding_config = [], - send_mod = ?MODULE}, - #megaco_receive_handle{local_mid = UserMid, - encoding_mod = ?MODULE, - encoding_config = [], - send_mod = ?MODULE}, - #megaco_receive_handle{local_mid = UserMid, - encoding_mod = ?MODULE, - encoding_config = [], - send_mod = ?MODULE} - ], - SendHandle = dummy_send_handle, - ControlPid = self(), + Name = transaction_id_counter_mgc, + Pre = fun() -> + i("starting config server"), + {ok, _ConfigPid} = megaco_config:start_link(), + + %% Basic user data + UserMid = {deviceName, "mgc"}, + UserConfig = [ + {min_trans_id, 1} + ], + + %% Basic connection data + RemoteMids = + [ + {deviceName, "mg01"}, + {deviceName, "mg02"}, + {deviceName, "mg03"}, + {deviceName, "mg04"}, + {deviceName, "mg05"}, + {deviceName, "mg06"}, + {deviceName, "mg07"}, + {deviceName, "mg08"}, + {deviceName, "mg09"}, + {deviceName, "mg10"} + ], + RecvHandles = + [ + #megaco_receive_handle{local_mid = UserMid, + encoding_mod = ?MODULE, + encoding_config = [], + send_mod = ?MODULE}, + #megaco_receive_handle{local_mid = UserMid, + encoding_mod = ?MODULE, + encoding_config = [], + send_mod = ?MODULE}, + #megaco_receive_handle{local_mid = UserMid, + encoding_mod = ?MODULE, + encoding_config = [], + send_mod = ?MODULE}, + #megaco_receive_handle{local_mid = UserMid, + encoding_mod = ?MODULE, + encoding_config = [], + send_mod = ?MODULE}, + #megaco_receive_handle{local_mid = UserMid, + encoding_mod = ?MODULE, + encoding_config = [], + send_mod = ?MODULE}, + #megaco_receive_handle{local_mid = UserMid, + encoding_mod = ?MODULE, + encoding_config = [], + send_mod = ?MODULE}, + #megaco_receive_handle{local_mid = UserMid, + encoding_mod = ?MODULE, + encoding_config = [], + send_mod = ?MODULE}, + #megaco_receive_handle{local_mid = UserMid, + encoding_mod = ?MODULE, + encoding_config = [], + send_mod = ?MODULE}, + #megaco_receive_handle{local_mid = UserMid, + encoding_mod = ?MODULE, + encoding_config = [], + send_mod = ?MODULE}, + #megaco_receive_handle{local_mid = UserMid, + encoding_mod = ?MODULE, + encoding_config = [], + send_mod = ?MODULE} + ], + SendHandle = dummy_send_handle, + ControlPid = self(), - %% Start user - i("start user"), - ok = megaco_config:start_user(UserMid, UserConfig), + %% Start user + i("start user"), + ok = megaco_config:start_user(UserMid, UserConfig), + + %% Create connection + i("create connection(s)"), + CDs = create_connections(RecvHandles, + RemoteMids, + SendHandle, + ControlPid), + + %% Set counter limits + i("set counter max limit(s)"), + set_counter_max_limits(CDs, 1000), + + {UserMid, CDs} + end, + Case = fun({_, CDs}) -> + %% Create the counter worker procs + i("create counter working procs"), + Pids = create_counter_working_procs(CDs, ?NUM_CNT_PROCS), + + %% Start the counter worker procs + i("release the counter working procs"), + start_counter_working_procs(Pids), + + %% Await the counter worker procs termination + i("await the counter working procs completion"), + ok = await_completion_counter_working_procs(Pids), + + %% Verify result + i("verify counter result"), + verify_counter_results(CDs) + end, + + Post = fun({UserMid, CDs}) -> + %% Stop test + i("disconnect"), + delete_connections(CDs), + i("stop user"), + ok = megaco_config:stop_user(UserMid), + i("stop megaco_config"), + ok = megaco_config:stop() + end, + try_tc(Name, Pre, Case, Post). - %% Create connection - i("create connection(s)"), - CDs = create_connections(RecvHandles, RemoteMids, SendHandle, ControlPid), - - %% Set counter limits - i("set counter max limit(s)"), - set_counter_max_limits(CDs, 1000), - - %% Create the counter worker procs - i("create counter working procs"), - Pids = create_counter_working_procs(CDs, ?NUM_CNT_PROCS), - - %% Start the counter worker procs - i("release the counter working procs"), - start_counter_working_procs(Pids), - - %% Await the counter worker procs termination - i("await the counter working procs completion"), - await_completion_counter_working_procs(Pids), - - %% Verify result - i("verify counter result"), - verify_counter_results(CDs), - - %% Stop test - i("disconnect"), - delete_connections(CDs), - i("stop user"), - ok = megaco_config:stop_user(UserMid), - i("stop megaco_config"), - ok = megaco_config:stop(), - - i("done"), - ok. create_connections(RecvHandles, RemoteMids, SendHandle, ControlPid) -> create_connections(RecvHandles, RemoteMids, SendHandle, ControlPid, []). @@ -1218,6 +1259,15 @@ otp_8183(Config) when is_list(Config) -> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +try_tc(TCName, Pre, Case, Post) -> + try_tc(TCName, "TEST", ?TEST_VERBOSITY, Pre, Case, Post). + +try_tc(TCName, Name, Verbosity, Pre, Case, Post) -> + ?TRY_TC(TCName, Name, Verbosity, Pre, Case, Post). + + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + p(F) -> p(F, []). @@ -1234,7 +1284,10 @@ i(F) -> i(F, []). i(F, A) -> - print(info, get(verbosity), get(tc), "INF", F, A). + print(info, get(verbosity), get(tc), "INFO", F, A). + +e(F, A) -> + print(info, get(verbosity), get(tc), "ERROR", F, A). printable(_, debug) -> true; printable(info, info) -> true; diff --git a/lib/megaco/test/megaco_segment_SUITE.erl b/lib/megaco/test/megaco_segment_SUITE.erl index a403c3309d..3763a20954 100644 --- a/lib/megaco/test/megaco_segment_SUITE.erl +++ b/lib/megaco/test/megaco_segment_SUITE.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2006-2020. All Rights Reserved. +%% Copyright Ericsson AB 2006-2021. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. @@ -2827,6 +2827,42 @@ do_send_segmented_msg_ooo1([MgcNode, MgNode]) -> d("[MG] start the simulation"), {ok, MgId} = megaco_test_megaco_generator:exec(Mg, MgEvSeq), + %% Await MGC ready for segments + d("await MGC trigger event"), + MgcPid = + receive + {ready_for_segmented_msg, mgc, Pid1} -> + d("received MGC trigger event"), + Pid1 + after 5000 -> + d("timeout waiting for MGC trigger event: ~p", + [megaco_test_lib:flush()]), + ?ERROR(timeout_MGC_trigger_event) + end, + + %% Await MG ready for segments + d("await MG trigger event"), + MgPid = + receive + {ready_for_segmented_msg, mg, Pid2} -> + d("received MG trigger event"), + Pid2 + after 5000 -> + d("timeout waiting for MG trigger event: ~p", + [megaco_test_lib:flush()]), + ?ERROR(timeout_MG_trigger_event) + end, + + %% Instruct the MG to continue + d("send continue to MG"), + MgPid ! {continue_with_segmented_msg, self()}, + + sleep(500), + + %% Instruct the MGC to continue + d("send continue to MGC"), + MgcPid ! {continue_with_segmented_msg, self()}, + d("await the generator reply(s)"), await_completion([MgcId, MgId]), @@ -2853,6 +2889,8 @@ ssmo1_mgc_event_sequence(text, tcp) -> Mid = {deviceName,"mgc"}, ScrVerifyFun = ssmo1_mgc_verify_service_change_req_msg_fun(), ServiceChangeRep = ssmo1_mgc_service_change_reply_msg(Mid, 1), + AnnounceReadySegs = ssmo1_mgc_announce_ready_for_segmented_msg_fun(), + AwaitContinueSegs = ssmo1_mgc_continue_with_segmented_msg_fun(), TermId1 = #megaco_term_id{id = ["00000000","00000000","00000001"]}, CtxId1 = 1, @@ -2923,7 +2961,13 @@ ssmo1_mgc_event_sequence(text, tcp) -> {expect_accept, any}, {expect_receive, "service-change-request", {ScrVerifyFun, 5000}}, {send, "service-change-reply", ServiceChangeRep}, - {expect_nothing, 1000}, + + {trigger, "announce ready for segmented message", + AnnounceReadySegs}, + {trigger, "await continue for segmented message", + AwaitContinueSegs}, + + %% {expect_nothing, 1000}, {send, "notify request", NotifyReq}, {expect_receive, "notify reply: segment 1", {NrVerifyFun1, 1000}}, {expect_receive, "notify reply: segment 2", {NrVerifyFun2, 1000}}, @@ -3053,6 +3097,23 @@ ssmo1_mgc_verify_service_change_req(#'MegacoMessage'{mess = Mess} = M) -> {error, {invalid_serviceChangeParms, Parms}} end. +ssmo1_mgc_announce_ready_for_segmented_msg_fun() -> + TC = self(), + fun() -> + TC ! {ready_for_segmented_msg, mgc, self()} + end. + +ssmo1_mgc_continue_with_segmented_msg_fun() -> + TC = self(), + fun() -> + p("[MGC] await continue with segmented message"), + receive + {continue_with_segmented_msg, TC} -> + p("[MGC] received continue with segmented message"), + ok + end + end. + ssmo1_mgc_verify_notify_reply_segment_msg_fun(SN, Last, TransId, TermId, Cid) -> fun(Msg) -> @@ -3219,6 +3280,8 @@ ssmo1_mg_event_sequence(text, tcp) -> ConnectVerify = ssmo1_mg_verify_handle_connect_fun(), ServiceChangeReq = ssmo1_mg_service_change_request_ar(Mid, 1), ServiceChangeReplyVerify = ssmo1_mg_verify_service_change_reply_fun(), + AnnounceReadySegs = ssmo1_mg_announce_ready_for_segmented_msg_fun(), + AwaitContinueSegs = ssmo1_mg_continue_with_segmented_msg_fun(), Tid1 = #megaco_term_id{id = ["00000000","00000000","00000001"]}, Tid2 = #megaco_term_id{id = ["00000000","00000000","00000002"]}, Tid3 = #megaco_term_id{id = ["00000000","00000000","00000003"]}, @@ -3247,8 +3310,13 @@ ssmo1_mg_event_sequence(text, tcp) -> {megaco_callback, handle_trans_reply, ServiceChangeReplyVerify}, {megaco_update_conn_info, protocol_version, ?VERSION}, {megaco_update_conn_info, segment_send, 3}, - {megaco_update_conn_info, max_pdu_size, 128}, - {sleep, 1000}, + {megaco_update_conn_info, max_pdu_size, 128}, + + {trigger, "announce ready for segmented message", + AnnounceReadySegs}, + {trigger, "await continue for segmented message", + AwaitContinueSegs}, + {megaco_callback, handle_trans_request, NotifyReqVerify}, {megaco_callback, handle_trans_ack, AckVerify, 15000}, megaco_stop_user, @@ -3257,7 +3325,6 @@ ssmo1_mg_event_sequence(text, tcp) -> ], EvSeq. - ssmo1_mg_verify_handle_connect_fun() -> fun(Ev) -> ssmo1_mg_verify_handle_connect(Ev) end. @@ -3333,6 +3400,23 @@ ssmo1_mg_do_verify_scr(AR) -> {error, Reason6, ok} end. +ssmo1_mg_announce_ready_for_segmented_msg_fun() -> + TC = self(), + fun() -> + TC ! {ready_for_segmented_msg, mg, self()} + end. + +ssmo1_mg_continue_with_segmented_msg_fun() -> + TC = self(), + fun() -> + p("[MG] await continue with segmented message"), + receive + {continue_with_segmented_msg, TC} -> + p("[MG] received continue with segmented message"), + ok + end + end. + ssmo1_mg_verify_notify_request_fun(Tids) -> fun(Req) -> ssmo1_mg_verify_notify_request(Req, Tids) end. @@ -7913,6 +7997,9 @@ try_tc(TCName, Name, Verbosity, Pre, Case, Post) -> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +p(F) -> + p(F, []). + p(F, A) -> io:format("*** [~s] ~p ***" "~n " ++ F ++ "~n", diff --git a/lib/megaco/test/megaco_test_lib.erl b/lib/megaco/test/megaco_test_lib.erl index 49168ea565..a04b27a061 100644 --- a/lib/megaco/test/megaco_test_lib.erl +++ b/lib/megaco/test/megaco_test_lib.erl @@ -494,7 +494,7 @@ init_per_suite(Config) -> SKIP end. -maybe_skip(HostInfo) -> +maybe_skip(_HostInfo) -> %% We have some crap machines that causes random test case failures %% for no obvious reason. So, attempt to identify those without actually @@ -553,14 +553,19 @@ maybe_skip(HostInfo) -> true end, SkipWindowsOnVirtual = + %% fun() -> + %% SysMan = win_sys_info_lookup(system_manufacturer, HostInfo), + %% case string:to_lower(SysMan) of + %% "vmware" ++ _ -> + %% true; + %% _ -> + %% false + %% end + %% end, fun() -> - SysMan = win_sys_info_lookup(system_manufacturer, HostInfo), - case string:to_lower(SysMan) of - "vmware" ++ _ -> - true; - _ -> - false - end + %% The host has been replaced and the VM has been reinstalled + %% so for now we give it a chance... + false end, COND = [ {unix, [{linux, LinuxVersionVerify}, diff --git a/lib/megaco/test/megaco_test_megaco_generator.erl b/lib/megaco/test/megaco_test_megaco_generator.erl index 4eedd8d731..f6ea57ab41 100644 --- a/lib/megaco/test/megaco_test_megaco_generator.erl +++ b/lib/megaco/test/megaco_test_megaco_generator.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2007-2020. All Rights Reserved. +%% Copyright Ericsson AB 2007-2021. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. @@ -313,6 +313,9 @@ handle_parse({megaco_callback, Verifiers0} = _Instruction, State) handle_parse({trigger, Trigger} = Instruction, State) when is_function(Trigger) -> {ok, Instruction, State}; +handle_parse({trigger, Desc, Trigger} = Instruction, State) + when is_list(Desc) andalso is_function(Trigger) -> + {ok, Instruction, State}; handle_parse(Instruction, _State) -> error({invalid_instruction, Instruction}). @@ -770,6 +773,10 @@ handle_exec({trigger, Trigger}, State) when is_function(Trigger) -> p("trigger"), (catch Trigger()), {ok, State}; +handle_exec({trigger, Desc, Trigger}, State) when is_function(Trigger) -> + p("trigger: ~s", [Desc]), + (catch Trigger()), + {ok, State}; handle_exec({sleep, To}, State) -> p("sleep ~p", [To]), diff --git a/lib/megaco/test/megaco_test_mgc.erl b/lib/megaco/test/megaco_test_mgc.erl index 8a9b182368..1204dbba07 100644 --- a/lib/megaco/test/megaco_test_mgc.erl +++ b/lib/megaco/test/megaco_test_mgc.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2003-2020. All Rights Reserved. +%% Copyright Ericsson AB 2003-2021. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. @@ -411,7 +411,7 @@ loop(S) -> server_reply(Parent, update_conn_info_ack, Res), loop(evs(S, {uci, {Tag, Val}})); - {{conn_info, Tag}, Parent} when S#mgc.parent == Parent -> + {{conn_info, Tag}, Parent} when S#mgc.parent =:= Parent -> i("loop -> got conn_info request for ~w", [Tag]), Conns = megaco:user_info(S#mgc.mid, connections), Fun = fun(CH) -> @@ -450,48 +450,63 @@ loop(S) -> %% Give me statistics {{statistics, 1}, Parent} when S#mgc.parent == Parent -> - i("loop -> got request for statistics 1"), + i("loop(stats1) -> got request for statistics 1"), {ok, Gen} = megaco:get_stats(), - GetTrans = + i("loop(stats1) -> gen stats: " + "~n ~p", [Gen]), + GetTrans = fun(CH) -> + i("loop(stats1):GetTrans -> " + "get stats for connection ~p", [CH]), Reason = {statistics, CH}, Pid = megaco:conn_info(CH, control_pid), + i("loop(stats1):GetTrans -> control pid: ~p", [Pid]), SendMod = megaco:conn_info(CH, send_mod), + i("loop(stats1):GetTrans -> " + "send module: ~p", [SendMod]), SendHandle = megaco:conn_info(CH, send_handle), + i("loop(stats1):GetTrans -> " + "send handle: ~p", [SendHandle]), {ok, Stats} = case SendMod of megaco_tcp -> megaco_tcp:get_stats(SendHandle); megaco_udp -> megaco_udp:get_stats(SendHandle); SendMod -> exit(Pid, Reason) end, + i("loop(stats1):GetTrans -> stats: " + "~n ~p", [Stats]), {SendHandle, Stats} end, - Mid = S#mgc.mid, - Trans = - lists:map(GetTrans, megaco:user_info(Mid, connections)), + Mid = S#mgc.mid, + Trans = lists:map(GetTrans, megaco:user_info(Mid, connections)), Reply = {ok, [{gen, Gen}, {trans, Trans}]}, + i("loop(stats1) -> send reply"), server_reply(Parent, {statistics_reply, 1}, Reply), + i("loop(stats1) -> done"), loop(evs(S, {stats, 1})); {{statistics, 2}, Parent} when S#mgc.parent == Parent -> - i("loop -> got request for statistics 2"), + i("loop(stats2) -> got request for statistics 2"), {ok, Gen} = megaco:get_stats(), #mgc{tcp_sup = TcpSup, udp_sup = UdpSup} = S, TcpStats = get_trans_stats(TcpSup, megaco_tcp), UdpStats = get_trans_stats(UdpSup, megaco_udp), Reply = {ok, [{gen, Gen}, {trans, [TcpStats, UdpStats]}]}, + i("loop(stats2) -> send reply"), server_reply(Parent, {statistics_reply, 2}, Reply), + i("loop(stats2) -> done"), loop(evs(S, {stats, 2})); %% Megaco callback messages {request, Request, From} -> - d("loop -> received megaco request from ~p:" + d("loop(request) -> received megaco request from ~p:" "~n ~p", [From, Request]), {Reply, S1} = handle_megaco_request(Request, S), - d("loop -> send request reply: ~n~p", [Reply]), + d("loop(request) -> send reply: ~n~p", [Reply]), reply(From, Reply), + d("loop(request) -> done"), loop(evs(S1, {req, Request})); @@ -557,9 +572,14 @@ loop(S) -> evs(#mgc{evs = EVS} = S, Ev) when (length(EVS) < ?EVS_MAX) -> - S#mgc{evs = [{?FTS(), Ev}|EVS]}; + echo_evs(S#mgc{evs = [{?FTS(), Ev}|EVS]}); evs(#mgc{evs = EVS} = S, Ev) -> - S#mgc{evs = [{?FTS(), Ev}|lists:droplast(EVS)]}. + echo_evs(S#mgc{evs = [{?FTS(), Ev}|lists:droplast(EVS)]}). + +echo_evs(#mgc{evs = EVS} = S) -> + i("Events: " + "~n ~p", [EVS]), + S. done(#mgc{evs = EVS}, Reason) -> info_msg("Exiting with latest event(s): " diff --git a/lib/mnesia/doc/src/Mnesia_chap1.xml b/lib/mnesia/doc/src/Mnesia_chap1.xml index fd78d01ab4..6e66132a52 100644 --- a/lib/mnesia/doc/src/Mnesia_chap1.xml +++ b/lib/mnesia/doc/src/Mnesia_chap1.xml @@ -32,13 +32,13 @@ <rev>C</rev> <file>Mnesia_chap1.xml</file> </header> - <p>The Mnesia application provides a heavy duty real-time + <p>The Mnesia application provides a heavy-duty real-time distributed database.</p> <section> <title>Scope</title> <p>This User's Guide describes how to - build Mnesia database applications, and how to integrate + build Mnesia-backed applications, and how to integrate and use the Mnesia database management system with OTP. Programming constructs are described, and numerous programming examples are included to illustrate the use of @@ -51,7 +51,7 @@ </item> <item><seeguide marker="Mnesia_chap2">Getting Started</seeguide> introduces Mnesia with an example database. Examples - are included how to start an Erlang session, specify a + are included on how to start an Erlang session, specify a Mnesia database directory, initialize a database schema, start Mnesia, and create tables. Initial prototyping of record definitions is also discussed. @@ -64,29 +64,29 @@ </item> <item><seeguide marker="Mnesia_chap4">Transactions and Other Access Contexts</seeguide> describes the transactions properties that make Mnesia into - a fault tolerant, real-time distributed database management + a fault-tolerant, real-time distributed database management system. This section also describes the concept of locking to ensure consistency in tables, and "dirty - operations", or short cuts, which bypass the transaction system + operations", or shortcuts, which bypass the transaction system to improve speed and reduce overheads. </item> <item><seeguide marker="Mnesia_chap5">Miscellaneous Mnesia Features</seeguide> describes features that enable the construction of more complex database applications. These features include indexing, checkpoints, distribution and fault - tolerance, disc-less nodes, replication manipulation, local + tolerance, disc-less nodes, replica manipulation, local content tables, concurrency, and object-based programming in Mnesia. </item> <item><seeguide marker="Mnesia_chap7">Mnesia System Information</seeguide> describes the files contained in the Mnesia database directory, database configuration data, - core and table dumps, as well as the important subject of - backup, fall-back, and disaster recovery principles. + core and table dumps, as well as the functions used for + backup, restore, fallback, and disaster recovery. </item> <item><seeguide marker="Mnesia_chap8">Combine Mnesia with - SNMP</seeguide> is a short section that outlines Mnesia - integrated with SNMP. + SNMP</seeguide> is a short section that outlines + the integration between Mnesia and SNMP. </item> <item><seeguide marker="Mnesia_App_A">Appendix A: Backup Callback Interface</seeguide> is a program listing of the @@ -110,4 +110,3 @@ database management systems.</p> </section> </chapter> - diff --git a/lib/mnesia/doc/src/Mnesia_overview.xml b/lib/mnesia/doc/src/Mnesia_overview.xml index a34548f3a0..a021181299 100644 --- a/lib/mnesia/doc/src/Mnesia_overview.xml +++ b/lib/mnesia/doc/src/Mnesia_overview.xml @@ -22,7 +22,7 @@ </legalnotice> - <title>Mnesia</title> + <title>Overview</title> <prepared>Claes Wikström, Hans Nilsson and Håkan Mattsson</prepared> <responsible>Bjarne Däcker</responsible> <docno></docno> @@ -33,36 +33,33 @@ <file>Mnesia_overview.xml</file> </header> - <p>The management of data in telecommunications system has many - aspects, thereof some, but not all, are addressed by traditional - commercial Database Management Systems (DBMSs). In particular the - high level of fault tolerance that is required in many nonstop + <p>The management of data in telecommunications systems has many + aspects of which some, but not all, are addressed by traditional + Database Management Systems (DBMSs). In particular, the + high level of fault tolerance required in many nonstop systems, combined with requirements on the DBMS to run in the same - address space as the application, have led us to implement a new + address space as the applications, have led us to implement a new DBMS, called Mnesia.</p> - <p>Mnesia is implemented in, and tightly connected to Erlang. + <p>Mnesia is implemented in, and tightly coupled to Erlang. It provides the functionality that is necessary for the - implementation of fault tolerant telecommunications systems.</p> - <p>Mnesia is a multiuser distributed DBMS specially made for - industrial telecommunications applications written in Erlang, + implementation of fault-tolerant telecommunications systems.</p> + <p>Mnesia is a multiuser distributed DBMS specifically designed for + industrial-grade telecommunications applications written in Erlang, which is also the intended target language. - Mnesia tries to address all the data - management issues required for typical telecommunications systems. - It has a number of features that are not normally found in traditional - databases.</p> - <p>In telecommunications applications, there are different needs - from the features provided by traditional DBMSs. The applications now - implemented in Erlang need a mixture of a broad range - of features, which generally are not satisfied by traditional DBMSs. - Mnesia is designed with requirements like the following in - mind:</p> - <list type="ordered"> - <item>Fast real-time key/value lookup + Mnesia tries to address all the data management issues required for + typical telecommunications systems and has a number of features not + normally found in traditional DBMSs.</p> + <p>Telecommunications applications need a mix of a broad range + of features generally not provided by traditional DBMSs. + Mnesia is designed to meet requirements such as:</p> + <list type="bulleted"> + <item>Fast real-time key-value lookup </item> - <item>Complicated non-real-time queries mainly for - operation and maintenance + <item>Complex non-real-time queries + (mainly for operation and maintenance tasks) </item> - <item>Distributed data because of distributed applications + <item>Distributed data + (due to the distributed nature of the applications) </item> <item>High fault tolerance </item> @@ -71,34 +68,33 @@ <item>Complex objects </item> </list> - <p>Mnesia is designed with the typical data management problems - of telecommunications applications in mind. This sets Mnesia - apart from most other DBMS. Hence Mnesia - combines many concepts found in traditional databases such as - transactions and queries with concepts found in data management - systems for telecommunications applications, for example:</p> + <p>Mnesia addresses the typical data management issues required for + telecommunications applications which sets it apart from most other DBMSs. + It combines many concepts found in traditional DBMSs, such as transactions + and queries, with concepts found in data management systems for + telecommunications applications such as:</p> <list type="bulleted"> <item>Fast real-time operations </item> - <item>Configurable degree of fault tolerance (by replication) + <item>Configurable replication for fault tolerance </item> - <item>The ability to reconfigure the system without stopping or - suspending it. + <item>Dynamic reconfiguration without service disruption </item> </list> - <p>Mnesia is also interesting because of its tight coupling to - Erlang, thus almost turning Erlang into a database programming - language. This has many benefits, the foremost is that + <p>Mnesia is also unique due to its tight coupling to + Erlang. It almost turns Erlang into a database programming + language, which yields many benefits. The foremost is that the impedance mismatch between the data format used by the DBMS and the data format used by the programming language, which is used to manipulate the data, completely disappears.</p> <section> - <title>Mnesia Database Management System (DBMS)</title> + <title>The Mnesia Database Management System</title> <section> <title>Features</title> - <p>Mnesia contains the following features that combine to - produce a fault-tolerant, distributed DBMS written in Erlang: + <p>Mnesia has the following features that combine to + produce a fault-tolerant distributed database management + system (DBMS) written in Erlang: </p> <list type="bulleted"> <item>Database schema can be dynamically reconfigured at runtime. @@ -107,85 +103,85 @@ replication, and persistence. </item> <item>Tables can be moved or replicated to several nodes to improve - fault tolerance. The rest of the system can still access the tables - to read, write, and delete records. + fault tolerance. Other nodes in the system can still access the + tables to read, write, and delete records. </item> <item>Table locations are transparent to the programmer. Programs address table names and the system itself keeps track of table locations. </item> - <item>Database transactions can be distributed, and many - functions can be called within one transaction. + <item>Transactions can be distributed and multiple + operations can be executed within a single transaction. </item> - <item>Several transactions can run concurrently, and their execution - is fully synchronized by the DBMS. Mnesia ensures that no - two processes manipulate data simultaneously. + <item>Multiple transactions can run concurrently and their execution + is fully synchronized by Mnesia, ensuring that no + two processes manipulate the same data simultaneously. </item> <item>Transactions can be assigned the property of being executed on - all nodes in the system, or on none. Transactions can also be - bypassed in favor of running "dirty operations", which reduce - overheads and run fast. + all nodes in the system, or on none. + </item> + <item>Transactions can be bypassed using dirty operations, + which reduce overheads and run fast. </item> </list> - <p>Details of these features are described in the following sections.</p> + <p>All of the above features are described in detail + in the coming sections.</p> </section> <section> - <title>Add-On Application</title> + <title>Query List Comprehension</title> <p>Query List Comprehension (QLC) can be used with Mnesia - to produce specialized functions that enhance the operational - ability of Mnesia. QLC has its own documentation as part - of the OTP documentation set. The main features of QLC - when used with Mnesia are as follows:</p> + to produce specialized functions that enhance its operational + ability. QLC has its own documentation as part + of the OTP documentation set. The main QLC advantages + when used with Mnesia are:</p> <list type="bulleted"> - <item>QLC can optimize the query compiler for the Mnesia - DBMS, essentially making the DBMS more efficient. + <item>QLC can optimize the query compiler for Mnesia, + essentially making the system more efficient. </item> <item>QLC can be used as a database programming - language for Mnesia. It includes a notation called "list - comprehensions" and can be used to make complex database + language for Mnesia. It includes a notation called list + comprehensions which can be used to execute complex database queries over a set of tables. </item> </list> - <p>For information about QLC, see the + <p>For more information about QLC, please see the <seeerl marker="stdlib:qlc">qlc</seeerl> manual page in STDLIB.</p> </section> <section> <title>When to Use Mnesia</title> - <p>Use Mnesia with the following types of applications:</p> + <p>Mnesia is a great fit for applications that:</p> <list type="bulleted"> - <item>Applications that need to replicate data. + <item>Need to replicate data. </item> - <item>Applications that perform complicated searches on data. + <item>Perform complex data queries. </item> - <item>Applications that need to use atomic transactions to - update several records simultaneously. + <item>Need to use atomic transactions to + safely update several records simultaneously. </item> - <item>Applications that use soft real-time characteristics. + <item>Require soft real-time characteristics. </item> </list> - <p>Mnesia is not as appropriate with the - following types of applications:</p> + <p>Mnesia is not as appropriate for applications that:</p> <list type="bulleted"> - <item>Programs that process plain text or binary data files. + <item>Process plain text or binary data files. </item> - <item>Applications that merely need a look-up dictionary that - can be stored to disc. Those applications use the standard + <item>Merely need a lookup dictionary that + can be stored on disc. Such applications may use the standard library module <c>dets</c>, which is a disc-based version - of the module <c>ets</c>. For information about <c>dets</c>, - see the <seeerl marker="stdlib:dets">dets</seeerl> + of the <c>ets</c> module. For more information about <c>dets</c>, + please see the <seeerl marker="stdlib:dets">dets</seeerl> manual page in STDLIB. </item> - <item>Applications that need disc logging facilities. - Those applications can - use the module <c>disk_log</c> by preference. For - information about <c>disk_log</c>, see the + <item>Need disc logging facilities. + Such applications may use the module <c>disk_log</c>. + For more information about <c>disk_log</c>, please see the <seeerl marker="kernel:disk_log">disk_log</seeerl> manual page in Kernel. </item> - <item>Hard real-time systems. + <item>Require hard real-time characteristics. </item> </list> </section> diff --git a/lib/mnesia/doc/src/notes.xml b/lib/mnesia/doc/src/notes.xml index c8c7c0c0a5..ba61efa7e9 100644 --- a/lib/mnesia/doc/src/notes.xml +++ b/lib/mnesia/doc/src/notes.xml @@ -39,7 +39,42 @@ thus constitutes one section in this document. The title of each section is the version number of Mnesia.</p> - <section><title>Mnesia 4.18.1</title> + <section><title>Mnesia 4.19</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Fixed the type spec for <c>disc_only_copies</c>.</p> + <p> + Own Id: OTP-17249 Aux Id: PR-4578 </p> + </item> + <item> + <p> + Do not crash in <c>mnesia:change_config/2</c> if mnesia + is stopping or starting.</p> + <p> + Own Id: OTP-17274 Aux Id: GH-4616 </p> + </item> + </list> + </section> + + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + Optimized table loading time for tables that are updated + during the loading.</p> + <p> + Own Id: OTP-17271 Aux Id: PR-4575 </p> + </item> + </list> + </section> + +</section> + +<section><title>Mnesia 4.18.1</title> <section><title>Fixed Bugs and Malfunctions</title> <list> diff --git a/lib/mnesia/info b/lib/mnesia/info index bfd0816a62..2fc77fc444 100644 --- a/lib/mnesia/info +++ b/lib/mnesia/info @@ -1,2 +1,2 @@ group: dat Database Applications -short: A heavy duty real-time distributed database +short: A heavy-duty real-time distributed database diff --git a/lib/mnesia/src/mnesia.erl b/lib/mnesia/src/mnesia.erl index 2efe739442..f9e452cd59 100644 --- a/lib/mnesia/src/mnesia.erl +++ b/lib/mnesia/src/mnesia.erl @@ -142,7 +142,7 @@ {'access_mode', 'read_write' | 'read_only'} | {'attributes', [atom()]} | {'disc_copies', [node()]} | - {'disc_only_copies', [node]} | + {'disc_only_copies', [node()]} | {'index', [index_attr()]} | {'load_order', non_neg_integer()} | {'majority', boolean()} | @@ -155,7 +155,7 @@ {'user_properties', proplists:proplist()}. -type t_result(Res) :: {'atomic', Res} | {'aborted', Reason::term()}. --type result() :: ok | {'error', Reason::term()}. +-type result() :: 'ok' | {'error', Reason::term()}. -type activity() :: 'ets' | 'async_dirty' | 'sync_dirty' | 'transaction' | 'sync_transaction' | {'transaction', Retries::non_neg_integer()} | {'sync_transaction', Retries::non_neg_integer()}. @@ -169,9 +169,9 @@ -type snmp_struct() :: [{atom(), snmp_type() | tuple_of(snmp_type())}]. -type snmp_type() :: 'fix_string' | 'string' | 'integer'. -type tuple_of(_T) :: tuple(). --type config_key() :: extra_db_nodes | dc_dump_limit. +-type config_key() :: 'extra_db_nodes' | 'dc_dump_limit'. -type config_value() :: [node()] | number(). --type config_result() :: {ok, config_value()} | {error, term()}. +-type config_result() :: {'ok', config_value()} | {'error', term()}. -type debug_level() :: 'none' | 'verbose' | 'debug' | 'trace'. -define(DEFAULT_ACCESS, ?MODULE). @@ -530,7 +530,7 @@ lock(LockItem, LockKind) -> -spec lock_table(Tab::table(), LockKind) -> [MnesiaNode] | no_return() when MnesiaNode :: node(), - LockKind :: lock_kind() | load. + LockKind :: lock_kind() | 'load'. lock_table(Tab, LockKind) -> lock({table, Tab}, LockKind). @@ -552,13 +552,13 @@ lock(Tid, Ts, LockItem, LockKind) -> end. %% Grab a read lock on a whole table --spec read_lock_table(Tab::table()) -> ok. +-spec read_lock_table(Tab::table()) -> 'ok'. read_lock_table(Tab) -> lock({table, Tab}, read), ok. %% Grab a write lock on a whole table --spec write_lock_table(Tab::table()) -> ok. +-spec write_lock_table(Tab::table()) -> 'ok'. write_lock_table(Tab) -> lock({table, Tab}, write), ok. @@ -2206,12 +2206,12 @@ bad_info_reply(_Tab, memory) -> 0; bad_info_reply(Tab, Item) -> abort({no_exists, Tab, Item}). %% Raw info about all tables --spec schema() -> ok. +-spec schema() -> 'ok'. schema() -> mnesia_schema:info(). %% Raw info about one tables --spec schema(Tab::table()) -> ok. +-spec schema(Tab::table()) -> 'ok'. schema(Tab) -> mnesia_schema:info(Tab). @@ -2219,7 +2219,7 @@ schema(Tab) -> error_description(Err) -> mnesia_lib:error_desc(Err). --spec info() -> ok. +-spec info() -> 'ok'. info() -> case mnesia_lib:is_running() of yes -> @@ -2659,8 +2659,8 @@ create_schema(Ns) -> -spec create_schema(Ns::[node()], [Prop]) -> result() when Prop :: BackendType | IndexPlugin, - BackendType :: {backend_types, [{Name::atom(), Module::module()}]}, - IndexPlugin :: {index_plugins, [{{Name::atom()}, Module::module(), Function::atom()}]}. + BackendType :: {'backend_types', [{Name::atom(), Module::module()}]}, + IndexPlugin :: {'index_plugins', [{{Name::atom()}, Module::module(), Function::atom()}]}. create_schema(Ns, Properties) -> mnesia_bup:create_schema(Ns, Properties). @@ -2760,29 +2760,29 @@ create_table(Name, Arg) -> delete_table(Tab) -> mnesia_schema:delete_table(Tab). --spec add_table_copy(Tab, N, ST) -> t_result(ok) when +-spec add_table_copy(Tab, N, ST) -> t_result('ok') when Tab :: table(), N::node(), ST::storage_type(). add_table_copy(Tab, N, S) -> mnesia_schema:add_table_copy(Tab, N, S). --spec del_table_copy(Tab::table(), N::node()) -> t_result(ok). +-spec del_table_copy(Tab::table(), N::node()) -> t_result('ok'). del_table_copy(Tab, N) -> mnesia_schema:del_table_copy(Tab, N). --spec move_table_copy(Tab::table(), From::node(), To::node()) -> t_result(ok). +-spec move_table_copy(Tab::table(), From::node(), To::node()) -> t_result('ok'). move_table_copy(Tab, From, To) -> mnesia_schema:move_table(Tab, From, To). --spec add_table_index(Tab, I) -> t_result(ok) when +-spec add_table_index(Tab, I) -> t_result('ok') when Tab :: table(), I :: index_attr(). add_table_index(Tab, Ix) -> mnesia_schema:add_table_index(Tab, Ix). --spec del_table_index(Tab, I) -> t_result(ok) when +-spec del_table_index(Tab, I) -> t_result('ok') when Tab::table(), I::index_attr(). del_table_index(Tab, Ix) -> mnesia_schema:del_table_index(Tab, Ix). --spec transform_table(Tab::table(), Fun, [Attr]) -> t_result(ok) when +-spec transform_table(Tab::table(), Fun, [Attr]) -> t_result('ok') when Attr :: atom(), Fun:: fun((Record::tuple()) -> Transformed::tuple()) | ignore. transform_table(Tab, Fun, NewA) -> @@ -2792,18 +2792,18 @@ transform_table(Tab, Fun, NewA) -> mnesia:abort(Reason) end. --spec transform_table(Tab::table(), Fun, [Attr], RecName) -> t_result(ok) when +-spec transform_table(Tab::table(), Fun, [Attr], RecName) -> t_result('ok') when RecName :: atom(), Attr :: atom(), Fun:: fun((Record::tuple()) -> Transformed::tuple()) | ignore. transform_table(Tab, Fun, NewA, NewRN) -> mnesia_schema:transform_table(Tab, Fun, NewA, NewRN). --spec change_table_copy_type(Tab::table(), Node::node(), To::storage_type()) -> t_result(ok). +-spec change_table_copy_type(Tab::table(), Node::node(), To::storage_type()) -> t_result('ok'). change_table_copy_type(T, N, S) -> mnesia_schema:change_table_copy_type(T, N, S). --spec clear_table(Tab::table()) -> t_result(ok). +-spec clear_table(Tab::table()) -> t_result('ok'). clear_table(Tab) -> case get(mnesia_activity_state) of State = {Mod, Tid, _Ts} when element(1, Tid) =/= tid -> @@ -2837,18 +2837,18 @@ clear_table(Tid, Ts, Tab, Obj) when element(1, Tid) =:= tid -> read_table_property(Tab, PropKey) -> val({Tab, user_property, PropKey}). --spec write_table_property(Tab::table(), Prop::tuple()) -> t_result(ok). +-spec write_table_property(Tab::table(), Prop::tuple()) -> t_result('ok'). write_table_property(Tab, Prop) -> mnesia_schema:write_table_property(Tab, Prop). --spec delete_table_property(Tab::table(), PropKey::term()) -> t_result(ok). +-spec delete_table_property(Tab::table(), PropKey::term()) -> t_result('ok'). delete_table_property(Tab, PropKey) -> mnesia_schema:delete_table_property(Tab, PropKey). %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Table mgt - user properties --spec change_table_frag(Tab::table(), FP::term()) -> t_result(ok). +-spec change_table_frag(Tab::table(), FP::term()) -> t_result('ok'). change_table_frag(Tab, FragProp) -> mnesia_schema:change_table_frag(Tab, FragProp). @@ -2856,7 +2856,7 @@ change_table_frag(Tab, FragProp) -> %% Table mgt - table load %% Dump a ram table to disc --spec dump_tables([Tab::table()]) -> t_result(ok). +-spec dump_tables([Tab::table()]) -> t_result('ok'). dump_tables(Tabs) -> mnesia_schema:dump_tables(Tabs). @@ -2873,17 +2873,17 @@ force_load_table(Tab) -> Other -> Other end. --spec change_table_access_mode(Tab::table(), Mode) -> t_result(ok) when +-spec change_table_access_mode(Tab::table(), Mode) -> t_result('ok') when Mode :: 'read_only'|'read_write'. change_table_access_mode(T, Access) -> mnesia_schema:change_table_access_mode(T, Access). --spec change_table_load_order(Tab::table(), Order) -> t_result(ok) when +-spec change_table_load_order(Tab::table(), Order) -> t_result('ok') when Order :: non_neg_integer(). change_table_load_order(T, O) -> mnesia_schema:change_table_load_order(T, O). --spec change_table_majority(Tab::table(), M::boolean()) -> t_result(ok). +-spec change_table_majority(Tab::table(), M::boolean()) -> t_result('ok'). change_table_majority(T, M) -> mnesia_schema:change_table_majority(T, M). @@ -3001,11 +3001,11 @@ report_event(Event) -> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Snmp --spec snmp_open_table(Tab::table(), Snmp::snmp_struct()) -> ok. +-spec snmp_open_table(Tab::table(), Snmp::snmp_struct()) -> 'ok'. snmp_open_table(Tab, Us) -> mnesia_schema:add_snmp(Tab, Us). --spec snmp_close_table(Tab::table()) -> ok. +-spec snmp_close_table(Tab::table()) -> 'ok'. snmp_close_table(Tab) -> mnesia_schema:del_snmp(Tab). @@ -3151,7 +3151,7 @@ snmp_filter_key(undefined, RowIndex, Tab, Store) -> %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %% Textfile access --spec load_textfile(File::file:filename()) -> t_result(ok) | {'error', term()}. +-spec load_textfile(File::file:filename()) -> t_result('ok') | {'error', term()}. load_textfile(F) -> mnesia_text:load_textfile(F). @@ -3169,7 +3169,7 @@ table(Tab) -> -spec table(Tab::table(), Options) -> qlc:query_handle() when Options :: Option | [Option], Option :: MnesiaOpt | QlcOption, - MnesiaOpt :: {'traverse', SelectOp} | {lock, lock_kind()} | {n_objects, non_neg_integer()}, + MnesiaOpt :: {'traverse', SelectOp} | {'lock', lock_kind()} | {'n_objects', non_neg_integer()}, SelectOp :: 'select' | {'select', ets:match_spec()}, QlcOption :: {'key_equality', '==' | '=:='}. table(Tab,Opts) -> diff --git a/lib/mnesia/src/mnesia_controller.erl b/lib/mnesia/src/mnesia_controller.erl index 5a8cb302b2..5bbd91184f 100644 --- a/lib/mnesia/src/mnesia_controller.erl +++ b/lib/mnesia/src/mnesia_controller.erl @@ -460,8 +460,6 @@ connect_nodes(Ns) -> connect_nodes(Ns, UserFun) -> case mnesia:system_info(is_running) of - no -> - {error, {node_not_running, node()}}; yes -> Pid = spawn_link(?MODULE,connect_nodes2,[self(),Ns, UserFun]), receive @@ -478,7 +476,9 @@ connect_nodes(Ns, UserFun) -> end; {'EXIT', Pid, Reason} -> {error, Reason} - end + end; + _ -> %% no, starting or stopping not ready to make a connection yet + {error, {node_not_running, node()}} end. connect_nodes2(Father, Ns, UserFun) -> diff --git a/lib/mnesia/src/mnesia_loader.erl b/lib/mnesia/src/mnesia_loader.erl index c34a83bb6c..a2fde5c808 100644 --- a/lib/mnesia/src/mnesia_loader.erl +++ b/lib/mnesia/src/mnesia_loader.erl @@ -320,6 +320,11 @@ start_remote_sender(Node,Tab,Storage) -> table_init_fun(SenderPid, Storage) -> fun(read) -> + % We want to store subscribed mnesia table events received during + % table copying for later processing to not let receiver message queue + % to grow too much (which in consequence would slow down the whole copying process) + SubscrCache = ets:new(subscr_cache, [private, ordered_set]), + put(mnesia_table_receiver_subscr_cache, SubscrCache), Receiver = self(), SenderPid ! {Receiver, more}, get_data(SenderPid, Receiver, Storage); @@ -471,6 +476,10 @@ get_data(Pid, TabRec, Storage) -> end; {'EXIT', Pid, Reason} -> handle_exit(Pid, Reason), + get_data(Pid, TabRec, Storage); + {mnesia_table_event, _} = SubscrEvent -> + SubscrCache = get(mnesia_table_receiver_subscr_cache), + ets:insert(SubscrCache, {erlang:unique_integer([monotonic]), SubscrEvent}), get_data(Pid, TabRec, Storage) end. @@ -542,7 +551,7 @@ init_table(Tab, _, Fun, _DetsInfo,_) -> finish_copy(Storage,Tab,Cs,SenderPid,DatBin,OrigTabRec) -> TabRef = {Storage, Tab}, - subscr_receiver(TabRef, Cs#cstruct.record_name), + subscr_postprocess(TabRef, Cs#cstruct.record_name), case handle_last(TabRef, Cs#cstruct.type, DatBin) of ok -> mnesia_index:init_index(Tab, Storage), @@ -558,8 +567,37 @@ finish_copy(Storage,Tab,Cs,SenderPid,DatBin,OrigTabRec) -> down(Tab, Storage) end. +subscr_postprocess(TabRef, RecName) -> + % process events received during table copying + case get(mnesia_table_receiver_subscr_cache) of + undefined -> + ok; + SubscrCache -> + ets:foldl( + fun({_, Event}, _Acc) -> + handle_subscr_event(Event, TabRef, RecName) + end, ok, SubscrCache), + ets:delete(SubscrCache) + end, + % and all remaining events + subscr_receiver(TabRef, RecName). + subscr_receiver(TabRef = {_, Tab}, RecName) -> receive + {mnesia_table_event, {_Op, Val, _Tid}} = Event + when element(1, Val) =:= Tab; element(1, Val) =:= schema -> + handle_subscr_event(Event, TabRef, RecName), + subscr_receiver(TabRef, RecName); + + {'EXIT', Pid, Reason} -> + handle_exit(Pid, Reason), + subscr_receiver(TabRef, RecName) + after 0 -> + ok + end. + +handle_subscr_event(Event, TabRef = {_, Tab}, RecName) -> + case Event of {mnesia_table_event, {Op, Val, _Tid}} when element(1, Val) =:= Tab -> if @@ -567,8 +605,7 @@ subscr_receiver(TabRef = {_, Tab}, RecName) -> handle_event(TabRef, Op, Val); true -> handle_event(TabRef, Op, setelement(1, Val, RecName)) - end, - subscr_receiver(TabRef, RecName); + end; {mnesia_table_event, {Op, Val, _Tid}} when element(1, Val) =:= schema -> %% clear_table is faked via two schema events @@ -576,14 +613,7 @@ subscr_receiver(TabRef = {_, Tab}, RecName) -> case Op of delete -> handle_event(TabRef, clear_table, {Tab, all}); _ -> ok - end, - subscr_receiver(TabRef, RecName); - - {'EXIT', Pid, Reason} -> - handle_exit(Pid, Reason), - subscr_receiver(TabRef, RecName) - after 0 -> - ok + end end. handle_event(TabRef, write, Rec) -> diff --git a/lib/mnesia/vsn.mk b/lib/mnesia/vsn.mk index 19a69fb714..a5832068a1 100644 --- a/lib/mnesia/vsn.mk +++ b/lib/mnesia/vsn.mk @@ -1 +1 @@ -MNESIA_VSN = 4.18.1 +MNESIA_VSN = 4.19 diff --git a/lib/odbc/c_src/odbcserver.c b/lib/odbc/c_src/odbcserver.c index ee5dc9cd0a..b22a6cb7af 100644 --- a/lib/odbc/c_src/odbcserver.c +++ b/lib/odbc/c_src/odbcserver.c @@ -178,7 +178,7 @@ static void encode_column_dyn(db_column column, int column_nr, db_state *state); static void encode_data_type(SQLSMALLINT sql_type, SQLINTEGER size, SQLSMALLINT decimal_digits, db_state *state); -static Boolean decode_params(db_state *state, byte *buffer, int *index, param_array **params, +static Boolean decode_params(db_state *state, char *buffer, int *index, param_array **params, int i, int j, int num_param_values); /*------------- Erlang port communication functions ----------------------*/ @@ -222,7 +222,7 @@ static SQLLEN* alloc_strlen_indptr(int n, int val); static void init_driver(int erl_auto_commit_mode, int erl_trace_driver, db_state *state); -static void init_param_column(param_array *params, byte *buffer, int *index, +static void init_param_column(param_array *params, char *buffer, int *index, int num_param_values, db_state* state); static void init_param_statement(int cols, @@ -235,7 +235,7 @@ static void map_dec_num_2_c_column(col_type *type, int precision, static db_result_msg map_sql_2_c_column(db_column* column, db_state *state); -static param_array * bind_parameter_arrays(byte *buffer, int *index, +static param_array * bind_parameter_arrays(char *buffer, int *index, int cols, int num_param_values, db_state *state); @@ -259,10 +259,10 @@ static void str_tolower(char *str, int len); /* ----------------------------- CODE ------------------------------------*/ #if defined(WIN32) -# define DO_EXIT(code) do { ExitProcess((code)); exit((code));} while (0) -/* exit() called only to avoid a warning */ +# define DO_EXIT(code) do { ExitProcess((code)); _exit((code));} while (0) +/* _exit() called only to avoid a warning */ #else -# define DO_EXIT(code) exit((code)) +# define DO_EXIT(code) _exit((code)) #endif /* ----------------- Main functions --------------------------------------*/ @@ -278,7 +278,7 @@ int main(void) msg = receive_erlang_port_msg(); - temp = strtok(msg, ";"); + temp = strtok((char*)msg, ";"); if (temp == NULL) DO_EXIT(EXIT_STDIN_BODY); length = strlen(temp); @@ -509,7 +509,9 @@ static db_result_msg db_connect(byte *args, db_state *state) diagnos = get_diagnos(SQL_HANDLE_DBC, connection_handle(state), extended_errors(state)); strcat((char *)diagnos.error_msg, " Connection to database failed."); - msg = encode_error_message(diagnos.error_msg, extended_error(state, diagnos.sqlState), diagnos.nativeError ); + msg = encode_error_message((char *)diagnos.error_msg, + extended_error(state, diagnos.sqlState), + diagnos.nativeError ); if(!sql_success(SQLFreeHandle(SQL_HANDLE_DBC, connection_handle(state)))) @@ -544,7 +546,9 @@ static db_result_msg db_connect(byte *args, db_state *state) diagnos = get_diagnos(SQL_HANDLE_DBC, connection_handle(state), extended_errors(state)); strcat((char *)diagnos.error_msg, " Set autocommit mode failed."); - msg = encode_error_message(diagnos.error_msg, extended_error(state, diagnos.sqlState), diagnos.nativeError); + msg = encode_error_message((char*)diagnos.error_msg, + extended_error(state, diagnos.sqlState), + diagnos.nativeError); if(!sql_success(SQLFreeHandle(SQL_HANDLE_DBC, connection_handle(state)))) @@ -576,7 +580,9 @@ static db_result_msg db_close_connection(db_state *state) if (!sql_success(result)) { diagnos = get_diagnos(SQL_HANDLE_DBC, connection_handle(state), extended_errors(state)); - return encode_error_message(diagnos.error_msg, extended_error(state, diagnos.sqlState), diagnos.nativeError); + return encode_error_message((char *)diagnos.error_msg, + extended_error(state, diagnos.sqlState), + diagnos.nativeError); } if(!sql_success(SQLFreeHandle(SQL_HANDLE_DBC, @@ -603,7 +609,9 @@ static db_result_msg db_end_tran(byte compleationtype, db_state *state) if (!sql_success(result)) { diagnos = get_diagnos(SQL_HANDLE_DBC, connection_handle(state), extended_errors(state)); - return encode_error_message(diagnos.error_msg, extended_error(state, diagnos.sqlState), diagnos.nativeError); + return encode_error_message((char *)diagnos.error_msg, + extended_error(state, diagnos.sqlState), + diagnos.nativeError); } else { return encode_atom_message("ok"); } @@ -645,7 +653,7 @@ static db_result_msg db_query(byte *sql, db_state *state) it as we want a nice and clean Erlang API */ if((strcmp((char *)is_error, "error") == 0)) { - msg = encode_error_message((char *)diagnos.error_msg, extended_error(state, diagnos.sqlState), diagnos.nativeError); + msg = encode_error_message((char *)diagnos.error_msg,extended_error(state, diagnos.sqlState), diagnos.nativeError); clean_state(state); return msg; } @@ -686,7 +694,7 @@ static db_result_msg db_query(byte *sql, db_state *state) ei_x_free(&(dynamic_buffer(state))); return msg; } else { - msg.buffer = dynamic_buffer(state).buff; + msg.buffer = (byte*)dynamic_buffer(state).buff; msg.length = dynamic_buffer(state).index; msg.dyn_alloc = TRUE; return msg; @@ -722,7 +730,9 @@ static db_result_msg db_select_count(byte *sql, db_state *state) if(!sql_success(SQLExecDirect(statement_handle(state), (SQLCHAR *)sql, SQL_NTS))) { diagnos = get_diagnos(SQL_HANDLE_STMT, statement_handle(state), extended_errors(state)); clean_state(state); - return encode_error_message(diagnos.error_msg, extended_error(state, diagnos.sqlState), diagnos.nativeError); + return encode_error_message((char *)diagnos.error_msg, + extended_error(state, diagnos.sqlState), + diagnos.nativeError); } if(!sql_success(SQLNumResultCols(statement_handle(state), @@ -810,7 +820,7 @@ static db_result_msg db_select(byte *args, db_state *state) ei_x_free(&(dynamic_buffer(state))); return msg; } else { - msg.buffer = dynamic_buffer(state).buff; + msg.buffer = (byte*)dynamic_buffer(state).buff; msg.length = dynamic_buffer(state).index; msg.dyn_alloc = TRUE; return msg; @@ -819,9 +829,10 @@ static db_result_msg db_select(byte *args, db_state *state) /* Description: Handles parameterized queries ex: INSERT INTO FOO VALUES(?, ?) */ -static db_result_msg db_param_query(byte *buffer, db_state *state) +static db_result_msg db_param_query(byte *byte_buffer, db_state *state) { - byte *sql; + char *sql; + char *buffer = (char *)byte_buffer; db_result_msg msg; SQLLEN num_param_values; int i, ver = 0, @@ -847,7 +858,7 @@ static db_result_msg db_param_query(byte *buffer, db_state *state) ei_get_type(buffer, &index, &erl_type, &size); - sql = (byte*)safe_malloc((sizeof(byte) * (size + 1))); + sql = safe_malloc((sizeof(byte) * (size + 1))); ei_decode_string(buffer, &index, sql); ei_decode_long(buffer, &index, &long_num_param_values); @@ -871,7 +882,9 @@ static db_result_msg db_param_query(byte *buffer, db_state *state) updates/deletes that affect no rows */ if(!sql_success(result) && !(result == SQL_NO_DATA && !strcmp((char *)diagnos.sqlState, INFO))) { - msg = encode_error_message(diagnos.error_msg, extended_error(state, diagnos.sqlState), diagnos.nativeError); + msg = encode_error_message((char*)diagnos.error_msg, + extended_error(state, diagnos.sqlState), + diagnos.nativeError); } else { for (i = 0; i < param_status.params_processed; i++) { switch (param_status.param_status_array[i]) { @@ -886,7 +899,9 @@ static db_result_msg db_param_query(byte *buffer, db_state *state) default: diagnos = get_diagnos(SQL_HANDLE_STMT, statement_handle(state), extended_errors(state)); - msg = encode_error_message(diagnos.error_msg, extended_error(state, diagnos.sqlState), diagnos.nativeError); + msg = encode_error_message((char*)diagnos.error_msg, + extended_error(state, diagnos.sqlState), + diagnos.nativeError); i = param_status.params_processed; break; } @@ -899,7 +914,7 @@ static db_result_msg db_param_query(byte *buffer, db_state *state) msg = encode_result(state); } if(msg.length == 0) { - msg.buffer = dynamic_buffer(state).buff; + msg.buffer = (byte *)dynamic_buffer(state).buff; msg.length = dynamic_buffer(state).index; msg.dyn_alloc = TRUE; } else { /* Error occurred */ @@ -956,7 +971,9 @@ static db_result_msg db_describe_table(byte *sql, db_state *state) if (!sql_success(SQLPrepare(statement_handle(state), (SQLCHAR *)sql, SQL_NTS))){ diagnos = get_diagnos(SQL_HANDLE_STMT, statement_handle(state), extended_errors(state)); - msg = encode_error_message(diagnos.error_msg, extended_error(state, diagnos.sqlState), diagnos.nativeError); + msg = encode_error_message((char *)diagnos.error_msg, + extended_error(state, diagnos.sqlState), + diagnos.nativeError); clean_state(state); return msg; } @@ -964,7 +981,9 @@ static db_result_msg db_describe_table(byte *sql, db_state *state) if(!sql_success(SQLNumResultCols(statement_handle(state), &num_of_columns))) { diagnos = get_diagnos(SQL_HANDLE_STMT, statement_handle(state), extended_errors(state)); - msg = encode_error_message(diagnos.error_msg, extended_error(state, diagnos.sqlState), diagnos.nativeError); + msg = encode_error_message((char *)diagnos.error_msg, + extended_error(state, diagnos.sqlState), + diagnos.nativeError); clean_state(state); return msg; } @@ -992,7 +1011,7 @@ static db_result_msg db_describe_table(byte *sql, db_state *state) ei_x_encode_empty_list(&dynamic_buffer(state)); clean_state(state); - msg.buffer = dynamic_buffer(state).buff; + msg.buffer = (byte *)dynamic_buffer(state).buff; msg.length = dynamic_buffer(state).index; msg.dyn_alloc = TRUE; return msg; @@ -1088,7 +1107,9 @@ static db_result_msg encode_result(db_state *state) if(!sql_success(SQLNumResultCols(statement_handle(state), &num_of_columns))) { diagnos = get_diagnos(SQL_HANDLE_STMT, statement_handle(state), extended_errors(state)); - msg = encode_error_message(diagnos.error_msg, extended_error(state, diagnos.sqlState), diagnos.nativeError); + msg = encode_error_message((char *)diagnos.error_msg, + extended_error(state, diagnos.sqlState), + diagnos.nativeError); clean_state(state); return msg; } @@ -1105,7 +1126,9 @@ static db_result_msg encode_result(db_state *state) if(!sql_success(SQLRowCount(statement_handle(state), &RowCountPtr))) { diagnos = get_diagnos(SQL_HANDLE_STMT, statement_handle(state), extended_errors(state)); - msg = encode_error_message(diagnos.error_msg, extended_error(state, diagnos.sqlState), diagnos.nativeError); + msg = encode_error_message((char *)diagnos.error_msg, + extended_error(state, diagnos.sqlState), + diagnos.nativeError); clean_state(state); return msg; } @@ -1652,7 +1675,7 @@ static void encode_data_type(SQLSMALLINT sql_type, SQLINTEGER size, } } -static Boolean decode_params(db_state *state, byte *buffer, int *index, param_array **params, +static Boolean decode_params(db_state *state, char *buffer, int *index, param_array **params, int i, int j, int num_param_values) { int erl_type, size; @@ -1688,7 +1711,7 @@ static Boolean decode_params(db_state *state, byte *buffer, int *index, param_ar if(erl_type != ERL_STRING_EXT) { return FALSE; } - ei_decode_string(buffer, index, &(param->values.string[param->offset])); + ei_decode_string(buffer, index, (char*)&(param->values.string[param->offset])); param->offset += param->type.len; } break; @@ -2197,7 +2220,7 @@ static void init_driver(int erl_auto_commit_mode, int erl_trace_driver, DO_EXIT(EXIT_CONNECTION); } -static void init_param_column(param_array *params, byte *buffer, int *index, +static void init_param_column(param_array *params, char *buffer, int *index, int num_param_values, db_state* state) { long user_type, precision, scale, length; @@ -2509,7 +2532,7 @@ static db_result_msg map_sql_2_c_column(db_column* column, db_state *state) return msg; } -static param_array * bind_parameter_arrays(byte *buffer, int *index, +static param_array * bind_parameter_arrays(char *buffer, int *index, int cols, int num_param_values, db_state *state) { @@ -2669,7 +2692,7 @@ static db_result_msg retrive_scrollable_cursor_support_info(db_state *state) ei_x_encode_atom(&dynamic_buffer(state), "false"); ei_x_encode_atom(&dynamic_buffer(state), "false"); } - msg.buffer = dynamic_buffer(state).buff; + msg.buffer = (byte *)dynamic_buffer(state).buff; msg.length = dynamic_buffer(state).index; msg.dyn_alloc = TRUE; return msg; @@ -2700,7 +2723,9 @@ static db_result_msg more_result_sets(db_state *state) diagnos = get_diagnos(SQL_HANDLE_STMT, statement_handle(state), extended_errors(state)); strcat((char *)diagnos.error_msg, "Failed to create on of the result sets"); - msg = encode_error_message(diagnos.error_msg, extended_error(state, diagnos.sqlState), diagnos.nativeError); + msg = encode_error_message((char*)diagnos.error_msg, + extended_error(state, diagnos.sqlState), + diagnos.nativeError); return msg; } } diff --git a/lib/odbc/doc/src/notes.xml b/lib/odbc/doc/src/notes.xml index 3533310e51..b219f79b87 100644 --- a/lib/odbc/doc/src/notes.xml +++ b/lib/odbc/doc/src/notes.xml @@ -32,7 +32,23 @@ <p>This document describes the changes made to the odbc application. </p> - <section><title>ODBC 2.13.2</title> + <section><title>ODBC 2.13.3</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Make sure odbc c-process exits when erlang process orders + it to shutdown.</p> + <p> + Own Id: OTP-17188 Aux Id: ERL-1448 </p> + </item> + </list> + </section> + +</section> + +<section><title>ODBC 2.13.2</title> <section><title>Fixed Bugs and Malfunctions</title> <list> diff --git a/lib/odbc/vsn.mk b/lib/odbc/vsn.mk index c08f40d233..717da2b77c 100644 --- a/lib/odbc/vsn.mk +++ b/lib/odbc/vsn.mk @@ -1 +1 @@ -ODBC_VSN = 2.13.2 +ODBC_VSN = 2.13.3 diff --git a/lib/public_key/doc/src/notes.xml b/lib/public_key/doc/src/notes.xml index 7cabfed09a..e831773871 100644 --- a/lib/public_key/doc/src/notes.xml +++ b/lib/public_key/doc/src/notes.xml @@ -35,6 +35,46 @@ <file>notes.xml</file> </header> +<section><title>Public_Key 1.10</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Fixed case insensitive hostname check.</p> + <p> + Own Id: OTP-17242 Aux Id: GH-4500 </p> + </item> + </list> + </section> + + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + Add sanity check of trusted anchor certificate expiration + to pkix_path_validation/3. Although the anchor is + considered a trusted input this sanity check does provide + extra security for the users of the public_key + application as this property needs to be checked at time + of usage and fits very well with the other checks + performed here.</p> + <p> + Own Id: OTP-16907</p> + </item> + <item> + <p> + Adjust generation of test certificates to conform to RFC + 5280 rules for formatting of the certificates validity</p> + <p> + Own Id: OTP-17111</p> + </item> + </list> + </section> + +</section> + <section><title>Public_Key 1.9.2</title> <section><title>Improvements and New Features</title> diff --git a/lib/public_key/src/pubkey_cert.erl b/lib/public_key/src/pubkey_cert.erl index 5966a15535..e84edffd53 100644 --- a/lib/public_key/src/pubkey_cert.erl +++ b/lib/public_key/src/pubkey_cert.erl @@ -68,11 +68,15 @@ -type test_root_cert() :: #{cert := binary(), key := public_key:private_key()}. %%==================================================================== -%% Internal application APIu +%% Internal application APIs %%==================================================================== %%-------------------------------------------------------------------- --spec verify_data(DER::binary()) -> {md5 | sha, binary(), binary()}. +-spec verify_data(DER::binary()) -> + {DigestType, PlainText, Signature} + when DigestType :: md5 | crypto:sha1() | crypto:sha2(), + PlainText :: binary(), + Signature :: binary(). %% %% Description: Extracts data from DerCert needed to call public_key:verify/4. %%-------------------------------------------------------------------- @@ -1214,13 +1218,28 @@ validity(Opts) -> DefFrom0 = calendar:gregorian_days_to_date(calendar:date_to_gregorian_days(date())-1), DefTo0 = calendar:gregorian_days_to_date(calendar:date_to_gregorian_days(date())+7), {DefFrom, DefTo} = proplists:get_value(validity, Opts, {DefFrom0, DefTo0}), - Format = + + GenFormat = fun({Y,M,D}) -> lists:flatten( io_lib:format("~4..0w~2..0w~2..0w130000Z",[Y,M,D])) end, - #'Validity'{notBefore={generalTime, Format(DefFrom)}, - notAfter ={generalTime, Format(DefTo)}}. + + UTCFormat = + fun({Y,M,D}) -> + [_, _, Y3, Y4] = integer_to_list(Y), + lists:flatten( + io_lib:format("~s~2..0w~2..0w130000Z",[[Y3, Y4],M,D])) + end, + + #'Validity'{notBefore = validity_format(DefFrom, GenFormat, UTCFormat), + notAfter = validity_format(DefTo, GenFormat, UTCFormat)}. + +validity_format({Year, _, _} = Validity, GenFormat, _UTCFormat) when Year >= 2049 -> + {generalTime, GenFormat(Validity)}; +validity_format(Validity, _GenFormat, UTCFormat) -> + {utcTime, UTCFormat(Validity)}. + sign_algorithm(#'RSAPrivateKey'{} = Key , Opts) -> case proplists:get_value(rsa_padding, Opts, rsa_pkcs1_pss_padding) of diff --git a/lib/public_key/src/public_key.erl b/lib/public_key/src/public_key.erl index 4176fce978..8c8b5585a0 100644 --- a/lib/public_key/src/public_key.erl +++ b/lib/public_key/src/public_key.erl @@ -992,10 +992,19 @@ pkix_path_validation(TrustedCert, CertChain, Options) pkix_path_validation(#'OTPCertificate'{} = TrustedCert, CertChain, Options) when is_list(CertChain), is_list(Options) -> MaxPathDefault = length(CertChain), - ValidationState = pubkey_cert:init_validation_state(TrustedCert, - MaxPathDefault, - Options), - path_validation(CertChain, ValidationState). + {VerifyFun, Userstat0} = + proplists:get_value(verify_fun, Options, ?DEFAULT_VERIFYFUN), + try pubkey_cert:validate_time(TrustedCert, Userstat0, VerifyFun) of + Userstate1 -> + ValidationState = pubkey_cert:init_validation_state(TrustedCert, + MaxPathDefault, + [{verify_fun, {VerifyFun, Userstate1}} | + proplists:delete(verify_fun, Options)]), + path_validation(CertChain, ValidationState) + catch + throw:{bad_cert, cert_expired} = Reason -> + {error, Reason} + end. %-------------------------------------------------------------------- -spec pkix_crls_validate(OTPcertificate, DPandCRLs, Options) -> @@ -1815,8 +1824,8 @@ verify_hostname_match_default0(_, _) -> verify_hostname_match_wildcard(FQDN, Name) -> - [F1|Fs] = string:tokens(FQDN, "."), - [N1|Ns] = string:tokens(Name, "."), + [F1|Fs] = string:tokens(to_lower_ascii(FQDN), "."), + [N1|Ns] = string:tokens(to_lower_ascii(Name), "."), match_wild(F1,N1) andalso Fs==Ns. diff --git a/lib/public_key/test/public_key_SUITE.erl b/lib/public_key/test/public_key_SUITE.erl index 438924375c..cf3b5ea0c8 100644 --- a/lib/public_key/test/public_key_SUITE.erl +++ b/lib/public_key/test/public_key_SUITE.erl @@ -1,4 +1,3 @@ - %% %% %CopyrightBegin% %% @@ -85,6 +84,8 @@ pkix_emailaddress/1, pkix_path_validation/0, pkix_path_validation/1, + pkix_path_validation_root_expired/0, + pkix_path_validation_root_expired/1, pkix_verify_hostname_cn/1, pkix_verify_hostname_subjAltName/1, pkix_verify_hostname_options/1, @@ -126,12 +127,18 @@ suite() -> []. all() -> - [app, appup, + [app, + appup, {group, pem_decode_encode}, encrypt_decrypt, {group, sign_verify}, - pkix, pkix_countryname, pkix_emailaddress, pkix_path_validation, - pkix_iso_rsa_oid, pkix_iso_dsa_oid, + pkix, + pkix_countryname, + pkix_emailaddress, + pkix_path_validation, + pkix_path_validation_root_expired, + pkix_iso_rsa_oid, + pkix_iso_dsa_oid, pkix_dsa_sha2_oid, pkix_crl, pkix_hash_type, @@ -142,7 +149,8 @@ all() -> pkix_verify_hostname_options, pkix_test_data_all_default, pkix_test_data, - short_cert_issuer_hash, short_crl_issuer_hash + short_cert_issuer_hash, + short_crl_issuer_hash ]. groups() -> @@ -737,9 +745,24 @@ pkix_path_validation(Config) when is_list(Config) -> {error, custom_reason} = public_key:pkix_path_validation(selfsigned_peer, [Trusted], [{verify_fun, - VerifyFunAndState2}]), - ok. - + VerifyFunAndState2}]). +pkix_path_validation_root_expired() -> + [{doc, "Test root expiration so that it does not fall between chairs"}]. +pkix_path_validation_root_expired(Config) when is_list(Config) -> + {Year, Month, Day} = date(), + SRoot = public_key:pkix_test_root_cert("OTP test server ROOT", [{validity, {{Year-2, Month, Day}, + {Year-1, Month, Day}}}]), + #{server_config := Conf} = public_key:pkix_test_data(#{server_chain => #{root => SRoot, + intermediates => [], + peer => []}, + client_chain => #{root => [], + intermediates => [], + peer => []}}), + [ICA, Root] = proplists:get_value(cacerts, Conf), + true = public_key:pkix_is_self_signed(Root), + Peer = proplists:get_value(cert, Conf), + {error, {bad_cert, cert_expired}} = public_key:pkix_path_validation(Root, [ICA, Peer], []). + %%-------------------------------------------------------------------- %% To generate the PEM file contents: %% @@ -800,24 +823,26 @@ pkix_verify_hostname_subjAltName(Config) -> %% Check that a dns_id matches a DNS subjAltName: true = public_key:pkix_verify_hostname(Cert, [{dns_id,"kb.example.org"}]), + true = public_key:pkix_verify_hostname(Cert, [{dns_id,"KB.EXAMPLE.ORG"}]), %% Check that a dns_id does not match a DNS subjAltName wiht wildcard false = public_key:pkix_verify_hostname(Cert, [{dns_id,"other.example.org"}]), %% Check that a dns_id does match a DNS subjAltName wiht wildcard with matchfun - true = public_key:pkix_verify_hostname(Cert, [{dns_id,"other.example.org"}], - [{match_fun, public_key:pkix_verify_hostname_match_fun(https)} - ] - ), + MatchFun = {match_fun, public_key:pkix_verify_hostname_match_fun(https)}, + true = public_key:pkix_verify_hostname(Cert, [{dns_id,"other.example.org"}], [MatchFun]), + true = public_key:pkix_verify_hostname(Cert, [{dns_id,"OTHER.EXAMPLE.ORG"}], [MatchFun]), %% Check that a uri_id does not match a DNS subjAltName wiht wildcard false = public_key:pkix_verify_hostname(Cert, [{uri_id,"https://other.example.org"}]), + false = public_key:pkix_verify_hostname(Cert, [{uri_id,"https://OTHER.EXAMPLE.ORG"}]), %% Check that a dns_id does match a DNS subjAltName wiht wildcard with matchfun - true = public_key:pkix_verify_hostname(Cert, [{uri_id,"https://other.example.org"}], - [{match_fun, public_key:pkix_verify_hostname_match_fun(https)} - ] - ). + true = public_key:pkix_verify_hostname(Cert, [{uri_id,"https://other.example.org"}], [MatchFun]), + true = public_key:pkix_verify_hostname(Cert, [{uri_id,"https://OTHER.EXAMPLE.ORG"}], [MatchFun]), + true = public_key:pkix_verify_hostname(Cert, [{uri_id,"https://OTHER.example.org"}], [MatchFun]), + + ok. %%-------------------------------------------------------------------- %% Uses the pem-file for pkix_verify_hostname_cn diff --git a/lib/public_key/vsn.mk b/lib/public_key/vsn.mk index 0de40d1d3f..442b3cc18e 100644 --- a/lib/public_key/vsn.mk +++ b/lib/public_key/vsn.mk @@ -1 +1 @@ -PUBLIC_KEY_VSN = 1.9.2 +PUBLIC_KEY_VSN = 1.10 diff --git a/lib/reltool/test/reltool_test_lib.erl b/lib/reltool/test/reltool_test_lib.erl index 88e5244b8b..4860f7ad66 100644 --- a/lib/reltool/test/reltool_test_lib.erl +++ b/lib/reltool/test/reltool_test_lib.erl @@ -27,10 +27,17 @@ init_per_suite(Config) when is_list(Config)-> global:register_name(reltool_global_logger, group_leader()), - incr_timetrap(Config, ?timeout). + ErlLibs = os:getenv("ERL_LIBS"), + os:unsetenv("ERL_LIBS"), + [{erl_libs,ErlLibs}|incr_timetrap(Config, ?timeout)]. end_per_suite(Config) when is_list(Config)-> global:unregister_name(reltool_global_logger), + case proplists:get_value(erl_libs, Config) of + false -> ok; + ErlLibs -> + os:putenv("ERL_LIBS", ErlLibs) + end, ok. incr_timetrap(Config, Times) -> diff --git a/lib/runtime_tools/doc/src/dbg.xml b/lib/runtime_tools/doc/src/dbg.xml index 8f197d1aa7..168e89b7f0 100644 --- a/lib/runtime_tools/doc/src/dbg.xml +++ b/lib/runtime_tools/doc/src/dbg.xml @@ -76,10 +76,10 @@ </type> <desc> <p>Pseudo function that by means of a <c>parse_transform</c> - translates the <em>literal</em><c>fun()</c> typed as parameter in + translates the <em>literal</em> <c>fun()</c> typed as parameter in the function call to a match specification as described in the <c>match_spec</c> manual of ERTS users guide. - (with literal I mean that the <c>fun()</c> needs to + (With literal I mean that the <c>fun()</c> needs to textually be written as the parameter of the function, it cannot be held in a variable which in turn is passed to the function). </p> diff --git a/lib/runtime_tools/doc/src/notes.xml b/lib/runtime_tools/doc/src/notes.xml index efd5939fe0..6303397f57 100644 --- a/lib/runtime_tools/doc/src/notes.xml +++ b/lib/runtime_tools/doc/src/notes.xml @@ -32,6 +32,21 @@ <p>This document describes the changes made to the Runtime_Tools application.</p> +<section><title>Runtime_Tools 1.16</title> + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + Clarify documentation of module 'scheduler'.</p> + <p> + Own Id: OTP-17208 Aux Id: GH-4502, PR-4532 </p> + </item> + </list> + </section> + +</section> + <section><title>Runtime_Tools 1.15.1</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/runtime_tools/doc/src/scheduler.xml b/lib/runtime_tools/doc/src/scheduler.xml index 713d70548b..d539ccb1c6 100644 --- a/lib/runtime_tools/doc/src/scheduler.xml +++ b/lib/runtime_tools/doc/src/scheduler.xml @@ -63,7 +63,10 @@ <taglist> <tag><c>{normal, SchedulerId, Util, Percent}</c></tag> <item>Scheduler utilization of a normal scheduler with number - <c>SchedulerId</c>.</item> + <c>SchedulerId</c>. Schedulers that are not online will also be + included. + <seeerl marker="erts:erlang#system_info_schedulers_online">Online + schedulers</seeerl> have the lowest <c>SchedulerId</c>.</item> <tag><c>{cpu, SchedulerId, Util, Percent}</c></tag> <item>Scheduler utilization of a dirty-cpu scheduler with number <c>SchedulerId</c>.</item> @@ -117,6 +120,29 @@ <p>Calculate scheduler utilizations for the time interval from when <c><anno>Sample</anno></c> was taken and "now". The same as calling <c>scheduler:utilization(Sample, scheduler:sample_all())</c>.</p> + <note> + <p> + Scheduler utilization is measured as an average value over a time + interval, calculated as the difference between two samples. To get + good useful utilization values at least a couple of seconds should + have passed between the two samples. For this reason, you should not + do + </p> +<pre> +scheduler:utilization(scheduler:sample()). % DO NOT DO THIS! +</pre> + <p> + The above example takes two samples in rapid succession and calculates + the scheduler utilization between them. The resulting values will + probably be more misleading than informative. + </p> + <p> + Instead use <seemfa marker="#utilization/1"> + <c>scheduler:utilization(Seconds)</c></seemfa> or let some time pass + between <c>Sample=scheduler:sample()</c> and + <c>scheduler:utilization(Sample)</c>. + </p> + </note> </desc> </func> diff --git a/lib/runtime_tools/test/erts_alloc_config_SUITE.erl b/lib/runtime_tools/test/erts_alloc_config_SUITE.erl index 6ae51d9a26..9ab61b89d2 100644 --- a/lib/runtime_tools/test/erts_alloc_config_SUITE.erl +++ b/lib/runtime_tools/test/erts_alloc_config_SUITE.erl @@ -25,7 +25,9 @@ -include_lib("common_test/include/ct.hrl"). %-compile(export_all). --export([all/0, suite/0, init_per_testcase/2, end_per_testcase/2]). +-export([all/0, suite/0, + init_per_suite/1, end_per_suite/1, + init_per_testcase/2, end_per_testcase/2]). %% Testcases -export([basic/1]). @@ -40,6 +42,18 @@ suite() -> all() -> [basic]. +init_per_suite(Config) -> + case test_server:is_asan() of + true -> + %% No point testing own allocators under address sanitizer. + {skip, "Address sanitizer"}; + false -> + Config + end. + +end_per_suite(_Config) -> + ok. + init_per_testcase(Case, Config) when is_list(Config) -> [{testcase, Case}, {erl_flags_env, save_env()} | Config]. diff --git a/lib/runtime_tools/vsn.mk b/lib/runtime_tools/vsn.mk index b4e5e953e4..e62d59acf6 100644 --- a/lib/runtime_tools/vsn.mk +++ b/lib/runtime_tools/vsn.mk @@ -1 +1 @@ -RUNTIME_TOOLS_VSN = 1.15.1 +RUNTIME_TOOLS_VSN = 1.16 diff --git a/lib/sasl/doc/src/notes.xml b/lib/sasl/doc/src/notes.xml index a15351cd23..46e6762b90 100644 --- a/lib/sasl/doc/src/notes.xml +++ b/lib/sasl/doc/src/notes.xml @@ -31,6 +31,31 @@ </header> <p>This document describes the changes made to the SASL application.</p> +<section><title>SASL 4.0.2</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Fix dependent application to be stopped after the primary + application when upgrading a release and the primary and + dependent application is removed.</p> + <p> + Example: In a release where app1 depends on app2 and we + should remove app1 and app2 using a release upgrade. When + the release upgrade is done app1 should be stopped and + purged before app2 as otherwise app1 could start crashing + when its dependency is removed.</p> + <p> + This bugfix changes the order of removal to be correct.</p> + <p> + Own Id: OTP-17113 Aux Id: ERL-1410 PR-2882 </p> + </item> + </list> + </section> + +</section> + <section><title>SASL 4.0.1</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/sasl/src/sasl.appup.src b/lib/sasl/src/sasl.appup.src index a8e1b85ff9..4fff8f79ab 100644 --- a/lib/sasl/src/sasl.appup.src +++ b/lib/sasl/src/sasl.appup.src @@ -38,7 +38,8 @@ {<<"^3\\.4\\.1(?:\\.[0-9]+)*$">>,[restart_new_emulator]}, {<<"^3\\.4\\.2(?:\\.[0-9]+)*$">>,[restart_new_emulator]}, {<<"^4\\.0$">>,[restart_new_emulator]}, - {<<"^4\\.0\\.0(?:\\.[0-9]+)+$">>,[restart_new_emulator]}], + {<<"^4\\.0\\.0(?:\\.[0-9]+)+$">>,[restart_new_emulator]}, + {<<"^4\\.0\\.1(?:\\.[0-9]+)*$">>,[restart_new_emulator]}], [{<<"^3\\.2$">>,[restart_new_emulator]}, {<<"^3\\.2\\.0(?:\\.[0-9]+)+$">>,[restart_new_emulator]}, {<<"^3\\.2\\.1(?:\\.[0-9]+)*$">>,[restart_new_emulator]}, @@ -49,4 +50,5 @@ {<<"^3\\.4\\.1(?:\\.[0-9]+)*$">>,[restart_new_emulator]}, {<<"^3\\.4\\.2(?:\\.[0-9]+)*$">>,[restart_new_emulator]}, {<<"^4\\.0$">>,[restart_new_emulator]}, - {<<"^4\\.0\\.0(?:\\.[0-9]+)+$">>,[restart_new_emulator]}]}. + {<<"^4\\.0\\.0(?:\\.[0-9]+)+$">>,[restart_new_emulator]}, + {<<"^4\\.0\\.1(?:\\.[0-9]+)*$">>,[restart_new_emulator]}]}. diff --git a/lib/sasl/src/systools_relup.erl b/lib/sasl/src/systools_relup.erl index 63e18d394b..19d25fef6e 100644 --- a/lib/sasl/src/systools_relup.erl +++ b/lib/sasl/src/systools_relup.erl @@ -290,9 +290,9 @@ foreach_baserel_up(TopRel, TopApps, [BaseRelDc|BaseRelDcs], Path, Opts, %% {RUs1, Ws1} = collect_appup_scripts(up, TopApps, BaseRel, Ws0++Ws, []), - {RUs2, Ws2} = create_add_app_scripts(BaseRel, TopRel, RUs1, Ws1), + {RUs2, Ws2} = prepend_add_app_scripts(BaseRel, TopRel, RUs1, Ws1), - {RUs3, Ws3} = create_remove_app_scripts(BaseRel, TopRel, RUs2, Ws2), + {RUs3, Ws3} = append_remove_app_scripts(BaseRel, TopRel, RUs2, Ws2), {RUs4, Ws4} = check_for_emulator_restart(TopRel, BaseRel, RUs3, Ws3, Opts), @@ -342,9 +342,9 @@ foreach_baserel_dn(TopRel, TopApps, [BaseRelDc|BaseRelDcs], Path, Opts, %% {RUs1, Ws1} = collect_appup_scripts(dn, TopApps, BaseRel, Ws0++Ws, []), - {RUs2, Ws2} = create_add_app_scripts(TopRel, BaseRel, RUs1, Ws1), + {RUs2, Ws2} = prepend_add_app_scripts(TopRel, BaseRel, RUs1, Ws1), - {RUs3, Ws3} = create_remove_app_scripts(TopRel, BaseRel, RUs2, Ws2), + {RUs3, Ws3} = append_remove_app_scripts(TopRel, BaseRel, RUs2, Ws2), {RUs4, Ws4} = check_for_emulator_restart(TopRel, BaseRel, RUs3, Ws3, Opts), @@ -439,7 +439,7 @@ collect_appup_scripts(_, [], _, Ws, RUs) -> {RUs, Ws}. %% FromRel = ToRel = #release %% ToApps = [#application] %% -create_add_app_scripts(FromRel, ToRel, RU0s, W0s) -> +prepend_add_app_scripts(FromRel, ToRel, RU0s, W0s) -> AddedNs = [{N, T} || {N, _V, T} <- ToRel#release.applications, not lists:keymember(N, 1, FromRel#release.applications)], %% io:format("Added apps: ~p~n", [AddedNs]), @@ -454,12 +454,12 @@ create_add_app_scripts(FromRel, ToRel, RU0s, W0s) -> %% %% XXX ToApps not used. %% -create_remove_app_scripts(FromRel, ToRel, RU0s, W0s) -> +append_remove_app_scripts(FromRel, ToRel, RU0s, W0s) -> RemovedNs = [N || {N, _V, _T} <- FromRel#release.applications, not lists:keymember(N, 1, ToRel#release.applications)], %% io:format("Removed apps: ~p~n", [RemovedNs]), RUs = [[{remove_application, N}] || N <- RemovedNs], - {RUs ++ RU0s, W0s}. + { RU0s ++ RUs, W0s}. %% get_script_from_appup(Mode, TopApp, BaseVsn, Ws, RUs) -> {NRUs, NWs} %% Mode = up | dn diff --git a/lib/sasl/test/release_handler_SUITE.erl b/lib/sasl/test/release_handler_SUITE.erl index d1bcc40049..5ce4050d30 100644 --- a/lib/sasl/test/release_handler_SUITE.erl +++ b/lib/sasl/test/release_handler_SUITE.erl @@ -1847,14 +1847,19 @@ otp_10463_upgrade_script_regexp(cleanup,Config) -> code:del_path(filename:join([DataDir,regexp_appup,app1,ebin])), ok. -no_dot_erlang(_Conf) -> - case init:get_argument(home) of - {ok,[[Home]]} when is_list(Home) -> - no_dot_erlang_1(Home); - _ -> ok +no_dot_erlang(Conf) -> + case {os:type(),init:get_argument(home)} of + {{unix,_},_} -> + %% On unix we set HOME to priv_dir so that we + %% do not have to change the users ~/.erlang + Home = ?config(priv_dir, Conf), + no_dot_erlang_1("HOME=\""++ Home ++"\" ",Home); + {{win32,_},{ok,[[Home]]}} when is_list(Home) -> + no_dot_erlang_1("",Home); + _ -> {skip,"Could not find home directory"} end. -no_dot_erlang_1(Home) -> +no_dot_erlang_1(Prefix, Home) -> DotErlang = filename:join(Home, ".erlang"), BupErlang = filename:join(Home, ".erlang_testbup"), try @@ -1869,7 +1874,7 @@ no_dot_erlang_1(Home) -> Args = " -noinput -run c pwd -run erlang halt", ok = file:write_file(DotErlang, <<"io:put_chars(\"DOT_ERLANG_READ\\n\").\n">>), - CMD1 = Quote ++ Erl ++ Quote ++ Args , + CMD1 = Prefix ++ Quote ++ Erl ++ Quote ++ Args , case os:cmd(CMD1) of "DOT_ERLANG_READ" ++ _ -> io:format("~p: Success~n", [?LINE]); @@ -1880,7 +1885,7 @@ no_dot_erlang_1(Home) -> exit({failed_to_start, test_error}) end, NO_DOT_ERL = " -boot no_dot_erlang", - CMD2 = Quote ++ Erl ++ Quote ++ NO_DOT_ERL ++ Args, + CMD2 = Prefix ++ Quote ++ Erl ++ Quote ++ NO_DOT_ERL ++ Args, case lists:prefix(Wd, Other2 = os:cmd(CMD2)) of true -> io:format("~p: Success~n", [?LINE]); false -> diff --git a/lib/sasl/test/systools_SUITE.erl b/lib/sasl/test/systools_SUITE.erl index 3c60c0fa21..4b67d406a6 100644 --- a/lib/sasl/test/systools_SUITE.erl +++ b/lib/sasl/test/systools_SUITE.erl @@ -72,7 +72,8 @@ groups() -> otp_9507_path_ebin, additional_files_tar, erts_tar]}, {relup, [], [normal_relup, restart_relup, abnormal_relup, no_sasl_relup, - no_appup_relup, bad_appup_relup, app_start_type_relup, regexp_relup + no_appup_relup, bad_appup_relup, app_start_type_relup, regexp_relup, + replace_app_relup ]}, {hybrid, [], [normal_hybrid,hybrid_no_old_sasl,hybrid_no_new_sasl]}, {options, [], [otp_6226_outdir,app_file_defaults]}]. @@ -1942,6 +1943,59 @@ regexp_relup(Config) -> ok. +%% make_relup: Replace an application dependency with another +%% The key part here is that the new application should be +%% started before the old one is stopped. +replace_app_relup(Config) when is_list(Config) -> + {ok, OldDir} = file:get_cwd(), + + {LatestDir,LatestName} = create_script(replace_app0,Config), + {_LatestDir1,LatestName1} = create_script(replace_app1,Config), + + DataDir = filename:absname(?copydir), + LibDir = [fname([DataDir, d_replace_app, lib])], + P = [fname([LibDir, '*', ebin]), + fname([DataDir, lib, kernel, ebin]), + fname([DataDir, lib, stdlib, ebin]), + fname([DataDir, lib, sasl, ebin])], + + ok = file:set_cwd(LatestDir), + + ok = systools:make_relup(LatestName, [LatestName1], [LatestName1], + [{path, P}]), + + check_start_stop_order([{start,gh},{stop,fe}], [{start,fe},{stop,gh}]), + + ok = file:set_cwd(OldDir), + ok. + + +check_start_stop_order(UpOrder, DownOrder) -> + + {ok, [{_V0, [{_V1, [], Up}], + [{_V1, [], Down}] + }]} = file:consult(relup), + + GetAppStartStop = fun(Instr) -> + [{Action,App} || {apply,{application,Action,[App|_]}} <- Instr, + lists:member(Action,[start,stop])] + end, + + case GetAppStartStop(Up) of + UpOrder -> ok; + ActualUpOrder -> + ct:fail("Incorrect upgrade order.~nExpected: ~p~nGot:~p", + [UpOrder,ActualUpOrder]) + end, + + case GetAppStartStop(Down) of + DownOrder -> ok; + ActualDownOrder -> + ct:fail("Incorrect down order.~nExpected: ~p~nGot:~p", + [DownOrder,ActualDownOrder]) + end, + + ok. %% make_hybrid_boot: Normal case. %% For upgrade of erts - create a boot file which is a hybrid between @@ -2497,7 +2551,13 @@ create_script({unicode,RelVsn},Config) -> do_create_script(unicode,RelVsn,Config,current,Apps); create_script(duplicate_modules,Config) -> Apps = core_apps(current) ++ [{app1,"1.0"},{app2,"1.0"}], - do_create_script(duplicate_modules,Config,current,Apps). + do_create_script(duplicate_modules,Config,current,Apps); +create_script(replace_app0,Config) -> + Apps = core_apps(current) ++ [{db,"1.1"},{gh,"1.0"}], + do_create_script(repace_app0,Config,current,Apps); +create_script(replace_app1,Config) -> + Apps = core_apps(current) ++ [{db,"1.0"},{fe,"2.1"}], + do_create_script(repace_app1,Config,current,Apps). do_create_script(Id,Config,ErtsVsn,AppVsns) -> diff --git a/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/db-1.0/ebin/db.app b/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/db-1.0/ebin/db.app new file mode 100644 index 0000000000..d12fcfaf7d --- /dev/null +++ b/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/db-1.0/ebin/db.app @@ -0,0 +1,8 @@ +{application, db, + [{description, "ERICSSON NR FOR DB"}, + {vsn, "1.0"}, + {modules, [db1, db2]}, + {registered, []}, + {applications, [fe]}, + {env, []}, + {start, {db1, start, []}}]}. diff --git a/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/db-1.0/src/db1.erl b/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/db-1.0/src/db1.erl new file mode 100644 index 0000000000..a17640316e --- /dev/null +++ b/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/db-1.0/src/db1.erl @@ -0,0 +1,2 @@ +-module(db2). +-vsn("1.0"). diff --git a/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/db-1.0/src/db2.erl b/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/db-1.0/src/db2.erl new file mode 100644 index 0000000000..a17640316e --- /dev/null +++ b/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/db-1.0/src/db2.erl @@ -0,0 +1,2 @@ +-module(db2). +-vsn("1.0"). diff --git a/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/db-1.1/ebin/db.app b/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/db-1.1/ebin/db.app new file mode 100644 index 0000000000..517a0810f9 --- /dev/null +++ b/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/db-1.1/ebin/db.app @@ -0,0 +1,8 @@ +{application, db, + [{description, "ERICSSON NR FOR DB"}, + {vsn, "1.1"}, + {modules, [db1, db2]}, + {registered, []}, + {applications, [gh]}, + {env, []}, + {start, {db1, start, []}}]}. diff --git a/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/db-1.1/ebin/db.appup b/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/db-1.1/ebin/db.appup new file mode 100644 index 0000000000..12d7ad4c9c --- /dev/null +++ b/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/db-1.1/ebin/db.appup @@ -0,0 +1,14 @@ +{ + "1.1", +%%% Upgrade from: + [ + {"1.0", [{update, db1, soft, soft_purge, soft_purge, []}, + {update, db2, soft, soft_purge, soft_purge, [db1]}]} + ], + +%%% Downgrade to: + [ + {"1.0", [{update, db1, soft, soft_purge, soft_purge, []}, + {update, db2, soft, soft_purge, soft_purge, [db1]}]} + ] +}. diff --git a/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/db-1.1/src/db1.erl b/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/db-1.1/src/db1.erl new file mode 100644 index 0000000000..ee7497f5c1 --- /dev/null +++ b/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/db-1.1/src/db1.erl @@ -0,0 +1,2 @@ +-module(db2). +-vsn("1.1"). diff --git a/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/db-1.1/src/db2.erl b/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/db-1.1/src/db2.erl new file mode 100644 index 0000000000..ee7497f5c1 --- /dev/null +++ b/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/db-1.1/src/db2.erl @@ -0,0 +1,2 @@ +-module(db2). +-vsn("1.1"). diff --git a/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/fe-2.1/ebin/fe.app b/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/fe-2.1/ebin/fe.app new file mode 100644 index 0000000000..717d30cf45 --- /dev/null +++ b/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/fe-2.1/ebin/fe.app @@ -0,0 +1,8 @@ +{application, fe, + [{description, "ERICSSON NR FOR FE"}, + {vsn, "2.1"}, + {modules, [fe1, fe2, fe3]}, + {registered, []}, + {applications, []}, + {env, []}, + {start, {fe2, start, []}}]}. diff --git a/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/fe-2.1/src/fe1.erl b/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/fe-2.1/src/fe1.erl new file mode 100644 index 0000000000..aa5bfa8098 --- /dev/null +++ b/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/fe-2.1/src/fe1.erl @@ -0,0 +1,2 @@ +-module(fe1). +-vsn("1.0"). diff --git a/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/fe-2.1/src/fe2.erl b/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/fe-2.1/src/fe2.erl new file mode 100644 index 0000000000..869f3b93c8 --- /dev/null +++ b/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/fe-2.1/src/fe2.erl @@ -0,0 +1,2 @@ +-module(fe2). +-vsn("1.0"). diff --git a/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/fe-2.1/src/fe3.erl b/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/fe-2.1/src/fe3.erl new file mode 100644 index 0000000000..6473342f52 --- /dev/null +++ b/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/fe-2.1/src/fe3.erl @@ -0,0 +1,2 @@ +-module(fe3). +-vsn("2.0"). diff --git a/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/gh-1.0/ebin/gh.app b/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/gh-1.0/ebin/gh.app new file mode 100644 index 0000000000..2823a7e592 --- /dev/null +++ b/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/gh-1.0/ebin/gh.app @@ -0,0 +1,8 @@ +{application, gh, + [{description, "ERICSSON NR FOR GH"}, + {vsn, "1.0"}, + {modules, [gh1]}, + {registered, []}, + {applications, []}, + {env, []}, + {start, {gh1, start, []}}]}. diff --git a/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/gh-1.0/src/gh1.erl b/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/gh-1.0/src/gh1.erl new file mode 100644 index 0000000000..acd0f43d6a --- /dev/null +++ b/lib/sasl/test/systools_SUITE_data/d_replace_app/lib/gh-1.0/src/gh1.erl @@ -0,0 +1,2 @@ +-module(gh1). +-vsn("1.0"). diff --git a/lib/sasl/vsn.mk b/lib/sasl/vsn.mk index f2b315c59e..a61b72d6f6 100644 --- a/lib/sasl/vsn.mk +++ b/lib/sasl/vsn.mk @@ -1 +1 @@ -SASL_VSN = 4.0.1 +SASL_VSN = 4.0.2 diff --git a/lib/snmp/doc/src/notes.xml b/lib/snmp/doc/src/notes.xml index 792090f3bf..f8caf6228b 100644 --- a/lib/snmp/doc/src/notes.xml +++ b/lib/snmp/doc/src/notes.xml @@ -34,7 +34,24 @@ </header> - <section><title>SNMP 5.7.3</title> + <section><title>SNMP 5.8</title> + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + Add function to get a list of configured agent + transports. Also improved agent info with regards to + transports.</p> + <p> + Own Id: OTP-17109 Aux Id: ERIERL-583 </p> + </item> + </list> + </section> + +</section> + +<section><title>SNMP 5.7.3</title> <section><title>Fixed Bugs and Malfunctions</title> <list> diff --git a/lib/snmp/doc/src/snmpa.xml b/lib/snmp/doc/src/snmpa.xml index 178f25ccb0..c3025dc945 100644 --- a/lib/snmp/doc/src/snmpa.xml +++ b/lib/snmp/doc/src/snmpa.xml @@ -4,7 +4,7 @@ <erlref> <header> <copyright> - <year>2004</year><year>2020</year> + <year>2004</year><year>2021</year> <holder>Ericsson AB. All Rights Reserved.</holder> </copyright> <legalnotice> @@ -503,6 +503,24 @@ notification_delivery_info() = #snmpa_notification_delivery_info{} <desc> <p>Retrieve all tables known to the agent.</p> + <marker id="which_transports"></marker> + </desc> + </func> + + <func> + <name since="">which_transports() -> Result</name> + <fsummary>Get all configured transports</fsummary> + <type> + <v>Result = [{TDomain, TAddress} | {TDomain, TAddress, Kind}]</v> + <v>TDomain = transportDomainUdpIpv4 | transportDomainUdpIpv6</v> + <v>TAddress = {IpAddr, IpPort}</v> + <v>IpAddr = inet:ip_address()</v> + <v>IpPort = pos_integer()</v> + <v>Kind = req_responder | trap_sender</v> + </type> + <desc> + <p>Retrieve all configured transports.</p> + <marker id="which_variables"></marker> </desc> </func> diff --git a/lib/snmp/src/agent/snmpa.erl b/lib/snmp/src/agent/snmpa.erl index 9e428466fa..729789d487 100644 --- a/lib/snmp/src/agent/snmpa.erl +++ b/lib/snmp/src/agent/snmpa.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2004-2016. All Rights Reserved. +%% Copyright Ericsson AB 2004-2021. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. @@ -63,6 +63,8 @@ register_subagent/3, unregister_subagent/2, + which_transports/0, + send_notification2/3, send_notification/3, send_notification/4, send_notification/5, send_notification/6, send_notification/7, @@ -832,6 +834,18 @@ sys_up_time() -> %%%----------------------------------------------------------------- +which_transports() -> + {value, Transports} = snmp_framework_mib:intAgentTransports(get), + [case Kind of + all -> + {Domain, Address}; + _ -> + {Domain, Address, Kind} + end || {Domain, Address, Kind, _} <- Transports]. + + +%%%----------------------------------------------------------------- + restart_worker() -> restart_worker(snmp_master_agent). diff --git a/lib/snmp/src/agent/snmpa_net_if.erl b/lib/snmp/src/agent/snmpa_net_if.erl index 835b8d4375..b6b115bd75 100644 --- a/lib/snmp/src/agent/snmpa_net_if.erl +++ b/lib/snmp/src/agent/snmpa_net_if.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2004-2020. All Rights Reserved. +%% Copyright Ericsson AB 2004-2021. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. @@ -130,6 +130,7 @@ {socket, kind = all :: all | transport_kind(), domain = snmpUDPDomain, + address :: inet:ip_address(), port_no :: pos_integer(), port_info :: port_info(), %% <EPHEMERAL-FOR-FUTUR-USE> @@ -273,8 +274,8 @@ do_init(Prio, NoteStore, MasterAgent, Parent, Opts) -> %% will be taken from the "global" socket options (which serve as %% default values). %% Also, note that Ephm are not actually used at this time. - {Ephm, PortInfo, SocketOpts} = socket_opts(Domain, Address, - RawSocketOpts, Opts), + {Ephm, IpAddr, PortInfo, SocketOpts} = socket_opts(Domain, Address, + RawSocketOpts, Opts), ?vtrace("socket opts processed:" "~n Ephm: ~p" "~n Port Info: ~p" @@ -296,6 +297,7 @@ do_init(Prio, NoteStore, MasterAgent, Parent, Opts) -> %% or a range), so it could have been "generated". %% Also, shall we push this into the transport (handled by the %% FRAMEWORK MIB)? Would not work for ephemeral sockets. + address = IpAddr, port_no = IpPort, port_info = PortInfo, ephm = Ephm, @@ -2034,7 +2036,7 @@ socket_opts(Domain, {IpAddr, PortInfo}, SocketOpts, DefaultOpts) -> %% Ephm = get_ephemeral(SocketOpts), %% {Ephm, PortInfo, Opts}. %% </EPHEMERAL-FOR-FUTUR-USE> - {none, PortInfo, Opts}. + {none, IpAddr, PortInfo, Opts}. %% ---------------------------------------------------------------- @@ -2150,10 +2152,21 @@ get_info(#state{transports = Transports, reqs = Reqs}) -> [{reqs, Reqs}, {counters, Counters}, {process_memory, ProcSize}, - {transport_info, [{PortNo, Kind, get_port_info(Socket)} || - #transport{socket = Socket, - port_no = PortNo, - kind = Kind} <- Transports]}]. + {transport_info, [#{tdomain => Domain, + taddress => {Address, PortNo}, + transport_kind => Kind, + port_info => PortInfo, + opts => Opts, + socket_info => get_port_info(Socket), + num_reqs => length(Refs)} || + #transport{socket = Socket, + domain = Domain, + address = Address, + port_no = PortNo, + port_info = PortInfo, + opts = Opts, + kind = Kind, + req_refs = Refs} <- Transports]}]. proc_mem(P) when is_pid(P) -> case (catch erlang:process_info(P, memory)) of diff --git a/lib/snmp/test/snmp_agent_SUITE.erl b/lib/snmp/test/snmp_agent_SUITE.erl index 3d8170cada..83bc05ff05 100644 --- a/lib/snmp/test/snmp_agent_SUITE.erl +++ b/lib/snmp/test/snmp_agent_SUITE.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2003-2020. All Rights Reserved. +%% Copyright Ericsson AB 2003-2021. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. @@ -5005,16 +5005,74 @@ command_handler([]) -> ok; command_handler([{_No, _Desc, Cmd}|Rest]) -> ?IPRINT("command_handler -> command ~w: ~n ~s", [_No, _Desc]), - case (catch Cmd()) of - ok -> - ?IPRINT("command_handler -> ~w: ok", [_No]), - command_handler(Rest); - {error, Reason} -> - ?EPRINT("command_handler -> ~w error: ~n~p", [_No, Reason]), - ?line ?FAIL(Reason); - Error -> - ?EPRINT("command_handler -> ~w unexpected: ~n~p", [_No, Error]), - ?line ?FAIL({unexpected_command_result, Error}) + %% case (catch Cmd()) of + %% ok -> + %% ?IPRINT("command_handler -> ~w: ok", [_No]), + %% command_handler(Rest); + %% {error, Reason} -> + %% ?EPRINT("command_handler -> ~w error: ~n~p", [_No, Reason]), + %% ?line ?FAIL(Reason); + %% Error -> + %% ?EPRINT("command_handler -> ~w unexpected: ~n~p", [_No, Error]), + %% ?line ?FAIL({unexpected_command_result, Error}) + %% end. + try Cmd() of + ok -> + ?IPRINT("command_handler -> ~w: ok", [_No]), + command_handler(Rest); + {error, Reason} -> + ?IPRINT("command_handler -> command ~w error", [_No]), + SysEvs = snmp_test_global_sys_monitor:events(), + if + (SysEvs =:= []) -> + ?EPRINT("command_handler -> ~w error: ~n~p", [_No, Reason]), + ?line ?FAIL(Reason); + true -> + ?WPRINT("command_handler -> " + "failed when we got system events: " + "~n Reason: ~p" + "~n Sys Events: ~p" + "~n", [Reason, SysEvs]), + ?SKIP([{reason, Reason}, {system_events, SysEvs}]) + end; + Error -> + ?IPRINT("command_handler -> command ~w unexpected", [_No]), + SysEvs = snmp_test_global_sys_monitor:events(), + if + (SysEvs =:= []) -> + ?EPRINT("command_handler -> " + "~w unexpected: ~n~p", [_No, Error]), + ?line ?FAIL({unexpected_command_result, Error}); + true -> + ?WPRINT("command_handler -> " + "unexpected when we got system events: " + "~n Unexpected: ~p" + "~n Sys Events: ~p" + "~n", [Error, SysEvs]), + ?SKIP([{unexpected, Error}, {system_events, SysEvs}]) + end + catch + C:E:S -> + ?IPRINT("command_handler -> command ~w catched", [_No]), + SysEvs = snmp_test_global_sys_monitor:events(), + if + (SysEvs =:= []) -> + ?EPRINT("command_handler -> ~w catched: " + "~n Class: ~p" + "~n Error: ~p" + "~n Stack: ~p", [_No, C, E, S]), + ?line ?FAIL({catched_command_result, {C, E, S}}); + true -> + ?WPRINT("command_handler -> " + "catched when we got system events: " + "~n Catched: " + "~n Class: ~p" + "~n Error: ~p" + "~n Stack: ~p" + "~n Sys Events: ~p" + "~n", [C, E, S, SysEvs]), + ?SKIP([{catched, {C, E, S}}, {system_events, SysEvs}]) + end end. @@ -5488,6 +5546,16 @@ snmp_framework_mib_3(Config) when is_list(Config) -> %% Therefor we must take that into account when we check if the %% Engine Time diff (between the two checks) is acceptably. snmp_framework_mib_test() -> + + ?IPRINT("transports: " + "~n ~p" + "~ninfo: " + "~n ~p", + [ + rpc:call(get(master_node), snmpa, which_transports, []), + rpc:call(get(master_node), snmpa, info, []) + ]), + Sleep = 5, ?line ["agentEngine"] = get_req(1, [[snmpEngineID,0]]), T1 = snmp_misc:now(ms), @@ -6182,81 +6250,129 @@ loop_mib_3(Config) when is_list(Config) -> %% Req. As many mibs all possible loop_mib_1_test() -> - ?DBG("loop_mib_1_test -> entry",[]), + ?IPRINT("loop_mib_1_test -> entry"), N = loop_it_1([1,1], 0), - io:format(user, "found ~w varibles\n", [N]), + ?IPRINT("found ~w varibles", [N]), ?line N = if N < 100 -> 100; true -> N end. loop_it_1(Oid, N) -> - ?DBG("loop_it_1_test -> entry with~n" - "\tOid: ~p~n" - "\tN: ~p",[Oid,N]), + ?IPRINT("loop_it_1_test -> entry with" + "~n Oid: ~p" + "~n N: ~p", [Oid, N]), case get_next_req([Oid]) of #pdu{type = 'get-response', error_status = noError, error_index = 0, varbinds = [#varbind{oid = NOid, value = _Value}]} when NOid > Oid -> - ?DBG("loop_it_1_test -> " - "~n NOid: ~p" - "~n Value: ~p", [NOid, _Value]), + ?IPRINT("loop_it_1_test -> " + "expected intermediate (get-next) result: " + "~n NOid: ~p" + "~n Value: ~p", [NOid, _Value]), ?line [_Value2] = get_req(1, [NOid]), % must not be same - ?DBG("loop_it_1_test -> " - "~n Value2: ~p", [_Value2]), + ?IPRINT("loop_it_1_test -> expected intermediate (get) result: " + "~n Value2: ~p", [_Value2]), loop_it_1(NOid, N+1); #pdu{type = 'get-response', error_status = noError, error_index = 0, varbinds = Vbs} -> - exit({unexpected_vbs, ?LINE, Vbs}); + ?EPRINT("loop_it_1_test -> unexpected (get-response) vbs: " + "~n Vbs: ~p", [Vbs]), + ?line ?FAIL({unexpected_vbs, + [{get_next_oid, Oid}, + {counter, N}, + {varbinds, Vbs}]}); #pdu{type = 'get-response', error_status = noSuchName, error_index = 1, varbinds = [_]} -> - ?DBG("loop_it_1_test -> done: ~p",[N]), + ?IPRINT("loop_it_1_test -> done: ~p", [N]), N; #pdu{type = 'get-response', error_status = Err, error_index = Idx, varbinds = Vbs} -> - exit({unexpected_pdu, ?LINE, Err, Idx, Vbs}); + ?EPRINT("loop_it_1_test -> unexpected (get-response) pdu: " + "~n Err: ~p" + "~n Idx: ~p" + "~n Vbs: ~p", [Err, Idx, Vbs]), + ?line ?FAIL({unexpected_pdu, + [{get_next_oid, Oid}, + {counter, N}, + {error_status, Err}, + {error_index, Idx}, + {varbinds, Vbs}]}); #pdu{type = Type, error_status = Err, error_index = Idx, varbinds = Vbs} -> - exit({unexpected_pdu, ?LINE, Type, Err, Idx, Vbs}); + ?EPRINT("loop_it_1_test -> unexpected pdu: " + "~n Type: ~p" + "~n Err: ~p" + "~n Idx: ~p" + "~n Vbs: ~p", [Type, Err, Idx, Vbs]), + ?line ?FAIL({unexpected_pdu, + [{get_next_oid, Oid}, + {counter, N}, + {type, Type}, + {error_status, Err}, + {error_index, Idx}, + {varbinds, Vbs}]}); {error, Reason} -> - exit({error, Reason, ?LINE}) + %% Regardless of the error here (its usually timeout), + %% if we have had system events we skip since the results + %% in those cases are simply not reliable. + %% There is just no point in trying to analyze the reason. + ?IPRINT("loop_it_1_test -> receive error: " + "~n ~p", [Reason]), + SysEvs = snmp_test_global_sys_monitor:events(), + if + (SysEvs =:= []) -> + ?EPRINT("loop_it_1_test -> error: " + "~n ~p", [Reason]), + ?line ?FAIL([{get_next_oid, Oid}, + {counter, N}, + {reason, Reason}]); + + true -> + ?WPRINT("loop_it_1_test -> " + "error when we got system events: " + "~n Reason: ~p" + "~n Sys Events: ~p" + "~n", [Reason, SysEvs]), + ?SKIP([{reason, Reason}, {system_events, SysEvs}]) + end end. %% Req. As many mibs all possible loop_mib_2_test() -> - ?DBG("loop_mib_2_test -> entry",[]), + ?IPRINT("loop_mib_2_test -> entry"), N = loop_it_2([1,1], 0), - io:format(user, "found ~w varibles\n", [N]), + ?IPRINT("found ~w varibles", [N]), ?line N = if N < 100 -> 100; true -> N end. loop_it_2(Oid, N) -> - ?DBG("loop_it_2 -> entry with" - "~n Oid: ~p" - "~n N: ~p",[Oid, N]), + ?IPRINT("loop_it_2 -> entry with" + "~n Oid: ~p" + "~n N: ~p", [Oid, N]), case get_next_req([Oid]) of #pdu{type = 'get-response', error_status = noError, error_index = 0, varbinds = [#varbind{oid = _NOid, value = endOfMibView}]} -> - ?DBG("loop_it_2 -> " - "~n NOid: ~p", [_NOid]), + ?IPRINT("loop_it_2 -> done: " + "~n NOid: ~p", [_NOid]), N; #pdu{type = 'get-response', @@ -6264,52 +6380,82 @@ loop_it_2(Oid, N) -> error_index = 0, varbinds = [#varbind{oid = NOid, value = _Value}]} when NOid > Oid -> - ?DBG("loop_it_2 -> " - "~n NOid: ~p" - "~n Value: ~p", [NOid, _Value]), + ?IPRINT("loop_it_2 -> " + "expected intermediate (get-next) result: " + "~n NOid: ~p" + "~n Value: ~p", [NOid, _Value]), ?line [_Value2] = get_req(1, [NOid]), % must not be same - ?DBG("loop_it_2 -> " - "~n Value2: ~p", [_Value2]), + ?IPRINT("loop_it_2 -> expected intermediate (get) result: " + "~n Value2: ~p", [_Value2]), loop_it_2(NOid, N+1); #pdu{type = 'get-response', error_status = noError, error_index = 0, varbinds = Vbs} -> - exit({unexpected_pdu, ?LINE, - [{varbinds, Vbs}, - {get_next_oid, Oid}, - {counter, N}]}); + ?EPRINT("loop_it_2 -> unexpected (get-response) vbs: " + "~n Vbs: ~p", [Vbs]), + ?line ?FAIL({unexpected_vbs, + [{get_next_oid, Oid}, + {counter, N}, + {varbinds, Vbs}]}); #pdu{type = 'get-response', error_status = ES, error_index = EI, varbinds = Vbs} -> - exit({unexpected_pdu, ?LINE, - [{error_status, ES}, - {error_index, EI}, - {varbinds, Vbs}, - {get_next_oid, Oid}, - {counter, N}]}); + ?EPRINT("loop_it_2 -> unexpected (get-response) pdu: " + "~n ES: ~p" + "~n EI: ~p" + "~n Vbs: ~p", [ES, EI, Vbs]), + ?line ?FAIL({unexpected_pdu, + [{get_next_oid, Oid}, + {counter, N}, + {error_status, ES}, + {error_index, EI}, + {varbinds, Vbs}]}); #pdu{type = Type, error_status = ES, error_index = EI, varbinds = Vbs} -> - exit({unexpected_pdu, ?LINE, - [{type, Type}, - {error_status, ES}, - {error_index, EI}, - {varbinds, Vbs}, - {get_next_oid, Oid}, - {counter, N}]}); + ?EPRINT("loop_it_2 -> unexpected pdu: " + "~n Type: ~p" + "~n ES: ~p" + "~n EI: ~p" + "~n Vbs: ~p", [Type, ES, EI, Vbs]), + ?line ?FAIL({unexpected_pdu, + [{get_next_oid, Oid}, + {counter, N}, + {type, Type}, + {error_status, ES}, + {error_index, EI}, + {varbinds, Vbs}]}); {error, Reason} -> - exit({unexpected_result, ?LINE, - [{reason, Reason}, - {get_next_oid, Oid}, - {counter, N}]}) - + %% Regardless of the error here (its usually timeout), + %% if we have had system events we skip since the results + %% in those cases are simply not reliable. + %% There is just no point in trying to analyze the reason. + ?IPRINT("loop_it_2 -> receive error: " + "~n ~p", [Reason]), + SysEvs = snmp_test_global_sys_monitor:events(), + if + (SysEvs =:= []) -> + ?EPRINT("loop_it_2 -> error: " + "~n ~p", [Reason]), + ?line ?FAIL([{get_next_oid, Oid}, + {counter, N}, + {reason, Reason}]); + + true -> + ?WPRINT("loop_it_2 -> " + "error when we got system events: " + "~n Reason: ~p" + "~n Sys Events: ~p" + "~n", [Reason, SysEvs]), + ?SKIP([{reason, Reason}, {system_events, SysEvs}]) + end end. loop_mib_3_test() -> @@ -7930,22 +8076,29 @@ otp16649_validate_transports([], []) -> ok; otp16649_validate_transports([AgentRawTransport|AgentRawTransports], [TI|TIs]) -> + ?IPRINT("validate transport:" + "~n AgentRawTransport: ~p" + "~n TI: ~p", [AgentRawTransport, TI]), otp16649_validate_transport(AgentRawTransport, TI), otp16649_validate_transports(AgentRawTransports, TIs). -otp16649_validate_transport({PortInfo, Kind}, {PortNo, Kind, _}) -> +otp16649_validate_transport({PortInfo, Kind}, #{taddress := {_, PortNo}, + transport_kind := Kind}) -> ?IPRINT("validate ~w transport:" "~n PortNo: ~w" "~n PortInfo: ~p", [Kind, PortNo, PortInfo]), otp16649_validate_port(PortInfo, PortNo); -otp16649_validate_transport({_, ConfKind}, {PortNo, ActualKind, _}) -> +otp16649_validate_transport({_, ConfKind}, #{taddress := {_, PortNo}, + transport_kind := ActualKind}) -> exit({invalid_transport_kind, {PortNo, ConfKind, ActualKind}}); -otp16649_validate_transport({PortInfo, Kind, _}, {PortNo, Kind, _}) -> +otp16649_validate_transport({PortInfo, Kind, _}, #{taddress := {_, PortNo}, + transport_kind := Kind}) -> ?IPRINT("validate ~w transport:" "~n PortNo: ~w" "~n PortInfo: ~p", [Kind, PortNo, PortInfo]), otp16649_validate_port(PortInfo, PortNo); -otp16649_validate_transport({_, ConfKind, _}, {PortNo, ActualKind, _}) -> +otp16649_validate_transport({_, ConfKind, _}, #{taddress := {_, PortNo}, + transport_kind := ActualKind}) -> exit({invalid_transport_kind, {PortNo, ConfKind, ActualKind}}). otp16649_validate_port(PortNo, PortNo) when is_integer(PortNo) -> @@ -8007,7 +8160,8 @@ otp16649_which_trap_port_no(TIs) -> otp16649_which_port_no([], Kind) -> exit({no_transport_port_no, Kind}); -otp16649_which_port_no([{PortNo, Kind, _}|_], Kind) -> +otp16649_which_port_no([#{taddress := {_, PortNo}, + transport_kind := Kind}|_], Kind) -> PortNo; otp16649_which_port_no([_|TIs], Kind) -> otp16649_which_port_no(TIs, Kind). @@ -8675,5 +8829,3 @@ rcall(Node, Mod, Func, Args) -> Else -> Else end. - - diff --git a/lib/snmp/test/snmp_agent_test_lib.erl b/lib/snmp/test/snmp_agent_test_lib.erl index da2762c3fb..96cc6add81 100644 --- a/lib/snmp/test/snmp_agent_test_lib.erl +++ b/lib/snmp/test/snmp_agent_test_lib.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2005-2020. All Rights Reserved. +%% Copyright Ericsson AB 2005-2021. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. @@ -395,7 +395,21 @@ await_tc_runner_done(Runner, OldFlag) -> unlink_and_flush_exit(Runner), case Ret of {error, Reason} -> - exit(Reason); + %% Any failures while we have system events are skipped + SysEvs = snmp_test_global_sys_monitor:events(), + if + (SysEvs =:= []) -> + ?EPRINT("TC failure: " + "~n ~p" + "~n", [Reason]), + exit(Reason); + true -> + ?WPRINT("TC failure when we got system events: " + "~n Reason: ~p" + "~n Sys Events: ~p" + "~n", [Reason, SysEvs]), + skip([{reason, Reason}, {system_events, SysEvs}]) + end; {skip, Reason} -> skip(Reason); OK -> @@ -498,34 +512,22 @@ tc_run(Mod, Func, Args, Opts) -> {dir, Dir}, {mibs, mibs(StdM, M)}]) of {ok, _Pid} -> - case (catch apply(Mod, Func, Args)) of - {'EXIT', {skip, Reason}} -> - ?WPRINT("apply skip detected: " - "~n ~p", [Reason]), - (catch snmp_test_mgr:stop()), - ?SKIP(Reason); - {'EXIT', Reason} -> - %% We have hosts (mostly *very* slooow VMs) that - %% can timeout anything. Since we are basically - %% testing communication, we therefor must check - %% for system events at every failure. Grrr! - SysEvs = snmp_test_global_sys_monitor:events(), - (catch snmp_test_mgr:stop()), - if - (SysEvs =:= []) -> - ?EPRINT("TC runner failed: " - "~n ~p~n", [Reason]), - ?FAIL({apply_failed, {Mod, Func, Args}, Reason}); - true -> - ?WPRINT("apply exit catched when we got system events: " - "~n Reason: ~p" - "~n Sys Events: ~p" - "~n", [Reason, SysEvs]), - ?SKIP([{reason, Reason}, {system_events, SysEvs}]) - end; - Res -> + try apply(Mod, Func, Args) of + Res -> (catch snmp_test_mgr:stop()), Res + catch + C:{skip, Reason} -> + ?WPRINT("apply (~w-) skip detected: " + "~n ~p", [C, Reason]), + (catch snmp_test_mgr:stop()), + ?SKIP(Reason); + + throw:{error, Reason} -> + tc_run_skip_sheck(Mod, Func, Args, Reason, throw); + + exit:Reason -> + tc_run_skip_sheck(Mod, Func, Args, Reason, exit) end; {error, Reason} -> @@ -541,6 +543,28 @@ tc_run(Mod, Func, Args, Opts) -> ?line ?FAIL({mgr_start_failure, Err}) end. +%% We have hosts (mostly *very* slooow VMs) that +%% can timeout anything. Since we are basically +%% testing communication, we therefor must check +%% for system events at every failure. Grrr! +tc_run_skip_sheck(Mod, Func, Args, Reason, Cat) -> + SysEvs = snmp_test_global_sys_monitor:events(), + (catch snmp_test_mgr:stop()), + if + (SysEvs =:= []) -> + ?EPRINT("TC runner (~w-) failed: " + "~n ~p~n", [Cat, Reason]), + ?FAIL({apply_failed, {Mod, Func, Args}, Reason}); + true -> + ?WPRINT("apply (~w) catched " + "when we got system events: " + "~n Reason: ~p" + "~n Sys Events: ~p" + "~n", [Cat, Reason, SysEvs]), + ?SKIP([{category, Cat}, + {reason, Reason}, {system_events, SysEvs}]) + end. + %% --------------------------------------------------------------- %% --- --- diff --git a/lib/snmp/test/snmp_test_global_sys_monitor.erl b/lib/snmp/test/snmp_test_global_sys_monitor.erl index 54cc7d588e..c3f2e24096 100644 --- a/lib/snmp/test/snmp_test_global_sys_monitor.erl +++ b/lib/snmp/test/snmp_test_global_sys_monitor.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 2019-2019. All Rights Reserved. +%% Copyright Ericsson AB 2019-2021. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. @@ -28,7 +28,8 @@ -include("snmp_test_lib.hrl"). --define(NAME, ?MODULE). +-define(NAME, ?MODULE). +-define(TIMEOUT, timer:seconds(6)). %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% @@ -43,10 +44,10 @@ stop() -> %% This does not reset the global counter but the "collector" %% See events for more info. reset_events() -> - call(reset_events). + call(reset_events, ?TIMEOUT). events() -> - call(events). + call(events, ?TIMEOUT). log(Event) -> cast({node(), Event}). @@ -198,23 +199,66 @@ cast(Msg) -> {error, {catched, C, E}} end. -call(Req) -> - call(Req, infinity). - -call(Req, Timeout) -> - Ref = make_ref(), - try global:send(?NAME, {?MODULE, Ref, self(), Req}) of - Pid when is_pid(Pid) -> - receive - {?MODULE, Ref, Rep} -> - Rep - after Timeout -> - {error, timeout} - end - catch - C:E:_ -> - {error, {catched, C, E}} +%% call(Req) -> +%% call(Req, infinity). + +%% call(Req, Timeout) -> +%% Ref = make_ref(), +%% try global:send(?NAME, {?MODULE, Ref, self(), Req}) of +%% Pid when is_pid(Pid) -> +%% receive +%% {?MODULE, Ref, Rep} -> +%% Rep +%% after Timeout -> +%% {error, timeout} +%% end +%% catch +%% C:E:_ -> +%% {error, {catched, C, E}} +%% end. + +call(Req, Timeout) when (Timeout =:= infinity) -> + call(Req, Timeout, Timeout); +call(Req, Timeout) when is_integer(Timeout) andalso (Timeout > 2000) -> + call(Req, Timeout, Timeout - 1000); +call(Req, Timeout) when is_integer(Timeout) andalso (Timeout > 1000) -> + call(Req, Timeout, Timeout - 500); +call(Req, Timeout) when is_integer(Timeout) -> + call(Req, Timeout, Timeout div 2). + +%% This peace of wierdness is because on some machines this call has +%% hung (in a call during end_per_testcase, which had a 1 min timeout, +%% or if that was the total time for the test case). +%% But because it hung there, we don't really know what where it git stuck. +%% So, by making the call in a tmp process, that we supervise, we can +%% keep control. Also, we change the default timeout from infinity to an +%% actual time (16 seconds). +call(Req, Timeout1, Timeout2) -> + F = fun() -> + Ref = make_ref(), + try global:send(?NAME, {?MODULE, Ref, self(), Req}) of + NamePid when is_pid(NamePid) -> + receive + {?MODULE, Ref, Rep} -> + Rep + after Timeout2 -> + {error, timeout} + end + catch + C:E:_ -> + {error, {catched, C, E}} + end + end, + {Pid, Mon} = spawn_monitor(F), + receive + {'DOWN', Mon, process, Pid, Result} -> + Result + after Timeout1 -> + PInfo = process_info(Pid), + exit(Pid, kill), + {error, {timeout, PInfo}} end. + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% diff --git a/lib/snmp/test/snmp_test_lib.erl b/lib/snmp/test/snmp_test_lib.erl index d4e7c53e0a..1828e62ffb 100644 --- a/lib/snmp/test/snmp_test_lib.erl +++ b/lib/snmp/test/snmp_test_lib.erl @@ -642,7 +642,7 @@ init_per_suite(Config) -> SKIP end. -maybe_skip(HostInfo) -> +maybe_skip(_HostInfo) -> %% We have some crap machines that causes random test case failures %% for no obvious reason. So, attempt to identify those without actually @@ -701,14 +701,19 @@ maybe_skip(HostInfo) -> true end, SkipWindowsOnVirtual = + %% fun() -> + %% SysMan = win_sys_info_lookup(system_manufacturer, HostInfo), + %% case string:to_lower(SysMan) of + %% "vmware" ++ _ -> + %% true; + %% _ -> + %% false + %% end + %% end, fun() -> - SysMan = win_sys_info_lookup(system_manufacturer, HostInfo), - case string:to_lower(SysMan) of - "vmware" ++ _ -> - true; - _ -> - false - end + %% The host has been replaced and the VM has been reinstalled + %% so for now we give it a chance... + false end, COND = [{unix, [{linux, LinuxVersionVerify}, {darwin, DarwinVersionVerify}]}, diff --git a/lib/snmp/test/snmp_test_mgr.erl b/lib/snmp/test/snmp_test_mgr.erl index d7db3a2b0d..fe2852c573 100644 --- a/lib/snmp/test/snmp_test_mgr.erl +++ b/lib/snmp/test/snmp_test_mgr.erl @@ -1,7 +1,7 @@ %% %% %CopyrightBegin% %% -%% Copyright Ericsson AB 1996-2020. All Rights Reserved. +%% Copyright Ericsson AB 1996-2021. All Rights Reserved. %% %% Licensed under the Apache License, Version 2.0 (the "License"); %% you may not use this file except in compliance with the License. @@ -168,6 +168,7 @@ get_timeout() -> get_timeout(_) -> 10000. % Trying to improve test results % 3500. + %%---------------------------------------------------------------------- %% Receives a trap from the agent. %% Returns: TrapPdu|{error, Reason} diff --git a/lib/snmp/vsn.mk b/lib/snmp/vsn.mk index e1e018878a..102de54127 100644 --- a/lib/snmp/vsn.mk +++ b/lib/snmp/vsn.mk @@ -19,6 +19,6 @@ # %CopyrightEnd% APPLICATION = snmp -SNMP_VSN = 5.7.3 +SNMP_VSN = 5.8 PRE_VSN = APP_VSN = "$(APPLICATION)-$(SNMP_VSN)$(PRE_VSN)" diff --git a/lib/ssh/doc/src/SSH_app.xml b/lib/ssh/doc/src/SSH_app.xml index 490920bd4e..f471971ab5 100644 --- a/lib/ssh/doc/src/SSH_app.xml +++ b/lib/ssh/doc/src/SSH_app.xml @@ -67,13 +67,13 @@ <item><c>known_hosts</c></item> <item><c>authorized_keys</c></item> <item><c>authorized_keys2</c></item> - <item><c>id_dsa</c> <i>(disabled by default)</i></item> - <item><c>id_rsa</c> <i>(disabled by default from OTP-24)</i></item> + <item><c>id_dsa</c> <i>(supported but disabled by default)</i></item> + <item><c>id_rsa</c> <i>(SHA1 sign/verify are supported but disabled by default from OTP-24)</i></item> <item><c>id_ecdsa</c></item> <item><c>id_ed25519</c></item> <item><c>id_ed448</c></item> - <item><c>ssh_host_dsa_key</c> <i>(disabled by default)</i></item> - <item><c>ssh_host_rsa_key</c> <i>(disabled by default from OTP-24)</i></item> + <item><c>ssh_host_dsa_key</c> <i>(supported but disabled by default)</i></item> + <item><c>ssh_host_rsa_key</c> <i>(SHA1 sign/verify are supported but disabled by default from OTP-24)</i></item> <item><c>ssh_host_ecdsa_key</c></item> <item><c>ssh_host_ed25519_key</c></item> <item><c>ssh_host_ed448_key</c></item> @@ -95,6 +95,15 @@ <p>See also the default callback module documentation in <seeerl marker="ssh_file">ssh_file</seeerl>. </p> + <p>Disabled public key algorithms can be enabled with the + <seetype marker="ssh:ssh#preferred_algorithms_common_option">preferred_algorithms</seetype> + or + <seetype marker="ssh:ssh#modify_algorithms_common_option">modify_algorithms</seetype> + options. + See <seeguide marker="configure_algos#example-9">Example 9</seeguide> in + <seeguide marker="configure_algos">Configuring algorithms in SSH</seeguide> + for a description. + </p> </section> <section> @@ -174,13 +183,13 @@ <item>curve25519-sha256@libssh.org</item> <item>curve448-sha512</item> </list> - <p>The following unsecure <c>sha1</c> algorithms are now disabled by default:</p> + <p>The following unsecure <c>SHA1</c> algorithms are now disabled by default:</p> <list> <item>(diffie-hellman-group14-sha1)</item> <item>(diffie-hellman-group-exchange-sha1)</item> <item>(diffie-hellman-group1-sha1)</item> </list> - <p>They can be enabled with the + <p>They can be enabled with the <seetype marker="ssh:ssh#preferred_algorithms_common_option">preferred_algorithms</seetype> or <seetype marker="ssh:ssh#modify_algorithms_common_option">modify_algorithms</seetype> @@ -197,17 +206,22 @@ <item>ssh-ed448</item> <item>rsa-sha2-256</item> <item>rsa-sha2-512</item> - <item>ssh-rsa <i>(disabled by default from OTP-24)</i></item> + <item>ssh-rsa <i>(SHA1 sign/verify are supported but disabled by default from OTP-24)</i></item> </list> - <p>The following unsecure <c>sha1</c> algorithm is now disabled by default:</p> + <p>The following unsecure <c>SHA1</c> algorithm is supported but disabled by default:</p> <list> <item>(ssh-dss)</item> </list> - <p>It can be enabled with the + <p>See + Disabled public key algorithms can be enabled with the <seetype marker="ssh:ssh#preferred_algorithms_common_option">preferred_algorithms</seetype> or <seetype marker="ssh:ssh#modify_algorithms_common_option">modify_algorithms</seetype> - options. Use for example the Option value <c>{modify_algorithms, [{append, [{public_key,['ssh-dss']}]}]}</c>)</p> + options. + See <seeguide marker="configure_algos#example-9">Example 9</seeguide> in + <seeguide marker="configure_algos">Configuring algorithms in SSH</seeguide> + for a description. + </p> </item> <tag>MAC algorithms</tag> @@ -220,11 +234,11 @@ <item>hmac-sha2-512</item> <item>hmac-sha1</item> </list> - <p>The following unsecure <c>sha1</c> algorithm is disabled by default:</p> + <p>The following unsecure <c>SHA1</c> algorithm is disabled by default:</p> <list> <item>(hmac-sha1-96)</item> </list> - <p>It can be enabled with the + <p>It can be enabled with the <seetype marker="ssh:ssh#preferred_algorithms_common_option">preferred_algorithms</seetype> or <seetype marker="ssh:ssh#modify_algorithms_common_option">modify_algorithms</seetype> @@ -250,7 +264,7 @@ <p>See the text at the description of <seeapp marker="#rfc5647_note">the rfc 5647 further down</seeapp> for more information regarding AEAD_AES_*_GCM. </p> - <p>Following the internet de-facto standard, the cipher and mac algorithm AEAD_AES_128_GCM is selected when the + <p>Following the internet de-facto standard, the cipher and mac algorithm AEAD_AES_128_GCM is selected when the cipher aes128-gcm@openssh.com is negotiated. The cipher and mac algorithm AEAD_AES_256_GCM is selected when the cipher aes256-gcm@openssh.com is negotiated. </p> @@ -307,11 +321,17 @@ </list> </item> </list> - <p>They are disabled by default, but can be enabled with the + <p>They are disabled by default as they now are regarded insecure, but they can be enabled with the <seetype marker="ssh:ssh#preferred_algorithms_common_option">preferred_algorithms</seetype> or <seetype marker="ssh:ssh#modify_algorithms_common_option">modify_algorithms</seetype> options. + See <seeguide marker="configure_algos#example-8">Example 8</seeguide> (diffie-hellman-group1-sha1) + and + <seeguide marker="configure_algos#example-9">Example 9</seeguide> (ssh-dss) + in + <seeguide marker="configure_algos">Configuring algorithms in SSH</seeguide> + for descriptions. </p> </item> @@ -341,7 +361,7 @@ <list type="bulleted"> <item>4.1. diffie-hellman-group-exchange-sha1</item> </list> - <p>It is disabled by default, but can be enabled with the + <p>It is disabled by defaultas as it now is regarded insecure, but it can be enabled with the <seetype marker="ssh:ssh#preferred_algorithms_common_option">preferred_algorithms</seetype> or <seetype marker="ssh:ssh#modify_algorithms_common_option">modify_algorithms</seetype> @@ -406,7 +426,7 @@ <item><c>diffie-hellman-group-exchange-sha1</c></item> <item><c>diffie-hellman-group14-sha1</c></item> </list> - <p>are not enabled by default, + <p>are not enabled by default as they now are regarded insecure, but are still supported and can be enabled with the options <seetype marker="ssh:ssh#preferred_algorithms_common_option">preferred_algorithms</seetype> or diff --git a/lib/ssh/doc/src/configure_algos.xml b/lib/ssh/doc/src/configure_algos.xml index 66426f3627..1b26cd2f25 100644 --- a/lib/ssh/doc/src/configure_algos.xml +++ b/lib/ssh/doc/src/configure_algos.xml @@ -429,7 +429,49 @@ but it is possible if an unforeseen need should arise.</p> </section> - + <section> + <title>Example 8</title> + <p>In this example, we need to use a diffie-hellman-group1-sha1 key exchange algorithm + although it is unsage and disabled by default. + </p> + <p>We use the + <seetype marker="ssh#modify_algorithms_common_option">modify_algorithms</seetype> + option, because we want to keep all other algorithm definitions. + </p> + <p>We add the option: + </p> + <code type="erl"> + {modify_algorithms, [{append, [{kex,['diffie-hellman-group1-sha1']}]}]} + </code> + <p>either to the Options list in a function call, in the <c>ssh.app</c> file or in a <c>.config</c> file for + the <c>erl</c> command. + See the chapter + <seeguide marker="ssh:configurations">Configuration in SSH</seeguide> + in the SSH User's Guide. + </p> + </section> + + <section> + <title>Example 9</title> + <p>In this example, we need to use a DSA key for sign and verify. + It might be either as a user's key, a host's key or both. + </p> + <p>To do that, we enable the 'ssh-dss' algorithm that is disabled by default by security reasons. We use the + <seetype marker="ssh#modify_algorithms_common_option">modify_algorithms</seetype> + option, because we want to keep all other algorithm definitions. + </p> + <p>We add the option: + </p> + <code type="erl"> + {modify_algorithms, [{append, [{public_key,['ssh-dss']}]}]} + </code> + <p>either to the Options list in a function call, in the <c>ssh.app</c> file or in a <c>.config</c> file for + the <c>erl</c> command. + See the chapter + <seeguide marker="ssh:configurations">Configuration in SSH</seeguide> + in the SSH User's Guide. + </p> + </section> </section> diff --git a/lib/ssh/doc/src/notes.xml b/lib/ssh/doc/src/notes.xml index 60c3a9b9a5..bd5bff57ab 100644 --- a/lib/ssh/doc/src/notes.xml +++ b/lib/ssh/doc/src/notes.xml @@ -30,6 +30,41 @@ <file>notes.xml</file> </header> +<section><title>Ssh 4.11.1</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + The idle_time timer was not cancelled when a channel was + opened within the timeout time on an empty connection + that have had channels previously.</p> + <p> + Own Id: OTP-17279</p> + </item> + </list> + </section> + +</section> + +<section><title>Ssh 4.11</title> + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + The long name field in SSH_FXP_NAME responses to display + file information in sftp version 3 now contains the + expanded format defined in the sftp draft. It is similar + to what is returned by "ls -l" on Unix systems.</p> + <p> + Own Id: OTP-17197 Aux Id: PR- 3049 </p> + </item> + </list> + </section> + +</section> + <section><title>Ssh 4.10.8</title> <section><title>Fixed Bugs and Malfunctions</title> @@ -486,6 +521,23 @@ </section> +<section><title>Ssh 4.9.1.3</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + The idle_time timer was not cancelled when a channel was + opened within the timeout time on an empty connection + that have had channels previously.</p> + <p> + Own Id: OTP-17279</p> + </item> + </list> + </section> + +</section> + <section><title>Ssh 4.9.1.2</title> <section><title>Fixed Bugs and Malfunctions</title> @@ -823,6 +875,23 @@ </section> +<section><title>Ssh 4.7.6.6</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + The idle_time timer was not cancelled when a channel was + opened within the timeout time on an empty connection + that have had channels previously.</p> + <p> + Own Id: OTP-17279</p> + </item> + </list> + </section> + +</section> + <section><title>Ssh 4.7.6.5</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/ssh/src/ssh_connection_handler.erl b/lib/ssh/src/ssh_connection_handler.erl index a198a95937..805ec042a6 100644 --- a/lib/ssh/src/ssh_connection_handler.erl +++ b/lib/ssh/src/ssh_connection_handler.erl @@ -1627,8 +1627,13 @@ handle_event(info, {'DOWN', _Ref, process, ChannelPid, _Reason}, _, D) -> end, [], Cache), {keep_state, D, cond_set_idle_timer(D)}; -handle_event({timeout,idle_time}, _Data, _StateName, _D) -> - {stop, {shutdown, "Timeout"}}; +handle_event({timeout,idle_time}, _Data, _StateName, D) -> + case ssh_client_channel:cache_info(num_entries, cache(D)) of + 0 -> + {stop, {shutdown, "Timeout"}}; + _ -> + keep_state_and_data + end; %%% So that terminate will be run when supervisor is shutdown handle_event(info, {'EXIT', _Sup, Reason}, StateName, _) -> diff --git a/lib/ssh/src/ssh_info.erl b/lib/ssh/src/ssh_info.erl index 91365205aa..583274d44c 100644 --- a/lib/ssh/src/ssh_info.erl +++ b/lib/ssh/src/ssh_info.erl @@ -34,8 +34,10 @@ -include("ssh_connect.hrl"). print() -> - io:format("~s", [string()]). + print(fun io:format/2). +print(F) when is_function(F,2) -> + F("~s", [string()]); print(File) when is_list(File) -> {ok,D} = file:open(File, [write]), print(D), @@ -51,10 +53,10 @@ string() -> print_general(), io_lib:nl(), underline("Client part", $=), - print_clients(), + lists:map(fun print_system/1, children(sshc_sup)), io_lib:nl(), underline("Server part", $=), - print_servers(), + lists:map(fun print_system/1, children(sshd_sup)), io_lib:nl(), underline("Supervisors", $=), walk_sups(ssh_sup), @@ -66,6 +68,8 @@ string() -> %%%================================================================ +-define(inc(N), (N+4)). + -define(INDENT, " "). print_general() -> @@ -74,116 +78,87 @@ print_general() -> io_lib:format('This printout is generated ~s. ~n',[datetime()]) ]. -print_clients() -> - try - lists:map(fun print_client/1, - supervisor:which_children(sshc_sup)) - catch - C:E:S -> - io_lib:format('***print_clients FAILED: ~p:~p,~n ~p~n',[C,E,S]) - end. -print_client({undefined,Pid,supervisor,[ssh_connection_handler]}) -> - {{Local,Remote},_Str} = ssh_connection_handler:get_print_info(Pid), - [io_lib:format(?INDENT"Local: ~s Remote: ~s ConnectionRef = ~p~n", - [fmt_host_port(Local), fmt_host_port(Remote), Pid]), - case channels(Pid) of - {ok,Channels=[_|_]} -> - [print_ch(ChPid) || #channel{user=ChPid} <- Channels]; - _ -> - io_lib:format(?INDENT?INDENT?INDENT"No channels~n",[]) - end]; +print_system({{server,ssh_system_sup,Addr,Port,Profile}, Pid, supervisor, [ssh_system_sup]}) -> + [io_lib:format(?INDENT"Local: ~s (~p children) Profile ~p~n", + [fmt_host_port({Addr,Port}), + ssh_acceptor:number_of_connections(Pid), + Profile + ]), + lists:map(fun print_subsystem/1, children(Pid)) + ]; +print_system({{client,ssh_system_sup,Addr,Port,Profile}, Pid, supervisor, [ssh_system_sup]}) -> + [io_lib:format(?INDENT"Local: ~s Profile ~p~n", + [fmt_host_port({Addr,Port}), + Profile + ]), + lists:map(fun print_subsystem/1, children(Pid)) + ]; +print_system({_, _Pid, worker, [ssh_controller]}) -> + ""; % io_lib:format(?INDENT"Controller~n",[]); +print_system(_X) -> + io_lib:format(?INDENT"nyi system ~p~n",[_X]). + + + + +print_subsystem({{ssh_acceptor_sup,_Addr,_Port,_Profile}, _Pid, supervisor, [ssh_acceptor_sup]}) -> + io_lib:format(?INDENT?INDENT"Acceptor~n",[]); +print_subsystem({Ref,Pid,supervisor,[ssh_subsystem_sup]}) when is_reference(Ref), + is_pid(Pid) -> + Cs = children(Pid), + [ + lists:map( + fun(Sup) -> + [print_conn(P) || {undefined,P,worker,[ssh_connection_handler]} <- children(Sup)] + end, + [P || {_Ref,P,supervisor,[ssh_connection_sup]} <- Cs]), + + lists:map( + fun(Sup) -> + [print_ch(M,P) || {_Ref,P,worker,[M]} <- children(Sup), + lists:member(M, [ssh_channel, + ssh_channel_sup, + ssh_client_channel, + ssh_daemon_channel, + ssh_server_channel + ])] + end, + [P || {_Ref,P,supervisor,[ssh_channel_sup]} <- Cs]), + + lists:map( + fun(Sup) -> + [io_lib:format(?INDENT?INDENT?INDENT"TCP/IP fwd acceptor: ~p~n", [Pa]) + || {undefined,Pa,worker,[ssh_tcpip_forward_acceptor]} <- children(Sup)] + end, + [P || {_Ref,P,supervisor,[ssh_tcpip_forward_acceptor_sup]} <- Cs]) + ]; -print_client({{client,ssh_system_sup,_,_,_},Pid,supervisor,[ssh_system_sup]}) when is_pid(Pid) -> - lists:map(fun print_system_sup/1, - supervisor:which_children(Pid)). +print_subsystem(_X) -> + io_lib:format(?INDENT?INDENT"nyi subsystem ~p~n",[_X]). -%%%================================================================ -print_servers() -> + +print_conn(Pid) -> try - lists:map(fun print_server/1, - supervisor:which_children(sshd_sup)) + {{_Local,Remote},StrM} = ssh_connection_handler:get_print_info(Pid), + io_lib:format(?INDENT?INDENT"Remote: ~s ConnectionRef = ~p ~s~n",[fmt_host_port(Remote),Pid,StrM]) catch - C:E:S -> - io_lib:format('***print_servers FAILED: ~p:~p,~n ~p~n',[C,E,S]) - end. + C:E -> + io_lib:format('****print_conn FAILED for ConnPid ~p: ~p:~p~n',[Pid, C, E]) + end. + - -print_server({{server,ssh_system_sup,LocalHost,LocalPort,Profile},Pid,supervisor,[ssh_system_sup]}) when is_pid(Pid) -> - Children = supervisor:which_children(Pid), - [io_lib:format(?INDENT"Listen: ~s (~p children) Profile ~p",[fmt_host_port({LocalHost,LocalPort}), - ssh_acceptor:number_of_connections(Pid), - Profile]), - case [AccPid - || {{ssh_acceptor_sup,_LocalHost,_LocalPort,_Profile}, AccPid, supervisor, [ssh_acceptor_sup]} - <- Children] of - AcceptorPids = [_|_] -> - [io_lib:format(" [Acceptor Pid", []), - [io_lib:format(" ~p",[AccPid]) || AccPid <- AcceptorPids], - io_lib:format("]~n", []) - ]; - [] -> - io_lib:nl() - end, - lists:map(fun print_system_sup/1, - supervisor:which_children(Pid)) - ]. - - -print_system_sup({Ref,Pid,supervisor,[ssh_subsystem_sup]}) when is_reference(Ref), - is_pid(Pid) -> - lists:map(fun print_channels/1, - supervisor:which_children(Pid)); - -print_system_sup({{ssh_acceptor_sup,_LocalHost,_LocalPort,_Profile}, Pid, supervisor, [ssh_acceptor_sup]}) when is_pid(Pid) -> - []. - - - -print_channels({{Role,ssh_channel_sup,_,_},Pid,supervisor,[ssh_channel_sup]}) when is_pid(Pid) -> - ChanBehaviour = - case Role of - server -> ssh_server_channel; - client -> ssh_client_channel - end, - Children = supervisor:which_children(Pid), - ChannelPids = [P || {R,P,worker,[Mod]} <- Children, - ChanBehaviour == Mod, - is_pid(P), - is_reference(R)], - case ChannelPids of - [] -> io_lib:format(?INDENT?INDENT"No channels~n",[]); - [Ch1Pid|_] -> - {{ConnManager,_}, _Str} = ChanBehaviour:get_print_info(Ch1Pid), - {{_,Remote},_} = ssh_connection_handler:get_print_info(ConnManager), - [io_lib:format(?INDENT?INDENT"Remote: ~s ConnectionRef = ~p~n",[fmt_host_port(Remote),ConnManager]), - lists:map(fun print_ch/1, ChannelPids) - ] - end; -print_channels({{_Role,ssh_connection_sup,_,_},Pid,supervisor,[ssh_connection_sup]}) when is_pid(Pid) -> - []; % The supervisor of the connections socket owning process - -print_channels({Ref,Pid,supervisor,[ssh_tcpip_forward_acceptor_sup]}) when is_pid(Pid), - is_reference(Ref) -> - []. % The supervisor of the forward_acceptor process - - -print_ch(Pid) -> +print_ch(CBmod, Pid) -> try - {{ConnManager,ChannelID}, Str} = ssh_server_channel:get_print_info(Pid), - {_LocalRemote,StrM} = ssh_connection_handler:get_print_info(ConnManager), - io_lib:format(?INDENT?INDENT?INDENT"ch ~p ~p: ~s ~s~n",[ChannelID, Pid, StrM, Str]) + {{_ConnManager,ChannelID}, Str} = ssh_server_channel:get_print_info(Pid), + io_lib:format(?INDENT?INDENT?INDENT"ch ~p ~p ~p: ~s~n",[ChannelID, Pid, CBmod, Str]) catch C:E -> io_lib:format('****print_ch FAILED for ChanPid ~p: ~p:~p~n',[Pid, C, E]) end. - %%%================================================================ --define(inc(N), (N+4)). - walk_sups(StartPid) -> - io_lib:format("Start at ~p, ~s.~n",[StartPid,dead_or_alive(StartPid)]), walk_sups(children(StartPid), _Indent=?inc(0)). walk_sups([H={_,Pid,_,_}|T], Indent) -> diff --git a/lib/ssh/src/ssh_message.erl b/lib/ssh/src/ssh_message.erl index fab9c50867..fa9176b61f 100644 --- a/lib/ssh/src/ssh_message.erl +++ b/lib/ssh/src/ssh_message.erl @@ -788,7 +788,35 @@ ssh_dbg_format(ssh_messages, {call, {?MODULE,decode,[_]}}) -> ssh_dbg_format(ssh_messages, {return_from,{?MODULE,decode,1},Msg}) -> Name = string:to_upper(atom_to_list(element(1,Msg))), ["Received ",Name,":\n", - wr_record(ssh_dbg:shrink_bin(Msg)) + wr_record(ssh_dbg:shrink_bin(Msg)), + case Msg of + #ssh_msg_userauth_request{service = "ssh-connection", + method = "publickey", + data = <<_,?DEC_BIN(Alg,__0),_/binary>>} -> + io_lib:format(" data decoded: ~s ... ~n", [Alg]); + + #ssh_msg_channel_request{request_type = "env", + data = <<?DEC_BIN(Var,__0),?DEC_BIN(Val,__1)>>} -> + io_lib:format(" data decoded: ~s = ~s~n", [Var, Val]); + + #ssh_msg_channel_request{request_type = "exec", + data = <<?DEC_BIN(Cmnd,__0)>>} -> + io_lib:format(" data decoded: ~s~n", [Cmnd]); + + #ssh_msg_channel_request{request_type = "pty-req", + data = <<?DEC_BIN(BTermName,_TermLen), + ?UINT32(Width),?UINT32(Height), + ?UINT32(PixWidth), ?UINT32(PixHeight), + Modes/binary>>} -> + io_lib:format(" data decoded: terminal = ~s~n" + " width x height = ~p x ~p~n" + " pix-width x pix-height = ~p x ~p~n" + " pty-opts = ~p~n", + [BTermName, Width,Height, PixWidth, PixHeight, + ssh_connection:decode_pty_opts(Modes)]); + _ -> + "" + end ]; ssh_dbg_format(raw_messages, {call,{?MODULE,decode,[BytesPT]}}) -> diff --git a/lib/ssh/src/ssh_sftpd.erl b/lib/ssh/src/ssh_sftpd.erl index 158ef3d5ae..1c53690ed0 100644 --- a/lib/ssh/src/ssh_sftpd.erl +++ b/lib/ssh/src/ssh_sftpd.erl @@ -32,6 +32,7 @@ -include("ssh_xfer.hrl"). -include("ssh_connect.hrl"). %% For ?DEFAULT_PACKET_SIZE and ?DEFAULT_WINDOW_SIZE + %%-------------------------------------------------------------------- %% External exports -export([subsystem_spec/1]). @@ -453,19 +454,19 @@ get_handle(Handles, BinHandle) -> %%% read_dir/5: read directory, send names, and return new state read_dir(State0 = #state{file_handler = FileMod, max_files = MaxLength, file_state = FS0}, - XF, ReqId, Handle, RelPath, {cache, Files}) -> + XF = #ssh_xfer{cm = _CM, channel = _Channel, vsn = Vsn}, ReqId, Handle, RelPath, {cache, Files}) -> AbsPath = relate_file_name(RelPath, State0), if length(Files) > MaxLength -> {ToSend, NewCache} = lists:split(MaxLength, Files), - {NamesAndAttrs, FS1} = get_attrs(AbsPath, ToSend, FileMod, FS0), + {NamesAndAttrs, FS1} = get_attrs(AbsPath, ToSend, FileMod, FS0, Vsn), ssh_xfer:xf_send_names(XF, ReqId, NamesAndAttrs), Handles = lists:keyreplace(Handle, 1, State0#state.handles, {Handle, directory, {RelPath,{cache, NewCache}}}), State0#state{handles = Handles, file_state = FS1}; true -> - {NamesAndAttrs, FS1} = get_attrs(AbsPath, Files, FileMod, FS0), + {NamesAndAttrs, FS1} = get_attrs(AbsPath, Files, FileMod, FS0, Vsn), ssh_xfer:xf_send_names(XF, ReqId, NamesAndAttrs), Handles = lists:keyreplace(Handle, 1, State0#state.handles, @@ -473,12 +474,12 @@ read_dir(State0 = #state{file_handler = FileMod, max_files = MaxLength, file_sta State0#state{handles = Handles, file_state = FS1} end; read_dir(State0 = #state{file_handler = FileMod, max_files = MaxLength, file_state = FS0}, - XF, ReqId, Handle, RelPath, _Status) -> + XF = #ssh_xfer{cm = _CM, channel = _Channel, vsn = Vsn}, ReqId, Handle, RelPath, _Status) -> AbsPath = relate_file_name(RelPath, State0), {Res, FS1} = FileMod:list_dir(AbsPath, FS0), case Res of {ok, Files} when MaxLength == 0 orelse MaxLength > length(Files) -> - {NamesAndAttrs, FS2} = get_attrs(AbsPath, Files, FileMod, FS1), + {NamesAndAttrs, FS2} = get_attrs(AbsPath, Files, FileMod, FS1, Vsn), ssh_xfer:xf_send_names(XF, ReqId, NamesAndAttrs), Handles = lists:keyreplace(Handle, 1, State0#state.handles, @@ -486,7 +487,7 @@ read_dir(State0 = #state{file_handler = FileMod, max_files = MaxLength, file_sta State0#state{handles = Handles, file_state = FS2}; {ok, Files} -> {ToSend, Cache} = lists:split(MaxLength, Files), - {NamesAndAttrs, FS2} = get_attrs(AbsPath, ToSend, FileMod, FS1), + {NamesAndAttrs, FS2} = get_attrs(AbsPath, ToSend, FileMod, FS1, Vsn), ssh_xfer:xf_send_names(XF, ReqId, NamesAndAttrs), Handles = lists:keyreplace(Handle, 1, State0#state.handles, @@ -497,21 +498,74 @@ read_dir(State0 = #state{file_handler = FileMod, max_files = MaxLength, file_sta send_status({error, Error}, ReqId, State1) end. +type_to_string(regular) -> "-"; +type_to_string(directory) -> "d"; +type_to_string(symlink) -> "s"; +type_to_string(device) -> "?"; +type_to_string(undefined) -> "?"; +type_to_string(other) -> "?". + +%% Converts a numeric mode to its human-readable representation +mode_to_string(Mode) -> + mode_to_string(Mode, "xwrxwrxwr", []). +mode_to_string(Mode, [C|T], Acc) when Mode band 1 =:= 1 -> + mode_to_string(Mode bsr 1, T, [C|Acc]); +mode_to_string(Mode, [_|T], Acc) -> + mode_to_string(Mode bsr 1, T, [$-|Acc]); +mode_to_string(_, [], Acc) -> + Acc. + +%% Converts a POSIX time to a readable string +time_to_string({{Y, Mon, Day}, {H, Min, _}}) -> + io_lib:format("~s ~2w ~s:~s ~w", [month(Mon), Day, two_d(H), two_d(Min), Y]). + +two_d(N) -> + tl(integer_to_list(N + 100)). + +month(1) -> "Jan"; +month(2) -> "Feb"; +month(3) -> "Mar"; +month(4) -> "Apr"; +month(5) -> "May"; +month(6) -> "Jun"; +month(7) -> "Jul"; +month(8) -> "Aug"; +month(9) -> "Sep"; +month(10) -> "Oct"; +month(11) -> "Nov"; +month(12) -> "Dec". + +longame({Name, Type, Size, Mtime, Mode, Uid, Gid}) -> + io_lib:format("~s~s ~4w/~-4w ~7w ~s ~s\n", + [type_to_string(Type), mode_to_string(Mode), + Uid, Gid, Size, time_to_string(Mtime), Name]). + +%%% get_long_name: get file longname (version 3) +%%% format output : -rwxr-xr-x 1 uid/gid 348911 Mar 25 14:29 t-filexfer +get_long_name(FileName, I) when is_record(I, file_info) -> + longame({FileName, I#file_info.type, I#file_info.size, I#file_info.mtime, + I#file_info.mode, I#file_info.uid, I#file_info.gid}). %%% get_attrs: get stat of each file and return -get_attrs(RelPath, Files, FileMod, FS) -> - get_attrs(RelPath, Files, FileMod, FS, []). +get_attrs(RelPath, Files, FileMod, FS, Vsn) -> + get_attrs(RelPath, Files, FileMod, FS, Vsn, []). -get_attrs(_RelPath, [], _FileMod, FS, Acc) -> +get_attrs(_RelPath, [], _FileMod, FS, _Vsn, Acc) -> {lists:reverse(Acc), FS}; -get_attrs(RelPath, [F | Rest], FileMod, FS0, Acc) -> +get_attrs(RelPath, [F | Rest], FileMod, FS0, Vsn, Acc) -> Path = filename:absname(F, RelPath), case FileMod:read_link_info(Path, FS0) of {{ok, Info}, FS1} -> + Name = if Vsn =< 3 -> + LongName = get_long_name(F, Info), + {F, LongName}; + true -> + F + end, Attrs = ssh_sftp:info_to_attr(Info), - get_attrs(RelPath, Rest, FileMod, FS1, [{F, Attrs} | Acc]); + get_attrs(RelPath, Rest, FileMod, FS1, Vsn, [{Name, Attrs} | Acc]); {{error, enoent}, FS1} -> - get_attrs(RelPath, Rest, FileMod, FS1, Acc); + get_attrs(RelPath, Rest, FileMod, FS1, Vsn, Acc); {Error, FS1} -> {Error, FS1} end. diff --git a/lib/ssh/src/ssh_xfer.erl b/lib/ssh/src/ssh_xfer.erl index e4d97b7393..d4fa0ccd9f 100644 --- a/lib/ssh/src/ssh_xfer.erl +++ b/lib/ssh/src/ssh_xfer.erl @@ -809,6 +809,15 @@ decode_names(Vsn, I, <<?UINT32(Len), FileName:Len/binary, encode_names(Vsn, NamesAndAttrs) -> lists:mapfoldl(fun(N, L) -> encode_name(Vsn, N, L) end, 0, NamesAndAttrs). +encode_name(Vsn, {{NameUC,LongNameUC},Attr}, Len) when Vsn =< 3 -> + Name = binary_to_list(unicode:characters_to_binary(NameUC)), + NLen = length(Name), + LongName = binary_to_list(unicode:characters_to_binary(LongNameUC)), + LNLen = length(LongName), + EncAttr = encode_ATTR(Vsn, Attr), + ALen = size(EncAttr), + NewLen = Len + NLen + LNLen + 4 + 4 + ALen, + {[<<?UINT32(NLen)>>, Name, <<?UINT32(LNLen)>>, LongName, EncAttr], NewLen}; encode_name(Vsn, {NameUC,Attr}, Len) when Vsn =< 3 -> Name = binary_to_list(unicode:characters_to_binary(NameUC)), NLen = length(Name), diff --git a/lib/ssh/test/property_test/ssh_eqc_client_server.erl b/lib/ssh/test/property_test/ssh_eqc_client_server.erl index 66a79c8a17..4bfc23d5ff 100644 --- a/lib/ssh/test/property_test/ssh_eqc_client_server.erl +++ b/lib/ssh/test/property_test/ssh_eqc_client_server.erl @@ -287,7 +287,7 @@ client_loop() -> client_loop() end. -do(Pid, Fun) -> do(Pid, Fun, 30?sec). +do(Pid, Fun) -> do(Pid, Fun, 60?sec). do(Pid, Fun, Timeout) when is_function(Fun,0) -> Pid ! {please_do,Fun,Ref=make_ref(),self()}, @@ -418,7 +418,7 @@ ssh_send(C=#chan{conn_ref=ConnectionRef, ref=ChannelRef, client_pid=Pid}, Type, ok -> receive {ssh_cm,ConnectionRef,{data,ChannelRef,Type,Answer}} -> Answer - after 15?sec -> + after 30?sec -> %% receive %% Other -> {error,{unexpected,Other}} %% after 0 -> diff --git a/lib/ssh/test/ssh_basic_SUITE.erl b/lib/ssh/test/ssh_basic_SUITE.erl index 5c6798bbcb..7c0d5fa9e5 100644 --- a/lib/ssh/test/ssh_basic_SUITE.erl +++ b/lib/ssh/test/ssh_basic_SUITE.erl @@ -430,43 +430,40 @@ exec_compressed(Config) when is_list(Config) -> end. %%-------------------------------------------------------------------- -%%% Idle timeout test, client -idle_time_client(Config) -> - SystemDir = filename:join(proplists:get_value(priv_dir, Config), system), - UserDir = proplists:get_value(priv_dir, Config), +%%% Idle timeout test +idle_time_client(Config) -> idle_time_common([], [{idle_time, 2000}], Config). - {Pid, Host, Port} = ssh_test_lib:daemon([{system_dir, SystemDir}, - {user_dir, UserDir}, - {failfun, fun ssh_test_lib:failfun/2}]), - ConnectionRef = - ssh_test_lib:connect(Host, Port, [{silently_accept_hosts, true}, - {user_dir, UserDir}, - {user_interaction, false}, - {idle_time, 2000}]), - {ok, Id} = ssh_connection:session_channel(ConnectionRef, 1000), - ssh_connection:close(ConnectionRef, Id), - receive - after 10000 -> - {error, closed} = ssh_connection:session_channel(ConnectionRef, 1000) - end, - ssh:stop_daemon(Pid). +idle_time_server(Config) -> idle_time_common([{idle_time, 2000}], [], Config). -%%-------------------------------------------------------------------- -%%% Idle timeout test, server -idle_time_server(Config) -> + +idle_time_common(DaemonExtraOpts, ClientExtraOpts, Config) -> SystemDir = filename:join(proplists:get_value(priv_dir, Config), system), UserDir = proplists:get_value(priv_dir, Config), {Pid, Host, Port} = ssh_test_lib:daemon([{system_dir, SystemDir}, {user_dir, UserDir}, - {idle_time, 2000}, - {failfun, fun ssh_test_lib:failfun/2}]), + {failfun, fun ssh_test_lib:failfun/2} + | DaemonExtraOpts + ]), ConnectionRef = ssh_test_lib:connect(Host, Port, [{silently_accept_hosts, true}, {user_dir, UserDir}, - {user_interaction, false}]), - {ok, Id} = ssh_connection:session_channel(ConnectionRef, 1000), - ssh_connection:close(ConnectionRef, Id), + {user_interaction, false} + | ClientExtraOpts + ]), + {ok, Id1} = ssh_sftp:start_channel(ConnectionRef), + {ok, Id2} = ssh_sftp:start_channel(ConnectionRef), + ssh_sftp:stop_channel(Id2), + timer:sleep(2500), + {ok, Id3} = ssh_sftp:start_channel(ConnectionRef), + ssh_sftp:stop_channel(Id1), + ssh_sftp:stop_channel(Id3), + timer:sleep(1000), + {ok, Id4} = ssh_sftp:start_channel(ConnectionRef), + timer:sleep(2500), + {ok, Id5} = ssh_sftp:start_channel(ConnectionRef), + ssh_sftp:stop_channel(Id4), + ssh_sftp:stop_channel(Id5), receive after 10000 -> {error, closed} = ssh_connection:session_channel(ConnectionRef, 1000) @@ -631,27 +628,7 @@ cli_exit_normal(Config) when is_list(Config) -> {ok, ChannelId} = ssh_connection:session_channel(ConnectionRef, infinity), ssh_connection:shell(ConnectionRef, ChannelId), - - receive - {ssh_cm, ConnectionRef,{eof, ChannelId}} -> - ok - after - 30000 -> ct:fail("timeout ~p:~p",[?MODULE,?LINE]) - end, - - receive - {ssh_cm, ConnectionRef,{exit_status,ChannelId,0}} -> - ok - after - 30000 -> ct:fail("timeout ~p:~p",[?MODULE,?LINE]) - end, - - receive - {ssh_cm, ConnectionRef,{closed, ChannelId}} -> - ok - after - 30000 -> ct:fail("timeout ~p:~p",[?MODULE,?LINE]) - end. + ssh_test_lib:receive_exec_end(ConnectionRef, ChannelId, _ExpectedExitStatus = 0). %%--------------------------------------------------------- %%% Test that SSH client receives user provided exit-status @@ -659,10 +636,13 @@ cli_exit_status(Config) when is_list(Config) -> process_flag(trap_exit, true), SystemDir = filename:join(proplists:get_value(priv_dir, Config), system), UserDir = proplists:get_value(priv_dir, Config), + NonZeroExitStatus = 7, {_Pid, Host, Port} = ssh_test_lib:daemon([{system_dir, SystemDir},{user_dir, UserDir}, {password, "morot"}, - {ssh_cli, {ssh_cli, [fun (_) -> spawn(fun () -> exit({exit_status, 7}) end) end]}}, + {ssh_cli, {ssh_cli, [fun (_) -> + spawn(fun () -> exit({exit_status, NonZeroExitStatus}) end) + end]}}, {subsystems, []}, {failfun, fun ssh_test_lib:failfun/2}]), ct:sleep(500), @@ -675,27 +655,7 @@ cli_exit_status(Config) when is_list(Config) -> {ok, ChannelId} = ssh_connection:session_channel(ConnectionRef, infinity), ssh_connection:shell(ConnectionRef, ChannelId), - - receive - {ssh_cm, ConnectionRef,{eof, ChannelId}} -> - ok - after - 30000 -> ct:fail("timeout ~p:~p",[?MODULE,?LINE]) - end, - - receive - {ssh_cm, ConnectionRef,{exit_status,ChannelId,7}} -> - ok - after - 30000 -> ct:fail("timeout ~p:~p",[?MODULE,?LINE]) - end, - - receive - {ssh_cm, ConnectionRef,{closed, ChannelId}} -> - ok - after - 30000 -> ct:fail("timeout ~p:~p",[?MODULE,?LINE]) - end. + ssh_test_lib:receive_exec_end(ConnectionRef, ChannelId, NonZeroExitStatus). %%-------------------------------------------------------------------- %%% Test that get correct error message if you try to start a daemon diff --git a/lib/ssh/test/ssh_connection_SUITE.erl b/lib/ssh/test/ssh_connection_SUITE.erl index 84f47803c4..49168d38bb 100644 --- a/lib/ssh/test/ssh_connection_SUITE.erl +++ b/lib/ssh/test/ssh_connection_SUITE.erl @@ -87,7 +87,8 @@ start_shell_sock_daemon_exec_multi/1, start_shell_sock_exec_fun/1, start_subsystem_on_closed_channel/1, - stop_listener/1 + stop_listener/1, + ssh_exec_echo/2 % called as an MFA ]). -define(SSH_DEFAULT_PORT, 22). @@ -1436,13 +1437,14 @@ test_shell_is_enabled(ConnectionRef, Expect) -> test_shell_is_enabled(ConnectionRef, Expect, PtyOpts) -> {ok, ChannelId} = ssh_connection:session_channel(ConnectionRef, infinity), - ok = ssh_connection:shell(ConnectionRef,ChannelId), case PtyOpts of [] -> no_alloc; _ -> success = ssh_connection:ptty_alloc(ConnectionRef, ChannelId, PtyOpts) end, + ok = ssh_connection:shell(ConnectionRef,ChannelId), + ExpSz = size(Expect), receive {ssh_cm,ConnectionRef, {data, ChannelId, 0, <<Expect:ExpSz/binary, _/binary>>}} -> diff --git a/lib/ssh/test/ssh_echo_server.erl b/lib/ssh/test/ssh_echo_server.erl index e039439f87..0e2519fc84 100644 --- a/lib/ssh/test/ssh_echo_server.erl +++ b/lib/ssh/test/ssh_echo_server.erl @@ -57,6 +57,7 @@ handle_ssh_msg({ssh_cm, CM, {data, ChannelId, 0, Data}}, #state{n = N} = State) case M > 0 of true -> ?DBG(State, "ssh_cm data Cid=~p size(Data)=~p M=~p",[ChannelId,size(Data),M]), + ssh_connection:adjust_window(CM, ChannelId, size(Data)), ssh_connection:send(CM, ChannelId, Data), {ok, State#state{n = M}}; false -> diff --git a/lib/ssh/test/ssh_options_SUITE.erl b/lib/ssh/test/ssh_options_SUITE.erl index 1dbfb0249b..09b1c7ccbb 100644 --- a/lib/ssh/test/ssh_options_SUITE.erl +++ b/lib/ssh/test/ssh_options_SUITE.erl @@ -1446,14 +1446,14 @@ try_to_connect(Connect, Host, Port, Pid, Tref, N) -> %%-------------------------------------------------------------------- max_sessions_drops_tcp_connects() -> - [{timetrap,{minutes,5}}]. + [{timetrap,{minutes,20}}]. max_sessions_drops_tcp_connects(Config) -> MaxSessions = 20, UseSessions = 2, % Must be =< MaxSessions FloodSessions = 1000, ParallelLogin = true, - NegTimeOut = 10*1000, + NegTimeOut = 8*1000, HelloTimeOut = 1*1000, %% Start a test daemon @@ -1470,8 +1470,8 @@ max_sessions_drops_tcp_connects(Config) -> {max_sessions, MaxSessions} ]), Host = ssh_test_lib:mangle_connect_address(Host0), - ct:log("~p Listen ~p:~p for max ~p sessions. Mangled Host = ~p", - [Pid,Host0,Port,MaxSessions,Host]), + ct:log("~p:~p ~p Listen ~p:~p for max ~p sessions. Mangled Host = ~p", + [?MODULE,?LINE,Pid,Host0,Port,MaxSessions,Host]), %% Log in UseSessions connections SSHconnect = fun(N) -> @@ -1482,7 +1482,7 @@ max_sessions_drops_tcp_connects(Config) -> {user, "carni"}, {password, "meat"} ]), - ct:log("~p: ssh:connect -> ~p", [N,R]), + ct:log("~p:~p ~p: ssh:connect -> ~p", [?MODULE,?LINE,N,R]), R end, @@ -1491,18 +1491,18 @@ max_sessions_drops_tcp_connects(Config) -> UseSessions -> %% As expected %% Try gen_tcp:connect - [ct:log("~p: gen_tcp:connect -> ~p", - [N, gen_tcp:connect(Host, Port, [])]) + [ct:log("~p:~p ~p: gen_tcp:connect -> ~p", + [?MODULE,?LINE, N, gen_tcp:connect(Host, Port, [])]) || N <- lists:seq(UseSessions+1, MaxSessions) ], - ct:log("Now try ~p gen_tcp:connect to be rejected", [FloodSessions]), - [ct:log("~p: gen_tcp:connect -> ~p", - [N, gen_tcp:connect(Host, Port, [])]) + ct:log("~p:~p Now try ~p gen_tcp:connect to be rejected", [?MODULE,?LINE,FloodSessions]), + [ct:log("~p:~p ~p: gen_tcp:connect -> ~p", + [?MODULE,?LINE, N, gen_tcp:connect(Host, Port, [])]) || N <- lists:seq(MaxSessions+1, MaxSessions+1+FloodSessions) ], - ct:log("try ~p ssh:connect", [MaxSessions - UseSessions]), + ct:log("~p:~p try ~p ssh:connect", [?MODULE,?LINE, MaxSessions - UseSessions]), try_ssh_connect(MaxSessions - UseSessions, NegTimeOut, SSHconnect); Len1 -> diff --git a/lib/ssh/test/ssh_test_lib.erl b/lib/ssh/test/ssh_test_lib.erl index a9591547dd..ae8f7abb70 100644 --- a/lib/ssh/test/ssh_test_lib.erl +++ b/lib/ssh/test/ssh_test_lib.erl @@ -55,6 +55,7 @@ rcv_lingering/1, receive_exec_result/1, receive_exec_result_or_fail/1, receive_exec_end/2, +receive_exec_end/3, receive_exec_result/3, failfun/2, hostname/0, @@ -443,10 +444,10 @@ receive_exec_result(Msgs) when is_list(Msgs) -> false -> case Msg of {ssh_cm,_,{data,_,1, Data}} -> - ct:log("~p:~p StdErr: ~p~n", [?MODULE,?FUNCTION_NAME,Data]), + ct:log("~p:~p unexpected StdErr: ~p~n~p~n", [?MODULE,?FUNCTION_NAME,Data,Msg]), receive_exec_result(Msgs); Other -> - ct:log("~p:~p Other ~p", [?MODULE,?FUNCTION_NAME,Other]), + ct:log("~p:~p unexpected Other ~p", [?MODULE,?FUNCTION_NAME,Other]), {unexpected_msg, Other} end end @@ -474,9 +475,12 @@ receive_exec_result_or_fail(Msg) -> end. receive_exec_end(ConnectionRef, ChannelId) -> + receive_exec_end(ConnectionRef, ChannelId, 0). + +receive_exec_end(ConnectionRef, ChannelId, ExitStatus) -> receive_exec_result( [{ssh_cm, ConnectionRef, {eof, ChannelId}}, - {optional, {ssh_cm, ConnectionRef, {exit_status, ChannelId, 0}}}, + {optional, {ssh_cm, ConnectionRef, {exit_status, ChannelId, ExitStatus}}}, {ssh_cm, ConnectionRef, {closed, ChannelId}} ]). diff --git a/lib/ssh/vsn.mk b/lib/ssh/vsn.mk index 4e74eeddde..2a4487a813 100644 --- a/lib/ssh/vsn.mk +++ b/lib/ssh/vsn.mk @@ -1,4 +1,4 @@ #-*-makefile-*- ; force emacs to enter makefile-mode -SSH_VSN = 4.10.8 +SSH_VSN = 4.11.1 APP_VSN = "ssh-$(SSH_VSN)" diff --git a/lib/ssl/doc/src/notes.xml b/lib/ssl/doc/src/notes.xml index e647bd1d94..7ee4b308e3 100644 --- a/lib/ssl/doc/src/notes.xml +++ b/lib/ssl/doc/src/notes.xml @@ -27,6 +27,73 @@ </header> <p>This document describes the changes made to the SSL application.</p> +<section><title>SSL 10.3</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Fix CRL handling that previously could fail to find the + issuer cert under some circumstances.</p> + <p> + Own Id: OTP-17261 Aux Id: GH-4589 </p> + </item> + <item> + <p> + TLS-1.3 client could, under some circumstances, select an + incorrect algorithm to sign the certificate verification + message causing a TLS Decrypt Alert being issued by the + server.</p> + <p> + Own Id: OTP-17281 Aux Id: GH-4620 </p> + </item> + <item> + <p> + Correct handling of default values for emulated socket + options and retain the order of the ssl options list to + ensure backwards compatible behavior if options should be + set more than once.</p> + <p> + Own Id: OTP-17282</p> + </item> + </list> + </section> + + + <section><title>Improvements and New Features</title> + <list> + <item> + <p> + Enhance pre TLS-1.3 session handling so the client and + server side handling is completely separated and client + disregards oldest session when reaching max limit of the + session table.</p> + <p> + Own Id: OTP-16876</p> + </item> + <item> + <p> + This change implements the early data feature for TLS 1.3 + clients.</p> + <p> + TLS 1.3 allows clients to send data in the first flight + using a Pre-Shared Key to authenticate the server and to + encrypt the early data.</p> + <p> + Own Id: OTP-16985</p> + </item> + <item> + <p> + This change implements the early data feature for TLS 1.3 + servers.</p> + <p> + Own Id: OTP-17042</p> + </item> + </list> + </section> + +</section> + <section><title>SSL 10.2.4.1</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/ssl/doc/src/ssl.xml b/lib/ssl/doc/src/ssl.xml index 5bd27e1d9a..c3dd3f4df0 100644 --- a/lib/ssl/doc/src/ssl.xml +++ b/lib/ssl/doc/src/ssl.xml @@ -1043,6 +1043,21 @@ fun(srp, Username :: binary(), UserState :: term()) -> </desc> </datatype> + <datatype> + <name name="client_early_data"/> + <desc> + <p>Configures the early data to be sent by the client.</p> + <p>In order to be able to verify + that the server has the intention to process the early data, the following 3-tuple is + sent to the user process:</p> + <p><c>{ssl, SslSocket, {early_data, Result}}</c></p> + <p>where <c>Result</c> is either <c>accepted</c> or <c>rejected</c>.</p> + <warning> + <p>It is the responsibility of the user to handle a rejected Early Data and + to resend when it is appropriate.</p></warning> + </desc> + </datatype> + <!-- <datatype> --> <!-- <name name="ocsp_stapling"/> --> <!-- <desc><p>If true, OCSP stapling will be enabled, an extension of type "status_request" will be --> @@ -1335,6 +1350,17 @@ fun(srp, Username :: binary(), UserState :: term()) -> </datatype> <datatype> + <name name="server_early_data"/> + <desc> + <p>Configures if the server accepts (<c>enabled</c>) or rejects (<c>rejects</c>) early + data sent by a client. The default value is <c>disabled</c>. + </p> + <warning><p>This option is a placeholder, early data is not yet implemented on the server side. + </p></warning> + </desc> + </datatype> + + <datatype> <name name="connection_info"/> </datatype> @@ -1403,7 +1429,7 @@ fun(srp, Username :: binary(), UserState :: term()) -> <desc><p>Lists all possible cipher suites corresponding to <c>Description</c> that are available. The <c>exclusive</c> option will exclusively list cipher suites - introduced in <c>Version</c> whereas the the other options + introduced in <c>Version</c> whereas the other options are inclusive from the lowest possible version to <c>Version</c>. The <c>all</c> options includes all suites except the anonymous. diff --git a/lib/ssl/doc/src/ssl_app.xml b/lib/ssl/doc/src/ssl_app.xml index f5905ecbab..0f09d873f9 100644 --- a/lib/ssl/doc/src/ssl_app.xml +++ b/lib/ssl/doc/src/ssl_app.xml @@ -65,14 +65,16 @@ <p><c>erl -ssl protocol_version "['tlsv1.2', 'tlsv1.1']"</c></p> <taglist> - <tag><c>protocol_version = </c><seetype marker="ssl#protocol">ssl:ssl_tls_protocol()</seetype><c><![CDATA[<optional>]]></c></tag> + <tag><c>protocol_version = </c><seetype marker="ssl#tls_version">ssl:tls_version()</seetype> | + [<seetype marker="ssl#tls_version">ssl:tls_version()</seetype>] <c><![CDATA[<optional>]]></c></tag> <item><p>Protocol supported by started clients and servers. If this option is not set, it defaults to all TLS protocols currently supported, more might be configurable, by the SSL application. This option can be overridden by the version option to <c>ssl:connect/[2,3]</c> and <c>ssl:listen/2</c>.</p></item> - <tag><c>dtls_protocol_version = </c><seetype marker="ssl#protocol">ssl:dtls_protocol()</seetype><c><![CDATA[<optional>]]></c></tag> + <tag><c>dtls_protocol_version = </c><seetype marker="ssl#dtls_version">ssl:dtls_version()</seetype> | + [<seetype marker="ssl#dtls_version">ssl:dtls_version()</seetype>] <c><![CDATA[<optional>]]></c></tag> <item><p>Protocol supported by started clients and servers. If this option is not set, it defaults to all DTLS protocols currently supported, more might be configurable, by the SSL application. @@ -86,15 +88,37 @@ </p></item> <tag><c><![CDATA[session_cb = atom() <optional>]]></c></tag> - <item><p>Name of the session cache callback module that implements + <item><p> Deprecated Since OTP-23.3 replaced by <c>client_session_cb</c> + and <c>server_session_cb</c> + </p></item> + + <tag><c><![CDATA[client_session_cb = atom() <optional>]]></c></tag> + <item><p> Since OTP-23.3 Name client of the session cache callback module that implements + the <c>ssl_session_cache_api</c> behavior. Defaults to + <c>ssl_client_session_cache_db</c>.</p></item> + + + <tag><c><![CDATA[server_session_cb = atom() <optional>]]></c></tag> + <item><p>Since OTP-23.3 Name of the server session cache callback module that implements the <c>ssl_session_cache_api</c> behavior. Defaults to - <c>ssl_session_cache</c>.</p></item> + <c>ssl_server_session_cache_db</c>.</p></item> <tag><c><![CDATA[session_cb_init_args = proplist:proplist() <optional>]]></c></tag> + <item><p>Deprecated Since OTP-23.3 replaced by <c>client_session_cb_init_args</c> + and <c>server_session_cb_init_args</c></p></item> + + <tag><c><![CDATA[client_session_cb_init_args = proplist:proplist() <optional>]]></c></tag> + <item><p>List of extra user-defined arguments to the <c>init</c> function in the session cache callback module. Defaults to <c>[]</c>.</p></item> + <tag><c><![CDATA[server_session_cb_init_args = proplist:proplist() <optional>]]></c></tag> + + <item><p>List of extra user-defined arguments to the <c>init</c> function + in the session cache callback module. Defaults to <c>[]</c>.</p></item> + + <tag><c><![CDATA[session_cache_client_max = integer() <optional>]]></c><br/></tag> <item><p>Limits the growth of the clients session cache, that is how many sessions towards servers that are cached to be used by @@ -177,6 +201,16 @@ </p> </item> + <tag><c><![CDATA[server_session_ticket_max_early_data = integer() <optional>]]></c></tag> + <item> + <p> + Sets the maximum size of the early data that the server accepts and also configures + its NewSessionTicket messages to include this same size limit in their + early_data_indication extension. + Defaults to 16384. Size limit is enforced by both client and server. + </p> + </item> + <tag><c><![CDATA[client_session_ticket_lifetime = integer() <optional>]]></c></tag> <item> <p> diff --git a/lib/ssl/doc/src/ssl_crl_cache_api.xml b/lib/ssl/doc/src/ssl_crl_cache_api.xml index 3f127676ce..69f1b3f4ae 100644 --- a/lib/ssl/doc/src/ssl_crl_cache_api.xml +++ b/lib/ssl/doc/src/ssl_crl_cache_api.xml @@ -77,8 +77,8 @@ <funcs> <func> - <name since="@maint@">fresh_crl(DistributionPoint, CRL) -> FreshCRL </name> - <name since="OTP 18.0">fresh_crl(DistributionPoint, CRL) -> FreshCRL | {LoggerInfo, FreshCRL}</name> + <name since="OTP 22.2">Module:fresh_crl(DistributionPoint, CRL) -> FreshCRL </name> + <name since="OTP 18.0">Module:fresh_crl(DistributionPoint, CRL) -> FreshCRL | {LoggerInfo, FreshCRL}</name> <fsummary> <c>fun fresh_crl/2 </c> will be used as input option <c>update_crl</c> to public_key:pkix_crls_validate/3 </fsummary> <type> @@ -100,10 +100,10 @@ </func> <func> - <name since="@maint@">lookup(DistributionPoint, Issuer, DbHandle) -> not_available | CRLs | + <name since="OTP 22.2">Module:lookup(DistributionPoint, Issuer, DbHandle) -> not_available | CRLs | {LoggerInfo, CRLs} </name> - <name since="OTP 19.0">lookup(DistributionPoint, Issuer, DbHandle) -> not_available | CRLs </name> - <name since="OTP 18.0">lookup(DistributionPoint, DbHandle) -> not_available | CRLs </name> + <name since="OTP 19.0">Module:lookup(DistributionPoint, Issuer, DbHandle) -> not_available | CRLs </name> + <name since="OTP 18.0">Module:lookup(DistributionPoint, DbHandle) -> not_available | CRLs </name> <fsummary> </fsummary> <type> <v> DistributionPoint = <seetype marker="#dist_point"> dist_point() </seetype> </v> @@ -137,8 +137,8 @@ </func> <func> - <name since="@maint@">select(Issuer, DbHandle) -> CRLs | {LoggerInfo, CRLs} </name> - <name since="OTP 18.0">select(Issuer, DbHandle) -> CRLs </name> + <name since="OTP 22.2">Module:select(Issuer, DbHandle) -> CRLs | {LoggerInfo, CRLs} </name> + <name since="OTP 18.0">Module:select(Issuer, DbHandle) -> CRLs </name> <fsummary>Select the CRLs in the cache that are issued by <c>Issuer</c></fsummary> <type> <v> Issuer = <seetype diff --git a/lib/ssl/doc/src/ssl_session_cache_api.xml b/lib/ssl/doc/src/ssl_session_cache_api.xml index 86fdc34fd8..3067d89b13 100644 --- a/lib/ssl/doc/src/ssl_session_cache_api.xml +++ b/lib/ssl/doc/src/ssl_session_cache_api.xml @@ -33,7 +33,7 @@ <description> <p> - Defines the API for the TLS session cache so + Defines the API for the TLS session cache (pre TLS-1.3) so that the data storage scheme can be replaced by defining a new callback module implementing this API. </p> @@ -77,7 +77,7 @@ <funcs> <func> - <name since="OTP R14B">delete(Cache, Key) -> _</name> + <name since="OTP R14B">Module:delete(Cache, Key) -> _</name> <fsummary>Deletes a cache entry.</fsummary> <type> <v>Cache = <seetype marker="#session_cache_ref"> session_cache_ref() </seetype></v> @@ -91,7 +91,7 @@ </func> <func> - <name since="OTP R14B">foldl(Fun, Acc0, Cache) -> Acc</name> + <name since="OTP R14B">Module:foldl(Fun, Acc0, Cache) -> Acc</name> <fsummary></fsummary> <type> <v>Fun = fun()</v> @@ -105,11 +105,16 @@ The function returns the final value of the accumulator. <c>Acc0</c> is returned if the cache is empty. </p> + + <note><p>Since OTP-23.3 this functions is only used on the client side + and does not need to implemented for a server cache. + </p></note> + </desc> </func> <func> - <name since="OTP 18.0">init(Args) -> Cache </name> + <name since="OTP 18.0">Module:init(Args) -> Cache </name> <fsummary>Returns cache reference.</fsummary> <type> <v>Cache = <seetype marker="#session_cache_ref"> session_cache_ref() </seetype></v> @@ -135,7 +140,7 @@ </func> <func> - <name since="OTP R14B">lookup(Cache, Key) -> Entry</name> + <name since="OTP R14B">Module:lookup(Cache, Key) -> Entry</name> <fsummary>Looks up a cache entry.</fsummary> <type> <v>Cache = <seetype marker="#session_cache_ref"> session_cache_ref() </seetype></v> @@ -150,7 +155,7 @@ </func> <func> - <name since="OTP R14B">select_session(Cache, PartialKey) -> [Session]</name> + <name since="OTP R14B">Module:select_session(Cache, PartialKey) -> [Session]</name> <fsummary>Selects sessions that can be reused.</fsummary> <type> <v>Cache = <seetype marker="#session_cache_ref"> session_cache_ref() </seetype></v> @@ -158,14 +163,17 @@ <v>Session = <seetype marker="#session">session()</seetype></v> </type> <desc> - <p>Selects sessions that can be reused. Is to be callable - from any process. - </p> + <p>Selects sessions that can be reused, that is sessions that + include <c>PartialKey</c> in its key. Is to be callable from + any process.</p> + <note><p>Since OTP-23.3 This functions is only used on the client side + and does not need to implemented for a server cache. + </p></note> </desc> </func> <func> - <name since="OTP 19.3">size(Cache) -> integer()</name> + <name since="OTP 19.3">Module:size(Cache) -> integer()</name> <fsummary>Returns the number of sessions in the cache.</fsummary> <type> <v>Cache = <seetype marker="#session_cache_ref"> session_cache_ref() </seetype></v> @@ -180,7 +188,7 @@ </func> <func> - <name since="OTP R14B">terminate(Cache) -> _</name> + <name since="OTP R14B">Module:terminate(Cache) -> _</name> <fsummary>Called by the process that handles the cache when it is about to terminate.</fsummary> <type> @@ -195,7 +203,7 @@ </func> <func> - <name since="OTP R14B">update(Cache, Key, Session) -> _</name> + <name since="OTP R14B">Module:update(Cache, Key, Session) -> _</name> <fsummary>Caches a new session or updates an already cached one.</fsummary> <type> <v>Cache = <seetype marker="#session_cache_ref"> session_cache_ref() </seetype></v> diff --git a/lib/ssl/doc/src/standards_compliance.xml b/lib/ssl/doc/src/standards_compliance.xml index 6cf25d726f..b2e7d17c45 100644 --- a/lib/ssl/doc/src/standards_compliance.xml +++ b/lib/ssl/doc/src/standards_compliance.xml @@ -140,7 +140,7 @@ <list type="bulleted"> <item>PSK and session resumption is supported (stateful and stateless tickets)</item> <item>Anti-replay protection using Bloom-filters with stateless tickets</item> - <item>Early data and 0-RTT not supported</item> + <item>Early data and 0-RTT is supported</item> <item>Key and Initialization Vector Update is supported</item> </list> <p>For more detailed information see the @@ -251,8 +251,8 @@ </url> </cell> <cell align="left" valign="middle"></cell> - <cell align="left" valign="middle"><em>NC</em></cell> - <cell align="left" valign="middle"></cell> + <cell align="left" valign="middle"><em>PC</em></cell> + <cell align="left" valign="middle"><em>23.3</em></cell> </row> <row> @@ -387,8 +387,8 @@ <row> <cell align="left" valign="middle"></cell> <cell align="left" valign="middle">early_data (RFC8446)</cell> - <cell align="left" valign="middle"><em>NC</em></cell> - <cell align="left" valign="middle"></cell> + <cell align="left" valign="middle"><em>C</em></cell> + <cell align="left" valign="middle"><em>23.3</em></cell> </row> <row> <cell align="left" valign="middle"></cell> @@ -526,8 +526,8 @@ <row> <cell align="left" valign="middle"></cell> <cell align="left" valign="middle">early_data (RFC8446)</cell> - <cell align="left" valign="middle"><em>NC</em></cell> - <cell align="left" valign="middle"></cell> + <cell align="left" valign="middle"><em>C</em></cell> + <cell align="left" valign="middle"><em>23.3</em></cell> </row> <row> <cell align="left" valign="middle"></cell> @@ -1135,14 +1135,14 @@ </url> </cell> <cell align="left" valign="middle"><em>Client</em></cell> - <cell align="left" valign="middle"><em>NC</em></cell> - <cell align="left" valign="middle"></cell> + <cell align="left" valign="middle"><em>C</em></cell> + <cell align="left" valign="middle"><em>23.3</em></cell> </row> <row> <cell align="left" valign="middle"></cell> <cell align="left" valign="middle"><em>Server</em></cell> - <cell align="left" valign="middle"><em>NC</em></cell> - <cell align="left" valign="middle"><em></em></cell> + <cell align="left" valign="middle"><em>C</em></cell> + <cell align="left" valign="middle"><em>23.3</em></cell> </row> <row> @@ -1274,8 +1274,8 @@ <row> <cell align="left" valign="middle"></cell> <cell align="left" valign="middle">early_data (RFC8446)</cell> - <cell align="left" valign="middle"><em>NC</em></cell> - <cell align="left" valign="middle"></cell> + <cell align="left" valign="middle"><em>C</em></cell> + <cell align="left" valign="middle"><em>23.3</em></cell> </row> <row> <cell align="left" valign="middle"></cell> @@ -1341,8 +1341,8 @@ <row> <cell align="left" valign="middle"></cell> <cell align="left" valign="middle">early_data (RFC8446)</cell> - <cell align="left" valign="middle"><em>NC</em></cell> - <cell align="left" valign="middle"></cell> + <cell align="left" valign="middle"><em>C</em></cell> + <cell align="left" valign="middle"><em>23.3</em></cell> </row> <row> <cell align="left" valign="middle"></cell> @@ -1667,14 +1667,14 @@ </url> </cell> <cell align="left" valign="middle"><em>Client</em></cell> - <cell align="left" valign="middle"><em>NC</em></cell> - <cell align="left" valign="middle"></cell> + <cell align="left" valign="middle"><em>C</em></cell> + <cell align="left" valign="middle"><em>23.3</em></cell> </row> <row> <cell align="left" valign="middle"></cell> <cell align="left" valign="middle"><em>Server</em></cell> - <cell align="left" valign="middle"><em>NC</em></cell> - <cell align="left" valign="middle"><em></em></cell> + <cell align="left" valign="middle"><em>C</em></cell> + <cell align="left" valign="middle"><em>23.3</em></cell> </row> <row> @@ -1684,27 +1684,27 @@ </url> </cell> <cell align="left" valign="middle"><em>Client</em></cell> - <cell align="left" valign="middle"><em>PC</em></cell> - <cell align="left" valign="middle"><em>22.2</em></cell> + <cell align="left" valign="middle"><em>C</em></cell> + <cell align="left" valign="middle"><em>23.3</em></cell> </row> <row> <cell align="left" valign="middle"></cell> <cell align="left" valign="middle">early_data (RFC8446)</cell> - <cell align="left" valign="middle"><em>NC</em></cell> - <cell align="left" valign="middle"></cell> + <cell align="left" valign="middle"><em>C</em></cell> + <cell align="left" valign="middle"><em>23.3</em></cell> </row> <row> <cell align="left" valign="middle"></cell> <cell align="left" valign="middle"><em>Server</em></cell> - <cell align="left" valign="middle"><em>PC</em></cell> - <cell align="left" valign="middle"><em>22.2</em></cell> + <cell align="left" valign="middle"><em>C</em></cell> + <cell align="left" valign="middle"><em>23.3</em></cell> </row> <row> <cell align="left" valign="middle"></cell> <cell align="left" valign="middle">early_data (RFC8446)</cell> - <cell align="left" valign="middle"><em>NC</em></cell> - <cell align="left" valign="middle"></cell> + <cell align="left" valign="middle"><em>C</em></cell> + <cell align="left" valign="middle"><em>23.3</em></cell> </row> <row> diff --git a/lib/ssl/doc/src/using_ssl.xml b/lib/ssl/doc/src/using_ssl.xml index 4a66bf9d90..7f45b72db9 100644 --- a/lib/ssl/doc/src/using_ssl.xml +++ b/lib/ssl/doc/src/using_ssl.xml @@ -559,6 +559,120 @@ ok </section> <section> + <title>Early Data in TLS 1.3</title> + <p>TLS 1.3 allows clients to send data on the first flight if the endpoints have + a shared crypographic secret (pre-shared key). This means that clients can send + early data if they have a valid session ticket received in a previous + successful handshake. For more information about session resumption see + <seeguide marker="ssl:using_ssl#session-tickets-and-session-resumption-in-tls-1.3"> + Session Tickets and Session Resumption in TLS 1.3</seeguide>. + </p> + <p>The security properties of Early Data are weaker than other kinds of TLS data. + This data is not forward secret, and it is vulnerable to replay attacks. For available + mitigation strategies see + <seeguide marker="ssl:using_ssl#anti-replay-protection-in-tls-1.3"> + Anti-Replay Protection in TLS 1.3</seeguide>.</p> + <p>In normal operation, clients will not know which, if any, of the available mitigation + strategies servers actually implement, and hence must only send early data which + they deem safe to be replayed. For example, idempotent HTTP operations, such as HEAD and + GET, can usually be regarded as safe but even they can be exploited by a large number of + replays causing resource limit exhaustion and other similar problems.</p> + <p>An example of sending early data with automatic and manual session ticket handling:</p> + <warning> + <p>The Early Data feature is experimental in this version of OTP. + </p> + </warning> + + <p><em>Server (with NSS key logging)</em></p> + <code type="none"> + early_data_server() -> + application:load(ssl), + {ok, _} = application:ensure_all_started(ssl), + Port = 11029, + LOpts = [{certfile, ?SERVER_CERT}, + {keyfile, ?SERVER_KEY}, + {reuseaddr, true}, + {versions, ['tlsv1.2','tlsv1.3']}, + {session_tickets, stateless}, + {early_data, enabled}, + {keep_secrets, true} %% Enable NSS key log (debug option) + ], + {ok, LSock} = ssl:listen(Port, LOpts), + %% Accept first connection + {ok, CSock0} = ssl:transport_accept(LSock), + {ok, _} = ssl:handshake(CSock0), + %% Accept second connection + {ok, CSock1} = ssl:transport_accept(LSock), + {ok, Sock} = ssl:handshake(CSock1), + Sock. + </code> + <p><em>Exporting the secrets (optional)</em></p> + <code type="none"> + {ok, [{keylog, KeylogItems}]} = ssl:connection_information(Sock, [keylog]). + file:write_file("key.log", [[KeylogItem,$\n] || KeylogItem <- KeylogItems]). + </code> + <p><em>Client (automatic ticket handling):</em></p> + <code type="erl"> + early_data_auto() -> + %% First handshake 1-RTT - get session tickets + application:load(ssl), + {ok, _} = application:ensure_all_started(ssl), + Port = 11029, + Data = <<"HEAD / HTTP/1.1\r\nHost: \r\nConnection: close\r\n">>, + COpts0 = [{cacertfile, ?CA_CERT}, + {versions, ['tlsv1.2', 'tlsv1.3']}, + {session_tickets, auto}], + {ok, Sock0} = ssl:connect("localhost", Port, COpts0), + + %% Wait for session tickets + timer:sleep(500), + %% Close socket if server cannot handle multiple connections e.g. openssl s_server + ssl:close(Sock0), + + %% Second handshake 0-RTT + COpts1 = [{cacertfile, ?CA_CERT}, + {versions, ['tlsv1.2', 'tlsv1.3']}, + {session_tickets, auto}, + {early_data, Data}], + {ok, Sock} = ssl:connect("localhost", Port, COpts1), + Sock. + </code> + <p><em>Client (manual ticket handling):</em></p> + <code type="erl"> + early_data_manual() -> + %% First handshake 1-RTT - get session tickets + application:load(ssl), + {ok, _} = application:ensure_all_started(ssl), + Port = 11029, + Data = <<"HEAD / HTTP/1.1\r\nHost: \r\nConnection: close\r\n">>, + COpts0 = [{cacertfile, ?CA_CERT}, + {versions, ['tlsv1.2', 'tlsv1.3']}, + {session_tickets, manual}], + {ok, Sock0} = ssl:connect("localhost", Port, COpts0), + + %% Wait for session tickets + Ticket = + receive + {ssl, session_ticket, Ticket0} -> + Ticket0 + end, + + %% Close socket if server cannot handle multiple connections + %% e.g. openssl s_server + ssl:close(Sock0), + + %% Second handshake 0-RTT + COpts1 = [{cacertfile, ?CA_CERT}, + {versions, ['tlsv1.2', 'tlsv1.3']}, + {session_tickets, manual}, + {use_ticket, [Ticket]}, + {early_data, Data}], + {ok, Sock} = ssl:connect("localhost", Port, COpts1), + Sock. + </code> + </section> + + <section> <title>Anti-Replay Protection in TLS 1.3</title> <p>The TLS 1.3 protocol does not provide inherent protection for replay of 0-RTT data but diff --git a/lib/ssl/src/Makefile b/lib/ssl/src/Makefile index 1a55ee7b83..5edd6cb4b9 100644 --- a/lib/ssl/src/Makefile +++ b/lib/ssl/src/Makefile @@ -65,6 +65,7 @@ MODULES= \ ssl_certificate \ ssl_cipher \ ssl_cipher_format \ + ssl_client_session_cache_db \ ssl_config \ ssl_connection_sup \ ssl_crl \ @@ -87,7 +88,6 @@ MODULES= \ ssl_server_session_cache_sup \ ssl_upgrade_server_session_cache_sup \ ssl_session \ - ssl_session_cache \ ssl_srp_primes \ ssl_sup \ tls_bloom_filter \ diff --git a/lib/ssl/src/dtls_connection.erl b/lib/ssl/src/dtls_connection.erl index fb389dcb4d..78348826e4 100644 --- a/lib/ssl/src/dtls_connection.erl +++ b/lib/ssl/src/dtls_connection.erl @@ -583,19 +583,8 @@ initial_state(Role, Host, Port, Socket, {CbModule, DataTag, CloseTag, ErrorTag, PassiveTag}) -> #{beast_mitigation := BeastMitigation} = SSLOptions, ConnectionStates = dtls_record:init_connection_states(Role, BeastMitigation), - - SessionCacheCb = case application:get_env(ssl, session_cb) of - {ok, Cb} when is_atom(Cb) -> - Cb; - _ -> - ssl_session_cache - end, - InternalActiveN = case application:get_env(ssl, internal_active_n) of - {ok, N} when is_integer(N) -> - N; - _ -> - ?INTERNAL_ACTIVE_N - end, + #{session_cb := SessionCacheCb} = ssl_config:pre_1_3_session_opts(Role), + InternalActiveN = ssl_config:get_internal_active_n(), Monitor = erlang:monitor(process, User), InitStatEnv = #static_env{ role = Role, diff --git a/lib/ssl/src/dtls_server_session_cache_sup.erl b/lib/ssl/src/dtls_server_session_cache_sup.erl index 457eb90167..65fbb34918 100644 --- a/lib/ssl/src/dtls_server_session_cache_sup.erl +++ b/lib/ssl/src/dtls_server_session_cache_sup.erl @@ -42,7 +42,7 @@ start_link() -> supervisor:start_link({local, ?MODULE}, ?MODULE, []). start_child(Listener) -> - supervisor:start_child(?MODULE, [Listener | ssl_config:pre_1_3_session_opts()]). + supervisor:start_child(?MODULE, [Listener | [ssl_config:pre_1_3_session_opts(server)]]). %%%========================================================================= %%% Supervisor callback diff --git a/lib/ssl/src/ssl.app.src b/lib/ssl/src/ssl.app.src index f3cc463cff..78bbcc8c04 100644 --- a/lib/ssl/src/ssl.app.src +++ b/lib/ssl/src/ssl.app.src @@ -58,7 +58,7 @@ tls_dist_server_sup, %% SSL/TLS session and cert handling ssl_session, - ssl_session_cache, + ssl_client_session_cache_db, ssl_server_session_cache, ssl_server_session_cache_db, ssl_server_session_cache_sup, diff --git a/lib/ssl/src/ssl.erl b/lib/ssl/src/ssl.erl index 5e10e254c4..f4f8f7cc9d 100644 --- a/lib/ssl/src/ssl.erl +++ b/lib/ssl/src/ssl.erl @@ -384,6 +384,8 @@ bloom_filter_bits()}. %% m - number of bits in bit vector -type use_ticket() :: [binary()]. -type middlebox_comp_mode() :: boolean(). +-type client_early_data() :: binary(). +-type server_early_data() :: disabled | enabled. %% ------------------------------------------------------------------------------------------------------- @@ -402,7 +404,8 @@ {signature_algs, client_signature_algs()} | {fallback, fallback()} | {session_tickets, client_session_tickets()} | - {use_ticket, use_ticket()}. %% | + {use_ticket, use_ticket()} | + {early_data, client_early_data()}. %% {ocsp_stapling, ocsp_stapling()} | %% {ocsp_responder_certs, ocsp_responder_certs()} | %% {ocsp_nonce, ocsp_nonce()}. @@ -453,7 +456,8 @@ {signature_algs, server_signature_algs()} | {session_tickets, server_session_tickets()} | {anti_replay, anti_replay()} | - {cookie, cookie()}. + {cookie, cookie()} | + {early_data, server_early_data()}. -type server_cacerts() :: [public_key:der_encoded()]. -type server_cafile() :: file:filename(). @@ -1699,6 +1703,32 @@ handle_option(client_renegotiation = Option, Value0, ['tlsv1','tlsv1.1','tlsv1.2']), Value = validate_option(Option, Value0), OptionsMap#{Option => Value}; +handle_option(early_data = Option, unbound, OptionsMap, #{rules := Rules}) -> + Value = validate_option(Option, default_value(Option, Rules)), + OptionsMap#{Option => Value}; +handle_option(early_data = Option, Value0, #{session_tickets := SessionTickets, + versions := Versions} = OptionsMap, + #{role := server = Role}) -> + assert_option_dependency(Option, versions, Versions, ['tlsv1.3']), + assert_option_dependency(Option, session_tickets, [SessionTickets], + [stateful, stateless]), + Value = validate_option(Option, Value0, Role), + OptionsMap#{Option => Value}; +handle_option(early_data = Option, Value0, #{session_tickets := SessionTickets, + use_ticket := UseTicket, + versions := Versions} = OptionsMap, + #{role := client = Role}) -> + assert_option_dependency(Option, versions, Versions, ['tlsv1.3']), + assert_option_dependency(Option, session_tickets, [SessionTickets], + [manual, auto]), + case UseTicket of + undefined when SessionTickets =/= auto -> + throw({error, {options, dependency, {Option, use_ticket}}}); + _ -> + ok + end, + Value = validate_option(Option, Value0, Role), + OptionsMap#{Option => Value}; handle_option(eccs = Option, unbound, #{versions := [HighestVersion|_]} = OptionsMap, #{rules := _Rules}) -> Value = handle_eccs_option(eccs(), HighestVersion), OptionsMap#{Option => Value}; @@ -1820,13 +1850,13 @@ handle_option(server_name_indication = Option, unbound, OptionsMap, #{host := Ho handle_option(server_name_indication = Option, Value0, OptionsMap, _Env) -> Value = validate_option(Option, Value0), OptionsMap#{Option => Value}; -handle_option(session_tickets = Option, unbound, OptionsMap, #{rules := Rules}) -> - Value = validate_option(Option, default_value(Option, Rules)), +handle_option(session_tickets = Option, unbound, OptionsMap, #{role := Role, + rules := Rules}) -> + Value = validate_option(Option, default_value(Option, Rules), Role), OptionsMap#{Option => Value}; handle_option(session_tickets = Option, Value0, #{versions := Versions} = OptionsMap, #{role := Role}) -> assert_option_dependency(Option, versions, Versions, ['tlsv1.3']), - assert_role_value(Role, Option, Value0, [disabled, stateful, stateless], [disabled, manual, auto]), - Value = validate_option(Option, Value0), + Value = validate_option(Option, Value0, Role), OptionsMap#{Option => Value}; handle_option(signature_algs = Option, unbound, #{versions := [HighestVersion|_]} = OptionsMap, #{role := Role}) -> Value = @@ -2036,24 +2066,6 @@ assert_role(server_only, _, _, undefined) -> assert_role(Type, _, Key, _) -> throw({error, {option, Type, Key}}). - -assert_role_value(client, Option, Value, _, ClientValues) -> - case lists:member(Value, ClientValues) of - true -> - ok; - false -> - %% throw({error, {option, client, Option, Value, ClientValues}}) - throw({error, {options, role, {Option, {Value, {client, ClientValues}}}}}) - end; -assert_role_value(server, Option, Value, ServerValues, _) -> - case lists:member(Value, ServerValues) of - true -> - ok; - false -> - %% throw({error, {option, server, Option, Value, ServerValues}}) - throw({error, {options, role, {Option, {Value, {server, ServerValues}}}}}) - end. - assert_option_dependency(Option, OptionDep, Values0, AllowedValues) -> case is_dtls_configured(Values0) of true -> @@ -2088,168 +2100,198 @@ is_dtls_configured(Versions) -> end, lists:any(Fun, Versions). -validate_option(versions, Versions) -> - validate_versions(Versions, Versions); -validate_option(verify, Value) - when Value == verify_none; Value == verify_peer -> +validate_option(Option, Value) -> + validate_option(Option, Value, undefined). +%% +validate_option(Opt, Value, _) + when Opt =:= alpn_advertised_protocols orelse + Opt =:= alpn_preferred_protocols, + is_list(Value) -> + validate_binary_list(Opt, Value), Value; -validate_option(verify_fun, undefined) -> +validate_option(Opt, Value, _) + when Opt =:= alpn_advertised_protocols orelse + Opt =:= alpn_preferred_protocols, + Value =:= undefined -> undefined; -%% Backwards compatibility -validate_option(verify_fun, Fun) when is_function(Fun) -> - {fun(_,{bad_cert, _} = Reason, OldFun) -> - case OldFun([Reason]) of - true -> - {valid, OldFun}; - false -> - {fail, Reason} - end; - (_,{extension, _}, UserState) -> - {unknown, UserState}; - (_, valid, UserState) -> - {valid, UserState}; - (_, valid_peer, UserState) -> - {valid, UserState} - end, Fun}; -validate_option(verify_fun, {Fun, _} = Value) when is_function(Fun) -> - Value; -validate_option(partial_chain, Value) when is_function(Value) -> +validate_option(anti_replay, '10k', _) -> + %% n = 10000 + %% p = 0.030003564 (1 in 33) + %% m = 72985 (8.91KiB) + %% k = 5 + {10, 5, 72985}; +validate_option(anti_replay, '100k', _) -> + %% n = 100000 + %% p = 0.03000428 (1 in 33) + %% m = 729845 (89.09KiB) + %% k = 5 + {10, 5, 729845}; +validate_option(anti_replay, Value, _) + when (is_tuple(Value) andalso + tuple_size(Value) =:= 3) -> + Value; +validate_option(beast_mitigation, Value, _) + when Value == one_n_minus_one orelse + Value == zero_n orelse + Value == disabled -> + Value; +%% certfile must be present in some cases otherwhise it can be set +%% to the empty string. +validate_option(cacertfile, undefined, _) -> + <<>>; +validate_option(cacertfile, Value, _) + when is_binary(Value) -> + Value; +validate_option(cacertfile, Value, _) + when is_list(Value), Value =/= ""-> + binary_filename(Value); +validate_option(cacerts, Value, _) + when Value == undefined; + is_list(Value) -> Value; -validate_option(fail_if_no_peer_cert, Value) when is_boolean(Value) -> +validate_option(cb_info, {V1, V2, V3, V4} = Value, _) + when is_atom(V1), + is_atom(V2), + is_atom(V3), + is_atom(V4) -> Value; -validate_option(depth, Value) when is_integer(Value), - Value >= 0, Value =< 255-> +validate_option(cb_info, {V1, V2, V3, V4, V5} = Value, _) + when is_atom(V1), + is_atom(V2), + is_atom(V3), + is_atom(V4), + is_atom(V5) -> Value; -validate_option(cert, Value) when Value == undefined; - is_list(Value)-> +validate_option(cert, Value, _) when Value == undefined; + is_list(Value)-> Value; -validate_option(cert, Value) when Value == undefined; - is_binary(Value)-> +validate_option(cert, Value, _) when Value == undefined; + is_binary(Value)-> [Value]; -validate_option(certfile, undefined = Value) -> +validate_option(certfile, undefined = Value, _) -> Value; -validate_option(certfile, Value) when is_binary(Value) -> +validate_option(certfile, Value, _) + when is_binary(Value) -> Value; -validate_option(certfile, Value) when is_list(Value) -> +validate_option(certfile, Value, _) + when is_list(Value) -> binary_filename(Value); - -validate_option(key, undefined) -> +validate_option(client_preferred_next_protocols, {Precedence, PreferredProtocols}, _) + when is_list(PreferredProtocols) -> + validate_binary_list(client_preferred_next_protocols, PreferredProtocols), + validate_npn_ordering(Precedence), + {Precedence, PreferredProtocols, ?NO_PROTOCOL}; +validate_option(client_preferred_next_protocols, + {Precedence, PreferredProtocols, Default} = Value, _) + when is_list(PreferredProtocols), is_binary(Default), + byte_size(Default) > 0, byte_size(Default) < 256 -> + validate_binary_list(client_preferred_next_protocols, PreferredProtocols), + validate_npn_ordering(Precedence), + Value; +validate_option(client_preferred_next_protocols, undefined, _) -> undefined; -validate_option(key, {KeyType, Value}) when is_binary(Value), - KeyType == rsa; %% Backwards compatibility - KeyType == dsa; %% Backwards compatibility - KeyType == 'RSAPrivateKey'; - KeyType == 'DSAPrivateKey'; - KeyType == 'ECPrivateKey'; - KeyType == 'PrivateKeyInfo' -> - {KeyType, Value}; -validate_option(key, #{algorithm := _} = Value) -> +validate_option(client_renegotiation, Value, _) + when is_boolean(Value) -> Value; -validate_option(keyfile, undefined) -> - <<>>; -validate_option(keyfile, Value) when is_binary(Value) -> +validate_option(cookie, Value, _) + when is_boolean(Value) -> Value; -validate_option(keyfile, Value) when is_list(Value), Value =/= "" -> - binary_filename(Value); -validate_option(key_update_at, Value) when is_integer(Value) andalso - Value > 0 -> +validate_option(crl_cache, {Cb, {_Handle, Options}} = Value, _) + when is_atom(Cb) and is_list(Options) -> Value; -validate_option(password, Value) when is_list(Value) -> +validate_option(crl_check, Value, _) + when is_boolean(Value) -> Value; - -validate_option(cacerts, Value) when Value == undefined; - is_list(Value) -> +validate_option(crl_check, Value, _) + when (Value == best_effort) or + (Value == peer) -> Value; -%% certfile must be present in some cases otherwhise it can be set -%% to the empty string. -validate_option(cacertfile, undefined) -> - <<>>; -validate_option(cacertfile, Value) when is_binary(Value) -> +validate_option(customize_hostname_check, Value, _) + when is_list(Value) -> Value; -validate_option(cacertfile, Value) when is_list(Value), Value =/= ""-> - binary_filename(Value); -validate_option(dh, Value) when Value == undefined; - is_binary(Value) -> +validate_option(depth, Value, _) + when is_integer(Value), + Value >= 0, Value =< 255-> Value; -validate_option(dhfile, undefined = Value) -> +validate_option(dh, Value, _) + when Value == undefined; + is_binary(Value) -> Value; -validate_option(dhfile, Value) when is_binary(Value) -> +validate_option(dhfile, undefined = Value, _) -> Value; -validate_option(dhfile, Value) when is_list(Value), Value =/= "" -> - binary_filename(Value); -validate_option(psk_identity, undefined) -> - undefined; -validate_option(psk_identity, Identity) - when is_list(Identity), Identity =/= "", length(Identity) =< 65535 -> - binary_filename(Identity); -validate_option(user_lookup_fun, undefined) -> - undefined; -validate_option(user_lookup_fun, {Fun, _} = Value) when is_function(Fun, 3) -> - Value; -validate_option(srp_identity, undefined) -> - undefined; -validate_option(srp_identity, {Username, Password}) - when is_list(Username), is_list(Password), Username =/= "", length(Username) =< 255 -> - {unicode:characters_to_binary(Username), - unicode:characters_to_binary(Password)}; - -validate_option(reuse_session, undefined) -> - undefined; -validate_option(reuse_session, Value) when is_function(Value) -> +validate_option(dhfile, Value, _) + when is_binary(Value) -> Value; -validate_option(reuse_session, Value) when is_binary(Value) -> +validate_option(dhfile, Value, _) + when is_list(Value), Value =/= "" -> + binary_filename(Value); +validate_option(early_data, Value, server) + when Value =:= disabled orelse + Value =:= enabled -> Value; -validate_option(reuse_session, {Id, Data} = Value) when is_binary(Id) andalso - is_binary(Data) -> +validate_option(early_data = Option, Value, server) -> + throw({error, + {options, role, {Option, {Value, {server, [disabled, enabled]}}}}}); +validate_option(early_data, Value, client) + when is_binary(Value) -> Value; -validate_option(reuse_sessions, Value) when is_boolean(Value) -> +validate_option(early_data = Option, Value, client) -> + throw({error, + {options, type, {Option, {Value, not_binary}}}}); +validate_option(erl_dist, Value, _) + when is_boolean(Value) -> Value; -validate_option(reuse_sessions, save = Value) -> +validate_option(fail_if_no_peer_cert, Value, _) + when is_boolean(Value) -> Value; -validate_option(secure_renegotiate, Value) when is_boolean(Value) -> +validate_option(fallback, Value, _) + when is_boolean(Value) -> Value; -validate_option(keep_secrets, Value) when is_boolean(Value) -> +validate_option(handshake, hello = Value, _) -> Value; -validate_option(client_renegotiation, Value) when is_boolean(Value) -> +validate_option(handshake, full = Value, _) -> Value; -validate_option(renegotiate_at, Value) when is_integer(Value) -> - erlang:min(Value, ?DEFAULT_RENEGOTIATE_AT); - -validate_option(hibernate_after, undefined) -> %% Backwards compatibility +validate_option(hibernate_after, undefined, _) -> %% Backwards compatibility infinity; -validate_option(hibernate_after, infinity) -> +validate_option(hibernate_after, infinity, _) -> infinity; -validate_option(hibernate_after, Value) when is_integer(Value), Value >= 0 -> +validate_option(hibernate_after, Value, _) + when is_integer(Value), Value >= 0 -> Value; - -validate_option(erl_dist,Value) when is_boolean(Value) -> +validate_option(honor_cipher_order, Value, _) + when is_boolean(Value) -> Value; -validate_option(Opt, Value) when Opt =:= alpn_advertised_protocols orelse Opt =:= alpn_preferred_protocols, - is_list(Value) -> - validate_binary_list(Opt, Value), +validate_option(honor_ecc_order, Value, _) + when is_boolean(Value) -> Value; -validate_option(Opt, Value) - when Opt =:= alpn_advertised_protocols orelse Opt =:= alpn_preferred_protocols, - Value =:= undefined -> - undefined; -validate_option(client_preferred_next_protocols, {Precedence, PreferredProtocols}) - when is_list(PreferredProtocols) -> - validate_binary_list(client_preferred_next_protocols, PreferredProtocols), - validate_npn_ordering(Precedence), - {Precedence, PreferredProtocols, ?NO_PROTOCOL}; -validate_option(client_preferred_next_protocols, {Precedence, PreferredProtocols, Default} = Value) - when is_list(PreferredProtocols), is_binary(Default), - byte_size(Default) > 0, byte_size(Default) < 256 -> - validate_binary_list(client_preferred_next_protocols, PreferredProtocols), - validate_npn_ordering(Precedence), +validate_option(keep_secrets, Value, _) when is_boolean(Value) -> Value; -validate_option(client_preferred_next_protocols, undefined) -> +validate_option(key, undefined, _) -> undefined; -validate_option(log_alert, true) -> - notice; -validate_option(log_alert, false) -> - warning; -validate_option(log_level, Value) when +validate_option(key, {KeyType, Value}, _) + when is_binary(Value), + KeyType == rsa; %% Backwards compatibility + KeyType == dsa; %% Backwards compatibility + KeyType == 'RSAPrivateKey'; + KeyType == 'DSAPrivateKey'; + KeyType == 'ECPrivateKey'; + KeyType == 'PrivateKeyInfo' -> + {KeyType, Value}; +validate_option(key, #{algorithm := _} = Value, _) -> + Value; +validate_option(keyfile, undefined, _) -> + <<>>; +validate_option(keyfile, Value, _) + when is_binary(Value) -> + Value; +validate_option(keyfile, Value, _) + when is_list(Value), Value =/= "" -> + binary_filename(Value); +validate_option(key_update_at, Value, _) + when is_integer(Value) andalso + Value > 0 -> + Value; +validate_option(log_level, Value, _) when is_atom(Value) andalso (Value =:= none orelse Value =:= all orelse @@ -2262,132 +2304,178 @@ validate_option(log_level, Value) when Value =:= info orelse Value =:= debug) -> Value; -validate_option(middlebox_comp_mode, Value) when is_boolean(Value) -> - Value; -validate_option(next_protocols_advertised, Value) when is_list(Value) -> - validate_binary_list(next_protocols_advertised, Value), - Value; -validate_option(next_protocols_advertised, undefined) -> - undefined; -validate_option(server_name_indication, Value) when is_list(Value) -> - %% RFC 6066, Section 3: Currently, the only server names supported are - %% DNS hostnames - %% case inet_parse:domain(Value) of - %% false -> - %% throw({error, {options, {{Opt, Value}}}}); - %% true -> - %% Value - %% end; - %% - %% But the definition seems very diffuse, so let all strings through - %% and leave it up to public_key to decide... - Value; -validate_option(server_name_indication, undefined) -> - undefined; -validate_option(server_name_indication, disable) -> - disable; - %% RFC 6066, Section 4 -validate_option(max_fragment_length, I) when I == ?MAX_FRAGMENT_LENGTH_BYTES_1; I == ?MAX_FRAGMENT_LENGTH_BYTES_2; - I == ?MAX_FRAGMENT_LENGTH_BYTES_3; I == ?MAX_FRAGMENT_LENGTH_BYTES_4 -> +validate_option(max_fragment_length, I, _) + when I == ?MAX_FRAGMENT_LENGTH_BYTES_1; + I == ?MAX_FRAGMENT_LENGTH_BYTES_2; + I == ?MAX_FRAGMENT_LENGTH_BYTES_3; + I == ?MAX_FRAGMENT_LENGTH_BYTES_4 -> I; -validate_option(max_fragment_length, undefined) -> - undefined; - -validate_option(sni_hosts, []) -> - []; -validate_option(sni_hosts, [{Hostname, SSLOptions} | Tail]) when is_list(Hostname) -> - RecursiveSNIOptions = proplists:get_value(sni_hosts, SSLOptions, undefined), - case RecursiveSNIOptions of - undefined -> - [{Hostname, validate_options(SSLOptions)} | validate_option(sni_hosts, Tail)]; - _ -> - throw({error, {options, {sni_hosts, RecursiveSNIOptions}}}) - end; -validate_option(sni_fun, undefined) -> +validate_option(max_fragment_length, undefined, _) -> undefined; -validate_option(sni_fun, Fun) when is_function(Fun) -> - Fun; -validate_option(honor_cipher_order, Value) when is_boolean(Value) -> +validate_option(max_handshake_size, Value, _) + when is_integer(Value) andalso + Value =< ?MAX_UNIT24 -> Value; -validate_option(honor_ecc_order, Value) when is_boolean(Value) -> +validate_option(middlebox_comp_mode, Value, _) + when is_boolean(Value) -> Value; -validate_option(padding_check, Value) when is_boolean(Value) -> +validate_option(next_protocols_advertised, Value, _) when is_list(Value) -> + validate_binary_list(next_protocols_advertised, Value), Value; -validate_option(fallback, Value) when is_boolean(Value) -> +validate_option(next_protocols_advertised, undefined, _) -> + undefined; +validate_option(ocsp_nonce, Value, _) + when Value =:= true orelse + Value =:= false -> Value; -validate_option(cookie, Value) when is_boolean(Value) -> +%% The OCSP responders' certificates can be given as a suggestion and +%% will be used to verify the OCSP response. +validate_option(ocsp_responder_certs, Value, _) + when is_list(Value) -> + [public_key:pkix_decode_cert(CertDer, plain) || CertDer <- Value, + is_binary(CertDer)]; +validate_option(ocsp_stapling, Value, _) + when Value =:= true orelse + Value =:= false -> Value; -validate_option(crl_check, Value) when is_boolean(Value) -> +validate_option(padding_check, Value, _) + when is_boolean(Value) -> Value; -validate_option(crl_check, Value) when (Value == best_effort) or (Value == peer) -> +validate_option(partial_chain, Value, _) + when is_function(Value) -> Value; -validate_option(crl_cache, {Cb, {_Handle, Options}} = Value) when is_atom(Cb) and is_list(Options) -> +validate_option(password, Value, _) + when is_list(Value) -> Value; -validate_option(beast_mitigation, Value) when Value == one_n_minus_one orelse - Value == zero_n orelse - Value == disabled -> - Value; -validate_option(max_handshake_size, Value) when is_integer(Value) andalso Value =< ?MAX_UNIT24 -> +validate_option(protocol, Value = tls, _) -> Value; -validate_option(protocol, Value = tls) -> +validate_option(protocol, Value = dtls, _) -> Value; -validate_option(protocol, Value = dtls) -> +validate_option(psk_identity, undefined, _) -> + undefined; +validate_option(psk_identity, Identity, _) + when is_list(Identity), Identity =/= "", length(Identity) =< 65535 -> + binary_filename(Identity); +validate_option(renegotiate_at, Value, _) when is_integer(Value) -> + erlang:min(Value, ?DEFAULT_RENEGOTIATE_AT); +validate_option(reuse_session, undefined, _) -> + undefined; +validate_option(reuse_session, Value, _) + when is_function(Value) -> Value; -validate_option(handshake, hello = Value) -> +validate_option(reuse_session, Value, _) + when is_binary(Value) -> Value; -validate_option(handshake, full = Value) -> +validate_option(reuse_session, {Id, Data} = Value, _) + when is_binary(Id) andalso + is_binary(Data) -> Value; -validate_option(customize_hostname_check, Value) when is_list(Value) -> +validate_option(reuse_sessions, Value, _) + when is_boolean(Value) -> Value; -validate_option(cb_info, {V1, V2, V3, V4} = Value) when is_atom(V1), - is_atom(V2), - is_atom(V3), - is_atom(V4) - -> +validate_option(reuse_sessions, save = Value, _) -> Value; -validate_option(cb_info, {V1, V2, V3, V4, V5} = Value) when is_atom(V1), - is_atom(V2), - is_atom(V3), - is_atom(V4), - is_atom(V5) - -> +validate_option(secure_renegotiate, Value, _) + when is_boolean(Value) -> Value; -validate_option(use_ticket, Value) when is_list(Value) -> +validate_option(server_name_indication, Value, _) + when is_list(Value) -> + %% RFC 6066, Section 3: Currently, the only server names supported are + %% DNS hostnames + %% case inet_parse:domain(Value) of + %% false -> + %% throw({error, {options, {{Opt, Value}}}}); + %% true -> + %% Value + %% end; + %% + %% But the definition seems very diffuse, so let all strings through + %% and leave it up to public_key to decide... Value; -validate_option(session_tickets, Value) when Value =:= disabled orelse - Value =:= manual orelse - Value =:= auto orelse - Value =:= stateless orelse - Value =:= stateful -> +validate_option(server_name_indication, undefined, _) -> + undefined; +validate_option(server_name_indication, disable, _) -> + disable; +validate_option(session_tickets, Value, server) + when Value =:= disabled orelse + Value =:= stateful orelse + Value =:= stateless -> Value; -validate_option(anti_replay, '10k') -> - %% n = 10000 - %% p = 0.030003564 (1 in 33) - %% m = 72985 (8.91KiB) - %% k = 5 - {10, 5, 72985}; -validate_option(anti_replay, '100k') -> - %% n = 100000 - %% p = 0.03000428 (1 in 33) - %% m = 729845 (89.09KiB) - %% k = 5 - {10, 5, 729845}; -validate_option(anti_replay, Value) when (is_tuple(Value) andalso - tuple_size(Value) =:= 3) -> +validate_option(session_tickets, Value, server) -> + throw({error, + {options, role, + {session_tickets, + {Value, {server, [disabled, stateful, stateless]}}}}}); +validate_option(session_tickets, Value, client) + when Value =:= disabled orelse + Value =:= manual orelse + Value =:= auto -> Value; -validate_option(ocsp_stapling, Value) when Value =:= true orelse - Value =:= false -> +validate_option(session_tickets, Value, client) -> + throw({error, + {options, role, + {session_tickets, + {Value, {client, [disabled, manual, auto]}}}}}); +validate_option(sni_fun, undefined, _) -> + undefined; +validate_option(sni_fun, Fun, _) + when is_function(Fun) -> + Fun; +validate_option(sni_hosts, [], _) -> + []; +validate_option(sni_hosts, [{Hostname, SSLOptions} | Tail], _) + when is_list(Hostname) -> + RecursiveSNIOptions = proplists:get_value(sni_hosts, SSLOptions, undefined), + case RecursiveSNIOptions of + undefined -> + [{Hostname, validate_options(SSLOptions)} | + validate_option(sni_hosts, Tail)]; + _ -> + throw({error, {options, {sni_hosts, RecursiveSNIOptions}}}) + end; +validate_option(srp_identity, undefined, _) -> + undefined; +validate_option(srp_identity, {Username, Password}, _) + when is_list(Username), + is_list(Password), Username =/= "", + length(Username) =< 255 -> + {unicode:characters_to_binary(Username), + unicode:characters_to_binary(Password)}; +validate_option(user_lookup_fun, undefined, _) -> + undefined; +validate_option(user_lookup_fun, {Fun, _} = Value, _) + when is_function(Fun, 3) -> + Value; +validate_option(use_ticket, Value, _) + when is_list(Value) -> Value; -%% The OCSP responders' certificates can be given as a suggestion and -%% will be used to verify the OCSP response. -validate_option(ocsp_responder_certs, Value) when is_list(Value) -> - [public_key:pkix_decode_cert(CertDer, plain) || CertDer <- Value, - is_binary(CertDer)]; -validate_option(ocsp_nonce, Value) when Value =:= true orelse - Value =:= false -> +validate_option(verify, Value, _) + when Value == verify_none; Value == verify_peer -> Value; -validate_option(Opt, undefined = Value) -> +validate_option(verify_fun, undefined, _) -> + undefined; +%% Backwards compatibility +validate_option(verify_fun, Fun, _) when is_function(Fun) -> + {fun(_,{bad_cert, _} = Reason, OldFun) -> + case OldFun([Reason]) of + true -> + {valid, OldFun}; + false -> + {fail, Reason} + end; + (_,{extension, _}, UserState) -> + {unknown, UserState}; + (_, valid, UserState) -> + {valid, UserState}; + (_, valid_peer, UserState) -> + {valid, UserState} + end, Fun}; +validate_option(verify_fun, {Fun, _} = Value, _) when is_function(Fun) -> + Value; +validate_option(versions, Versions, _) -> + validate_versions(Versions, Versions); +validate_option(Opt, undefined = Value, _) -> AllOpts = maps:keys(?RULES), case lists:member(Opt, AllOpts) of true -> @@ -2395,7 +2483,7 @@ validate_option(Opt, undefined = Value) -> false -> throw({error, {options, {Opt, Value}}}) end; -validate_option(Opt, Value) -> +validate_option(Opt, Value, _) -> throw({error, {options, {Opt, Value}}}). handle_cb_info({V1, V2, V3, V4}) -> diff --git a/lib/ssl/src/ssl_certificate.erl b/lib/ssl/src/ssl_certificate.erl index b5b0a23d85..0b4211e1c3 100644 --- a/lib/ssl/src/ssl_certificate.erl +++ b/lib/ssl/src/ssl_certificate.erl @@ -75,7 +75,7 @@ trusted_cert_and_paths(Chain, CertDbHandle, CertDbRef, PartialChainHandler) -> %% If the chain contains extraneous certificates there could be %% more than one possible path such chains might be used to phase out %% an old certificate. - Paths = paths(Chain, CertDbHandle, CertDbRef), + Paths = paths(Chain, CertDbHandle), lists:map(fun(Path) -> case handle_partial_chain(Path, PartialChainHandler, CertDbHandle, CertDbRef) of {unknown_ca, _} = Result -> @@ -86,10 +86,9 @@ trusted_cert_and_paths(Chain, CertDbHandle, CertDbRef, PartialChainHandler) -> Result -> Result end - end, Paths). - + end, Paths). %%-------------------------------------------------------------------- --spec certificate_chain(undefined | binary() | #'OTPCertificate'{} , db_handle(), certdb_ref()) -> +-spec certificate_chain(undefined | binary() | #'OTPCertificate'{} , db_handle(), certdb_ref() | {extracted, list()}) -> {error, no_cert} | {ok, der_cert() | undefined, [der_cert()]}. %% %% Description: Return the certificate chain to send to peer. @@ -439,38 +438,38 @@ pre_1_3_hash(sha1) -> pre_1_3_hash(Hash) -> Hash. -paths(Chain, CertDbHandle, CertDbRef) -> - paths(Chain, Chain, CertDbHandle, CertDbRef, []). +paths(Chain, CertDbHandle) -> + paths(Chain, Chain, CertDbHandle, []). -paths([Root], _, _, _, Path) -> - [[Root | Path]]; -paths([Cert1, Cert2 | Rest], Chain, CertDbHandle, CertDbRef, Path) -> +paths([Root], _, _, Path) -> + [[Root | Path]]; +paths([Cert1, Cert2 | Rest], Chain, CertDbHandle, Path) -> case public_key:pkix_is_issuer(Cert1, Cert2) of true -> %% Chain orded so far - paths([Cert2 | Rest], Chain, CertDbHandle, CertDbRef, [Cert1 | Path]); + paths([Cert2 | Rest], Chain, CertDbHandle, [Cert1 | Path]); false -> %% Chain is unorded and/or contains extraneous certificates - unorded_or_extraneous(Chain, CertDbHandle, CertDbRef) + unorded_or_extraneous(Chain, CertDbHandle) end. - -unorded_or_extraneous([Peer | FalseChain], CertDbHandle, CertDbRef) -> - ChainCandidates = extraneous_chains(FalseChain), - lists:map(fun(Candidate) -> - path_candidate(Peer, Candidate, CertDbHandle, CertDbRef) - end, + +unorded_or_extraneous([Peer | UnorderedChain], CertDbHandle) -> + ChainCandidates = extraneous_chains(UnorderedChain), + lists:map(fun(Candidate) -> + path_candidate(Peer, Candidate, CertDbHandle) + end, ChainCandidates). -path_candidate(Peer, ChainCandidateCAs, CertDbHandle, _CertDbRef) -> +path_candidate(Peer, ChainCandidateCAs, CertDbHandle) -> {ok, ExtractedCerts} = ssl_pkix_db:extract_trusted_certs({der, ChainCandidateCAs}), - %% certificate_chain/4 will make sure the chain is ordered - case certificate_chain(Peer, CertDbHandle, ExtractedCerts, []) of + %% certificate_chain/4 will make sure the chain is ordered + case certificate_chain(Peer, CertDbHandle, ExtractedCerts, []) of {ok, undefined, Chain} -> lists:reverse(Chain); {ok, Root, Chain} -> [Root | lists:reverse(Chain)] - end. - + end. + handle_partial_chain([IssuerCert| Rest] = Path, PartialChainHandler, CertDbHandle, CertDbRef) -> case public_key:pkix_is_self_signed(IssuerCert) of true -> %% IssuerCert = ROOT (That is ROOT was included in chain) @@ -533,7 +532,12 @@ handle_incomplete_chain([PeerCert| _] = Chain0, PartialChainHandler, Default, Ce %% See if we have the certificates to rebuild it. case certificate_chain(PeerCert, CertDbHandle, CertDbRef) of {ok, _, [PeerCert | _] = Chain} when Chain =/= Chain0 -> %% Chain candidate found - handle_partial_chain(lists:reverse(Chain), PartialChainHandler, CertDbHandle, CertDbRef); + case lists:prefix(Chain0, Chain) of + true -> + handle_partial_chain(lists:reverse(Chain), PartialChainHandler, CertDbHandle, CertDbRef); + false -> + Default + end; _ -> Default end. @@ -546,8 +550,8 @@ extraneous_chains(Certs) -> Subjects = [{subject(Cert), Cert} || Cert <- Certs], Duplicates = find_duplicates(Subjects), %% Number of certs with duplicates (same subject) has been limited - %% to two and the maximum number of combinations is limited to 4. - build_candidates(Duplicates, 2, 4). + %% to 4 and the maximum number of combinations is limited to 16. + build_candidates(Duplicates, 4, 16). build_candidates(Map, Duplicates, Combinations) -> Subjects = maps:keys(Map), @@ -568,7 +572,7 @@ build_candidates([H|T], Map, Duplicates, Combinations, Max, Acc0) -> Acc = [[Cert|L] || Cert <- Certs, L <- Acc0], build_candidates(T, Map, Duplicates - 1, Combinations * Counter, Max, Acc) end; - {[Cert|_], _} -> + {[Cert|_Throw], _Counter} -> case Acc0 of [] -> Acc = [[Cert]], diff --git a/lib/ssl/src/ssl_cipher.erl b/lib/ssl/src/ssl_cipher.erl index a7fac8722b..85042e8612 100644 --- a/lib/ssl/src/ssl_cipher.erl +++ b/lib/ssl/src/ssl_cipher.erl @@ -70,7 +70,8 @@ hash_size/1, effective_key_bits/1, key_material/1, - signature_algorithm_to_scheme/1]). + signature_algorithm_to_scheme/1, + bulk_cipher_algorithm/1]). %% RFC 8446 TLS 1.3 -export([generate_client_shares/1, diff --git a/lib/ssl/src/ssl_session_cache.erl b/lib/ssl/src/ssl_client_session_cache_db.erl index b2b33b9af3..d344294231 100644 --- a/lib/ssl/src/ssl_session_cache.erl +++ b/lib/ssl/src/ssl_client_session_cache_db.erl @@ -19,24 +19,30 @@ %% %% --module(ssl_session_cache). +-module(ssl_client_session_cache_db). -behaviour(ssl_session_cache_api). -include("ssl_handshake.hrl"). -include("ssl_internal.hrl"). --export([init/1, terminate/1, lookup/2, update/3, delete/2, foldl/3, - select_session/2, size/1, take_oldest/1]). +-export([init/1, + terminate/1, + lookup/2, + update/3, + delete/2, + foldl/3, + select_session/2, + size/1]). %%-------------------------------------------------------------------- -%% Description: Return table reference. Called by ssl_manager process. +%% Description: Return table reference. Called by ssl_manager process. %%-------------------------------------------------------------------- init(Options) -> ets:new(cache_name(proplists:get_value(role, Options)), [ordered_set, protected]). %%-------------------------------------------------------------------- -%% Description: Handles cache table at termination of ssl manager. +%% Description: Handles cache table at termination of ssl manager. %%-------------------------------------------------------------------- terminate(Cache) -> ets:delete(Cache). @@ -85,17 +91,17 @@ foldl(Fun, Acc0, Cache) -> _:_ -> Acc0 end. - + %%-------------------------------------------------------------------- %% Description: Selects a session that could be reused. Should be callable %% from any process. %%-------------------------------------------------------------------- -select_session(Cache, PartialKey) -> - try ets:select(Cache, +select_session(Cache, PartialKey) -> + try ets:select(Cache, [{{{PartialKey,'_'}, '$1'},[],['$1']}]) of Result -> Result - catch + catch _:_ -> [] end. @@ -107,14 +113,6 @@ size(Cache) -> ets:info(Cache, size). %%-------------------------------------------------------------------- -%% Description: Returns the oldest entry -%%-------------------------------------------------------------------- -take_oldest(Cache) -> - {Key, Oldest} = ets:first(Cache), - delete(Cache, Key), - {Oldest, Cache}. - -%%-------------------------------------------------------------------- %%% Internal functions %%-------------------------------------------------------------------- cache_name(Name) -> diff --git a/lib/ssl/src/ssl_config.erl b/lib/ssl/src/ssl_config.erl index 6d09af9b1c..2832d76d42 100644 --- a/lib/ssl/src/ssl_config.erl +++ b/lib/ssl/src/ssl_config.erl @@ -29,7 +29,11 @@ -define(DEFAULT_MAX_SESSION_CACHE, 1000). -export([init/2, - pre_1_3_session_opts/0 + pre_1_3_session_opts/1, + get_max_early_data_size/0, + get_ticket_lifetime/0, + get_ticket_store_size/0, + get_internal_active_n/0 ]). %%==================================================================== @@ -51,20 +55,46 @@ init(#{erl_dist := ErlDist, DHParams = init_diffie_hellman(PemCache, DH, DHFile, Role), {ok, Config#{private_key => PrivateKey, dh_params => DHParams}}. -pre_1_3_session_opts() -> - CbOpts = case application:get_env(ssl, session_cb) of - {ok, Cb} when is_atom(Cb) -> - InitArgs = session_cb_init_args(), - #{session_cb => Cb, - session_cb_init_args => InitArgs}; - _ -> - #{session_cb => ssl_server_session_cache_db, - session_cb_init_args => []} - end, - LifeTime = session_lifetime(), - Max = max_session_cache_size(), - [CbOpts#{lifetime => LifeTime, max => Max}]. +pre_1_3_session_opts(Role) -> + {Cb, InitArgs} = session_cb_opts(Role), + CbOpts = #{session_cb => Cb, + session_cb_init_args => InitArgs}, + LifeTime = session_lifetime(Role), + Max = max_session_cache_size(Role), + CbOpts#{lifetime => LifeTime, max => Max}. +get_ticket_lifetime() -> + case application:get_env(ssl, server_session_ticket_lifetime) of + {ok, Seconds} when is_integer(Seconds) andalso + Seconds =< 604800 -> %% MUST be less than 7 days + Seconds; + _ -> + 7200 %% Default 2 hours + end. + +get_ticket_store_size() -> + case application:get_env(ssl, server_session_ticket_store_size) of + {ok, Size} when is_integer(Size) -> + Size; + _ -> + 1000 + end. + +get_max_early_data_size() -> + case application:get_env(ssl, server_session_ticket_max_early_data) of + {ok, Size} when is_integer(Size) -> + Size; + _ -> + ?DEFAULT_MAX_EARLY_DATA_SIZE + end. + +get_internal_active_n() -> + case application:get_env(ssl, internal_active_n) of + {ok, N} when is_integer(N) -> + N; + _ -> + ?INTERNAL_ACTIVE_N + end. %%==================================================================== %% Internal functions @@ -202,15 +232,33 @@ init_diffie_hellman(DbHandle,_, DHParamFile, server) -> file_error(DHParamFile, {dhfile, Reason}) end. -session_cb_init_args() -> - case application:get_env(ssl, session_cb_init_args) of - {ok, Args} when is_list(Args) -> - Args; - _ -> - [] + +session_cb_init_args(client) -> + case application:get_env(ssl, client_session_cb_init_args) of + undefined -> + case application:get_env(ssl, session_cb_init_args) of + {ok, Args} when is_list(Args) -> + Args; + _ -> + [] + end; + {ok, Args} -> + Args + end; +session_cb_init_args(server) -> + case application:get_env(ssl, server_session_cb_init_args) of + undefined -> + case application:get_env(ssl, session_cb_init_args) of + {ok, Args} when is_list(Args) -> + Args; + _ -> + [] + end; + {ok, Args} -> + Args end. -session_lifetime() -> +session_lifetime(_Role) -> case application:get_env(ssl, session_lifetime) of {ok, Time} when is_integer(Time) -> Time; @@ -218,10 +266,32 @@ session_lifetime() -> ?'24H_in_sec' end. -max_session_cache_size() -> +max_session_cache_size(client) -> + case application:get_env(ssl, session_cache_client_max) of + {ok, Size} when is_integer(Size) -> + Size; + _ -> + ?DEFAULT_MAX_SESSION_CACHE + end; +max_session_cache_size(server) -> case application:get_env(ssl, session_cache_server_max) of {ok, Size} when is_integer(Size) -> Size; _ -> ?DEFAULT_MAX_SESSION_CACHE end. + +session_cb_opts(client = Role)-> + case application:get_env(ssl, session_cb, ssl_client_session_cache_db) of + ssl_client_session_cache_db = ClientCb -> + {ClientCb, []}; + ClientCb -> + {ClientCb, session_cb_init_args(Role)} + end; +session_cb_opts(server = Role) -> + case application:get_env(ssl, session_cb, ssl_server_session_cache_db) of + ssl_server_session_cache_db = ServerCb -> + {ServerCb, []}; + ServerCb -> + {ServerCb, session_cb_init_args(Role)} + end. diff --git a/lib/ssl/src/ssl_connection.hrl b/lib/ssl/src/ssl_connection.hrl index 371599bbe8..4f9584bb9f 100644 --- a/lib/ssl/src/ssl_connection.hrl +++ b/lib/ssl/src/ssl_connection.hrl @@ -64,6 +64,7 @@ resumption = false :: boolean(), %% TLS 1.3 change_cipher_spec_sent = false :: boolean(), %% TLS 1.3 sni_guided_cert_selection = false :: boolean(), %% TLS 1.3 + early_data_accepted = false :: boolean(), %% TLS 1.3 allow_renegotiate = true ::boolean(), %% Ext handling hello, %%:: #client_hello{} | #server_hello{} diff --git a/lib/ssl/src/ssl_crl.erl b/lib/ssl/src/ssl_crl.erl index 888a75bfd6..12d261bcdc 100644 --- a/lib/ssl/src/ssl_crl.erl +++ b/lib/ssl/src/ssl_crl.erl @@ -27,35 +27,40 @@ -include("ssl_internal.hrl"). -include_lib("public_key/include/public_key.hrl"). --export([trusted_cert_and_path/3]). +-export([trusted_cert_and_path/4]). -trusted_cert_and_path(CRL, {SerialNumber, Issuer},{_, {Db, DbRef}} = DbHandle) -> +trusted_cert_and_path(CRL, {SerialNumber, Issuer}, CertPath, {Db, DbRef}) -> + %% CRL issuer cert ID is known case ssl_pkix_db:lookup_trusted_cert(Db, DbRef, SerialNumber, Issuer) of undefined -> - trusted_cert_and_path(CRL, issuer_not_found, DbHandle); + %% But not found in our database + search_certpath(CRL, CertPath, Db, DbRef); {ok, {_, OtpCert}} -> {ok, Root, Chain} = ssl_certificate:certificate_chain(OtpCert, Db, DbRef), {ok, Root, lists:reverse(Chain)} end; -trusted_cert_and_path(CRL, issuer_not_found, {CertPath, {Db, DbRef}}) -> - case find_issuer(CRL, {certpath, - [{Der, public_key:pkix_decode_cert(Der,otp)} || Der <- CertPath]}) of - {ok, OtpCert} -> - {ok, Root, Chain} = ssl_certificate:certificate_chain(OtpCert, Db, DbRef), - {ok, Root, lists:reverse(Chain)}; - {error, issuer_not_found} -> - trusted_cert_and_path(CRL, issuer_not_found, {Db, DbRef}) - end; -trusted_cert_and_path(CRL, issuer_not_found, {Db, DbRef} = DbInfo) -> - case find_issuer(CRL, DbInfo) of - {ok, OtpCert} -> - {ok, Root, Chain} = ssl_certificate:certificate_chain(OtpCert, Db, DbRef), - {ok, Root, lists:reverse(Chain)}; - {error, issuer_not_found} -> - {error, unknown_ca} - end. +trusted_cert_and_path(CRL, issuer_not_found, CertPath, {Db, DbRef}) -> + case search_certpath(CRL, CertPath, Db, DbRef) of + {error, unknown_ca} -> + Issuer = public_key:pkix_normalize_name(public_key:pkix_crl_issuer(CRL)), + IsIssuerFun = + fun({_Key, {_Der,ErlCertCandidate}}, Acc) -> + verify_crl_issuer(CRL, ErlCertCandidate, Issuer, Acc); + (_, Acc) -> + Acc + end, + case search_db(IsIssuerFun, Db, DbRef) of + {ok, OtpCert} -> + {ok, Root, Chain} = ssl_certificate:certificate_chain(OtpCert, Db, DbRef), + {ok, Root, lists:reverse(Chain)}; + {error, issuer_not_found} -> + {error, unknown_ca} + end; + Result -> + Result + end. -find_issuer(CRL, {certpath = Db, DbRef}) -> +search_certpath(CRL, CertPath, Db, DbRef) -> Issuer = public_key:pkix_normalize_name(public_key:pkix_crl_issuer(CRL)), IsIssuerFun = fun({_Der,ErlCertCandidate}, Acc) -> @@ -63,15 +68,18 @@ find_issuer(CRL, {certpath = Db, DbRef}) -> (_, Acc) -> Acc end, - find_issuer(IsIssuerFun, Db, DbRef); -find_issuer(CRL, {Db, DbRef}) -> - Issuer = public_key:pkix_normalize_name(public_key:pkix_crl_issuer(CRL)), - IsIssuerFun = - fun({_Key, {_Der,ErlCertCandidate}}, Acc) -> - verify_crl_issuer(CRL, ErlCertCandidate, Issuer, Acc); - (_, Acc) -> - Acc - end, + case find_issuer(IsIssuerFun, certpath, + [{Der, public_key:pkix_decode_cert(Der,otp)} || Der <- CertPath]) of + {ok, OtpCert} -> + {ok, Root, Chain} = ssl_certificate:certificate_chain(OtpCert, Db, DbRef), + {ok, Root, lists:reverse(Chain)}; + {error, issuer_not_found} -> + {error, unknown_ca} + end. + +search_db(IsIssuerFun, _, {extracted, ExtractedCerts})-> + find_issuer(IsIssuerFun, extracted, ExtractedCerts); +search_db(IsIssuerFun, Db, DbRef) -> find_issuer(IsIssuerFun, Db, DbRef). find_issuer(IsIssuerFun, certpath, Certs) -> diff --git a/lib/ssl/src/ssl_gen_statem.erl b/lib/ssl/src/ssl_gen_statem.erl index aa2a5541a1..e6268b4876 100644 --- a/lib/ssl/src/ssl_gen_statem.erl +++ b/lib/ssl/src/ssl_gen_statem.erl @@ -438,50 +438,50 @@ initial_hello({call, From}, {start, Timeout}, #state{static_env = #static_env{role = client = Role, host = Host, port = Port, - protocol_cb = Connection, - transport_cb = Transport, - socket = Socket}, + protocol_cb = Connection}, handshake_env = #handshake_env{renegotiation = {Renegotiation, _}, - ocsp_stapling_state = OcspState0} = HsEnv, + ocsp_stapling_state = OcspState0}, connection_env = CEnv, - ssl_options = #{log_level := LogLevel, - %% Use highest version in initial ClientHello. + ssl_options = #{%% Use highest version in initial ClientHello. %% Versions is a descending list of supported versions. versions := [HelloVersion|_] = Versions, session_tickets := SessionTickets, ocsp_stapling := OcspStaplingOpt, - ocsp_nonce := OcspNonceOpt} = SslOpts, + ocsp_nonce := OcspNonceOpt, + early_data := EarlyData} = SslOpts, session = Session, connection_states = ConnectionStates0 } = State0) -> KeyShare = maybe_generate_client_shares(SslOpts), - %% Update UseTicket in case of automatic session resumption + %% Update UseTicket in case of automatic session resumption. The automatic ticket handling + %% also takes it into account if the ticket is suitable for sending early data not exceeding + %% the max_early_data_size or if it can only be used for session resumption. {UseTicket, State1} = tls_handshake_1_3:maybe_automatic_session_resumption(State0), TicketData = tls_handshake_1_3:get_ticket_data(self(), SessionTickets, UseTicket), OcspNonce = tls_handshake:ocsp_nonce(OcspNonceOpt, OcspStaplingOpt), - Hello = tls_handshake:client_hello(Host, Port, ConnectionStates0, SslOpts, - Session#session.session_id, - Renegotiation, - Session#session.own_certificates, - KeyShare, - TicketData, - OcspNonce), - - Handshake0 = ssl_handshake:init_handshake_history(), + Hello0 = tls_handshake:client_hello(Host, Port, ConnectionStates0, SslOpts, + Session#session.session_id, + Renegotiation, + Session#session.own_certificates, + KeyShare, + TicketData, + OcspNonce), + + %% Early Data Indication + Hello1 = tls_handshake_1_3:maybe_add_early_data_indication(Hello0, + EarlyData, + HelloVersion), %% Update pre_shared_key extension with binders (TLS 1.3) - Hello1 = tls_handshake_1_3:maybe_add_binders(Hello, TicketData, HelloVersion), + Hello2 = tls_handshake_1_3:maybe_add_binders(Hello1, TicketData, HelloVersion), MaxFragEnum = maps:get(max_frag_enum, Hello1#client_hello.extensions, undefined), ConnectionStates1 = ssl_record:set_max_fragment_length(MaxFragEnum, ConnectionStates0), + State2 = State1#state{connection_states = ConnectionStates1, + connection_env = CEnv#connection_env{negotiated_version = HelloVersion}}, - {BinMsg, ConnectionStates, Handshake} = - Connection:encode_handshake(Hello1, HelloVersion, ConnectionStates1, Handshake0), - - tls_socket:send(Transport, Socket, BinMsg), - ssl_logger:debug(LogLevel, outbound, 'handshake', Hello1), - ssl_logger:debug(LogLevel, outbound, 'record', BinMsg), + State3 = Connection:queue_handshake(Hello2, State2), %% RequestedVersion is used as the legacy record protocol version and shall be %% {3,3} in case of TLS 1.2 and higher. In all other cases it defaults to the @@ -490,18 +490,31 @@ initial_hello({call, From}, {start, Timeout}, %% negotiated_version is also used by the TLS 1.3 state machine and is set after %% ServerHello is processed. RequestedVersion = tls_record:hello_version(Versions), - State = State1#state{connection_states = ConnectionStates, - connection_env = CEnv#connection_env{ - negotiated_version = RequestedVersion}, - session = Session, - handshake_env = HsEnv#handshake_env{ - tls_handshake_history = Handshake, - ocsp_stapling_state = OcspState0#{ocsp_nonce => OcspNonce}}, - start_or_recv_from = From, - key_share = KeyShare}, - NextState = next_statem_state(Versions, Role), - Connection:next_event(NextState, no_record, State, - [{{timeout, handshake}, Timeout, close}]); + + {Ref,Maybe} = tls_handshake_1_3:maybe(), + try + %% Send Early Data + State4 = Maybe(tls_handshake_1_3:maybe_send_early_data(State3)), + + {#state{handshake_env = HsEnv1} = State5, _} = + Connection:send_handshake_flight(State4), + + State = State5#state{ + connection_env = CEnv#connection_env{ + negotiated_version = RequestedVersion}, + session = Session, + handshake_env = HsEnv1#handshake_env{ + ocsp_stapling_state = OcspState0#{ocsp_nonce => OcspNonce}}, + start_or_recv_from = From, + key_share = KeyShare}, + NextState = next_statem_state(Versions, Role), + Connection:next_event(NextState, no_record, State, + [{{timeout, handshake}, Timeout, close}]) + catch + {Ref, #alert{} = Alert} -> + handle_own_alert(Alert, RequestedVersion, init, + State0#state{start_or_recv_from = From}) + end; initial_hello({call, From}, {start, Timeout}, #state{static_env = #static_env{role = Role, protocol_cb = Connection}, ssl_options = #{versions := Versions}} = State0) -> @@ -1421,11 +1434,17 @@ read_application_data_deliver(State, Front, BufferSize, Rear, SocketOpts0, RecvF State#state{ user_data_buffer = {Front,BufferSize,Rear}, start_or_recv_from = undefined, - bytes_to_read = undefined, + bytes_to_read = undefined, socket_options = SocketOpts }}; true -> %% Try to deliver more data - read_application_data(State, Front, BufferSize, Rear, SocketOpts, undefined, undefined) + %% Process early data if it is accepted. + case (State#state.handshake_env)#handshake_env.early_data_accepted of + false -> + read_application_data(State, Front, BufferSize, Rear, SocketOpts, undefined, undefined); + true -> + read_application_data(State, Front, BufferSize, Rear, SocketOpts, RecvFrom, undefined) + end end. @@ -1782,20 +1801,37 @@ security_info(#state{connection_states = ConnectionStates, #security_parameters{client_random = ClientRand, server_random = ServerRand, master_secret = MasterSecret, - application_traffic_secret = AppTrafSecretRead}} = ReadState, + application_traffic_secret = AppTrafSecretRead, + client_early_data_secret = ServerEarlyData + }} = ReadState, BaseSecurityInfo = [{client_random, ClientRand}, {server_random, ServerRand}, {master_secret, MasterSecret}], if KeepSecrets =/= true -> BaseSecurityInfo; true -> #{security_parameters := - #security_parameters{application_traffic_secret = AppTrafSecretWrite}} = + #security_parameters{application_traffic_secret = AppTrafSecretWrite, + client_early_data_secret = ClientEarlyData + }} = ssl_record:current_connection_state(ConnectionStates, write), - BaseSecurityInfo ++ - if Role == server -> - [{server_traffic_secret_0, AppTrafSecretWrite}, {client_traffic_secret_0, AppTrafSecretRead}]; - true -> - [{client_traffic_secret_0, AppTrafSecretWrite}, {server_traffic_secret_0, AppTrafSecretRead}] - end ++ + if Role == server -> + if ServerEarlyData =/= undefined -> + [{server_traffic_secret_0, AppTrafSecretWrite}, + {client_traffic_secret_0, AppTrafSecretRead}, + {client_early_data_secret, ServerEarlyData}]; + true -> + [{server_traffic_secret_0, AppTrafSecretWrite}, + {client_traffic_secret_0, AppTrafSecretRead}] + end; + true -> + if ClientEarlyData =/= undefined -> + [{client_traffic_secret_0, AppTrafSecretWrite}, + {server_traffic_secret_0, AppTrafSecretRead}, + {client_early_data_secret, ClientEarlyData}]; + true -> + [{client_traffic_secret_0, AppTrafSecretWrite}, + {server_traffic_secret_0, AppTrafSecretRead}] + end + end ++ case ReadState of #{client_handshake_traffic_secret := ClientHSTrafficSecret, server_handshake_traffic_secret := ServerHSTrafficSecret} -> @@ -1803,7 +1839,7 @@ security_info(#state{connection_states = ConnectionStates, {server_handshake_traffic_secret, ServerHSTrafficSecret}]; _ -> [] - end + end ++ BaseSecurityInfo end. record_cb(tls) -> @@ -1981,10 +2017,18 @@ maybe_add_keylog({_, 'tlsv1.3'}, Info) -> ServerTrafficSecret0 = keylog_secret(ServerTrafficSecret0Bin, Prf), ClientHSecret = keylog_secret(ClientHSecretBin, Prf), ServerHSecret = keylog_secret(ServerHSecretBin, Prf), - Keylog = [io_lib:format("CLIENT_HANDSHAKE_TRAFFIC_SECRET ~64.16.0B ", [ClientRandom]) ++ ClientHSecret, - io_lib:format("SERVER_HANDSHAKE_TRAFFIC_SECRET ~64.16.0B ", [ClientRandom]) ++ ServerHSecret, - io_lib:format("CLIENT_TRAFFIC_SECRET_0 ~64.16.0B ", [ClientRandom]) ++ ClientTrafficSecret0, - io_lib:format("SERVER_TRAFFIC_SECRET_0 ~64.16.0B ", [ClientRandom]) ++ ServerTrafficSecret0], + Keylog0 = [io_lib:format("CLIENT_HANDSHAKE_TRAFFIC_SECRET ~64.16.0B ", [ClientRandom]) ++ ClientHSecret, + io_lib:format("SERVER_HANDSHAKE_TRAFFIC_SECRET ~64.16.0B ", [ClientRandom]) ++ ServerHSecret, + io_lib:format("CLIENT_TRAFFIC_SECRET_0 ~64.16.0B ", [ClientRandom]) ++ ClientTrafficSecret0, + io_lib:format("SERVER_TRAFFIC_SECRET_0 ~64.16.0B ", [ClientRandom]) ++ ServerTrafficSecret0], + Keylog = case lists:keyfind(client_early_data_secret, 1, Info) of + {client_early_data_secret, EarlySecret} -> + ClientEarlySecret = keylog_secret(EarlySecret, Prf), + [io_lib:format("CLIENT_EARLY_TRAFFIC_SECRET ~64.16.0B ", [ClientRandom]) ++ ClientEarlySecret + | Keylog0]; + _ -> + Keylog0 + end, Info ++ [{keylog,Keylog}] catch _Cxx:_Exx -> diff --git a/lib/ssl/src/ssl_handshake.erl b/lib/ssl/src/ssl_handshake.erl index 7d6c21438e..de5490d232 100644 --- a/lib/ssl/src/ssl_handshake.erl +++ b/lib/ssl/src/ssl_handshake.erl @@ -43,7 +43,7 @@ -type ssl_handshake() :: #server_hello{} | #server_hello_done{} | #certificate{} | #certificate_request{} | #client_key_exchange{} | #finished{} | #certificate_verify{} | - #hello_request{} | #next_protocol{}. + #hello_request{} | #next_protocol{} | #end_of_early_data{}. %% Create handshake messages -export([hello_request/0, server_hello/4, server_hello_done/0, @@ -768,7 +768,13 @@ encode_extensions([#cookie{cookie = Cookie} | Rest], Acc) -> CookieLen = byte_size(Cookie), Len = CookieLen + 2, encode_extensions(Rest, <<?UINT16(?COOKIE_EXT), ?UINT16(Len), ?UINT16(CookieLen), - Cookie/binary, Acc/binary>>). + Cookie/binary, Acc/binary>>); +encode_extensions([#early_data_indication{} | Rest], Acc) -> + encode_extensions(Rest, <<?UINT16(?EARLY_DATA_EXT), + ?UINT16(0), Acc/binary>>); +encode_extensions([#early_data_indication_nst{indication = MaxSize} | Rest], Acc) -> + encode_extensions(Rest, <<?UINT16(?EARLY_DATA_EXT), + ?UINT16(4), ?UINT32(MaxSize), Acc/binary>>). encode_cert_status_req( StatusType, @@ -1309,7 +1315,9 @@ get_identities_binders(TicketData) -> %% get_identities_binders([], {Identities, Binders}, _) -> {lists:reverse(Identities), lists:reverse(Binders)}; -get_identities_binders([{Key, _, Identity, _, _, HKDF}|T], {I0, B0}, N) -> +get_identities_binders([#ticket_data{key = Key, + identity = Identity, + cipher_suite = {_, HKDF}}|T], {I0, B0}, N) -> %% Use dummy binder for proper calculation of packet size when creating %% the real binder value. Binder = dummy_binder(HKDF), @@ -1970,15 +1978,15 @@ cert_status_check(OtpCert, #{ocsp_state := #{ocsp_stapling := best_effort, %%TOD maybe_check_crl(_, #{crl_check := false}, _, _, _) -> valid; -maybe_check_crl(_, #{crl_check := peer}, _, valid, _) -> %% Do not check CAs with this option. +maybe_check_crl(_, #{crl_check := peer}, valid, _, _) -> %% Do not check CAs with this option. valid; maybe_check_crl(OtpCert, #{crl_check := Check, certdb := CertDbHandle, certdb_ref := CertDbRef, crl_db := {Callback, CRLDbHandle}}, _, CertPath, LogLevel) -> Options = [{issuer_fun, {fun(_DP, CRL, Issuer, DBInfo) -> - ssl_crl:trusted_cert_and_path(CRL, Issuer, {CertPath, - DBInfo}) + ssl_crl:trusted_cert_and_path(CRL, Issuer, CertPath, + DBInfo) end, {CertDbHandle, CertDbRef}}}, {update_crl, fun(DP, CRL) -> case Callback:fresh_crl(DP, CRL) of @@ -2813,6 +2821,7 @@ decode_extensions(<<?UINT16(?COOKIE_EXT), ?UINT16(Len), ?UINT16(CookieLen), when Len == CookieLen + 2 -> decode_extensions(Rest, Version, MessageType, Acc#{cookie => #cookie{cookie = Cookie}}); + %% RFC6066, if a server returns a "CertificateStatus" message, then %% the server MUST have included an extension of type "status_request" %% with empty "extension_data" in the extended server hello. @@ -2838,6 +2847,18 @@ decode_extensions(<<?UINT16(?STATUS_REQUEST), ?UINT16(Len), decode_extensions(Rest, Version, MessageType, Acc) end; +decode_extensions(<<?UINT16(?EARLY_DATA_EXT), ?UINT16(0), Rest/binary>>, + Version, MessageType, Acc) -> + decode_extensions(Rest, Version, MessageType, + Acc#{early_data => #early_data_indication{}}); + +decode_extensions(<<?UINT16(?EARLY_DATA_EXT), ?UINT16(4), ?UINT32(MaxSize), + Rest/binary>>, + Version, MessageType, Acc) -> + decode_extensions(Rest, Version, MessageType, + Acc#{early_data => + #early_data_indication_nst{indication = MaxSize}}); + %% Ignore data following the ClientHello (i.e., %% extensions) if not understood. decode_extensions(<<?UINT16(_), ?UINT16(Len), _Unknown:Len/binary, Rest/binary>>, Version, MessageType, Acc) -> diff --git a/lib/ssl/src/ssl_handshake.hrl b/lib/ssl/src/ssl_handshake.hrl index 790a115983..3f06eaa095 100644 --- a/lib/ssl/src/ssl_handshake.hrl +++ b/lib/ssl/src/ssl_handshake.hrl @@ -38,19 +38,20 @@ -define(ECDSA, 3). -record(session, { - session_id, - peer_certificate, - own_certificates, - compression_method, - cipher_suite, - master_secret, - srp_username, - is_resumable, - time_stamp, - ecc, %% TLS 1.3 Group - sign_alg, %% TLS 1.3 Signature Algorithm - dh_public_value %% TLS 1.3 DH Public Value from peer - }). + session_id, + internal_id, + peer_certificate, + own_certificates, + compression_method, + cipher_suite, + master_secret, + srp_username, + is_resumable, + time_stamp, + ecc, %% TLS 1.3 Group + sign_alg, %% TLS 1.3 Signature Algorithm + dh_public_value %% TLS 1.3 DH Public Value from peer + }). -define(NUM_OF_SESSION_ID_BYTES, 32). % TSL 1.1 & SSL 3 -define(NUM_OF_PREMASTERSECRET_BYTES, 48). diff --git a/lib/ssl/src/ssl_internal.hrl b/lib/ssl/src/ssl_internal.hrl index 6477f5ab57..b080016458 100644 --- a/lib/ssl/src/ssl_internal.hrl +++ b/lib/ssl/src/ssl_internal.hrl @@ -115,6 +115,8 @@ %% 2^24.5 * 2^14 = 2^38.5 -define(KEY_USAGE_LIMIT_AES_GCM, 388736063997). +-define(DEFAULT_MAX_EARLY_DATA_SIZE, 16384). + %% This map stores all supported options with default values and %% list of dependencies: %% #{<option> => {<default_value>, [<option>]}, @@ -140,6 +142,9 @@ depth => {10, [versions]}, dh => {undefined, [versions]}, dhfile => {undefined, [versions]}, + early_data => {undefined, [versions, + session_tickets, + use_ticket]}, eccs => {undefined, [versions]}, erl_dist => {false, [versions]}, fail_if_no_peer_cert => {false, [versions]}, @@ -236,6 +241,17 @@ {stop, any(), any()}. -type ssl_options() :: map(). +%% Internal ticket data record holding pre-processed ticket data. +-record(ticket_data, + {key, %% key in client ticket store + pos, %% ticket position in binders list + identity, %% opaque ticket binary + psk, %% pre-shared key + nonce, %% ticket nonce + cipher_suite, %% cipher suite - hash, bulk cipher algorithm + max_size %% max early data size allowed by this ticket + }). + -endif. % -ifdef(ssl_internal). diff --git a/lib/ssl/src/ssl_logger.erl b/lib/ssl/src/ssl_logger.erl index d8bf5b3a2b..027dbe8732 100644 --- a/lib/ssl/src/ssl_logger.erl +++ b/lib/ssl/src/ssl_logger.erl @@ -254,9 +254,13 @@ parse_handshake(Direction, #key_update{} = KeyUpdate) -> Header = io_lib:format("~s Post-Handshake, KeyUpdate", [header_prefix(Direction)]), Message = io_lib:format("~p", [?rec_info(key_update, KeyUpdate)]), + {Header, Message}; +parse_handshake(Direction, #end_of_early_data{} = EndOfEarlyData) -> + Header = io_lib:format("~s Handshake, EndOfEarlyData", + [header_prefix(Direction)]), + Message = io_lib:format("~p", [?rec_info(end_of_early_data, EndOfEarlyData)]), {Header, Message}. - parse_cipher_suites([_|_] = Ciphers) -> [format_cipher(C) || C <- Ciphers]. diff --git a/lib/ssl/src/ssl_manager.erl b/lib/ssl/src/ssl_manager.erl index 4f6f12163f..9d0dbc2d9b 100644 --- a/lib/ssl/src/ssl_manager.erl +++ b/lib/ssl/src/ssl_manager.erl @@ -48,13 +48,14 @@ -record(state, { session_cache_client :: db_handle(), - session_cache_cb :: atom(), + session_cache_client_cb :: atom(), session_lifetime :: integer(), certificate_db :: db_handle(), session_validation_timer :: reference(), session_cache_client_max :: integer(), session_client_invalidator :: undefined | pid(), - options :: list() + options :: list(), + client_session_order :: gb_trees:tree() }). -define(GEN_UNIQUE_ID_MAX_TRIES, 10). @@ -219,25 +220,31 @@ init([ManagerName, PemCacheName, Opts]) -> put(ssl_manager, ManagerName), put(ssl_pem_cache, PemCacheName), process_flag(trap_exit, true), - CacheCb = proplists:get_value(session_cb, Opts, ssl_session_cache), + + #{session_cb := DefaultCacheCb, + session_cb_init_args := DefaultCacheCbInitArgs, + lifetime := DefaultSessLifeTime, + max := ClientSessMax + } = ssl_config:pre_1_3_session_opts(client), + CacheCb = proplists:get_value(session_cb, Opts, DefaultCacheCb), SessionLifeTime = - proplists:get_value(session_lifetime, Opts, ?'24H_in_sec'), - CertDb = ssl_pkix_db:create(PemCacheName), + proplists:get_value(session_lifetime, Opts, DefaultSessLifeTime), ClientSessionCache = CacheCb:init([{role, client} | - proplists:get_value(session_cb_init_args, Opts, [])]), + proplists:get_value(session_cb_init_args, Opts, DefaultCacheCbInitArgs)]), + CertDb = ssl_pkix_db:create(PemCacheName), Timer = erlang:send_after(SessionLifeTime * 1000 + 5000, self(), validate_sessions), {ok, #state{certificate_db = CertDb, session_cache_client = ClientSessionCache, - session_cache_cb = CacheCb, + session_cache_client_cb = CacheCb, session_lifetime = SessionLifeTime, session_validation_timer = Timer, - session_cache_client_max = - max_session_cache_size(session_cache_client_max), + session_cache_client_max = ClientSessMax, session_client_invalidator = undefined, - options = Opts + options = Opts, + client_session_order = gb_trees:empty() }}. %%-------------------------------------------------------------------- @@ -306,7 +313,7 @@ handle_cast({register_session, Host, Port, Session, true}, State0) -> handle_cast({invalidate_session, Host, Port, #session{session_id = ID} = Session}, #state{session_cache_client = Cache, - session_cache_cb = CacheCb} = State) -> + session_cache_client_cb = CacheCb} = State) -> invalidate_session(Cache, CacheCb, {{Host, Port}, ID}, Session, State); handle_cast({insert_crls, Path, CRLs}, #state{certificate_db = Db} = State) -> @@ -326,7 +333,7 @@ handle_cast({delete_crls, CRLsOrPath}, %% %% Description: Handling all non call/cast messages %%------------------------------------------------------------------- -handle_info(validate_sessions, #state{session_cache_cb = CacheCb, +handle_info(validate_sessions, #state{session_cache_client_cb = CacheCb, session_cache_client = ClientCache, session_lifetime = LifeTime, session_client_invalidator = Client @@ -362,7 +369,7 @@ handle_info(_Info, State) -> %%-------------------------------------------------------------------- terminate(_Reason, #state{certificate_db = Db, session_cache_client = ClientSessionCache, - session_cache_cb = CacheCb, + session_cache_client_cb = CacheCb, session_validation_timer = Timer}) -> erlang:cancel_timer(Timer), ssl_pkix_db:remove(Db), @@ -420,21 +427,15 @@ session_validation({{Port, _}, Session}, LifeTime) -> validate_session(Port, Session, LifeTime), LifeTime. -max_session_cache_size(CacheType) -> - case application:get_env(ssl, CacheType) of - {ok, Size} when is_integer(Size) -> - Size; - _ -> - ?DEFAULT_MAX_SESSION_CACHE - end. - -invalidate_session(Cache, CacheCb, Key, _Session, State) -> +invalidate_session(Cache, CacheCb, Key, _Session, + #state{client_session_order = Order} = State) -> case CacheCb:lookup(Cache, Key) of undefined -> %% Session is already invalidated {noreply, State}; - #session{} -> + #session{internal_id = InternalId} -> CacheCb:delete(Cache, Key), - {noreply, State} + {noreply, State#state{session_cache_client = Cache, + client_session_order = gb_trees:delete(InternalId, Order)}} end. clean_cert_db(Ref, CertDb, RefDb, FileMapDb, File) -> @@ -448,49 +449,61 @@ clean_cert_db(Ref, CertDb, RefDb, FileMapDb, File) -> end. client_register_unique_session(Host, Port, Session, #state{session_cache_client = Cache0, - session_cache_cb = CacheCb, + session_cache_client_cb = CacheCb, session_cache_client_max = Max, - options = Options} = State) -> + options = Options, + client_session_order = Order0} = State) -> TimeStamp = erlang:monotonic_time(), NewSession = Session#session{time_stamp = TimeStamp}, case CacheCb:select_session(Cache0, {Host, Port}) of no_session -> - Cache = do_register_session({{Host, Port}, - NewSession#session.session_id}, - NewSession, Max, Cache0, CacheCb, Options), - State#state{session_cache_client = Cache}; + {Cache, Order} = do_register_session({{Host, Port}, + NewSession#session.session_id}, + NewSession, Max, Cache0, CacheCb, Options, Order0), + State#state{session_cache_client = Cache, client_session_order = Order}; Sessions -> register_unique_session(Sessions, NewSession, {Host, Port}, State) end. client_register_session(Host, Port, Session, #state{session_cache_client = Cache0, - session_cache_cb = CacheCb, + session_cache_client_cb = CacheCb, session_cache_client_max = Max, - options = Options} = State) -> + options = Options, + client_session_order = Order0} = State) -> TimeStamp = erlang:monotonic_time(), NewSession = Session#session{time_stamp = TimeStamp}, - Cache = do_register_session({{Host, Port}, - NewSession#session.session_id}, - NewSession, Max, Cache0, CacheCb, Options), - State#state{session_cache_client = Cache}. - -do_register_session(Key, Session, Max, Cache0, CacheCb, Options) -> + SessionId = NewSession#session.session_id, + {Cache, Order} = do_register_session({{Host, Port}, SessionId}, + NewSession, Max, Cache0, CacheCb, Options, Order0), + State#state{session_cache_client = Cache, + client_session_order = Order}. + +do_register_session(Key, #session{time_stamp = TimeStamp} = Session0, + Max, Cache, CacheCb, Options, Order0) -> try - case CacheCb:size(Cache0) of + case CacheCb:size(Cache) of Max -> - {_, Cache} = CacheCb:take_oldest(Cache0), + InternalId = {TimeStamp, erlang:unique_integer([monotonic])}, + Session = Session0#session{internal_id = InternalId}, + {_, OldKey, Order1} = gb_trees:take_smallest(Order0), + Order = gb_trees:insert(InternalId, Key, Order1), + CacheCb:delete(Cache, OldKey), + CacheCb:update(Cache, Key, Session), + {Cache, Order}; + _ -> + InternalId = {TimeStamp, erlang:unique_integer([monotonic])}, + Session = Session0#session{internal_id = InternalId}, + Order = gb_trees:insert(InternalId, Key, Order0), CacheCb:update(Cache, Key, Session), - Cache; - _ -> - CacheCb:update(Cache0, Key, Session), - Cache0 + {Cache, Order} end catch _:_ -> + %% Backwards compatibility if size functions is not implemented by callback Args = proplists:get_value(session_cb_init_args, Options, []), - CacheCb:terminate(Cache0), - CacheCb:init(Args) + CacheCb:terminate(Cache), + {CacheCb:init(Args), gb_trees:empty()} end. @@ -499,16 +512,18 @@ do_register_session(Key, Session, Max, Cache0, CacheCb, Options) -> register_unique_session(Sessions, Session, PartialKey, #state{session_cache_client_max = Max, session_cache_client = Cache0, - session_cache_cb = CacheCb, - options = Options} = State) -> + session_cache_client_cb = CacheCb, + options = Options, + client_session_order = Order0} = State) -> case exists_equivalent(Session , Sessions) of true -> State; false -> - Cache = do_register_session({PartialKey, - Session#session.session_id}, - Session, Max, Cache0, CacheCb, Options), - State#state{session_cache_client = Cache} + {Cache, Order} = do_register_session({PartialKey, + Session#session.session_id}, + Session, Max, Cache0, CacheCb, Options, Order0), + State#state{session_cache_client = Cache, + client_session_order = Order} end. exists_equivalent(_, []) -> diff --git a/lib/ssl/src/ssl_record.erl b/lib/ssl/src/ssl_record.erl index 47a9f11829..040c4f1ebc 100644 --- a/lib/ssl/src/ssl_record.erl +++ b/lib/ssl/src/ssl_record.erl @@ -41,8 +41,12 @@ set_client_verify_data/3, set_server_verify_data/3, set_max_fragment_length/2, - empty_connection_state/2, initial_connection_state/2, record_protocol_role/1, - step_encryption_state/1]). + empty_connection_state/2, + empty_connection_state/3, + record_protocol_role/1, + step_encryption_state/1, + step_encryption_state_read/1, + step_encryption_state_write/1]). %% Compression -export([compress/3, uncompress/3, compressions/0]). @@ -138,6 +142,17 @@ step_encryption_state(#state{connection_states = ConnStates#{current_read => NewRead, current_write => NewWrite}}. +step_encryption_state_read(#state{connection_states = + #{pending_read := PendingRead} = ConnStates} = State) -> + NewRead = PendingRead#{sequence_number => 0}, + State#state{connection_states = + ConnStates#{current_read => NewRead}}. + +step_encryption_state_write(#state{connection_states = + #{pending_write := PendingWrite} = ConnStates} = State) -> + NewWrite = PendingWrite#{sequence_number => 0}, + State#state{connection_states = + ConnStates#{current_write => NewWrite}}. %%-------------------------------------------------------------------- -spec set_security_params(#security_parameters{}, #security_parameters{}, @@ -445,6 +460,10 @@ nonce_seed(_,_, CipherState) -> %%-------------------------------------------------------------------- empty_connection_state(ConnectionEnd, BeastMitigation) -> + MaxEarlyDataSize = ssl_config:get_max_early_data_size(), + empty_connection_state(ConnectionEnd, BeastMitigation, MaxEarlyDataSize). +%% +empty_connection_state(ConnectionEnd, BeastMitigation, MaxEarlyDataSize) -> SecParams = empty_security_params(ConnectionEnd), #{security_parameters => SecParams, beast_mitigation => BeastMitigation, @@ -454,7 +473,10 @@ empty_connection_state(ConnectionEnd, BeastMitigation) -> secure_renegotiation => undefined, client_verify_data => undefined, server_verify_data => undefined, - max_fragment_length => undefined + max_early_data_size => MaxEarlyDataSize, + max_fragment_length => undefined, + trial_decryption => false, + early_data_limit => false }. empty_security_params(ConnectionEnd = ?CLIENT) -> @@ -481,20 +503,6 @@ record_protocol_role(client) -> record_protocol_role(server) -> ?SERVER. -initial_connection_state(ConnectionEnd, BeastMitigation) -> - #{security_parameters => - initial_security_params(ConnectionEnd), - sequence_number => 0, - beast_mitigation => BeastMitigation, - compression_state => undefined, - cipher_state => undefined, - mac_secret => undefined, - secure_renegotiation => undefined, - client_verify_data => undefined, - server_verify_data => undefined, - max_fragment_length => undefined - }. - initial_security_params(ConnectionEnd) -> SecParams = #security_parameters{connection_end = ConnectionEnd, compression_algorithm = ?NULL}, diff --git a/lib/ssl/src/ssl_record.hrl b/lib/ssl/src/ssl_record.hrl index 9aa598daed..d142ecf6da 100644 --- a/lib/ssl/src/ssl_record.hrl +++ b/lib/ssl/src/ssl_record.hrl @@ -68,6 +68,7 @@ master_secret, % opaque 48 resumption_master_secret, application_traffic_secret, + client_early_data_secret, client_random, % opaque 32 server_random, % opaque 32 exportable % boolean diff --git a/lib/ssl/src/ssl_server_session_cache.erl b/lib/ssl/src/ssl_server_session_cache.erl index 44862e5cad..b444cf8865 100644 --- a/lib/ssl/src/ssl_server_session_cache.erl +++ b/lib/ssl/src/ssl_server_session_cache.erl @@ -36,7 +36,6 @@ new_session_id/1, register_session/2, reuse_session/2 - %%invalidate_session/2 ]). %% gen_server callbacks @@ -53,7 +52,7 @@ lifetime, db, max, - session_index, + session_order, id_generator, listner }). @@ -121,7 +120,7 @@ init([Listner, #{lifetime := Lifetime, lifetime = Lifetime, db = DbRef, max = Max, - session_index = #{}, + session_order = gb_trees:empty(), id_generator = crypto:strong_rand_bytes(16), listner = Monitor }, @@ -135,64 +134,59 @@ handle_call(new_session_id, _From, #state{id_generator = IdGen} = State) -> handle_call({reuse_session, SessionId}, _From, #state{store_cb = Cb, db = Store0, lifetime = Lifetime, - session_index = Index0} = State0) -> - case maps:get(SessionId, Index0, undefined) of + session_order = Order0} = State0) -> + case lookup(Cb, Store0, SessionId) of undefined -> {reply, not_reusable, State0}; - Key -> - case lookup(Cb, Store0, Key) of - undefined -> - {reply, not_reusable, State0}; - Session -> - case ssl_session:valid_session(Session, Lifetime) of - true -> - {reply, Session, State0}; - false -> - {Store, Index} = invalidate_session(Cb, Store0, Index0, SessionId, Key), - {reply, not_reusable, State0#state{db = Store, session_index = Index}} - end + #session{internal_id = InId} = Session -> + case ssl_session:valid_session(Session, Lifetime) of + true -> + {reply, Session, State0}; + false -> + Order = invalidate_session(Cb, Store0, Order0, SessionId, InId), + {reply, not_reusable, State0#state{session_order = Order}} end end. -spec handle_cast(Request :: term(), State :: term()) -> {noreply, NewState :: term()}. -handle_cast({register_session, #session{session_id = SessionId} = Session}, +handle_cast({register_session, #session{session_id = SessionId, time_stamp = TimeStamp} = Session0}, #state{store_cb = Cb, db = Store0, max = Max, lifetime = Lifetime, - session_index = Index0} + session_order = Order0} = State0) -> - TimeStamp = erlang:monotonic_time(), - Id = {TimeStamp, erlang:unique_integer([monotonic])}, + InternalId = {TimeStamp, erlang:unique_integer([monotonic])}, + Session = Session0#session{internal_id = InternalId}, State = case size(Cb, Store0) of Max -> - {#session{session_id = OldSessionId}, Store1} - = oldest_session(Cb, Store0), %% Throw away oldest session table may not grow larger than max - Store = update(Cb, Store1, Id, Session), - Index = maps:without([OldSessionId], Index0), - State0#state{db = Store, - session_index = Index#{SessionId => Id}}; + {_, OldSessId, Order1} = gb_trees:take_smallest(Order0), + Store1 = delete(Cb, Store0, OldSessId), + %% Insert new session + Order = gb_trees:insert(InternalId, SessionId, Order1), + Store = update(Cb, Store1, SessionId, Session), + Store#state{db = Store, session_order = Order}; Size when Size > 0 -> - {#session{session_id = OldSessionId} = OldestSession, Store1} - = oldest_session(Cb, Store0), + {_, OldSessId, Order1} = gb_trees:take_smallest(Order0), + OldestSession = lookup(Cb, Store0, OldSessId), case ssl_session:valid_session(OldestSession, Lifetime) of true -> - Store = update(Cb, Store0, Id, Session#session{time_stamp = TimeStamp}), + Store = update(Cb, Store0, SessionId, Session#session{time_stamp = TimeStamp}), State0#state{db = Store, - session_index = Index0#{SessionId => Id}}; + session_order = gb_trees:insert(InternalId, SessionId, Order0)}; false -> %% Throw away oldest session as it is not valid anymore - Store = update(Cb, Store1, Id, Session#session{time_stamp = TimeStamp}), - Index = maps:without([OldSessionId], Index0), + Store1 = delete(Cb, Store0, OldSessId), + Store = update(Cb, Store1, SessionId, Session#session{time_stamp = TimeStamp}), State0#state{db = Store, - session_index = Index#{SessionId => Id}} + session_order = gb_trees:insert(InternalId, SessionId, Order1)} end; 0 -> - Store = update(Cb, Store0, Id, Session#session{time_stamp = TimeStamp}), + Store = update(Cb, Store0, SessionId, Session#session{time_stamp = TimeStamp}), State0#state{db = Store, - session_index = Index0#{SessionId => Id}} + session_order = gb_trees:insert(InternalId, SessionId, Order0)} end, {noreply, State}. @@ -226,25 +220,9 @@ session_id(Key) -> Bin2 = crypto:crypto_one_time(aes_128_ecb, Key, <<Unique2:128>>, true), <<Bin1/binary, Bin2/binary>>. -invalidate_session(Cb, Store, Index, SessionId, Key) -> - {delete(Cb, Store, Key), - maps:without([SessionId], Index)}. - -oldest_session(Cb, Store0) -> - try Cb:take_oldest(Store0) of - {_, Element, Store} -> - {Element, Store} - catch - _:_ -> %% Backwards compatible - {Key, OldApiElement} = Cb:foldl(fun(E, []) -> - E; - (_, Acc)-> - Acc - end, [], - Store0), - OldAPIStore = delete(Cb, Store0, Key), - {OldApiElement, OldAPIStore} - end. +invalidate_session(Cb, Store, Order, SessionId, InternalId) -> + Cb:delete(Store, SessionId), + gb_trees:delete(InternalId, Order). init(Cb, Options) -> Cb:init(Options). diff --git a/lib/ssl/src/ssl_server_session_cache_db.erl b/lib/ssl/src/ssl_server_session_cache_db.erl index cab018c5c2..b55d2e306f 100644 --- a/lib/ssl/src/ssl_server_session_cache_db.erl +++ b/lib/ssl/src/ssl_server_session_cache_db.erl @@ -24,15 +24,15 @@ -module(ssl_server_session_cache_db). -%%-behaviour(ssl_session_cache_api). +-behaviour(ssl_session_cache_api). %% API -export([init/1, + terminate/1, lookup/2, update/3, delete/2, - size/1, - take_oldest/1]). + size/1]). %%%=================================================================== %%% API @@ -76,8 +76,5 @@ delete(Cache, Key) -> size(Cache) -> gb_trees:size(Cache). -%%-------------------------------------------------------------- -%% Description: Returns the oldest cache entry -%%-------------------------------------------------------------------- -take_oldest(Cache) -> - gb_trees:take_smallest(Cache). +terminate(_) -> + ok. diff --git a/lib/ssl/src/ssl_server_session_cache_sup.erl b/lib/ssl/src/ssl_server_session_cache_sup.erl index 2f0c3dc823..88f068a319 100644 --- a/lib/ssl/src/ssl_server_session_cache_sup.erl +++ b/lib/ssl/src/ssl_server_session_cache_sup.erl @@ -42,7 +42,7 @@ start_link() -> supervisor:start_link({local, ?MODULE}, ?MODULE, []). start_child(Listner) -> - supervisor:start_child(?MODULE, [Listner | ssl_config:pre_1_3_session_opts()]). + supervisor:start_child(?MODULE, [Listner | [ssl_config:pre_1_3_session_opts(server)]]). %%%========================================================================= diff --git a/lib/ssl/src/ssl_session_cache_api.erl b/lib/ssl/src/ssl_session_cache_api.erl index d438a9dafd..59fbd9b3c3 100644 --- a/lib/ssl/src/ssl_session_cache_api.erl +++ b/lib/ssl/src/ssl_session_cache_api.erl @@ -40,3 +40,5 @@ -callback foldl(fun(), term(), session_cache_ref()) -> term(). -callback select_session(session_cache_ref(), {ssl:host(), inet:port_number()} | inet:port_number()) -> [#session{}]. -callback size(session_cache_ref()) -> integer(). + +-optional_callbacks([select_session/2, foldl/3]). diff --git a/lib/ssl/src/ssl_upgrade_server_session_cache_sup.erl b/lib/ssl/src/ssl_upgrade_server_session_cache_sup.erl index 936ffcc0ac..69169cca0d 100644 --- a/lib/ssl/src/ssl_upgrade_server_session_cache_sup.erl +++ b/lib/ssl/src/ssl_upgrade_server_session_cache_sup.erl @@ -55,7 +55,7 @@ start_child(Type) -> %% only one will be able to grab the local name and we will use %% that process for handling pre TLS-1.3 sessions for %% servers with to us unknown listeners. - case supervisor:start_child(SupName, [ssl_unknown_listener | ssl_config:pre_1_3_session_opts()]) of + case supervisor:start_child(SupName, [ssl_unknown_listener, ssl_config:pre_1_3_session_opts(server)]) of {error, {already_started, Child}} -> {ok, Child}; {ok, _} = Return -> diff --git a/lib/ssl/src/tls_client_ticket_store.erl b/lib/ssl/src/tls_client_ticket_store.erl index 179c86ba8d..eb10adc9f1 100644 --- a/lib/ssl/src/tls_client_ticket_store.erl +++ b/lib/ssl/src/tls_client_ticket_store.erl @@ -25,10 +25,11 @@ -module(tls_client_ticket_store). -behaviour(gen_server). +-include("ssl_internal.hrl"). -include("tls_handshake_1_3.hrl"). %% API --export([find_ticket/3, +-export([find_ticket/5, get_tickets/2, lock_tickets/2, remove_tickets/1, @@ -51,7 +52,7 @@ -record(data, { pos = undefined, - hkdf, + cipher_suite, sni, psk, timestamp, @@ -69,8 +70,8 @@ start_link(Max, Lifetime) -> gen_server:start_link({local, ?MODULE}, ?MODULE, [Max, Lifetime], []). -find_ticket(Pid, HashAlgos, SNI) -> - gen_server:call(?MODULE, {find_ticket, Pid, HashAlgos, SNI}, infinity). +find_ticket(Pid, Ciphers, HashAlgos, SNI, EarlyDataSize) -> + gen_server:call(?MODULE, {find_ticket, Pid, Ciphers, HashAlgos, SNI, EarlyDataSize}, infinity). get_tickets(Pid, Keys) -> gen_server:call(?MODULE, {get_tickets, Pid, Keys}, infinity). @@ -85,8 +86,8 @@ remove_tickets([]) -> remove_tickets(Keys) -> gen_server:cast(?MODULE, {remove_tickets, Keys}). -store_ticket(Ticket, HKDF, SNI, PSK) -> - gen_server:call(?MODULE, {store_ticket, Ticket, HKDF, SNI, PSK}, infinity). +store_ticket(Ticket, CipherSuite, SNI, PSK) -> + gen_server:call(?MODULE, {store_ticket, Ticket, CipherSuite, SNI, PSK}, infinity). unlock_tickets(Pid, Keys) -> gen_server:call(?MODULE, {unlock, Pid, Keys}, infinity). @@ -107,8 +108,8 @@ init(Args) -> -spec handle_call(Request :: term(), From :: {pid(), term()}, State :: term()) -> {reply, Reply :: term(), NewState :: term()} . -handle_call({find_ticket, Pid, HashAlgos, SNI}, _From, State) -> - Key = do_find_ticket(State, Pid, HashAlgos, SNI), +handle_call({find_ticket, Pid, Ciphers, HashAlgos, SNI, EarlyDataSize}, _From, State) -> + Key = do_find_ticket(State, Pid, Ciphers, HashAlgos, SNI, EarlyDataSize), {reply, Key, State}; handle_call({get_tickets, Pid, Keys}, _From, State) -> Data = get_tickets(State, Pid, Keys), @@ -116,8 +117,8 @@ handle_call({get_tickets, Pid, Keys}, _From, State) -> handle_call({lock, Pid, Keys}, _From, State0) -> State = lock_tickets(State0, Pid, Keys), {reply, ok, State}; -handle_call({store_ticket, Ticket, HKDF, SNI, PSK}, _From, State0) -> - State = store_ticket(State0, Ticket, HKDF, SNI, PSK), +handle_call({store_ticket, Ticket, CipherSuite, SNI, PSK}, _From, State0) -> + State = store_ticket(State0, Ticket, CipherSuite, SNI, PSK), {reply, ok, State}; handle_call({unlock, Pid, Keys}, _From, State0) -> State = unlock_tickets(State0, Pid, Keys), @@ -170,42 +171,78 @@ inital_state([Max, Lifetime]) -> max = Max }. - -do_find_ticket(_, _, [], _) -> - undefined; +do_find_ticket(Iter, Pid, Ciphers, HashAlgos, SNI, EarlyDataSize) -> + do_find_ticket(Iter, Pid, Ciphers, HashAlgos, SNI, EarlyDataSize, []). +%% +do_find_ticket(_, _, _, [], _, _, []) -> + {undefined, undefined}; +do_find_ticket(_, _, _, [], _, _, Acc) -> + {undefined, last_elem(Acc)}; do_find_ticket(#state{db = Db, - lifetime = Lifetime} = State, Pid, [Hash|T], SNI) -> - case iterate_tickets(gb_trees:iterator(Db), Pid, Hash, SNI, Lifetime) of - none -> - do_find_ticket(State, Pid, T, SNI); + lifetime = Lifetime} = State, Pid, Ciphers, [Hash|T], SNI, EarlyDataSize, Acc) -> + case iterate_tickets(gb_trees:iterator(Db), Pid, Ciphers, Hash, SNI, Lifetime, EarlyDataSize) of + {undefined, undefined} -> + do_find_ticket(State, Pid, Ciphers, T, SNI, EarlyDataSize, Acc); + {undefined, Key} -> + do_find_ticket(State, Pid, Ciphers, T, SNI, EarlyDataSize, [Key|Acc]); Key -> Key end. -iterate_tickets(Iter0, Pid, Hash, SNI, Lifetime) -> +iterate_tickets(Iter0, Pid, Ciphers, Hash, SNI, Lifetime, EarlyDataSize) -> + iterate_tickets(Iter0, Pid, Ciphers, Hash, SNI, Lifetime, EarlyDataSize, []). +%% +iterate_tickets(Iter0, Pid, Ciphers, Hash, SNI, Lifetime, EarlyDataSize, Acc) -> case gb_trees:next(Iter0) of - {Key, #data{hkdf = Hash, + {Key, #data{cipher_suite = {Cipher, Hash}, sni = TicketSNI, + ticket = #new_session_ticket{ + extensions = Extensions}, timestamp = Timestamp, lock = Lock}, Iter} when Lock =:= undefined orelse Lock =:= Pid -> + MaxEarlyData = tls_handshake_1_3:get_max_early_data(Extensions), Age = erlang:system_time(seconds) - Timestamp, if Age < Lifetime -> case verify_ticket_sni(SNI, TicketSNI) of match -> - Key; + case lists:member(Cipher, Ciphers) of + true -> + Front = last_elem(Acc), + %% 'Key' can be used with early_data as both + %% block cipher and hash algorithm matches. + %% 'Front' can only be used for session + %% resumption. + case EarlyDataSize =:= undefined orelse + EarlyDataSize =< MaxEarlyData of + true -> + {Key, Front}; + false -> + %% 'Key' cannot be used for early_data as the data + %% to be sent exceeds the max limit for this ticket. + iterate_tickets(Iter, Pid, Ciphers, Hash, SNI, + Lifetime, EarlyDataSize,[Key|Acc]) + end; + false -> + iterate_tickets(Iter, Pid, Ciphers, Hash, SNI, Lifetime, EarlyDataSize, [Key|Acc]) + end; nomatch -> - iterate_tickets(Iter, Pid, Hash, SNI, Lifetime) + iterate_tickets(Iter, Pid, Ciphers, Hash, SNI, Lifetime, EarlyDataSize, Acc) end; true -> - iterate_tickets(Iter, Pid, Hash, SNI, Lifetime) + iterate_tickets(Iter, Pid, Ciphers, Hash, SNI, Lifetime, EarlyDataSize, Acc) end; {_, _, Iter} -> - iterate_tickets(Iter, Pid, Hash, SNI, Lifetime); + iterate_tickets(Iter, Pid, Ciphers, Hash, SNI, Lifetime, EarlyDataSize, Acc); none -> - none + {undefined, last_elem(Acc)} end. +last_elem([_|_] = L) -> + lists:last(L); +last_elem([]) -> + undefined. + verify_ticket_sni(undefined, _) -> match; verify_ticket_sni(SNI, SNI) -> @@ -224,7 +261,7 @@ get_tickets(_, _, [], Acc) -> get_tickets(#state{db = Db} = State, Pid, [Key|T], Acc) -> try gb_trees:get(Key, Db) of #data{pos = Pos, - hkdf = HKDF, + cipher_suite = CipherSuite, psk = PSK, timestamp = Timestamp, ticket = NewSessionTicket, @@ -235,14 +272,23 @@ get_tickets(#state{db = Db} = State, Pid, [Key|T], Acc) -> ticket_age_add = AgeAdd, ticket_nonce = Nonce, ticket = Ticket, - extensions = _Extensions + extensions = Extensions } = NewSessionTicket, TicketAge = erlang:system_time(seconds) - Timestamp, ObfuscatedTicketAge = obfuscate_ticket_age(TicketAge, AgeAdd), Identity = #psk_identity{ identity = Ticket, obfuscated_ticket_age = ObfuscatedTicketAge}, - get_tickets(State, Pid, T, [{Key, Pos, Identity, PSK, Nonce, HKDF}|Acc]) + MaxEarlyData = tls_handshake_1_3:get_max_early_data(Extensions), + TicketData = #ticket_data{ + key = Key, + pos = Pos, + identity = Identity, + psk = PSK, + nonce = Nonce, + cipher_suite = CipherSuite, + max_size = MaxEarlyData}, + get_tickets(State, Pid, T, [TicketData|Acc]) catch _:_ -> get_tickets(State, Pid, T, Acc) @@ -296,7 +342,7 @@ collect_invalid_tickets(Iter0, Lifetime, Acc) -> end. -store_ticket(#state{db = Db0, max = Max} = State, Ticket, HKDF, SNI, PSK) -> +store_ticket(#state{db = Db0, max = Max} = State, Ticket, CipherSuite, SNI, PSK) -> Timestamp = erlang:system_time(seconds), Size = gb_trees:size(Db0), Db1 = if Size =:= Max -> @@ -306,7 +352,7 @@ store_ticket(#state{db = Db0, max = Max} = State, Ticket, HKDF, SNI, PSK) -> end, Key = {erlang:monotonic_time(), erlang:unique_integer([monotonic])}, Db = gb_trees:insert(Key, - #data{hkdf = HKDF, + #data{cipher_suite = CipherSuite, sni = SNI, psk = PSK, timestamp = Timestamp, diff --git a/lib/ssl/src/tls_connection.erl b/lib/ssl/src/tls_connection.erl index e96e0e2cf6..8f25e5a3cd 100644 --- a/lib/ssl/src/tls_connection.erl +++ b/lib/ssl/src/tls_connection.erl @@ -452,17 +452,12 @@ initial_state(Role, Sender, Host, Port, Socket, {SSLOptions, SocketOptions, Trac erl_dist := IsErlDist, client_renegotiation := ClientRenegotiation} = SSLOptions, ConnectionStates = tls_record:init_connection_states(Role, BeastMitigation), - SessionCacheCb = case application:get_env(ssl, session_cb) of - {ok, Cb} when is_atom(Cb) -> - Cb; - _ -> - ssl_session_cache - end, - InternalActiveN = case application:get_env(ssl, internal_active_n) of - {ok, N} when is_integer(N) andalso (not IsErlDist) -> - N; - _ -> - ?INTERNAL_ACTIVE_N + #{session_cb := SessionCacheCb} = ssl_config:pre_1_3_session_opts(Role), + InternalActiveN = case IsErlDist of + true -> + ?INTERNAL_ACTIVE_N; + false -> + ssl_config:get_internal_active_n() end, UserMonitor = erlang:monitor(process, User), InitStatEnv = #static_env{ diff --git a/lib/ssl/src/tls_connection_1_3.erl b/lib/ssl/src/tls_connection_1_3.erl index fc4b4f673f..6c81933c23 100644 --- a/lib/ssl/src/tls_connection_1_3.erl +++ b/lib/ssl/src/tls_connection_1_3.erl @@ -132,6 +132,7 @@ wait_sh/3, wait_ee/3, wait_cert_cr/3, + wait_eoed/3, connection/3, downgrade/3 ]). @@ -168,8 +169,8 @@ update_cipher_key(ConnStateName, CS0) -> ApplicationTrafficSecret = tls_v1:update_traffic_secret(HKDF, ApplicationTrafficSecret0), %% Calculate traffic keys - #{cipher := Cipher} = ssl_cipher_format:suite_bin_to_map(CipherSuite), - {Key, IV} = tls_v1:calculate_traffic_keys(HKDF, Cipher, ApplicationTrafficSecret), + KeyLength = tls_v1:key_length(CipherSuite), + {Key, IV} = tls_v1:calculate_traffic_keys(HKDF, KeyLength, ApplicationTrafficSecret), SecParams = SecParams0#security_parameters{application_traffic_secret = ApplicationTrafficSecret}, CipherState = CipherState0#cipher_state{key = Key, iv = IV}, @@ -307,8 +308,8 @@ negotiated(internal, Message, State0) -> negotiated(info, Msg, State) -> tls_gen_connection:handle_info(Msg, ?FUNCTION_NAME, State). -wait_cert(internal, #change_cipher_spec{}, State) -> - tls_gen_connection:next_event(?FUNCTION_NAME, no_record, State); +wait_cert(internal, #change_cipher_spec{}, State0) -> + tls_gen_connection:next_event(?FUNCTION_NAME, no_record, State0); wait_cert(internal, #certificate_1_3{} = Certificate, State0) -> case tls_handshake_1_3:do_wait_cert(Certificate, State0) of @@ -337,9 +338,8 @@ wait_cv(info, Msg, State) -> wait_cv(Type, Msg, State) -> ssl_gen_statem:handle_common_event(Type, Msg, ?FUNCTION_NAME, State). - -wait_finished(internal, #change_cipher_spec{}, State) -> - tls_gen_connection:next_event(?FUNCTION_NAME, no_record, State); +wait_finished(internal, #change_cipher_spec{}, State0) -> + tls_gen_connection:next_event(?FUNCTION_NAME, no_record, State0); wait_finished(internal, #finished{} = Finished, State0) -> case tls_handshake_1_3:do_wait_finished(Finished, State0) of @@ -417,6 +417,20 @@ wait_cert_cr(info, Msg, State) -> wait_cert_cr(Type, Msg, State) -> ssl_gen_statem:handle_common_event(Type, Msg, ?FUNCTION_NAME, State). +wait_eoed(internal, #change_cipher_spec{}, State) -> + tls_gen_connection:next_event(?FUNCTION_NAME, no_record, State); +wait_eoed(internal, #end_of_early_data{} = EOED, State0) -> + case tls_handshake_1_3:do_wait_eoed(EOED, State0) of + {#alert{} = Alert, State} -> + ssl_gen_statem:handle_own_alert(Alert, {3,4}, wait_eoed, State); + {State1, NextState} -> + tls_gen_connection:next_event(NextState, no_record, State1) + end; +wait_eoed(info, Msg, State) -> + tls_gen_connection:handle_info(Msg, ?FUNCTION_NAME, State); +wait_eoed(Type, Msg, State) -> + ssl_gen_statem:handle_common_event(Type, Msg, ?FUNCTION_NAME, State). + connection(internal, #new_session_ticket{} = NewSessionTicket, State) -> handle_new_session_ticket(NewSessionTicket, State), tls_gen_connection:next_event(?FUNCTION_NAME, no_record, State); @@ -450,7 +464,8 @@ initial_state(Role, Sender, Host, Port, Socket, {SSLOptions, SocketOptions, Trac {CbModule, DataTag, CloseTag, ErrorTag, PassiveTag}) -> #{erl_dist := IsErlDist, client_renegotiation := ClientRenegotiation} = SSLOptions, - ConnectionStates = tls_record:init_connection_states(Role, disabled), + MaxEarlyDataSize = init_max_early_data_size(Role), + ConnectionStates = tls_record:init_connection_states(Role, disabled, MaxEarlyDataSize), InternalActiveN = case application:get_env(ssl, internal_active_n) of {ok, N} when is_integer(N) andalso (not IsErlDist) -> N; @@ -504,10 +519,12 @@ handle_new_session_ticket(#new_session_ticket{ticket_nonce = Nonce} = NewSession when SessionTickets =:= manual -> #{security_parameters := SecParams} = ssl_record:current_connection_state(ConnectionStates, read), + CipherSuite = SecParams#security_parameters.cipher_suite, + #{cipher := Cipher} = ssl_cipher_format:suite_bin_to_map(CipherSuite), HKDF = SecParams#security_parameters.prf_algorithm, RMS = SecParams#security_parameters.resumption_master_secret, PSK = tls_v1:pre_shared_key(RMS, Nonce, HKDF), - send_ticket_data(User, NewSessionTicket, HKDF, SNI, PSK); + send_ticket_data(User, NewSessionTicket, {Cipher, HKDF}, SNI, PSK); handle_new_session_ticket(#new_session_ticket{ticket_nonce = Nonce} = NewSessionTicket, #state{connection_states = ConnectionStates, ssl_options = #{session_tickets := SessionTickets, @@ -515,21 +532,21 @@ handle_new_session_ticket(#new_session_ticket{ticket_nonce = Nonce} = NewSession when SessionTickets =:= auto -> #{security_parameters := SecParams} = ssl_record:current_connection_state(ConnectionStates, read), + CipherSuite = SecParams#security_parameters.cipher_suite, + #{cipher := Cipher} = ssl_cipher_format:suite_bin_to_map(CipherSuite), HKDF = SecParams#security_parameters.prf_algorithm, RMS = SecParams#security_parameters.resumption_master_secret, PSK = tls_v1:pre_shared_key(RMS, Nonce, HKDF), - tls_client_ticket_store:store_ticket(NewSessionTicket, HKDF, SNI, PSK). - + tls_client_ticket_store:store_ticket(NewSessionTicket, {Cipher, HKDF}, SNI, PSK). -%% Send ticket data to user as opaque binary -send_ticket_data(User, NewSessionTicket, HKDF, SNI, PSK) -> +send_ticket_data(User, NewSessionTicket, CipherSuite, SNI, PSK) -> Timestamp = erlang:system_time(seconds), - TicketData = #{hkdf => HKDF, + TicketData = #{cipher_suite => CipherSuite, sni => SNI, psk => PSK, timestamp => Timestamp, ticket => NewSessionTicket}, - User ! {ssl, session_ticket, {SNI, erlang:term_to_binary(TicketData)}}. + User ! {ssl, session_ticket, TicketData}. handle_key_update(#key_update{request_update = update_not_requested}, State0) -> %% Update read key in connection @@ -545,3 +562,12 @@ handle_key_update(#key_update{request_update = update_requested}, {error, Reason} -> {error, State1, ?ALERT_REC(?FATAL, ?INTERNAL_ERROR, Reason)} end. + +init_max_early_data_size(client) -> + %% Disable trial decryption on the client side + %% Servers do trial decryption of max_early_data bytes of plain text. + %% Setting it to 0 means that a decryption error will result in an Alert. + 0; +init_max_early_data_size(server) -> + ssl_config:get_max_early_data_size(). + diff --git a/lib/ssl/src/tls_gen_connection.erl b/lib/ssl/src/tls_gen_connection.erl index 7c16bfa3de..5da87e79d6 100644 --- a/lib/ssl/src/tls_gen_connection.erl +++ b/lib/ssl/src/tls_gen_connection.erl @@ -659,6 +659,17 @@ next_record(#state{connection_env = #connection_env{negotiated_version = {3,4} = [_|_] -> next_record(State, CipherTexts, ConnectionStates, Check, [Fragment|Acc]) end; + {trial_decryption_failed, ConnectionStates} -> + case CipherTexts of + [] -> + %% End of cipher texts - build and deliver an ?APPLICATION_DATA record + %% from the accumulated fragments + next_record_done(State, [], ConnectionStates, + #ssl_tls{type = ?APPLICATION_DATA, + fragment = iolist_to_binary(lists:reverse(Acc))}); + [_|_] -> + next_record(State, CipherTexts, ConnectionStates, Check, Acc) + end; {Record, ConnectionStates} when Acc =:= [] -> %% Singelton non-?APPLICATION_DATA record - deliver next_record_done(State, CipherTexts, ConnectionStates, Record); diff --git a/lib/ssl/src/tls_handshake_1_3.erl b/lib/ssl/src/tls_handshake_1_3.erl index dbde7ad476..cd9beecb3f 100644 --- a/lib/ssl/src/tls_handshake_1_3.erl +++ b/lib/ssl/src/tls_handshake_1_3.erl @@ -51,12 +51,19 @@ do_wait_sh/2, do_wait_ee/2, do_wait_cert_cr/2, + do_wait_eoed/2, + early_data_size/1, get_ticket_data/3, maybe_add_binders/3, maybe_add_binders/4, - maybe_automatic_session_resumption/1]). + maybe_add_early_data_indication/3, + maybe_automatic_session_resumption/1, + maybe_send_early_data/1, + update_current_read/3]). --export([is_valid_binder/4]). +-export([get_max_early_data/1, + is_valid_binder/4, + maybe/0]). %% crypto:hash(sha256, "HelloRetryRequest"). -define(HELLO_RETRY_REQUEST_RANDOM, <<207,33,173,116,229,154,97,17, @@ -178,12 +185,18 @@ encrypted_extensions(#state{handshake_env = HandshakeEnv}) -> MaxFragEnum -> E1#{max_frag_enum => MaxFragEnum} end, - E = case HandshakeEnv#handshake_env.sni_guided_cert_selection of + E3 = case HandshakeEnv#handshake_env.sni_guided_cert_selection of false -> E2; true -> E2#{sni => #sni{hostname = ""}} end, + E = case HandshakeEnv#handshake_env.early_data_accepted of + false -> + E3; + true -> + E3#{early_data => #early_data_indication{}} + end, #encrypted_extensions{ extensions = E }. @@ -595,9 +608,11 @@ do_start(#client_hello{cipher_suites = ClientCiphers, supported_groups := ServerGroups0, alpn_preferred_protocols := ALPNPreferredProtocols, keep_secrets := KeepSecrets, - honor_cipher_order := HonorCipherOrder}} = State0) -> + honor_cipher_order := HonorCipherOrder, + early_data := EarlyDataEnabled}} = State0) -> SNI = maps:get(sni, Extensions, undefined), ClientGroups0 = maps:get(elliptic_curves, Extensions, undefined), + EarlyDataIndication = maps:get(early_data, Extensions, undefined), {Ref,Maybe} = maybe(), try ClientGroups = Maybe(get_supported_groups(ClientGroups0)), @@ -618,7 +633,7 @@ do_start(#client_hello{cipher_suites = ClientCiphers, CookieExt = maps:get(cookie, Extensions, undefined), Cookie = get_cookie(CookieExt), - + #state{connection_states = ConnectionStates0, session = #session{own_certificates = [Cert | _]}} = State1 = Maybe(ssl_gen_statem:handle_sni_extension(SNI, State0)), @@ -668,14 +683,14 @@ do_start(#client_hello{cipher_suites = ClientCiphers, State2 end, - State = update_start_state(State3, - #{cipher => Cipher, - key_share => KeyShare, - session_id => SessionId, - group => Group, - sign_alg => SelectedSignAlg, - peer_public_key => ClientPubKey, - alpn => ALPNProtocol}), + State4 = update_start_state(State3, + #{cipher => Cipher, + key_share => KeyShare, + session_id => SessionId, + group => Group, + sign_alg => SelectedSignAlg, + peer_public_key => ClientPubKey, + alpn => ALPNProtocol}), %% 4.1.4. Hello Retry Request %% @@ -683,13 +698,15 @@ do_start(#client_hello{cipher_suites = ClientCiphers, %% message if it is able to find an acceptable set of parameters but the %% ClientHello does not contain sufficient information to proceed with %% the handshake. - case Maybe(send_hello_retry_request(State, ClientPubKey, KeyShare, SessionId)) of + case Maybe(send_hello_retry_request(State4, ClientPubKey, KeyShare, SessionId)) of {_, start} = NextStateTuple -> NextStateTuple; - {_, negotiated} = NextStateTuple -> + {State5, negotiated} -> + %% Determine if early data is accepted + State = handle_early_data(State5, EarlyDataEnabled, EarlyDataIndication), %% Exclude any incompatible PSKs. PSK = Maybe(handle_pre_shared_key(State, OfferedPSKs, Cipher)), - Maybe(session_resumption(NextStateTuple, PSK)) + Maybe(session_resumption({State, negotiated}, PSK)) end catch {Ref, #alert{} = Alert} -> @@ -790,6 +807,9 @@ do_start(#server_hello{cipher_suite = SelectedCipherSuite, do_negotiated({start_handshake, PSK0}, #state{connection_states = ConnectionStates0, + handshake_env = + #handshake_env{ + early_data_accepted = EarlyDataAccepted}, static_env = #static_env{protocol_cb = Connection}, session = #session{session_id = SessionId, ecc = SelectedGroup, @@ -802,7 +822,6 @@ do_negotiated({start_handshake, PSK0}, ssl_record:pending_connection_state(ConnectionStates0, read), #security_parameters{prf_algorithm = HKDF} = SecParamsR, - {Ref,Maybe} = maybe(), try %% Create server_hello @@ -810,39 +829,52 @@ do_negotiated({start_handshake, PSK0}, State1 = Connection:queue_handshake(ServerHello, State0), %% D.4. Middlebox Compatibility Mode State2 = maybe_queue_change_cipher_spec(State1, last), - {State3, _} = Connection:send_handshake_flight(State2), PSK = get_pre_shared_key(PSK0, HKDF), - State4 = + State3 = calculate_handshake_secrets(ClientPublicKey, ServerPrivateKey, SelectedGroup, - PSK, State3), + PSK, State2), - State5 = ssl_record:step_encryption_state(State4), + %% Step only write state if early_data is accepted + State4 = + case EarlyDataAccepted of + true -> + ssl_record:step_encryption_state_write(State3); + false -> + %% Read state is overwritten when hanshake secrets are set. + %% Trial_decryption and early_data_limit must be set here! + update_current_read( + ssl_record:step_encryption_state(State3), + true, %% trial_decryption + false %% early data limit + ) + + end, %% Create EncryptedExtensions - EncryptedExtensions = encrypted_extensions(State5), + EncryptedExtensions = encrypted_extensions(State4), %% Encode EncryptedExtensions - State6 = Connection:queue_handshake(EncryptedExtensions, State5), + State5 = Connection:queue_handshake(EncryptedExtensions, State4), %% Create and send CertificateRequest ({verify, verify_peer}) - {State7, NextState} = maybe_send_certificate_request(State6, SslOpts, PSK0), + {State6, NextState} = maybe_send_certificate_request(State5, SslOpts, PSK0), %% Create and send Certificate (if PSK is undefined) - State8 = Maybe(maybe_send_certificate(State7, PSK0)), + State7 = Maybe(maybe_send_certificate(State6, PSK0)), %% Create and send CertificateVerify (if PSK is undefined) - State9 = Maybe(maybe_send_certificate_verify(State8, PSK0)), + State8 = Maybe(maybe_send_certificate_verify(State7, PSK0)), %% Create Finished - Finished = finished(State9), + Finished = finished(State8), %% Encode Finished - State10= Connection:queue_handshake(Finished, State9), + State9 = Connection:queue_handshake(Finished, State8), %% Send first flight - {State, _} = Connection:send_handshake_flight(State10), + {State, _} = Connection:send_handshake_flight(State9), {State, NextState} @@ -864,10 +896,15 @@ do_wait_cert(#certificate_1_3{} = Certificate, State0) -> end. -do_wait_cv(#certificate_verify_1_3{} = CertificateVerify, State0) -> +do_wait_cv(#certificate_verify_1_3{} = CertificateVerify, #state{static_env = #static_env{role = Role}} = State0) -> {Ref,Maybe} = maybe(), try - State1 = Maybe(verify_signature_algorithm(State0, CertificateVerify)), + State1 = case Role of + server -> + Maybe(verify_signature_algorithm(State0, CertificateVerify)); + client -> + State0 + end, Maybe(verify_certificate_verify(State1, CertificateVerify)) catch {Ref, {#alert{} = Alert, State}} -> @@ -907,18 +944,20 @@ do_wait_finished(#finished{verify_data = VerifyData}, Maybe(validate_finished(State0, VerifyData)), %% D.4. Middlebox Compatibility Mode State1 = maybe_queue_change_cipher_spec(State0, first), + %% Signal change of cipher + State2 = maybe_send_end_of_early_data(State1), %% Maybe send Certificate + CertificateVerify - State2 = Maybe(maybe_queue_cert_cert_cv(State1)), - Finished = finished(State2), + State3 = Maybe(maybe_queue_cert_cert_cv(State2)), + Finished = finished(State3), %% Encode Finished - State3 = Connection:queue_handshake(Finished, State2), + State4 = Connection:queue_handshake(Finished, State3), %% Send first flight - {State4, _} = Connection:send_handshake_flight(State3), - State5 = calculate_traffic_secrets(State4), - State6 = maybe_calculate_resumption_master_secret(State5), - State7 = forget_master_secret(State6), + {State5, _} = Connection:send_handshake_flight(State4), + State6 = calculate_traffic_secrets(State5), + State7 = maybe_calculate_resumption_master_secret(State6), + State8 = forget_master_secret(State7), %% Configure traffic keys - ssl_record:step_encryption_state(State7) + ssl_record:step_encryption_state(State8) catch {Ref, #alert{} = Alert} -> Alert @@ -973,8 +1012,8 @@ do_wait_sh(#server_hello{cipher_suite = SelectedCipherSuite, PSK = Maybe(get_pre_shared_key(SessionTickets, UseTicket, HKDFAlgo, SelectedIdentity)), State3 = calculate_handshake_secrets(ServerPublicKey, ClientPrivateKey, SelectedGroup, PSK, State2), - State4 = ssl_record:step_encryption_state(State3), - + %% State4 = ssl_record:step_encryption_state(State3), + State4 = ssl_record:step_encryption_state_read(State3), {State4, wait_ee} catch @@ -989,6 +1028,7 @@ do_wait_ee(#encrypted_extensions{extensions = Extensions}, State0) -> ALPNProtocol0 = maps:get(alpn, Extensions, undefined), ALPNProtocol = get_alpn(ALPNProtocol0), + EarlyDataIndication = maps:get(early_data, Extensions, undefined), {Ref, Maybe} = maybe(), @@ -996,14 +1036,17 @@ do_wait_ee(#encrypted_extensions{extensions = Extensions}, State0) -> %% RFC 6066: handle received/expected maximum fragment length Maybe(maybe_max_fragment_length(Extensions, State0)), + %% Check if early_data is accepted/rejected + State1 = maybe_check_early_data_indication(EarlyDataIndication, State0), + %% Go to state 'wait_finished' if using PSK. - Maybe(maybe_resumption(State0)), + Maybe(maybe_resumption(State1)), %% Update state - #state{handshake_env = HsEnv} = State0, - State1 = State0#state{handshake_env = HsEnv#handshake_env{alpn = ALPNProtocol}}, + #state{handshake_env = HsEnv} = State1, + State2 = State1#state{handshake_env = HsEnv#handshake_env{alpn = ALPNProtocol}}, - {State1, wait_cert_cr} + {State2, wait_cert_cr} catch {Ref, {State, StateName}} -> {State, StateName}; @@ -1032,6 +1075,25 @@ do_wait_cert_cr(#certificate_request_1_3{} = CertificateRequest, State0) -> end. +do_wait_eoed(#end_of_early_data{}, State0) -> + {Ref,_Maybe} = maybe(), + try + %% Step read state to enable reading handshake messages from the client. + %% Write state is already stepped in state 'negotiated'. + State1 = ssl_record:step_encryption_state_read(State0), + + %% Early data has been received, no more early data is expected. + HsEnv = (State1#state.handshake_env)#handshake_env{early_data_accepted = false}, + State2 = State1#state{handshake_env = HsEnv}, + {State2, wait_finished} + catch + {Ref, #alert{} = Alert} -> + {Alert, State0}; + {Ref, {#alert{} = Alert, State}} -> + {Alert, State} + end. + + %% For reasons of backward compatibility with middleboxes (see %% Appendix D.4), the HelloRetryRequest message uses the same structure %% as the ServerHello, but with Random set to the special value of the @@ -1225,12 +1287,33 @@ session_resumption({#state{ssl_options = #{session_tickets := disabled}} = State session_resumption({#state{ssl_options = #{session_tickets := Tickets}} = State, negotiated}, undefined) when Tickets =/= disabled -> {ok, {State, negotiated}}; -session_resumption({#state{ssl_options = #{session_tickets := Tickets}} = State0, negotiated}, PSK) +session_resumption({#state{ssl_options = #{session_tickets := Tickets}, + handshake_env = #handshake_env{ + early_data_accepted = false}} = State0, negotiated}, PSK) when Tickets =/= disabled -> State = handle_resumption(State0, ok), - {ok, {State, negotiated, PSK}}. - - + {ok, {State, negotiated, PSK}}; +session_resumption({#state{ssl_options = #{session_tickets := Tickets}, + handshake_env = #handshake_env{ + early_data_accepted = true}} = State0, negotiated}, PSK0) + when Tickets =/= disabled -> + State1 = handle_resumption(State0, ok), + %% TODO Refactor PSK-tuple {Index, PSK}, index might not be needed. + {_ , PSK} = PSK0, + State2 = calculate_client_early_traffic_secret(State1, PSK), + %% Set 0-RTT traffic keys for reading early_data + State3 = ssl_record:step_encryption_state_read(State2), + State = update_current_read(State3, true, true), + {ok, {State, negotiated, PSK0}}. + +%% Session resumption with early_data +maybe_send_certificate_request(#state{ + handshake_env = + #handshake_env{ + early_data_accepted = true}} = State, + _, PSK) when PSK =/= undefined -> + %% Go wait for End of Early Data + {State, wait_eoed}; %% Do not send CR during session resumption maybe_send_certificate_request(State, _, PSK) when PSK =/= undefined -> {State, wait_finished}; @@ -1325,27 +1408,34 @@ process_certificate_request(#certificate_request_1_3{}, {ok, {State#state{client_certificate_requested = true}, wait_cert}}; process_certificate_request(#certificate_request_1_3{ - extensions = Extensions}, - #state{session = #session{own_certificates = [Cert|_]} = Session} = + extensions = Extensions}, + #state{ssl_options = #{signature_algs := ClientSignAlgs}, + session = #session{own_certificates = [Cert|_]} = Session} = State) -> ServerSignAlgs = get_signature_scheme_list( maps:get(signature_algs, Extensions, undefined)), ServerSignAlgsCert = get_signature_scheme_list( maps:get(signature_algs_cert, Extensions, undefined)), - {_PublicKeyAlgo, SignAlgo, SignHash, _} = get_certificate_params(Cert), - - %% Check if server supports signature algorithm of client certificate - case check_cert_sign_algo(SignAlgo, SignHash, ServerSignAlgs, ServerSignAlgsCert) of - ok -> - {ok, {State#state{client_certificate_requested = true}, wait_cert}}; - {error, _} -> - %% Certificate not supported: send empty certificate in state 'wait_finished' + {PublicKeyAlgo, SignAlgo, SignHash, MaybeRSAKeySize} = get_certificate_params(Cert), + {Ref, Maybe} = maybe(), + try + SelectedSignAlg = Maybe(select_sign_algo(PublicKeyAlgo, MaybeRSAKeySize, ServerSignAlgs, ClientSignAlgs)), + %% Check if server supports signature algorithm of client certificate + case check_cert_sign_algo(SignAlgo, SignHash, ServerSignAlgs, ServerSignAlgsCert) of + ok -> {ok, {State#state{client_certificate_requested = true, - session = Session#session{own_certificates = undefined}}, wait_cert}} + session = Session#session{sign_alg = SelectedSignAlg}}, wait_cert}}; + {error, _} -> + %% Certificate not supported: send empty certificate in state 'wait_finished' + {ok, {State#state{client_certificate_requested = true, + session = Session#session{own_certificates = undefined}}, wait_cert}} + end + catch + {Ref, #alert{} = Alert} -> + Alert end. - process_certificate(#certificate_1_3{ certificate_request_context = <<>>, certificate_list = []}, @@ -1508,16 +1598,15 @@ calculate_handshake_secrets(PublicKey, PrivateKey, SelectedGroup, PSK, %% Calculate [sender]_handshake_traffic_secret {Messages, _} = HHistory, - ClientHSTrafficSecret = tls_v1:client_handshake_traffic_secret(HKDFAlgo, HandshakeSecret, lists:reverse(Messages)), ServerHSTrafficSecret = tls_v1:server_handshake_traffic_secret(HKDFAlgo, HandshakeSecret, lists:reverse(Messages)), %% Calculate traffic keys - #{cipher := Cipher} = ssl_cipher_format:suite_bin_to_map(CipherSuite), - {ReadKey, ReadIV} = tls_v1:calculate_traffic_keys(HKDFAlgo, Cipher, ClientHSTrafficSecret), - {WriteKey, WriteIV} = tls_v1:calculate_traffic_keys(HKDFAlgo, Cipher, ServerHSTrafficSecret), + KeyLength = tls_v1:key_length(CipherSuite), + {ReadKey, ReadIV} = tls_v1:calculate_traffic_keys(HKDFAlgo, KeyLength, ClientHSTrafficSecret), + {WriteKey, WriteIV} = tls_v1:calculate_traffic_keys(HKDFAlgo, KeyLength, ServerHSTrafficSecret), %% Calculate Finished Keys ReadFinishedKey = tls_v1:finished_key(ClientHSTrafficSecret, HKDFAlgo), @@ -1530,6 +1619,67 @@ calculate_handshake_secrets(PublicKey, PrivateKey, SelectedGroup, PSK, ReadKey, ReadIV, ReadFinishedKey, WriteKey, WriteIV, WriteFinishedKey). +%% Server +calculate_client_early_traffic_secret(#state{connection_states = ConnectionStates, + handshake_env = + #handshake_env{ + tls_handshake_history = {Hist, _}}} = State, PSK) -> + + #{security_parameters := SecParamsR} = + ssl_record:pending_connection_state(ConnectionStates, read), + #security_parameters{cipher_suite = CipherSuite} = SecParamsR, + #{cipher := Cipher, + prf := HKDF} = ssl_cipher_format:suite_bin_to_map(CipherSuite), + calculate_client_early_traffic_secret(Hist, PSK, Cipher, HKDF, State). + +%% Client +calculate_client_early_traffic_secret( + ClientHello, PSK, Cipher, HKDFAlgo, + #state{connection_states = ConnectionStates, + ssl_options = #{keep_secrets := KeepSecrets}, + static_env = #static_env{role = Role}} = State0) -> + EarlySecret = tls_v1:key_schedule(early_secret, HKDFAlgo , {psk, PSK}), + ClientEarlyTrafficSecret = + tls_v1:client_early_traffic_secret(HKDFAlgo, EarlySecret, ClientHello), + %% Calculate traffic key + KeyLength = ssl_cipher:key_material(Cipher), + {Key, IV} = + tls_v1:calculate_traffic_keys(HKDFAlgo, KeyLength, ClientEarlyTrafficSecret), + %% Update pending connection states + case Role of + client -> + PendingWrite0 = ssl_record:pending_connection_state(ConnectionStates, write), + PendingWrite1 = maybe_store_early_data_secret(KeepSecrets, ClientEarlyTrafficSecret, + PendingWrite0), + PendingWrite = update_connection_state(PendingWrite1, undefined, undefined, + undefined, + Key, IV, undefined), + State0#state{connection_states = ConnectionStates#{pending_write => PendingWrite}}; + server -> + PendingRead0 = ssl_record:pending_connection_state(ConnectionStates, read), + PendingRead1 = maybe_store_early_data_secret(KeepSecrets, ClientEarlyTrafficSecret, + PendingRead0), + PendingRead2 = update_connection_state(PendingRead1, undefined, undefined, + undefined, + Key, IV, undefined), + %% Signal start of early data. This is to prevent handshake messages to be + %% counted in max_early_data_size. + PendingRead = PendingRead2#{count_early_data => true}, + State0#state{connection_states = ConnectionStates#{pending_read => PendingRead}} + end. + +update_current_read(#state{connection_states = CS} = State, TrialDecryption, EarlyDataLimit) -> + Read0 = ssl_record:current_connection_state(CS, read), + Read = Read0#{trial_decryption => TrialDecryption, + early_data_limit => EarlyDataLimit}, + State#state{connection_states = CS#{current_read => Read}}. + +maybe_store_early_data_secret(true, EarlySecret, State) -> + #{security_parameters := SecParams0} = State, + SecParams = SecParams0#security_parameters{client_early_data_secret = EarlySecret}, + State#{security_parameters := SecParams}; +maybe_store_early_data_secret(false, _, State) -> + State. %% Server get_pre_shared_key(undefined, HKDFAlgo) -> @@ -1554,7 +1704,7 @@ get_pre_shared_key(manual = SessionTickets, UseTicket, HKDFAlgo, SelectedIdentit {ok, binary:copy(<<0>>, ssl_cipher:hash_size(HKDFAlgo))}; illegal_parameter -> {error, ?ALERT_REC(?FATAL, ?ILLEGAL_PARAMETER)}; - {_, PSK} -> + {_, PSK, _, _, _} -> {ok, PSK} end; get_pre_shared_key(auto = SessionTickets, UseTicket, HKDFAlgo, SelectedIdentity) -> @@ -1566,19 +1716,35 @@ get_pre_shared_key(auto = SessionTickets, UseTicket, HKDFAlgo, SelectedIdentity) illegal_parameter -> tls_client_ticket_store:unlock_tickets(self(), UseTicket), {error, ?ALERT_REC(?FATAL, ?ILLEGAL_PARAMETER)}; - {Key, PSK} -> + {Key, PSK, _, _, _} -> tls_client_ticket_store:remove_tickets([Key]), %% Remove single-use ticket tls_client_ticket_store:unlock_tickets(self(), UseTicket -- [Key]), {ok, PSK} end. - +%% +%% Early Data +get_pre_shared_key_early_data(SessionTickets, UseTicket) -> + TicketData = get_ticket_data(self(), SessionTickets, UseTicket), + case choose_psk(TicketData, 0) of + undefined -> %% Should not happen + {error, ?ALERT_REC(?FATAL, ?ILLEGAL_PARAMETER)}; + illegal_parameter -> + {error, ?ALERT_REC(?FATAL, ?ILLEGAL_PARAMETER)}; + {_Key, PSK, Cipher, HKDF, MaxSize} -> + {ok, {PSK, Cipher, HKDF, MaxSize}} + end. choose_psk(undefined, _) -> undefined; choose_psk([], _) -> illegal_parameter; -choose_psk([{Key, SelectedIdentity, _, PSK, _, _}|_], SelectedIdentity) -> - {Key, PSK}; +choose_psk([#ticket_data{ + key = Key, + pos = SelectedIdentity, + psk = PSK, + cipher_suite = {Cipher, HKDF}, + max_size = MaxSize}|_], SelectedIdentity) -> + {Key, PSK, Cipher, HKDF, MaxSize}; choose_psk([_|T], SelectedIdentity) -> choose_psk(T, SelectedIdentity). @@ -1608,9 +1774,9 @@ calculate_traffic_secrets(#state{ tls_v1:server_application_traffic_secret_0(HKDFAlgo, MasterSecret, lists:reverse(Messages)), %% Calculate traffic keys - #{cipher := Cipher} = ssl_cipher_format:suite_bin_to_map(CipherSuite), - {ReadKey, ReadIV} = tls_v1:calculate_traffic_keys(HKDFAlgo, Cipher, ClientAppTrafficSecret0), - {WriteKey, WriteIV} = tls_v1:calculate_traffic_keys(HKDFAlgo, Cipher, ServerAppTrafficSecret0), + KeyLength = tls_v1:key_length(CipherSuite), + {ReadKey, ReadIV} = tls_v1:calculate_traffic_keys(HKDFAlgo, KeyLength, ClientAppTrafficSecret0), + {WriteKey, WriteIV} = tls_v1:calculate_traffic_keys(HKDFAlgo, KeyLength, ServerAppTrafficSecret0), update_pending_connection_states(State0, MasterSecret, undefined, ClientAppTrafficSecret0, ServerAppTrafficSecret0, @@ -1925,13 +2091,12 @@ maybe_update_selected_sign_alg(State, _, _) -> State. -verify_certificate_verify(#state{ - static_env = #static_env{role = Role}, - connection_states = ConnectionStates, - handshake_env = - #handshake_env{ - public_key_info = PublicKeyInfo, - tls_handshake_history = HHistory}} = State0, +verify_certificate_verify(#state{static_env = #static_env{role = Role}, + connection_states = ConnectionStates, + handshake_env = + #handshake_env{ + public_key_info = PublicKeyInfo, + tls_handshake_history = HHistory}} = State0, #certificate_verify_1_3{ algorithm = SignatureScheme, signature = Signature}) -> @@ -2157,12 +2322,12 @@ check_cert_sign_algo(SignAlgo, SignHash, _, ClientSignAlgsCert) -> %% DSA keys are not supported by TLS 1.3 -select_sign_algo(dsa, _RSAKeySize, _ClientSignAlgs, _ServerSignAlgs) -> +select_sign_algo(dsa, _RSAKeySize, _PeerSignAlgs, _OwnSignAlgs) -> {error, ?ALERT_REC(?FATAL, ?INSUFFICIENT_SECURITY, no_suitable_public_key)}; select_sign_algo(_, _RSAKeySize, [], _) -> {error, ?ALERT_REC(?FATAL, ?INSUFFICIENT_SECURITY, no_suitable_signature_algorithm)}; -select_sign_algo(PublicKeyAlgo, RSAKeySize, [C|ClientSignAlgs], ServerSignAlgs) -> - {_, S, _} = ssl_cipher:scheme_to_components(C), +select_sign_algo(PublicKeyAlgo, RSAKeySize, [PeerSignAlg|PeerSignAlgs], OwnSignAlgs) -> + {_, S, _} = ssl_cipher:scheme_to_components(PeerSignAlg), %% RSASSA-PKCS1-v1_5 and Legacy algorithms are not defined for use in signed %% TLS handshake messages: filter sha-1 and rsa_pkcs1. %% @@ -2174,25 +2339,25 @@ select_sign_algo(PublicKeyAlgo, RSAKeySize, [C|ClientSignAlgs], ServerSignAlgs) orelse (PublicKeyAlgo =:= rsa_pss_pss andalso S =:= rsa_pss_pss) orelse (PublicKeyAlgo =:= ecdsa andalso S =:= ecdsa)) andalso - lists:member(C, ServerSignAlgs) of + lists:member(PeerSignAlg, OwnSignAlgs) of true -> validate_key_compatibility(PublicKeyAlgo, RSAKeySize, - [C|ClientSignAlgs], ServerSignAlgs); + [PeerSignAlg|PeerSignAlgs], OwnSignAlgs); false -> - select_sign_algo(PublicKeyAlgo, RSAKeySize, ClientSignAlgs, ServerSignAlgs) + select_sign_algo(PublicKeyAlgo, RSAKeySize, PeerSignAlgs, OwnSignAlgs) end. -validate_key_compatibility(PublicKeyAlgo, RSAKeySize, [C|ClientSignAlgs], ServerSignAlgs) +validate_key_compatibility(PublicKeyAlgo, RSAKeySize, [PeerSignAlg|PeerSignAlgs], OwnSignAlgs) when PublicKeyAlgo =:= rsa orelse PublicKeyAlgo =:= rsa_pss_pss -> - case is_rsa_key_compatible(RSAKeySize, C) of + case is_rsa_key_compatible(RSAKeySize, PeerSignAlg) of true -> - {ok, C}; + {ok, PeerSignAlg}; false -> - select_sign_algo(PublicKeyAlgo, RSAKeySize, ClientSignAlgs, ServerSignAlgs) + select_sign_algo(PublicKeyAlgo, RSAKeySize, PeerSignAlgs, OwnSignAlgs) end; -validate_key_compatibility(_, _, [C|_], _) -> - {ok, C}. +validate_key_compatibility(_, _, [PeerSignAlg|_], _) -> + {ok, PeerSignAlg}. is_rsa_key_compatible(KeySize, SigAlg) -> {Hash, _, _} = ssl_cipher:scheme_to_components(SigAlg), @@ -2396,18 +2561,18 @@ maybe_add_binders(Hello0, {[HRR,MessageHash|_], _}, TicketData, Version) when Ve maybe_add_binders(Hello, _, _, Version) when Version =< {3,3} -> Hello. - create_binders(Context, TicketData) -> create_binders(Context, TicketData, []). %% create_binders(_, [], Acc) -> lists:reverse(Acc); -create_binders(Context, [{_, _, _, PSK, _, HKDF}|T], Acc) -> +create_binders(Context, [#ticket_data{ + psk = PSK, + cipher_suite = {_, HKDF}}|T], Acc) -> FinishedKey = calculate_finished_key(PSK, HKDF), Binder = calculate_binder(FinishedKey, HKDF, Context), create_binders(Context, T, [Binder|Acc]). - %% Removes the binders list from the ClientHello. %% opaque PskBinderEntry<32..255>; %% @@ -2441,6 +2606,18 @@ truncate_client_hello(HelloBin0) -> {Truncated, _} = split_binary(HelloBin0, size(HelloBin0) - BindersSize - 2), Truncated. +maybe_add_early_data_indication(#client_hello{ + extensions = Extensions0} = ClientHello, + EarlyData, + Version) + when Version =:= {3,4} andalso + is_binary(EarlyData) andalso + size(EarlyData) > 0 -> + Extensions = Extensions0#{early_data => + #early_data_indication{}}, + ClientHello#client_hello{extensions = Extensions}; +maybe_add_early_data_indication(ClientHello, _, _) -> + ClientHello. %% The PskBinderEntry is computed in the same way as the Finished %% message (Section 4.4.4) but with the BaseKey being the binder_key @@ -2475,6 +2652,7 @@ update_binders(#client_hello{extensions = maybe_automatic_session_resumption(#state{ ssl_options = #{versions := [Version|_], ciphers := UserSuites, + early_data := EarlyData, session_tickets := SessionTickets, server_name_indication := SNI} = SslOpts0 } = State0) @@ -2482,7 +2660,13 @@ maybe_automatic_session_resumption(#state{ SessionTickets =:= auto -> AvailableCipherSuites = ssl_handshake:available_suites(UserSuites, Version), HashAlgos = cipher_hash_algos(AvailableCipherSuites), - UseTicket = tls_client_ticket_store:find_ticket(self(), HashAlgos, SNI), + Ciphers = ciphers_for_early_data(AvailableCipherSuites), + %% Find a pair of tickets KeyPair = {Ticket0, Ticket2} where Ticket0 satisfies + %% requirements for early_data and session resumption while Ticket2 can only + %% be used for session resumption. + EarlyDataSize = early_data_size(EarlyData), + KeyPair = tls_client_ticket_store:find_ticket(self(), Ciphers, HashAlgos, SNI, EarlyDataSize), + UseTicket = choose_ticket(KeyPair, EarlyData), tls_client_ticket_store:lock_tickets(self(), [UseTicket]), State = State0#state{ssl_options = SslOpts0#{use_ticket => [UseTicket]}}, {[UseTicket], State}; @@ -2491,6 +2675,144 @@ maybe_automatic_session_resumption(#state{ } = State) -> {UseTicket, State}. +early_data_size(undefined) -> + undefined; +early_data_size(EarlyData) when is_binary(EarlyData) -> + byte_size(EarlyData). + +%% Choose a ticket based on the intention of the user. The first argument is +%% a 2-tuple of ticket keys where the first element refers to a ticket that +%% fulfills all criteria for sending early_data (hash, cipher, early data size). +%% Second argument refers to a ticket that can only be used for session +%% resumption. +choose_ticket({Key, _}, _) when Key =/= undefined -> + Key; +choose_ticket({_, Key}, EarlyData) when EarlyData =:= undefined -> + Key; +choose_ticket(_, _) -> + %% No tickets found that fulfills the original intention of the user + %% (sending early_data). It is possible to do session resumption but + %% in that case the configured early data would have to be removed + %% and that would contradict the will of the user. Returning undefined + %% here prevents session resumption instead. + undefined. + +maybe_send_early_data(#state{ + handshake_env = #handshake_env{tls_handshake_history = {Hist, _}}, + protocol_specific = #{sender := _Sender}, + ssl_options = #{versions := [Version|_], + use_ticket := UseTicket, + session_tickets := SessionTickets, + early_data := EarlyData} = _SslOpts0 + } = State0) when Version =:= {3,4} andalso + UseTicket =/= [undefined] andalso + EarlyData =/= undefined -> + %% D.4. Middlebox Compatibility Mode + State1 = maybe_queue_change_cipher_spec(State0, last), + %% Early traffic secret + EarlyDataSize = early_data_size(EarlyData), + case get_pre_shared_key_early_data(SessionTickets, UseTicket) of + {ok, {PSK, Cipher, HKDF, MaxSize}} when EarlyDataSize =< MaxSize -> + State2 = calculate_client_early_traffic_secret(Hist, PSK, Cipher, HKDF, State1), + %% Set 0-RTT traffic keys for sending early_data and EndOfEarlyData + State3 = ssl_record:step_encryption_state_write(State2), + {ok, encode_early_data(Cipher, State3)}; + {ok, {_, _, _, _MaxSize}} -> + {error, ?ALERT_REC(?FATAL, ?ILLEGAL_PARAMETER, too_much_early_data)}; + {error, Alert} -> + {error, Alert} + end; +maybe_send_early_data(State) -> + {ok, State}. + +encode_early_data(Cipher, + #state{ + flight_buffer = Flight0, + protocol_specific = #{sender := _Sender}, + ssl_options = #{versions := [Version|_], + early_data := EarlyData} = _SslOpts0 + } = State0) -> + #state{connection_states = + #{current_write := + #{security_parameters := SecurityParameters0} = Write0} = ConnectionStates0} = State0, + BulkCipherAlgo = ssl_cipher:bulk_cipher_algorithm(Cipher), + SecurityParameters = SecurityParameters0#security_parameters{ + cipher_type = ?AEAD, + bulk_cipher_algorithm = BulkCipherAlgo}, + Write = Write0#{security_parameters => SecurityParameters}, + ConnectionStates1 = ConnectionStates0#{current_write => Write}, + {BinEarlyData, ConnectionStates} = tls_record:encode_data([EarlyData], Version, ConnectionStates1), + State0#state{connection_states = ConnectionStates, + flight_buffer = Flight0 ++ [BinEarlyData]}. + +maybe_send_end_of_early_data( + #state{ + handshake_env = #handshake_env{early_data_accepted = true}, + protocol_specific = #{sender := _Sender}, + ssl_options = #{versions := [Version|_], + use_ticket := UseTicket, + early_data := EarlyData}, + static_env = #static_env{protocol_cb = Connection} + } = State0) when Version =:= {3,4} andalso + UseTicket =/= [undefined] andalso + EarlyData =/= undefined -> + %% EndOfEarlydata is encrypted with the 0-RTT traffic keys + State1 = Connection:queue_handshake(#end_of_early_data{}, State0), + %% Use handshake keys after EndOfEarlyData is sent + ssl_record:step_encryption_state_write(State1); +maybe_send_end_of_early_data(State) -> + State. + +maybe_check_early_data_indication(EarlyDataIndication, + #state{ + handshake_env = HsEnv, + ssl_options = #{versions := [Version|_], + use_ticket := UseTicket, + early_data := EarlyData} + } = State) when Version =:= {3,4} andalso + UseTicket =/= [undefined] andalso + EarlyData =/= undefined andalso + EarlyDataIndication =/= undefined -> + signal_user_early_data(State, accepted), + State#state{handshake_env = HsEnv#handshake_env{early_data_accepted = true}}; +maybe_check_early_data_indication(EarlyDataIndication, + #state{ + protocol_specific = #{sender := _Sender}, + ssl_options = #{versions := [Version|_], + use_ticket := UseTicket, + early_data := EarlyData} = _SslOpts0 + } = State) when Version =:= {3,4} andalso + UseTicket =/= [undefined] andalso + EarlyData =/= undefined andalso + EarlyDataIndication =:= undefined -> + signal_user_early_data(State, rejected), + %% Use handshake keys if early_data is rejected. + ssl_record:step_encryption_state_write(State); +maybe_check_early_data_indication(_, State) -> + %% Use handshake keys if there is no early_data. + ssl_record:step_encryption_state_write(State). + +signal_user_early_data(#state{ + connection_env = + #connection_env{ + user_application = {_, User}}, + static_env = + #static_env{ + socket = Socket, + protocol_cb = Connection, + transport_cb = Transport, + trackers = Trackers}} = State, + Result) -> + CPids = Connection:pids(State), + SslSocket = Connection:socket(CPids, Transport, Socket, Trackers), + User ! {ssl, SslSocket, {early_data, Result}}. + +handle_early_data(State, enabled, #early_data_indication{}) -> + %% Accept early data + HsEnv = (State#state.handshake_env)#handshake_env{early_data_accepted = true}, + State#state{handshake_env = HsEnv}; +handle_early_data(State, _, _) -> + State. cipher_hash_algos(Ciphers) -> Fun = fun(Cipher) -> @@ -2499,6 +2821,14 @@ cipher_hash_algos(Ciphers) -> end, lists:map(Fun, Ciphers). +ciphers_for_early_data(CipherSuites0) -> + %% Use only supported TLS 1.3 cipher suites + Supported = lists:filter(fun(CipherSuite) -> + lists:member(CipherSuite, tls_v1:exclusive_suites(4)) end, + CipherSuites0), + %% Return supported block cipher algorithms + lists:map(fun(#{cipher := Cipher}) -> Cipher end, + lists:map(fun ssl_cipher_format:suite_bin_to_map/1, Supported)). get_ticket_data(_, undefined, _) -> undefined; @@ -2523,30 +2853,43 @@ process_user_tickets([H|T], Acc, N) -> process_user_tickets(T, [TicketData|Acc], N + 1) end. -process_ticket(Bin, N) when is_binary(Bin) -> - try erlang:binary_to_term(Bin, [safe]) of - #{hkdf := HKDF, - sni := _SNI, %% TODO: Handle SNI? - psk := PSK, - timestamp := Timestamp, - ticket := NewSessionTicket} -> - #new_session_ticket{ - ticket_lifetime = _LifeTime, - ticket_age_add = AgeAdd, - ticket_nonce = Nonce, - ticket = Ticket, - extensions = _Extensions - } = NewSessionTicket, - TicketAge = erlang:system_time(seconds) - Timestamp, - ObfuscatedTicketAge = obfuscate_ticket_age(TicketAge, AgeAdd), - Identity = #psk_identity{ - identity = Ticket, - obfuscated_ticket_age = ObfuscatedTicketAge}, - {undefined, N, Identity, PSK, Nonce, HKDF}; - _Else -> - error - catch error:badarg -> - error +%% Used when session_tickets = manual +process_ticket(#{cipher_suite := CipherSuite, + sni := _SNI, %% TODO user's responsibility to handle SNI? + psk := PSK, + timestamp := Timestamp, + ticket := NewSessionTicket}, N) -> + #new_session_ticket{ + ticket_lifetime = _LifeTime, + ticket_age_add = AgeAdd, + ticket_nonce = Nonce, + ticket = Ticket, + extensions = Extensions + } = NewSessionTicket, + TicketAge = erlang:system_time(seconds) - Timestamp, + ObfuscatedTicketAge = obfuscate_ticket_age(TicketAge, AgeAdd), + Identity = #psk_identity{ + identity = Ticket, + obfuscated_ticket_age = ObfuscatedTicketAge}, + MaxEarlyData = get_max_early_data(Extensions), + #ticket_data{ + key = undefined, + pos = N, + identity = Identity, + psk = PSK, + nonce = Nonce, + cipher_suite = CipherSuite, + max_size = MaxEarlyData}; +process_ticket(_, _) -> + error. + +get_max_early_data(Extensions) -> + EarlyDataIndication = maps:get(early_data, Extensions, undefined), + case EarlyDataIndication of + undefined -> + undefined; + #early_data_indication_nst{indication = MaxSize} -> + MaxSize end. %% The "obfuscated_ticket_age" diff --git a/lib/ssl/src/tls_handshake_1_3.hrl b/lib/ssl/src/tls_handshake_1_3.hrl index 9f41c84708..d506821f6c 100644 --- a/lib/ssl/src/tls_handshake_1_3.hrl +++ b/lib/ssl/src/tls_handshake_1_3.hrl @@ -87,9 +87,11 @@ %% RFC 8446 4.2.10. Early Data Indication -record(empty, { }). --record(early_data_indication, { - indication % uint32 max_early_data_size (new_session_ticket) | - %% #empty{} (client_hello, encrypted_extensions) + +%% #empty{} (client_hello, encrypted_extensions) +-record(early_data_indication, {}). +-record(early_data_indication_nst, { + indication % uint32 max_early_data_size (new_session_ticket) }). %% RFC 8446 4.2.11. Pre-Shared Key Extension diff --git a/lib/ssl/src/tls_record.erl b/lib/ssl/src/tls_record.erl index 6e5c30760d..9ec5490aa6 100644 --- a/lib/ssl/src/tls_record.erl +++ b/lib/ssl/src/tls_record.erl @@ -33,7 +33,9 @@ -include_lib("kernel/include/logger.hrl"). %% Handling of incoming data --export([get_tls_records/5, init_connection_states/2]). +-export([get_tls_records/5, + init_connection_states/2, + init_connection_states/3]). %% Encoding TLS records -export([encode_handshake/3, encode_alert_record/3, @@ -64,16 +66,29 @@ %% Handling of incoming data %%==================================================================== %%-------------------------------------------------------------------- --spec init_connection_states(client | server, one_n_minus_one | zero_n | disabled) -> - ssl_record:connection_states(). +-spec init_connection_states(Role, BeastMitigation) -> + ssl_record:connection_states() when + Role :: client | server, + BeastMitigation :: one_n_minus_one | zero_n | disabled. + %% %% Description: Creates a connection_states record with appropriate %% values for the initial SSL connection setup. %%-------------------------------------------------------------------- init_connection_states(Role, BeastMitigation) -> + MaxEarlyDataSize = ssl_config:get_max_early_data_size(), + init_connection_states(Role, BeastMitigation, MaxEarlyDataSize). +%% +-spec init_connection_states(Role, BeastMitigation, MaxEarlyDataSize) -> + ssl_record:connection_states() when + Role :: client | server, + BeastMitigation :: one_n_minus_one | zero_n | disabled, + MaxEarlyDataSize :: non_neg_integer(). + +init_connection_states(Role, BeastMitigation, MaxEarlyDataSize) -> ConnectionEnd = ssl_record:record_protocol_role(Role), - Current = initial_connection_state(ConnectionEnd, BeastMitigation), - Pending = ssl_record:empty_connection_state(ConnectionEnd, BeastMitigation), + Current = initial_connection_state(ConnectionEnd, BeastMitigation, MaxEarlyDataSize), + Pending = ssl_record:empty_connection_state(ConnectionEnd, BeastMitigation, MaxEarlyDataSize), #{current_read => Current, pending_read => Pending, current_write => Current, @@ -181,7 +196,8 @@ encode_data(Data, Version, %%-------------------------------------------------------------------- -spec decode_cipher_text(tls_version(), #ssl_tls{}, ssl_record:connection_states(), boolean()) -> - {#ssl_tls{}, ssl_record:connection_states()}| #alert{}. + {#ssl_tls{} | trial_decryption_failed, + ssl_record:connection_states()}| #alert{}. %% %% Description: Decode cipher text %%-------------------------------------------------------------------- @@ -465,7 +481,7 @@ split_iovec(Data, MaximumFragmentLength) -> %%-------------------------------------------------------------------- %%% Internal functions %%-------------------------------------------------------------------- -initial_connection_state(ConnectionEnd, BeastMitigation) -> +initial_connection_state(ConnectionEnd, BeastMitigation, MaxEarlyDataSize) -> #{security_parameters => ssl_record:initial_security_params(ConnectionEnd), sequence_number => 0, @@ -476,7 +492,10 @@ initial_connection_state(ConnectionEnd, BeastMitigation) -> secure_renegotiation => undefined, client_verify_data => undefined, server_verify_data => undefined, - max_fragment_length => undefined + max_early_data_size => MaxEarlyDataSize, + max_fragment_length => undefined, + trial_decryption => false, + early_data_limit => false }. %% Used by logging to recreate the received bytes diff --git a/lib/ssl/src/tls_record_1_3.erl b/lib/ssl/src/tls_record_1_3.erl index a9ba415099..3e42e3bf97 100644 --- a/lib/ssl/src/tls_record_1_3.erl +++ b/lib/ssl/src/tls_record_1_3.erl @@ -107,7 +107,8 @@ encode_iolist(Type, Data, ConnectionStates0) -> %%-------------------------------------------------------------------- -spec decode_cipher_text(#ssl_tls{}, ssl_record:connection_states()) -> - {#ssl_tls{}, ssl_record:connection_states()}| #alert{}. + {#ssl_tls{} | trial_decryption_failed, + ssl_record:connection_states()}| #alert{}. %% %% Description: Decode cipher text, use legacy type ssl_tls instead of tls_cipher_text %% in decoding context so that we can reuse the code from erlier versions. @@ -124,12 +125,25 @@ decode_cipher_text(#ssl_tls{type = ?OPAQUE_TYPE, #security_parameters{ cipher_type = ?AEAD, bulk_cipher_algorithm = - BulkCipherAlgo} + BulkCipherAlgo}, + max_early_data_size := MaxEarlyDataSize0, + trial_decryption := TrialDecryption, + early_data_limit := EarlyDataLimit } = ReadState0} = ConnectionStates0) -> case decipher_aead(CipherFragment, BulkCipherAlgo, Key, Seq, IV, TagLen) of + #alert{} when TrialDecryption =:= true andalso + MaxEarlyDataSize0 > 0 -> %% Trial decryption + trial_decrypt(ConnectionStates0, ReadState0, MaxEarlyDataSize0, + BulkCipherAlgo, CipherFragment); #alert{} = Alert -> Alert; - PlainFragment -> + PlainFragment0 when EarlyDataLimit =:= true andalso + MaxEarlyDataSize0 > 0 -> + PlainFragment = remove_padding(PlainFragment0), + process_early_data(ConnectionStates0, ReadState0, MaxEarlyDataSize0, Seq, + BulkCipherAlgo, CipherFragment, PlainFragment); + PlainFragment0 -> + PlainFragment = remove_padding(PlainFragment0), ConnectionStates = ConnectionStates0#{current_read => ReadState0#{sequence_number => Seq + 1}}, @@ -189,9 +203,55 @@ decode_cipher_text(#ssl_tls{type = Type}, _) -> %% Version mismatch is already asserted ?ALERT_REC(?FATAL, ?BAD_RECORD_MAC, {record_type_mismatch, Type}). + + %%-------------------------------------------------------------------- %%% Internal functions %%-------------------------------------------------------------------- +trial_decrypt(ConnectionStates0, ReadState0, MaxEarlyDataSize0, + BulkCipherAlgo, CipherFragment) -> + MaxEarlyDataSize = update_max_early_date_size(MaxEarlyDataSize0, BulkCipherAlgo, CipherFragment), + ConnectionStates = + ConnectionStates0#{current_read => + ReadState0#{max_early_data_size => MaxEarlyDataSize}}, + if MaxEarlyDataSize < 0 -> + %% More early data is trial decrypted as the configured limit + ?ALERT_REC(?FATAL, ?BAD_RECORD_MAC, decryption_failed); + true -> + {trial_decryption_failed, ConnectionStates} + end. + +process_early_data(ConnectionStates0, ReadState0, _MaxEarlyDataSize0, Seq, + _BulkCipherAlgo, _CipherFragment, PlainFragment) + when PlainFragment =:= <<5,0,0,0,22>> -> + %% struct { + %% opaque content[TLSPlaintext.length]; <<5,0,0,0>> - 5 = EndOfEarlyData + %% 0 = (uint24) size + %% ContentType type; <<22>> - Handshake + %% uint8 zeros[length_of_padding]; <<>> - no padding + %% } TLSInnerPlaintext; + %% EndOfEarlyData should not be counted into early data + ConnectionStates = + ConnectionStates0#{current_read => + ReadState0#{sequence_number => Seq + 1}}, + {decode_inner_plaintext(PlainFragment), ConnectionStates}; +process_early_data(ConnectionStates0, ReadState0, MaxEarlyDataSize0, Seq, + BulkCipherAlgo, CipherFragment, PlainFragment) -> + %% First packet is deciphered anyway so we must check if more early data is received + %% than the configured limit (max_early_data_size). + MaxEarlyDataSize = + update_max_early_date_size(MaxEarlyDataSize0, BulkCipherAlgo, CipherFragment), + if MaxEarlyDataSize < 0 -> + %% Too much early data received, send alert unexpected_message + ?ALERT_REC(?FATAL, ?UNEXPECTED_MESSAGE, too_much_early_data); + true -> + ConnectionStates = + ConnectionStates0#{current_read => + ReadState0#{sequence_number => Seq + 1, + max_early_data_size => MaxEarlyDataSize}}, + {decode_inner_plaintext(PlainFragment), ConnectionStates} + end. + inner_plaintext(Type, Data, Length) -> #inner_plaintext{ content = Data, @@ -299,8 +359,6 @@ aead_ciphertext_split(CipherTextFragment, TagLen) decode_inner_plaintext(PlainText) -> case binary:last(PlainText) of - 0 -> - decode_inner_plaintext(init_binary(PlainText)); Type when Type =:= ?APPLICATION_DATA orelse Type =:= ?HANDSHAKE orelse Type =:= ?ALERT -> @@ -315,3 +373,30 @@ init_binary(B) -> {Init, _} = split_binary(B, byte_size(B) - 1), Init. + +remove_padding(InnerPlainText) -> + case binary:last(InnerPlainText) of + 0 -> + remove_padding(init_binary(InnerPlainText)); + _ -> + InnerPlainText + end. + +update_max_early_date_size(MaxEarlyDataSize, BulkCipherAlgo, CipherFragment) -> + %% CipherFragment is the binary encoded form of a TLSInnerPlaintext: + %% + %% struct { + %% opaque content[TLSPlaintext.length]; + %% ContentType type; + %% uint8 zeros[length_of_padding]; + %% } TLSInnerPlaintext; + %% + TypeLen = 1, + PaddingLen = 0, %% TODO Update formula when padding is implemented! + MaxEarlyDataSize - (byte_size(CipherFragment) - TypeLen - PaddingLen - + bca_tag_len(BulkCipherAlgo)). + +bca_tag_len(?AES_CCM_8) -> + 8; +bca_tag_len(_) -> + 16. diff --git a/lib/ssl/src/tls_server_session_ticket.erl b/lib/ssl/src/tls_server_session_ticket.erl index 5f278cb939..b9bafa6e36 100644 --- a/lib/ssl/src/tls_server_session_ticket.erl +++ b/lib/ssl/src/tls_server_session_ticket.erl @@ -31,7 +31,7 @@ -include("ssl_cipher.hrl"). %% API --export([start_link/4, +-export([start_link/5, new/3, use/4 ]). @@ -46,18 +46,19 @@ stateless, stateful, nonce, - lifetime + lifetime, + max_early_data_size }). %%%=================================================================== %%% API %%%=================================================================== --spec start_link(atom(), integer(), integer(), tuple()) -> {ok, Pid :: pid()} | +-spec start_link(atom(), integer(), integer(), integer(), tuple()) -> {ok, Pid :: pid()} | {error, Error :: {already_started, pid()}} | {error, Error :: term()} | ignore. -start_link(Mode, Lifetime, TicketStoreSize, AntiReplay) -> - gen_server:start_link(?MODULE, [Mode, Lifetime, TicketStoreSize, AntiReplay], []). +start_link(Mode, Lifetime, TicketStoreSize, MaxEarlyDataSize, AntiReplay) -> + gen_server:start_link(?MODULE, [Mode, Lifetime, TicketStoreSize, MaxEarlyDataSize, AntiReplay], []). new(Pid, Prf, MasterSecret) -> gen_server:call(Pid, {new_session_ticket, Prf, MasterSecret}, infinity). @@ -81,10 +82,11 @@ init(Args) -> handle_call({new_session_ticket, Prf, MasterSecret}, _From, #state{nonce = Nonce, lifetime = LifeTime, + max_early_data_size = MaxEarlyDataSize, stateful = #{id_generator := IdGen}} = State0) -> Id = stateful_psk_ticket_id(IdGen), PSK = tls_v1:pre_shared_key(MasterSecret, ticket_nonce(Nonce), Prf), - SessionTicket = new_session_ticket(Id, Nonce, LifeTime), + SessionTicket = new_session_ticket(Id, Nonce, LifeTime, MaxEarlyDataSize), State = stateful_ticket_store(Id, SessionTicket, Prf, PSK, State0), {reply, SessionTicket, State}; handle_call({new_session_ticket, Prf, MasterSecret}, _From, @@ -142,27 +144,30 @@ format_status(_Opt, Status) -> %%% Internal functions %%%=================================================================== -inital_state([stateless, Lifetime, _, undefined]) -> +inital_state([stateless, Lifetime, _, MaxEarlyDataSize, undefined]) -> #state{nonce = 0, stateless = #{seed => {crypto:strong_rand_bytes(16), crypto:strong_rand_bytes(32)}, window => undefined}, - lifetime = Lifetime + lifetime = Lifetime, + max_early_data_size = MaxEarlyDataSize }; -inital_state([stateless, Lifetime, _, {Window, K, M}]) -> +inital_state([stateless, Lifetime, _, MaxEarlyDataSize, {Window, K, M}]) -> erlang:send_after(Window * 1000, self(), rotate_bloom_filters), #state{nonce = 0, stateless = #{bloom_filter => tls_bloom_filter:new(K, M), seed => {crypto:strong_rand_bytes(16), crypto:strong_rand_bytes(32)}, window => Window}, - lifetime = Lifetime + lifetime = Lifetime, + max_early_data_size = MaxEarlyDataSize }; -inital_state([stateful, Lifetime, TicketStoreSize|_]) -> +inital_state([stateful, Lifetime, TicketStoreSize, MaxEarlyDataSize|_]) -> %% statfeful servers replay %% protection is that it saves %% all valid tickets #state{lifetime = Lifetime, + max_early_data_size = MaxEarlyDataSize, nonce = 0, stateful = #{db => stateful_store(), max => TicketStoreSize, @@ -187,17 +192,21 @@ ticket_nonce(I) -> <<?UINT64(I)>>. new_session_ticket_base(#state{nonce = Nonce, - lifetime = Lifetime}) -> - new_session_ticket(undefined, Nonce, Lifetime). + lifetime = Lifetime, + max_early_data_size = MaxEarlyDataSize}) -> + new_session_ticket(undefined, Nonce, Lifetime, MaxEarlyDataSize). -new_session_ticket(Id, Nonce, Lifetime) -> +new_session_ticket(Id, Nonce, Lifetime, MaxEarlyDataSize) -> TicketAgeAdd = ticket_age_add(), + Extensions = #{early_data => + #early_data_indication_nst{ + indication = MaxEarlyDataSize}}, #new_session_ticket{ ticket = Id, ticket_lifetime = Lifetime, ticket_age_add = TicketAgeAdd, ticket_nonce = ticket_nonce(Nonce), - extensions = #{} + extensions = Extensions }. @@ -322,7 +331,7 @@ generate_stateless_ticket(#new_session_ticket{ticket_nonce = Nonce, timestamp = Timestamp }, Shard, IV), Ticket#new_session_ticket{ticket = Encrypted}. - + stateless_use(#offered_psks{ identities = Identities, binders = Binders diff --git a/lib/ssl/src/tls_socket.erl b/lib/ssl/src/tls_socket.erl index 48f1935e81..91fdad4e44 100644 --- a/lib/ssl/src/tls_socket.erl +++ b/lib/ssl/src/tls_socket.erl @@ -79,10 +79,12 @@ listen(Transport, Port, #config{transport_info = {Transport, _, _, _, _}, case Transport:listen(Port, Options ++ internal_inet_values()) of {ok, ListenSocket} -> {ok, Tracker} = inherit_tracker(ListenSocket, EmOpts, SslOpts), - LifeTime = get_ticket_lifetime(), - TicketStoreSize = get_ticket_store_size(), + LifeTime = ssl_config:get_ticket_lifetime(), + TicketStoreSize = ssl_config:get_ticket_store_size(), + MaxEarlyDataSize = ssl_config:get_max_early_data_size(), %% TLS-1.3 session handling - {ok, SessionHandler} = session_tickets_tracker(LifeTime, TicketStoreSize, SslOpts), + {ok, SessionHandler} = + session_tickets_tracker(LifeTime, TicketStoreSize, MaxEarlyDataSize, SslOpts), %% PRE TLS-1.3 session handling {ok, SessionIdHandle} = session_id_tracker(ListenSocket, SslOpts), Trackers = [{option_tracker, Tracker}, {session_tickets_tracker, SessionHandler}, @@ -261,15 +263,15 @@ inherit_tracker(ListenSocket, EmOpts, #{erl_dist := false} = SslOpts) -> inherit_tracker(ListenSocket, EmOpts, #{erl_dist := true} = SslOpts) -> ssl_listen_tracker_sup:start_child_dist([ListenSocket, EmOpts, SslOpts]). -session_tickets_tracker(_, _, #{erl_dist := false, - session_tickets := disabled}) -> +session_tickets_tracker(_, _, _, #{erl_dist := false, + session_tickets := disabled}) -> {ok, disabled}; -session_tickets_tracker(Lifetime, TicketStoreSize, +session_tickets_tracker(Lifetime, TicketStoreSize, MaxEarlyDataSize, #{erl_dist := false, session_tickets := Mode, anti_replay := AntiReplay}) -> - tls_server_session_ticket_sup:start_child([Mode, Lifetime, TicketStoreSize, AntiReplay]); -session_tickets_tracker(Lifetime, TicketStoreSize, + tls_server_session_ticket_sup:start_child([Mode, Lifetime, TicketStoreSize, MaxEarlyDataSize, AntiReplay]); +session_tickets_tracker(Lifetime, TicketStoreSize, MaxEarlyDataSize, #{erl_dist := true, session_tickets := Mode, anti_replay := AntiReplay}) -> @@ -278,7 +280,7 @@ session_tickets_tracker(Lifetime, TicketStoreSize, Workers = proplists:get_value(workers, Children), case Workers of 0 -> - tls_server_session_ticket_sup:start_child([Mode, Lifetime, TicketStoreSize, AntiReplay]); + tls_server_session_ticket_sup:start_child([Mode, Lifetime, TicketStoreSize, MaxEarlyDataSize, AntiReplay]); 1 -> [{_,Child,_, _}] = supervisor:which_children(SupName), {ok, Child} @@ -504,19 +506,3 @@ validate_inet_option(active, Value) validate_inet_option(_, _) -> ok. -get_ticket_lifetime() -> - case application:get_env(ssl, server_session_ticket_lifetime) of - {ok, Seconds} when is_integer(Seconds) andalso - Seconds =< 604800 -> %% MUST be less than 7 days - Seconds; - _ -> - 7200 %% Default 2 hours - end. - -get_ticket_store_size() -> - case application:get_env(ssl, server_session_ticket_store_size) of - {ok, Size} when is_integer(Size) -> - Size; - _ -> - 1000 - end. diff --git a/lib/ssl/src/tls_v1.erl b/lib/ssl/src/tls_v1.erl index cbba413ee2..59c425ecbe 100644 --- a/lib/ssl/src/tls_v1.erl +++ b/lib/ssl/src/tls_v1.erl @@ -56,6 +56,7 @@ hkdf_expand_label/5, hkdf_extract/3, hkdf_expand/4, + key_length/1, key_schedule/3, key_schedule/4, create_info/3, @@ -455,13 +456,20 @@ update_traffic_secret(Algo, Secret) -> %% %% [sender]_write_key = HKDF-Expand-Label(Secret, "key", "", key_length) %% [sender]_write_iv = HKDF-Expand-Label(Secret, "iv", "", iv_length) --spec calculate_traffic_keys(atom(), atom(), binary()) -> {binary(), binary()}. -calculate_traffic_keys(HKDFAlgo, Cipher, Secret) -> - Key = hkdf_expand_label(Secret, <<"key">>, <<>>, ssl_cipher:key_material(Cipher), HKDFAlgo), +-spec calculate_traffic_keys(atom(), integer(), binary()) -> {binary(), binary()}. +calculate_traffic_keys(HKDFAlgo, KeyLength, Secret) -> + Key = hkdf_expand_label(Secret, <<"key">>, <<>>, KeyLength, HKDFAlgo), %% TODO: remove hard coded IV size IV = hkdf_expand_label(Secret, <<"iv">>, <<>>, 12, HKDFAlgo), {Key, IV}. +-spec key_length(CipherSuite) -> KeyLength when + CipherSuite :: binary(), + KeyLength :: 0 | 8 | 16 | 24 | 32. +key_length(CipherSuite) -> + #{cipher := Cipher} = ssl_cipher_format:suite_bin_to_map(CipherSuite), + ssl_cipher:key_material(Cipher). + %% TLS v1.3 --------------------------------------------------- %% TLS 1.0 -1.2 --------------------------------------------------- diff --git a/lib/ssl/test/Makefile b/lib/ssl/test/Makefile index ea59bc9ffb..4e81a761f6 100644 --- a/lib/ssl/test/Makefile +++ b/lib/ssl/test/Makefile @@ -76,6 +76,7 @@ MODULES = \ ssl_pem_cache_SUITE \ ssl_session_SUITE \ ssl_session_cache_SUITE \ + ssl_session_cache_api_SUITE\ \ ssl_session_ticket_SUITE \ openssl_session_ticket_SUITE \ openssl_session_SUITE \ diff --git a/lib/ssl/test/openssl_cipher_suite_SUITE.erl b/lib/ssl/test/openssl_cipher_suite_SUITE.erl index eb823471c8..fb1f28aa4a 100644 --- a/lib/ssl/test/openssl_cipher_suite_SUITE.erl +++ b/lib/ssl/test/openssl_cipher_suite_SUITE.erl @@ -95,10 +95,13 @@ %% Common Test interface functions ----------------------------------- %%-------------------------------------------------------------------- all() -> - [ - {group, openssl_server}, - {group, openssl_client} - ]. + case ssl_test_lib:working_openssl_client() of + true -> + [{group, openssl_server}, + {group, openssl_client}]; + false -> + [{group, openssl_server}] + end. all_protocol_groups() -> [ @@ -955,7 +958,7 @@ cipher_suite_test(CipherSuite, Version, Config) -> [{ciphers, [CipherSuite]} | SOpts], Config); _ -> ssl_test_lib:basic_test([{versions, [Version]}, {ciphers, [CipherSuite]} | COpts], - [{ciphers, ssl:cipher_suites(all, Version)} | SOpts], Config) + [{ciphers, ssl_test_lib:openssl_ciphers()} | SOpts], Config) end. test_ciphers(Kex, Cipher, Version) -> @@ -978,3 +981,5 @@ test_ciphers(Kex, Cipher, Version) -> end, Ciphers). +openssl_suitestr_to_map(OpenSSLSuiteStrs) -> + [ssl_cipher_format:suite_openssl_str_to_map(SuiteStr) || SuiteStr <- OpenSSLSuiteStrs]. diff --git a/lib/ssl/test/openssl_client_cert_SUITE.erl b/lib/ssl/test/openssl_client_cert_SUITE.erl index 7e8d842f14..0248956056 100644 --- a/lib/ssl/test/openssl_client_cert_SUITE.erl +++ b/lib/ssl/test/openssl_client_cert_SUITE.erl @@ -156,10 +156,15 @@ init_per_suite(Config) -> catch crypto:stop(), try crypto:start() of ok -> - ssl_test_lib:clean_start(), - Config + case ssl_test_lib:working_openssl_client() of + true -> + ssl_test_lib:clean_start(), + Config; + false -> + {skip, "Broken OpenSSL s_client"} + end catch _:_ -> - {skip, "Crypto did not start"} + {skip, "Crypto did not start"} end. end_per_suite(_Config) -> @@ -167,9 +172,9 @@ end_per_suite(_Config) -> application:unload(ssl), application:stop(crypto). -init_per_group(openssl_client, Config0) -> - Config = proplists:delete(server_type, proplists:delete(client_type, Config0)), +init_per_group(openssl_client, Config) -> [{client_type, openssl}, {server_type, erlang} | Config]; + init_per_group(Group, Config0) when Group == rsa; Group == rsa_1_3 -> Config = ssl_test_lib:make_rsa_cert(Config0), diff --git a/lib/ssl/test/openssl_session_ticket_SUITE.erl b/lib/ssl/test/openssl_session_ticket_SUITE.erl index 9986a492b1..effcedc7ee 100644 --- a/lib/ssl/test/openssl_session_ticket_SUITE.erl +++ b/lib/ssl/test/openssl_session_ticket_SUITE.erl @@ -40,7 +40,21 @@ openssl_client_basic/0, openssl_client_basic/1, openssl_client_hrr/0, - openssl_client_hrr/1]). + openssl_client_hrr/1, + openssl_server_early_data_basic/0, + openssl_server_early_data_basic/1, + openssl_server_early_data_big/0, + openssl_server_early_data_big/1, + openssl_server_early_data_manual/0, + openssl_server_early_data_manual/1, + openssl_server_early_data_manual_big/0, + openssl_server_early_data_manual_big/1, + openssl_server_early_data_manual_2_tickets/0, + openssl_server_early_data_manual_2_tickets/1, + openssl_server_early_data_manual_2_chacha_tickets/0, + openssl_server_early_data_manual_2_chacha_tickets/1, + openssl_client_early_data_basic/0, + openssl_client_early_data_basic/1]). -include("tls_handshake.hrl"). @@ -62,14 +76,21 @@ groups() -> {group, openssl_server}]}, {openssl_server, [], [openssl_server_basic, openssl_server_hrr, - openssl_server_hrr_multiple_tickets + openssl_server_hrr_multiple_tickets, + openssl_server_early_data_basic, + openssl_server_early_data_big, + openssl_server_early_data_manual, + openssl_server_early_data_manual_big, + openssl_server_early_data_manual_2_tickets, + openssl_server_early_data_manual_2_chacha_tickets ]}, {stateful, [], session_tests()}, {stateless, [], session_tests()}]. session_tests() -> [openssl_client_basic, - openssl_client_hrr]. + openssl_client_hrr, + openssl_client_early_data_basic]. init_per_suite(Config0) -> catch crypto:stop(), @@ -316,7 +337,7 @@ openssl_server_hrr_multiple_tickets(Config) when is_list(Config) -> {versions, ['tlsv1.2','tlsv1.3']}, {supported_groups,[secp256r1, x25519]}|ClientOpts0], - + Server = ssl_test_lib:start_server(openssl, [{groups, "X448:X25519"}], [{server_opts, ServerOpts} | Config]), @@ -350,3 +371,348 @@ openssl_server_hrr_multiple_tickets(Config) when is_list(Config) -> ssl_test_lib:close(Client1), ssl_test_lib:close(Server). + +openssl_server_early_data_basic() -> + [{doc,"Test early data (erlang client - openssl server)"}]. +openssl_server_early_data_basic(Config) when is_list(Config) -> + ClientOpts0 = ssl_test_lib:ssl_options(client_rsa_verify_opts, Config), + ServerOpts = ssl_test_lib:ssl_options(server_rsa_verify_opts, Config), + {ClientNode, _, Hostname} = ssl_test_lib:run_where(Config), + + %% Configure session tickets + ClientOpts1 = [{session_tickets, auto}, {log_level, debug}, + {versions, ['tlsv1.2','tlsv1.3']}|ClientOpts0], + ClientOpts2 = [{early_data, <<"SampleData">>}|ClientOpts1], + + Server = ssl_test_lib:start_server(openssl, [{early_data, 16384}], + [{server_opts, ServerOpts} | Config]), + + Port = ssl_test_lib:inet_port(Server), + + %% Store ticket from first connection + Client0 = ssl_test_lib:start_client([{node, ClientNode}, + {port, Port}, {host, Hostname}, + {mfa, {ssl_test_lib, + verify_active_session_resumption, + [false, no_reply]}}, + {from, self()}, {options, ClientOpts1}]), + %% Wait for session ticket + ct:sleep(100), + + %% Close previous connection as s_server can only handle one at a time + ssl_test_lib:close(Client0), + + %% Use ticket + Client1 = ssl_test_lib:start_client([{node, ClientNode}, + {port, Port}, {host, Hostname}, + {mfa, {ssl_test_lib, + verify_active_session_resumption, + [true, no_reply, no_tickets, + {verify_early_data, accepted}]}}, + {from, self()}, + {options, ClientOpts2}]), + ssl_test_lib:close(Client1), + ssl_test_lib:close(Server). + +openssl_server_early_data_big() -> + [{doc,"Send more early data than the max_early_data_size (erlang client - openssl server)"}]. +openssl_server_early_data_big(Config) when is_list(Config) -> + ClientOpts0 = ssl_test_lib:ssl_options(client_rsa_verify_opts, Config), + ServerOpts = ssl_test_lib:ssl_options(server_rsa_verify_opts, Config), + {ClientNode, _, Hostname} = ssl_test_lib:run_where(Config), + + %% Configure session tickets + ClientOpts1 = [{session_tickets, auto}, {log_level, debug}, + {versions, ['tlsv1.2','tlsv1.3']}|ClientOpts0], + ClientOpts2 = [{early_data, <<"SampleData">>}|ClientOpts1], + + Server = ssl_test_lib:start_server(openssl, [{early_data, 5}], + [{server_opts, ServerOpts} | Config]), + + Port = ssl_test_lib:inet_port(Server), + + %% Store ticket from first connection + Client0 = ssl_test_lib:start_client([{node, ClientNode}, + {port, Port}, {host, Hostname}, + {mfa, {ssl_test_lib, + verify_active_session_resumption, + [false, no_reply]}}, + {from, self()}, {options, ClientOpts1}]), + %% Wait for session ticket + ct:sleep(100), + + %% Close previous connection as s_server can only handle one at a time + ssl_test_lib:close(Client0), + + %% Use ticket + %% The tickets received cannot be used for sending more early data than the + %% max_early_data_size. They are filtered by the automatic ticket handling + %% mechanism and there will be no session resumption. + Client1 = ssl_test_lib:start_client([{node, ClientNode}, + {port, Port}, {host, Hostname}, + {mfa, {ssl_test_lib, + verify_active_session_resumption, + [false, no_reply, no_tickets]}}, + {from, self()}, + {options, ClientOpts2}]), + ssl_test_lib:close(Client1), + ssl_test_lib:close(Server). + +openssl_server_early_data_manual() -> + [{doc,"Test sending early data - manual ticket handling (erlang client - openssl server)"}]. +openssl_server_early_data_manual(Config) when is_list(Config) -> + process_flag(trap_exit, true), + ClientOpts0 = ssl_test_lib:ssl_options(client_rsa_verify_opts, Config), + ServerOpts = ssl_test_lib:ssl_options(server_rsa_verify_opts, Config), + {ClientNode, _, Hostname} = ssl_test_lib:run_where(Config), + + %% Configure session tickets + ClientOpts1 = [{session_tickets, manual}, {log_level, debug}, + {versions, ['tlsv1.2','tlsv1.3']}|ClientOpts0], + ClientOpts2 = [{early_data, <<"SampleData">>}|ClientOpts1], + + Server = ssl_test_lib:start_server(openssl, [{early_data, 16384}], + [{server_opts, ServerOpts} | Config]), + + Port = ssl_test_lib:inet_port(Server), + + %% Store ticket from first connection + Client0 = ssl_test_lib:start_client([{node, ClientNode}, + {port, Port}, {host, Hostname}, + {mfa, {ssl_test_lib, + verify_active_session_resumption, + [false, no_reply, {tickets, 1}]}}, + {from, self()}, {options, ClientOpts1}]), + + Tickets0 = ssl_test_lib:check_tickets(Client0), + + ct:pal("Received tickets: ~p~n", [Tickets0]), + + %% Close previous connection as s_server can only handle one at a time + ssl_test_lib:close(Client0), + + %% Use tickets + Client1 = ssl_test_lib:start_client([{node, ClientNode}, + {port, Port}, {host, Hostname}, + {mfa, {ssl_test_lib, + verify_active_session_resumption, + [true, no_reply, no_tickets, + {verify_early_data, accepted}]}}, + {from, self()}, + {options, [{use_ticket, Tickets0}|ClientOpts2]}]), + + process_flag(trap_exit, false), + + ssl_test_lib:close(Client1), + ssl_test_lib:close(Server). + +openssl_server_early_data_manual_big() -> + [{doc,"Test sending more early data than the max_early_data_size - manual ticket handling " + "(erlang client - openssl server)"}]. +openssl_server_early_data_manual_big(Config) when is_list(Config) -> + process_flag(trap_exit, true), + ClientOpts0 = ssl_test_lib:ssl_options(client_rsa_verify_opts, Config), + ServerOpts = ssl_test_lib:ssl_options(server_rsa_verify_opts, Config), + {ClientNode, _, Hostname} = ssl_test_lib:run_where(Config), + + %% Configure session tickets + ClientOpts1 = [{session_tickets, manual}, {log_level, debug}, + {versions, ['tlsv1.2','tlsv1.3']}|ClientOpts0], + ClientOpts2 = [{early_data, <<"SampleData">>}|ClientOpts1], + + Server = ssl_test_lib:start_server(openssl, [{early_data, 5}], + [{server_opts, ServerOpts} | Config]), + + Port = ssl_test_lib:inet_port(Server), + + %% Store ticket from first connection + Client0 = ssl_test_lib:start_client([{node, ClientNode}, + {port, Port}, {host, Hostname}, + {mfa, {ssl_test_lib, + verify_active_session_resumption, + [false, no_reply, {tickets, 1}]}}, + {from, self()}, {options, ClientOpts1}]), + + Tickets0 = ssl_test_lib:check_tickets(Client0), + + ct:pal("Received tickets: ~p~n", [Tickets0]), + + %% Close previous connection as s_server can only handle one at a time + ssl_test_lib:close(Client0), + + %% Use tickets + Client1 = ssl_test_lib:start_client_error([{node, ClientNode}, + {port, Port}, {host, Hostname}, + {mfa, {ssl_test_lib, + verify_active_session_resumption, + [true, no_reply, no_tickets]}}, + {from, self()}, + {options, [{use_ticket, Tickets0}|ClientOpts2]}]), + ssl_test_lib:check_client_alert(Client1, illegal_parameter), + process_flag(trap_exit, false), + + ssl_test_lib:close(Client1), + ssl_test_lib:close(Server). + +openssl_server_early_data_manual_2_tickets() -> + [{doc,"Test sending early data - manual ticket handling, 2 tickets (erlang client - openssl server)"}]. +openssl_server_early_data_manual_2_tickets(Config) when is_list(Config) -> + process_flag(trap_exit, true), + ClientOpts0 = ssl_test_lib:ssl_options(client_rsa_verify_opts, Config), + ServerOpts = ssl_test_lib:ssl_options(server_rsa_verify_opts, Config), + {ClientNode, _, Hostname} = ssl_test_lib:run_where(Config), + + %% Configure session tickets + ClientOpts1 = [{session_tickets, manual}, {log_level, debug}, + {versions, ['tlsv1.2','tlsv1.3']}|ClientOpts0], + ClientOpts2 = [{early_data, <<"SampleData">>}|ClientOpts1], + + Server = ssl_test_lib:start_server(openssl, [{early_data, 16384}], + [{server_opts, ServerOpts} | Config]), + + Port = ssl_test_lib:inet_port(Server), + + %% Store ticket from first connection + Client0 = ssl_test_lib:start_client([{node, ClientNode}, + {port, Port}, {host, Hostname}, + {mfa, {ssl_test_lib, + verify_active_session_resumption, + [false, no_reply, {tickets, 2}]}}, + {from, self()}, {options, ClientOpts1}]), + + Tickets0 = ssl_test_lib:check_tickets(Client0), + + ct:pal("Received tickets: ~p~n", [Tickets0]), + + %% Close previous connection as s_server can only handle one at a time + ssl_test_lib:close(Client0), + + %% Use tickets + Client1 = ssl_test_lib:start_client([{node, ClientNode}, + {port, Port}, {host, Hostname}, + {mfa, {ssl_test_lib, + verify_active_session_resumption, + [true, no_reply, no_tickets, + {verify_early_data, accepted}]}}, + {from, self()}, + {options, [{use_ticket, Tickets0}|ClientOpts2]}]), + process_flag(trap_exit, false), + + ssl_test_lib:close(Client1), + ssl_test_lib:close(Server). + +openssl_server_early_data_manual_2_chacha_tickets() -> + [{doc,"Test sending early data - manual ticket handling, 2 tickets - chacha (erlang client - openssl server)"}]. +openssl_server_early_data_manual_2_chacha_tickets(Config) when is_list(Config) -> + process_flag(trap_exit, true), + ClientOpts0 = ssl_test_lib:ssl_options(client_rsa_verify_opts, Config), + ServerOpts = ssl_test_lib:ssl_options(server_rsa_verify_opts, Config), + {ClientNode, _, Hostname} = ssl_test_lib:run_where(Config), + + %% Configure session tickets + ClientOpts1 = [{session_tickets, manual}, {log_level, debug}, + {ciphers, ["TLS_CHACHA20_POLY1305_SHA256"]}, + {versions, ['tlsv1.2','tlsv1.3']}|ClientOpts0], + ClientOpts2 = [{early_data, <<"SampleData">>}|ClientOpts1], + + %% openssl s_server seems to select a cipher_suite that satisfies the requirements + %% for early_data. + Server = ssl_test_lib:start_server(openssl, [{early_data, 16384}], + [{server_opts, ServerOpts} | Config]), + + Port = ssl_test_lib:inet_port(Server), + + %% Store ticket from first connection + Client0 = ssl_test_lib:start_client([{node, ClientNode}, + {port, Port}, {host, Hostname}, + {mfa, {ssl_test_lib, + verify_active_session_resumption, + [false, no_reply, {tickets, 2}]}}, + {from, self()}, {options, ClientOpts1}]), + + %% Receive 2 tickets that used Chacha20-Poly1305 and sha256 + Tickets0 = ssl_test_lib:check_tickets(Client0), + + ct:pal("Received tickets: ~p~n", [Tickets0]), + + %% Close previous connection as s_server can only handle one at a time + ssl_test_lib:close(Client0), + + %% Use tickets + Client1 = ssl_test_lib:start_client([{node, ClientNode}, + {port, Port}, {host, Hostname}, + {mfa, {ssl_test_lib, + verify_active_session_resumption, + [true, no_reply, no_tickets, + {verify_early_data, accepted}]}}, + {from, self()}, + {options, [{use_ticket, Tickets0}|ClientOpts2]}]), + process_flag(trap_exit, false), + + ssl_test_lib:close(Client1), + ssl_test_lib:close(Server). + +openssl_client_early_data_basic() -> + [{doc,"Test early data (openssl client - erlang server)"}]. +openssl_client_early_data_basic(Config) when is_list(Config) -> + ServerOpts0 = ssl_test_lib:ssl_options(server_rsa_verify_opts, Config), + ClientOpts = proplists:get_value(client_rsa_opts, Config), + + {_, ServerNode, _Hostname} = ssl_test_lib:run_where(Config), + TicketFile0 = filename:join([proplists:get_value(priv_dir, Config), "session_ticket0"]), + TicketFile1 = filename:join([proplists:get_value(priv_dir, Config), "session_ticket1"]), + RequestFile = filename:join([proplists:get_value(priv_dir, Config), "request"]), + ServerTicketMode = proplists:get_value(server_ticket_mode, Config), + %% Create request file to be used with early data + EarlyData = <<"HEAD / HTTP/1.1\nHost: \nConnection: close\n\n">>, + create_request(RequestFile, EarlyData), + + %% Configure session tickets + ServerOpts = [{session_tickets, ServerTicketMode}, + {early_data, enabled}, + {log_level, debug}, + {versions, ['tlsv1.2','tlsv1.3']}|ServerOpts0], + + Server0 = + ssl_test_lib:start_server([{node, ServerNode}, {port, 0}, + {from, self()}, + {mfa, {ssl_test_lib, + verify_active_session_resumption, + [false]}}, + {options, ServerOpts}]), + + Port0 = ssl_test_lib:inet_port(Server0), + + Client0 = ssl_test_lib:start_client(openssl, [{port, Port0}, + {options, ClientOpts}, + {session_args, ["-sess_out", TicketFile0]}], Config), + ssl_test_lib:check_result(Server0, ok), + + Server0 ! {listen, {mfa, {ssl_test_lib, + verify_server_early_data, + [wait_reply, EarlyData]}}}, + + %% %% Wait for session ticket + ct:sleep(100), + ssl_test_lib:close(Client0), + + Client1 = ssl_test_lib:start_client(openssl, [{port, Port0}, + {options, ClientOpts}, + {session_args, ["-sess_in", TicketFile0, + "-sess_out", TicketFile1, + "-early_data", RequestFile]}], + Config), + + ssl_test_lib:check_result(Server0, ok), + ssl_test_lib:close(Server0), + ssl_test_lib:close(Client1). + +%%-------------------------------------------------------------------- +%% Internal functions ------------------------------------------------ +%%-------------------------------------------------------------------- + +create_request(File, EarlyData) -> + {ok, S} = file:open(File, [write]), + io:format(S, "~s", [binary_to_list(EarlyData)]), + file:close(S). + diff --git a/lib/ssl/test/property_test/ssl_eqc_chain.erl b/lib/ssl/test/property_test/ssl_eqc_chain.erl index e78dc3fc0e..e108591776 100644 --- a/lib/ssl/test/property_test/ssl_eqc_chain.erl +++ b/lib/ssl/test/property_test/ssl_eqc_chain.erl @@ -124,6 +124,23 @@ prop_tls_extraneous_and_unordered_path() -> end ). +prop_client_cert_auth() -> + ?FORALL({ClientOptions, ServerOptions}, ?LET(Version, tls_version(), client_cert_auth_opts(Version)), + try + [TLSVersion] = proplists:get_value(versions, ClientOptions), + ssl_test_lib:basic_test(ClientOptions, ServerOptions, [{server_type, erlang}, + {client_type, erlang}, + {version, TLSVersion} + ]) + of + _ -> + true + catch + _:_ -> + false + end + ). + %%-------------------------------------------------------------------- %% Chain Generators ----------------------------------------------- %%-------------------------------------------------------------------- @@ -232,6 +249,9 @@ unordered_extraneous_options(Version) -> der_extraneous_and_unorder_options(Version) -> ?LET(Alg, key_alg(Version), der_extraneous_and_unorder_chain(Version, Alg)). +client_cert_auth_opts(Version) -> + ?LET({SAlg, CAlg}, {key_alg(Version), key_alg(Version)}, der_cert_chains(Version, CAlg,SAlg)). + extraneous_der_cert_chain_opts(Version, Alg) -> #{cert := OrgSRoot} = SRoot = public_key:pkix_test_root_cert("OTP test server ROOT", root_key(Alg)), #{cert := OrgCRoot} = CRoot = public_key:pkix_test_root_cert("OTP test client ROOT", root_key(Alg)), @@ -275,7 +295,6 @@ extraneous_pem_cert_chain_opts(Version, Alg, PrivDir) -> extraneous_pem_conf(ServerChain, ClientRoot, OrgCRoot, ServerConf0, PrivDir)]}. extra_extraneous_der_cert_chain_opts(Version, Alg) -> - #{cert := OrgSRoot} = SRoot = public_key:pkix_test_root_cert("OTP test server ROOT", root_key(Alg)), #{cert := OrgCRoot} = CRoot = public_key:pkix_test_root_cert("OTP test client ROOT", root_key(Alg)), @@ -304,7 +323,6 @@ extra_extraneous_der_cert_chain_opts(Version, Alg) -> der_extraneous_and_unorder_chain(Version, Alg) -> - #{cert := OrgSRoot} = SRoot = public_key:pkix_test_root_cert("OTP test server ROOT", root_key(Alg)), #{cert := OrgCRoot} = CRoot = public_key:pkix_test_root_cert("OTP test client ROOT", root_key(Alg)), @@ -331,6 +349,20 @@ der_extraneous_and_unorder_chain(Version, Alg) -> server_options(Version) ++ [protocol(Version), {versions, [Version]} | extraneous_der_conf(ServerChain, ClientRoot1, [OrgCRoot, ClientRoot0], ServerConf0)]}. +der_cert_chains(Version, CAlg, SAlg) -> + SRoot = public_key:pkix_test_root_cert("OTP test server ROOT", root_key(SAlg)), + CRoot = public_key:pkix_test_root_cert("OTP test client ROOT", root_key(CAlg)), + + #{server_config := ServerConf, + client_config := ClientConf} = public_key:pkix_test_data(#{server_chain => #{root => SRoot, + intermediates => intermediates(SAlg, 1), + peer => peer_key(SAlg)}, + client_chain => #{root => CRoot, + intermediates => intermediates(CAlg, 1), + peer => peer_key(CAlg)}}), + {client_options(Version) ++ [protocol(Version), {versions, [Version]} | ClientConf], + server_options(Version) ++ [protocol(Version), {versions, [Version]} | ServerConf]}. + chain_and_root(Config) -> OwnCert = proplists:get_value(cert, Config), {ok, ExtractedCAs} = ssl_pkix_db:extract_trusted_certs({der, proplists:get_value(cacerts, Config)}), diff --git a/lib/ssl/test/ssl_api_SUITE.erl b/lib/ssl/test/ssl_api_SUITE.erl index 07aaf41ef5..b3be655cdc 100644 --- a/lib/ssl/test/ssl_api_SUITE.erl +++ b/lib/ssl/test/ssl_api_SUITE.erl @@ -160,6 +160,10 @@ client_options_negative_dependency_stateless/1, client_options_negative_dependency_role/0, client_options_negative_dependency_role/1, + client_options_negative_early_data/0, + client_options_negative_early_data/1, + server_options_negative_early_data/0, + server_options_negative_early_data/1, server_options_negative_version_gap/0, server_options_negative_version_gap/1, server_options_negative_dependency_role/0, @@ -319,6 +323,8 @@ tls13_group() -> client_options_negative_dependency_version, client_options_negative_dependency_stateless, client_options_negative_dependency_role, + client_options_negative_early_data, + server_options_negative_early_data, server_options_negative_version_gap, server_options_negative_dependency_role, invalid_options_tls13, @@ -1876,7 +1882,7 @@ new_options_in_handshake(Config) when is_list(Config) -> (ecdh_rsa) -> true; (rsa) -> - true; + false; (_) -> false end @@ -2226,6 +2232,91 @@ client_options_negative_dependency_role(Config) when is_list(Config) -> {session_tickets,{stateless,{client,[disabled,manual,auto]}}}}). %%-------------------------------------------------------------------- +client_options_negative_early_data() -> + [{doc,"Test client option early_data."}]. +client_options_negative_early_data(Config) when is_list(Config) -> + start_client_negative(Config, [{versions, ['tlsv1.2']}, + {early_data, "test"}], + {options,dependency, + {early_data,{versions,['tlsv1.3']}}}), + start_client_negative(Config, [{versions, ['tlsv1.2', 'tlsv1.3']}, + {early_data, "test"}], + {options,dependency, + {early_data,{session_tickets,[manual,auto]}}}), + + start_client_negative(Config, [{versions, ['tlsv1.2', 'tlsv1.3']}, + {session_tickets, stateful}, + {early_data, "test"}], + {options,role, + {session_tickets, + {stateful,{client,[disabled,manual,auto]}}}}), + start_client_negative(Config, [{versions, ['tlsv1.2', 'tlsv1.3']}, + {session_tickets, disabled}, + {early_data, "test"}], + {options,dependency, + {early_data,{session_tickets,[manual,auto]}}}), + + start_client_negative(Config, [{versions, ['tlsv1.2', 'tlsv1.3']}, + {session_tickets, manual}, + {early_data, "test"}], + {options,dependency, + {early_data, use_ticket}}), + start_client_negative(Config, [{versions, ['tlsv1.2', 'tlsv1.3']}, + {session_tickets, manual}, + {use_ticket, [<<"ticket">>]}, + {early_data, "test"}], + {options, type, + {early_data, {"test", not_binary}}}), + %% All options are ok but there is no server + start_client_negative(Config, [{versions, ['tlsv1.2', 'tlsv1.3']}, + {session_tickets, manual}, + {use_ticket, [<<"ticket">>]}, + {early_data, <<"test">>}], + econnrefused), + + start_client_negative(Config, [{versions, ['tlsv1.2', 'tlsv1.3']}, + {session_tickets, auto}, + {early_data, "test"}], + {options, type, + {early_data, {"test", not_binary}}}), + %% All options are ok but there is no server + start_client_negative(Config, [{versions, ['tlsv1.2', 'tlsv1.3']}, + {session_tickets, auto}, + {early_data, <<"test">>}], + econnrefused). + +%%-------------------------------------------------------------------- +server_options_negative_early_data() -> + [{doc,"Test server option early_data."}]. +server_options_negative_early_data(Config) when is_list(Config) -> + start_server_negative(Config, [{versions, ['tlsv1.2']}, + {early_data, "test"}], + {options,dependency, + {early_data,{versions,['tlsv1.3']}}}), + start_server_negative(Config, [{versions, ['tlsv1.2', 'tlsv1.3']}, + {early_data, "test"}], + {options,dependency, + {early_data,{session_tickets,[stateful,stateless]}}}), + + start_server_negative(Config, [{versions, ['tlsv1.2', 'tlsv1.3']}, + {session_tickets, manual}, + {early_data, "test"}], + {options,role, + {session_tickets, + {manual,{server,[disabled,stateful,stateless]}}}}), + start_server_negative(Config, [{versions, ['tlsv1.2', 'tlsv1.3']}, + {session_tickets, disabled}, + {early_data, "test"}], + {options,dependency, + {early_data,{session_tickets,[stateful,stateless]}}}), + + start_server_negative(Config, [{versions, ['tlsv1.2', 'tlsv1.3']}, + {session_tickets, stateful}, + {early_data, "test"}], + {options,role, + {early_data,{"test",{server,[disabled,enabled]}}}}). + +%%-------------------------------------------------------------------- server_options_negative_version_gap() -> [{doc,"Test server options with faulty version gap."}]. server_options_negative_version_gap(Config) when is_list(Config) -> @@ -2782,10 +2873,12 @@ cookie_extension(Config, Cookie) -> start_client_negative(Config, Options, Error) -> ClientOpts = ssl_test_lib:ssl_options(client_rsa_opts, Config), - {ClientNode, _, Hostname} = ssl_test_lib:run_where(Config), - Client = ssl_test_lib:start_client([{node, ClientNode}, {port, 0}, + {ClientNode, ServerNode, Hostname} = ssl_test_lib:run_where(Config), + Port = ssl_test_lib:inet_port(ServerNode), + Client = ssl_test_lib:start_client([{node, ClientNode}, {port, Port}, {host, Hostname}, {from, self()}, + {return_error, econnrefused}, {mfa, {?MODULE, connection_info_result, []}}, {options, Options ++ ClientOpts}]), ct:pal("Actual: ~p~nExpected: ~p", [Client, {connect_failed, Error}]), diff --git a/lib/ssl/test/ssl_basic_SUITE.erl b/lib/ssl/test/ssl_basic_SUITE.erl index a5ece4fad8..ce0a2d0902 100644 --- a/lib/ssl/test/ssl_basic_SUITE.erl +++ b/lib/ssl/test/ssl_basic_SUITE.erl @@ -75,7 +75,9 @@ fake_root_no_intermediate_legacy/0, fake_root_no_intermediate_legacy/1, fake_intermediate_cert/0, - fake_intermediate_cert/1 + fake_intermediate_cert/1, + incompleat_chain_length/0, + incompleat_chain_length/1 ]). %% Apply export @@ -127,7 +129,8 @@ basic_tests() -> fake_root_no_intermediate, fake_root_legacy, fake_root_no_intermediate_legacy, - fake_intermediate_cert + fake_intermediate_cert, + incompleat_chain_length ]. options_tests() -> @@ -756,6 +759,74 @@ fake_intermediate_cert(Config) when is_list(Config) -> ssl_test_lib:check_client_alert(Client1, bad_certificate). +incompleat_chain_length() -> + [{doc,"Test that attempts to reconstruct incomplete chains does not make shorter incomplete chains"}]. +incompleat_chain_length(Config) when is_list(Config)-> + {ClientNode, ServerNode, Hostname} = ssl_test_lib:run_where(Config), + Ext = x509_test:extensions([{key_usage, [keyCertSign, cRLSign, digitalSignature, keyAgreement]}]), + ROOT = public_key:pkix_test_root_cert("SERVER ROOT CA", [{key, ssl_test_lib:hardcode_rsa_key(6)}, + {extensions, Ext}]), + + OtherROOT = public_key:pkix_test_root_cert("OTHER SERVER ROOT CA", [{key, ssl_test_lib:hardcode_rsa_key(3)}, + {extensions, Ext}]), + + + #{client_config := ClientConf} = public_key:pkix_test_data(#{server_chain => + #{root => ROOT, + intermediates => [[{key, ssl_test_lib:hardcode_rsa_key(5)}]], + peer => [{key, ssl_test_lib:hardcode_rsa_key(4)}]}, + client_chain => + #{root => [{key, ssl_test_lib:hardcode_rsa_key(1)}], + intermediates => [[{key, ssl_test_lib:hardcode_rsa_key(2)}]], + peer => [{key, ssl_test_lib:hardcode_rsa_key(3)}]}} + ), + + #{server_config := ServerConf} = public_key:pkix_test_data(#{server_chain => + #{root => OtherROOT, + intermediates => [[{key, ssl_test_lib:hardcode_rsa_key(2)}], + [{key, ssl_test_lib:hardcode_rsa_key(3)}] + ], + peer => [{key, ssl_test_lib:hardcode_rsa_key(1)}]}, + client_chain => + #{root => [{key, ssl_test_lib:hardcode_rsa_key(1)}], + intermediates => [[{key, ssl_test_lib:hardcode_rsa_key(2)}]], + peer => [{key, ssl_test_lib:hardcode_rsa_key(3)}]}} + ), + + + VerifyFun = {fun(_,{bad_cert, unknown_ca}, UserState) -> + %% accept this error to provoke the + %% building of an shorter incomplete chain + %% than the one recived + {valid, UserState}; + (_,{extension, _} = Extension, #{ext := N} = UserState) -> + ct:pal("~p", [Extension]), + {unknown, UserState#{ext => N +1}}; + (_, valid, #{intermediates := N} = UserState) -> + {valid, UserState#{intermediates => N +1}}; + (_, valid_peer, #{intermediates := 2, + ext := 1} = UserState) -> + {valid, UserState}; + (_, valid_peer, UserState) -> + ct:pal("~p", [UserState]), + {error, {bad_cert, too_short_path}} + end, #{intermediates => 0, + ext => 0}}, + + Server = ssl_test_lib:start_server([{node, ServerNode}, {port, 0}, + {from, self()}, + {mfa, {ssl_test_lib, send_recv_result_active, []}}, + {options, ServerConf} + ]), + Port = ssl_test_lib:inet_port(Server), + + Client = ssl_test_lib:start_client([{node, ClientNode}, {port, Port}, + {host, Hostname}, + {from, self()}, + {mfa, {ssl_test_lib, send_recv_result_active, []}}, + {options, [{verify, verify_peer}, {verify_fun, VerifyFun} | ClientConf]}]), + ssl_test_lib:check_result(Client, ok, Server, ok). + %%-------------------------------------------------------------------- %% callback functions ------------------------------------------------ %%-------------------------------------------------------------------- diff --git a/lib/ssl/test/ssl_cert_SUITE.erl b/lib/ssl/test/ssl_cert_SUITE.erl index efa01fda76..a142115b55 100644 --- a/lib/ssl/test/ssl_cert_SUITE.erl +++ b/lib/ssl/test/ssl_cert_SUITE.erl @@ -98,6 +98,8 @@ unsupported_sign_algo_cert_client_auth/1, longer_chain/0, longer_chain/1, + duplicate_chain/0, + duplicate_chain/1, key_auth_ext_sign_only/0, key_auth_ext_sign_only/1, hello_retry_request/0, @@ -870,6 +872,62 @@ longer_chain(Config) when is_list(Config) -> proplists:delete(cacerts, ClientOpts0)], Config), ssl_test_lib:basic_test(ClientOpts, ServerOpts, Config). +duplicate_chain() -> + [{doc, "Manual test of chain with duplicate entries"}]. +duplicate_chain(Config) + when is_list(Config) -> + Key1 = ssl_test_lib:hardcode_rsa_key(1), + Key2 = ssl_test_lib:hardcode_rsa_key(2), + Key3 = ssl_test_lib:hardcode_rsa_key(3), + Key4 = ssl_test_lib:hardcode_rsa_key(4), + Key5 = ssl_test_lib:hardcode_rsa_key(5), + + #{server_config := ServerOpts0, client_config := ClientOpts0} = + public_key:pkix_test_data(#{server_chain => #{root => [{key, Key1}], + peer => [{key, Key5}]}, + client_chain => #{root => [{key, Key3}], + intermediates => [[{key, Key2}], [{key, Key3}]], + peer => [{key, Key1}]}}), + + #{client_config := ClientOptsNew} = + public_key:pkix_test_data(#{server_chain => #{root => [{key, Key1}], + peer => [{key, Key5}]}, + client_chain => #{root => [{key, Key4}], + intermediates => [[{key, Key2}], [{key, Key1}]], + peer => [{key, Key1}]}}), + + ServerCas0 = proplists:get_value(cacerts, ServerOpts0), + ClientCas0 = proplists:get_value(cacerts, ClientOpts0), + + {[Peer,CI1,CI2,CROld], CROld} = chain_and_root(ClientOpts0), + {[_Peer,CI1New,CI2New,CRNew], CRNew} = chain_and_root(ClientOptsNew), + + ServerCas = [CRNew|ServerCas0 -- [CROld]], + ServerOpts = ssl_test_lib:ssl_options([{verify, verify_peer} | + lists:keyreplace(cacerts, 1, ServerOpts0, {cacerts, ServerCas})], + Config), + ClientOpts = ssl_test_lib:ssl_options([{verify, verify_peer} | + lists:keyreplace(cacerts, 1, + lists:keyreplace(cert, 1, ClientOpts0, + {cert, [Peer,CI1New,CI2New,CI1,CI2,CRNew,CROld]}), + {cacerts, ClientCas0})], + Config), + ssl_test_lib:basic_test(ClientOpts, ServerOpts, Config), + ClientOpts2 = ssl_test_lib:ssl_options([{verify, verify_peer} | + lists:keyreplace(cacerts, 1, + lists:keyreplace(cert, 1, ClientOpts0, + {cert, [Peer,CI1,CI1New,CI2,CI2New,CROld,CRNew]}), + {cacerts, ClientCas0})], + Config), + ssl_test_lib:basic_test(ClientOpts2, ServerOpts, Config), + ok. + +chain_and_root(Config) -> + OwnCert = proplists:get_value(cert, Config), + {ok, ExtractedCAs} = ssl_pkix_db:extract_trusted_certs({der, proplists:get_value(cacerts, Config)}), + {ok, Root, Chain} = ssl_certificate:certificate_chain(OwnCert, ets:new(foo, []), ExtractedCAs, []), + {Chain, Root}. + %%-------------------------------------------------------------------- %% TLS 1.3 Test cases ----------------------------------------------- %%-------------------------------------------------------------------- diff --git a/lib/ssl/test/ssl_crl_SUITE.erl b/lib/ssl/test/ssl_crl_SUITE.erl index f9fac82962..c7897d9404 100644 --- a/lib/ssl/test/ssl_crl_SUITE.erl +++ b/lib/ssl/test/ssl_crl_SUITE.erl @@ -43,6 +43,10 @@ crl_verify_valid/1, crl_verify_revoked/0, crl_verify_revoked/1, + crl_verify_valid_derCAs/0, + crl_verify_valid_derCAs/1, + crl_verify_revoked_derCAs/0, + crl_verify_revoked_derCAs/1, crl_verify_no_crl/0, crl_verify_no_crl/1, crl_hash_dir_collision/0, @@ -84,7 +88,11 @@ groups() -> {crl_verify_crldp_crlissuer, [], [crl_verify_valid]}]. basic_tests() -> - [crl_verify_valid, crl_verify_revoked, crl_verify_no_crl]. + [crl_verify_valid, + crl_verify_revoked, + crl_verify_valid_derCAs, + crl_verify_revoked_derCAs, + crl_verify_no_crl]. crl_hash_dir_tests() -> [crl_hash_dir_collision, crl_hash_dir_expired]. @@ -220,13 +228,13 @@ crl_verify_valid(Config) when is_list(Config) -> {cacertfile, filename:join([PrivDir, "server", "cacerts.pem"])}], ClientOpts = case proplists:get_value(idp_crl, Config) of true -> - [{cacertfile, filename:join([PrivDir, "server", "cacerts.pem"])}, + [{cacertfile, filename:join([PrivDir, "client", "cacerts.pem"])}, {crl_check, Check}, {crl_cache, {ssl_crl_cache, {internal, [{http, 5000}]}}}, {verify, verify_peer}]; false -> proplists:get_value(crl_cache_opts, Config) ++ - [{cacertfile, filename:join([PrivDir, "server", "cacerts.pem"])}, + [{cacertfile, filename:join([PrivDir, "client", "cacerts.pem"])}, {crl_check, Check}, {verify, verify_peer}] end, @@ -266,15 +274,79 @@ crl_verify_revoked(Config) when is_list(Config) -> crl_verify_error(Hostname, ServerNode, ServerOpts, ClientNode, ClientOpts, certificate_revoked). +crl_verify_valid_derCAs() -> + [{doc,"Verify a simple valid CRL chain"}]. +crl_verify_valid_derCAs(Config) when is_list(Config) -> + PrivDir = proplists:get_value(cert_dir, Config), + Check = proplists:get_value(crl_check, Config), + + CaCerts = der_cas(filename:join([PrivDir, "client", "cacerts.pem"])), + + ServerOpts = [{keyfile, filename:join([PrivDir, "server", "key.pem"])}, + {certfile, filename:join([PrivDir, "server", "cert.pem"])}, + {cacerts, der_cas(filename:join([PrivDir, "server", "cacerts.pem"]))} + ], + ClientOpts = case proplists:get_value(idp_crl, Config) of + true -> + [{cacerts, CaCerts}, + {crl_check, Check}, + {crl_cache, {ssl_crl_cache, {internal, [{http, 5000}]}}}, + {verify, verify_peer}]; + false -> + proplists:get_value(crl_cache_opts, Config) ++ + [{cacerts, CaCerts}, + {crl_check, Check}, + {verify, verify_peer}] + end, + {ClientNode, ServerNode, Hostname} = ssl_test_lib:run_where(Config), + + ssl_crl_cache:insert({file, filename:join([PrivDir, "erlangCA", "crl.pem"])}), + ssl_crl_cache:insert({file, filename:join([PrivDir, "otpCA", "crl.pem"])}), + + crl_verify_valid(Hostname, ServerNode, ServerOpts, ClientNode, ClientOpts). + +crl_verify_revoked_derCAs() -> + [{doc,"Verify a simple CRL chain when peer cert is reveoked"}]. +crl_verify_revoked_derCAs(Config) when is_list(Config) -> + PrivDir = proplists:get_value(cert_dir, Config), + Check = proplists:get_value(crl_check, Config), + + CaCerts = der_cas(filename:join([PrivDir, "revoked", "cacerts.pem"])), + + ServerOpts = [{keyfile, filename:join([PrivDir, "revoked", "key.pem"])}, + {certfile, filename:join([PrivDir, "revoked", "cert.pem"])}, + {cacerts, der_cas(filename:join([PrivDir, "server", "cacerts.pem"]))}], + + {ClientNode, ServerNode, Hostname} = ssl_test_lib:run_where(Config), + + ssl_crl_cache:insert({file, filename:join([PrivDir, "erlangCA", "crl.pem"])}), + ssl_crl_cache:insert({file, filename:join([PrivDir, "otpCA", "crl.pem"])}), + + ClientOpts = case proplists:get_value(idp_crl, Config) of + true -> + [{cacerts, CaCerts}, + {crl_cache, {ssl_crl_cache, {internal, [{http, 5000}]}}}, + {crl_check, Check}, + {verify, verify_peer}]; + false -> + proplists:get_value(crl_cache_opts, Config) ++ + [{cacerts, CaCerts}, + {crl_check, Check}, + {verify, verify_peer}] + end, + + crl_verify_error(Hostname, ServerNode, ServerOpts, ClientNode, ClientOpts, + certificate_revoked). crl_verify_no_crl() -> [{doc,"Verify a simple CRL chain when the CRL is missing"}]. crl_verify_no_crl(Config) when is_list(Config) -> PrivDir = proplists:get_value(cert_dir, Config), Check = proplists:get_value(crl_check, Config), + ServerOpts = [{keyfile, filename:join([PrivDir, "server", "key.pem"])}, - {certfile, filename:join([PrivDir, "server", "cert.pem"])}, - {cacertfile, filename:join([PrivDir, "server", "cacerts.pem"])}], + {certfile, filename:join([PrivDir, "server", "cert.pem"])}, + {cacertfile, filename:join([PrivDir, "server", "cacerts.pem"])}], ClientOpts = case proplists:get_value(idp_crl, Config) of true -> [{cacertfile, filename:join([PrivDir, "server", "cacerts.pem"])}, @@ -552,3 +624,9 @@ new_ca(FileName, CA1, CA2) -> Pem = public_key:pem_encode(E1 ++E2), file:write_file(FileName, Pem), FileName. + + +der_cas(CAcertsFile) -> + {ok, Pem} = file:read_file(CAcertsFile), + Decoded = public_key:pem_decode(Pem), + [DER || {_, DER, _} <- Decoded]. diff --git a/lib/ssl/test/ssl_dist_SUITE.erl b/lib/ssl/test/ssl_dist_SUITE.erl index 019e22eaa8..3b0c4d8c09 100644 --- a/lib/ssl/test/ssl_dist_SUITE.erl +++ b/lib/ssl/test/ssl_dist_SUITE.erl @@ -56,17 +56,7 @@ verify_fun_fail/0, verify_fun_fail/1, verify_fun_pass/0, - verify_fun_pass/1, - crl_check_pass/0, - crl_check_pass/1, - crl_check_fail/0, - crl_check_fail/1, - crl_check_best_effort/0, - crl_check_best_effort/1, - crl_cache_check_pass/0, - crl_cache_check_pass/1, - crl_cache_check_fail/0, - crl_cache_check_fail/1 + verify_fun_pass/1 ]). %% Apply export @@ -80,19 +70,9 @@ connect_options_test/3, verify_fun_fail_test/3, verify_fun_pass_test/3, - crl_check_fail_test/3, - crl_check_best_effort_test/3, - crl_check_pass_test/3, - crl_cache_check_fail_test/3, - crl_cache_check_pass_test/3, verify_pass_always/3, verify_fail_always/3]). -%% CRL API --export([lookup/2, - select/2, - fresh_crl/2 - ]). -define(DEFAULT_TIMETRAP_SECS, 240). -define(AWAIT_SSL_NODE_UP_TIMEOUT, 30000). @@ -119,14 +99,11 @@ all() -> connect_options, use_interface, verify_fun_fail, - verify_fun_pass, - crl_check_pass, - crl_check_fail, - crl_check_best_effort, - crl_cache_check_pass, - crl_cache_check_fail]. + verify_fun_pass + ]. init_per_suite(Config0) -> + _ = end_per_suite(Config0), try crypto:start() of ok -> %% Currently no ct function avilable for is_cover! @@ -142,18 +119,17 @@ init_per_suite(Config0) -> {skip, "Crypto did not start"} end. -end_per_suite(Config) -> - application:stop(crypto), - Config. +end_per_suite(_Config) -> + application:stop(crypto). init_per_testcase(plain_verify_options = Case, Config) when is_list(Config) -> - SslFlags = setup_dist_opts([{many_verify_opts, true} | Config]), + SslFlags = setup_tls_opts(Config), Flags = case os:getenv("ERL_FLAGS") of false -> os:putenv("ERL_FLAGS", SslFlags), ""; OldFlags -> - os:putenv("ERL_FLAGS", OldFlags ++ "" ++ SslFlags), + os:putenv("ERL_FLAGS", OldFlags ++ " " ++ SslFlags), OldFlags end, common_init(Case, [{old_flags, Flags} | Config]); @@ -184,32 +160,30 @@ basic(Config) when is_list(Config) -> %%-------------------------------------------------------------------- payload() -> - [{doc,"Test that send a lot of data between the ssl distributed noes"}]. + [{doc,"Test that send a lot of data between the ssl distributed nodes"}]. payload(Config) when is_list(Config) -> gen_dist_test(payload_test, Config). %%-------------------------------------------------------------------- plain_options() -> - [{doc,"Test specifying additional options"}]. + [{doc,"Test specifying tls options not related to certificate verification"}]. plain_options(Config) when is_list(Config) -> - DistOpts = "-ssl_dist_opt server_secure_renegotiate true " + TLSOpts = "-ssl_dist_opt server_secure_renegotiate true " "client_secure_renegotiate true " - "server_reuse_sessions true client_reuse_sessions true " - "client_verify verify_none server_verify verify_none " - "server_depth 1 client_depth 1 " "server_hibernate_after 500 client_hibernate_after 500", - gen_dist_test(plain_options_test, [{additional_dist_opts, DistOpts} | Config]). + gen_dist_test(plain_options_test, [{tls_only_basic_opts, TLSOpts} | Config]). %%-------------------------------------------------------------------- plain_verify_options() -> - [{doc,"Test specifying additional options"}]. + [{doc,"Test specifying tls options including certificate verification options"}]. plain_verify_options(Config) when is_list(Config) -> - DistOpts = "-ssl_dist_opt server_secure_renegotiate true " + TLSOpts = "-ssl_dist_opt server_secure_renegotiate true " "client_secure_renegotiate true " + "server_hibernate_after 500 client_hibernate_after 500" "server_reuse_sessions true client_reuse_sessions true " - "server_hibernate_after 500 client_hibernate_after 500", - gen_dist_test(plain_verify_options_test, [{additional_dist_opts, DistOpts} | Config]). + "server_depth 1 client_depth 1 ", + gen_dist_test(plain_verify_options_test, [{tls_verify_opts, TLSOpts} | Config]). %%-------------------------------------------------------------------- nodelay_option() -> @@ -239,7 +213,7 @@ listen_port_options(Config) when is_list(Config) -> PortOpt1 = "-kernel inet_dist_listen_min " ++ integer_to_list(Port1) ++ " inet_dist_listen_max " ++ integer_to_list(Port1), - try start_ssl_node([{additional_dist_opts, PortOpt1} | Config]) of + try start_ssl_node([{tls_verify_opts, PortOpt1} | proplists:delete(tls_verify_opts, Config)]) of #node_handle{} -> %% If the node was able to start, it didn't take the port %% option into account. @@ -254,7 +228,7 @@ listen_port_options(Config) when is_list(Config) -> %% Try again, now specifying a high max port. PortOpt2 = "-kernel inet_dist_listen_min " ++ integer_to_list(Port1) ++ " inet_dist_listen_max 65535", - NH2 = start_ssl_node([{additional_dist_opts, PortOpt2} | Config]), + NH2 = start_ssl_node([{tls_verify_opts, PortOpt2} | proplists:delete(tls_verify_opts, Config)]), try Node2 = NH2#node_handle.nodename, @@ -300,7 +274,7 @@ use_interface(Config) when is_list(Config) -> Options = "-kernel inet_dist_use_interface " ++ IpString, %% Start a node, and get the port number it's listening on. - NH1 = start_ssl_node([{additional_dist_opts, Options} | Config]), + NH1 = start_ssl_node([{tls_verify_opts, Options} | Config]), try Node1 = NH1#node_handle.nodename, @@ -330,89 +304,24 @@ use_interface(Config) when is_list(Config) -> verify_fun_fail() -> [{doc,"Test specifying verify_fun with a function that always fails"}]. verify_fun_fail(Config) when is_list(Config) -> - DistOpts = "-ssl_dist_opt " - "server_verify verify_peer server_verify_fun " + AddTLSVerifyOpts = "-ssl_dist_opt " + "server_verify_fun " "\"{ssl_dist_SUITE,verify_fail_always,{}}\" " - "client_verify verify_peer client_verify_fun " + "client_verify_fun " "\"{ssl_dist_SUITE,verify_fail_always,{}}\" ", - gen_dist_test(verify_fun_fail_test, [{additional_dist_opts, DistOpts} | Config]). + gen_dist_test(verify_fun_fail_test, [{tls_verify_opts, AddTLSVerifyOpts} | Config]). %%-------------------------------------------------------------------- verify_fun_pass() -> [{doc,"Test specifying verify_fun with a function that always succeeds"}]. verify_fun_pass(Config) when is_list(Config) -> - DistOpts = "-ssl_dist_opt " - "server_verify verify_peer server_verify_fun " + AddTLSVerifyOpts = "-ssl_dist_opt " + "server_verify_fun " "\"{ssl_dist_SUITE,verify_pass_always,{}}\" " - "server_fail_if_no_peer_cert true " - "client_verify verify_peer client_verify_fun " + "client_verify_fun " "\"{ssl_dist_SUITE,verify_pass_always,{}}\" ", - gen_dist_test(verify_fun_pass_test, [{additional_dist_opts, DistOpts} | Config]). - - -%%-------------------------------------------------------------------- -crl_check_pass() -> - [{doc,"Test crl_check with non-revoked certificate"}]. -crl_check_pass(Config) when is_list(Config) -> - DistOpts = "-ssl_dist_opt client_crl_check true", - NewConfig = - [{many_verify_opts, true}, {additional_dist_opts, DistOpts}] ++ Config, - gen_dist_test(crl_check_pass_test, NewConfig). - -%%-------------------------------------------------------------------- -crl_check_fail() -> - [{doc,"Test crl_check with revoked certificate"}]. -crl_check_fail(Config) when is_list(Config) -> - DistOpts = "-ssl_dist_opt client_crl_check true", - NewConfig = - [{many_verify_opts, true}, - %% The server uses a revoked certificate. - {server_cert_dir, "revoked"}, - {additional_dist_opts, DistOpts}] ++ Config, - gen_dist_test(crl_check_fail_test, NewConfig). - -%%-------------------------------------------------------------------- -crl_check_best_effort() -> - [{doc,"Test specifying crl_check as best_effort"}]. -crl_check_best_effort(Config) when is_list(Config) -> - DistOpts = "-ssl_dist_opt " - "server_verify verify_peer server_crl_check best_effort", - NewConfig = - [{many_verify_opts, true}, {additional_dist_opts, DistOpts}] ++ Config, - gen_dist_test(crl_check_best_effort_test, NewConfig). - -%%-------------------------------------------------------------------- -crl_cache_check_pass() -> - [{doc,"Test specifying crl_check with custom crl_cache module"}]. -crl_cache_check_pass(Config) when is_list(Config) -> - PrivDir = ?config(priv_dir, Config), - NodeDir = filename:join([PrivDir, "Certs"]), - DistOpts = "-ssl_dist_opt " - "client_crl_check true " - "client_crl_cache " - "\"{ssl_dist_SUITE,{\\\"" ++ NodeDir ++ "\\\",[]}}\"", - NewConfig = - [{many_verify_opts, true}, {additional_dist_opts, DistOpts}] ++ Config, - gen_dist_test(crl_cache_check_pass_test, NewConfig). - -%%-------------------------------------------------------------------- -crl_cache_check_fail() -> - [{doc,"Test custom crl_cache module with revoked certificate"}]. -crl_cache_check_fail(Config) when is_list(Config) -> - PrivDir = ?config(priv_dir, Config), - NodeDir = filename:join([PrivDir, "Certs"]), - DistOpts = "-ssl_dist_opt " - "client_crl_check true " - "client_crl_cache " - "\"{ssl_dist_SUITE,{\\\"" ++ NodeDir ++ "\\\",[]}}\"", - NewConfig = - [{many_verify_opts, true}, - %% The server uses a revoked certificate. - {server_cert_dir, "revoked"}, - {additional_dist_opts, DistOpts}] ++ Config, - - gen_dist_test(crl_cache_check_fail_test, NewConfig). + gen_dist_test(verify_fun_pass_test, [{tls_verify_opts, AddTLSVerifyOpts} | Config]). %%-------------------------------------------------------------------- %%% Internal functions ----------------------------------------------- @@ -573,7 +482,7 @@ do_listen_options(Prio, Config) -> end, Options = "-kernel inet_dist_listen_options " ++ PriorityString, - gen_dist_test(listen_options_test, [{prio, Prio}, {additional_dist_opts, Options} | Config]). + gen_dist_test(listen_options_test, [{prio, Prio}, {tls_only_basic_opts, Options} | Config]). listen_options_test(NH1, NH2, Config) -> Prio = proplists:get_value(prio, Config), @@ -605,7 +514,7 @@ do_connect_options(Prio, Config) -> Options = "-kernel inet_dist_connect_options " ++ PriorityString, gen_dist_test(connect_options_test, - [{prio, Prio}, {additional_dist_opts, Options} | Config]). + [{prio, Prio}, {tls_only_basic_opts, Options} | Config]). connect_options_test(NH1, NH2, Config) -> Prio = proplists:get_value(prio, Config), @@ -662,58 +571,7 @@ verify_fun_pass_test(NH1, NH2, _) -> [{verify_pass_always_ran, true}] = apply_on_ssl_node(NH2, fun () -> ets:tab2list(verify_fun_ran) end). -crl_check_fail_test(NH1, NH2, Config) -> - Node2 = NH2#node_handle.nodename, - - PrivDir = ?config(priv_dir, Config), - cache_crls_on_ssl_nodes(PrivDir, ["erlangCA", "otpCA"], [NH1, NH2]), - - %% The server's certificate is revoked, so connection fails. - pang = apply_on_ssl_node(NH1, fun () -> net_adm:ping(Node2) end), - - [] = apply_on_ssl_node(NH1, fun () -> nodes() end), - [] = apply_on_ssl_node(NH2, fun () -> nodes() end). -crl_check_best_effort_test(NH1, NH2, _Config) -> - %% We don't have the correct CRL at hand, but since crl_check is - %% best_effort, we accept it anyway. - Node1 = NH1#node_handle.nodename, - Node2 = NH2#node_handle.nodename, - - pong = apply_on_ssl_node(NH1, fun () -> net_adm:ping(Node2) end), - - [Node2] = apply_on_ssl_node(NH1, fun () -> nodes() end), - [Node1] = apply_on_ssl_node(NH2, fun () -> nodes() end). - -crl_check_pass_test(NH1, NH2, Config) -> - Node1 = NH1#node_handle.nodename, - Node2 = NH2#node_handle.nodename, - - PrivDir = ?config(priv_dir, Config), - cache_crls_on_ssl_nodes(PrivDir, ["erlangCA", "otpCA"], [NH1, NH2]), - - %% The server's certificate is not revoked, so connection succeeds. - pong = apply_on_ssl_node(NH1, fun () -> net_adm:ping(Node2) end), - - [Node2] = apply_on_ssl_node(NH1, fun () -> nodes() end), - [Node1] = apply_on_ssl_node(NH2, fun () -> nodes() end). - -crl_cache_check_pass_test(NH1, NH2, _) -> - Node1 = NH1#node_handle.nodename, - Node2 = NH2#node_handle.nodename, - - pong = apply_on_ssl_node(NH1, fun () -> net_adm:ping(Node2) end), - - [Node2] = apply_on_ssl_node(NH1, fun () -> nodes() end), - [Node1] = apply_on_ssl_node(NH2, fun () -> nodes() end). - - -crl_cache_check_fail_test(NH1, NH2, _) -> - Node2 = NH2#node_handle.nodename, - pang = apply_on_ssl_node(NH1, fun () -> net_adm:ping(Node2) end), - - [] = apply_on_ssl_node(NH1, fun () -> nodes() end), - [] = apply_on_ssl_node(NH2, fun () -> nodes() end). get_socket_priorities() -> [Priority || {ok,[{priority,Priority}]} <- @@ -723,36 +581,16 @@ inet_ports() -> [Port || Port <- erlang:ports(), element(2, erlang:port_info(Port, name)) =:= "tcp_inet"]. -%% -%% test_server side api -%% - start_ssl_node(Config) -> start_ssl_node(Config, ""). start_ssl_node(Config, XArgs) -> Name = mk_node_name(Config), - SSL = proplists:get_value(ssl_opts, Config), - SSLDistOpts = setup_dist_opts(Config), + App = proplists:get_value(app_opts, Config), + SSLOpts = setup_tls_opts(Config), start_ssl_node_name( - Name, SSL ++ " " ++ SSLDistOpts ++ XArgs). - -cache_crls_on_ssl_nodes(PrivDir, CANames, NHs) -> - [begin - File = filename:join([PrivDir, "Certs", CAName, "crl.pem"]), - {ok, PemBin} = file:read_file(File), - PemEntries = public_key:pem_decode(PemBin), - CRLs = [ CRL || {'CertificateList', CRL, not_encrypted} - <- PemEntries], - ok = apply_on_ssl_node(NH, ssl_manager, insert_crls, - ["no_distribution_point", CRLs, dist]) - end - || NH <- NHs, CAName <- CANames], - ok. + Name, App ++ " " ++ SSLOpts ++ XArgs). -%% -%% command line creation -%% mk_node_name(Config) -> N = erlang:unique_integer([positive]), @@ -763,107 +601,51 @@ mk_node_name(Config) -> ++ "_" ++ integer_to_list(N). -%% -%% Setup ssl dist info -%% - -rand_bin(N) -> - rand_bin(N, []). - -rand_bin(0, Acc) -> - Acc; -rand_bin(N, Acc) -> - rand_bin(N-1, [rand:uniform(256)-1|Acc]). - -make_randfile(Dir) -> - {ok, IoDev} = file:open(filename:join([Dir, "RAND"]), [write]), - ok = file:write(IoDev, rand_bin(1024)), - file:close(IoDev). - -append_files(FileNames, ResultFileName) -> - {ok, ResultFile} = file:open(ResultFileName, [write]), - do_append_files(FileNames, ResultFile). - -do_append_files([], RF) -> - ok = file:close(RF); -do_append_files([F|Fs], RF) -> - {ok, Data} = file:read_file(F), - ok = file:write(RF, Data), - do_append_files(Fs, RF). - setup_certs(Config) -> PrivDir = proplists:get_value(priv_dir, Config), - NodeDir = filename:join([PrivDir, "Certs"]), - RGenDir = filename:join([NodeDir, "rand_gen"]), - ok = file:make_dir(NodeDir), - ok = file:make_dir(RGenDir), - make_randfile(RGenDir), - [Hostname|_] = string:split(net_adm:localhost(), ".", all), - {ok, _} = make_certs:all(RGenDir, NodeDir, [{hostname,Hostname}]), - SDir = filename:join([NodeDir, "server"]), - SC = filename:join([SDir, "cert.pem"]), - SK = filename:join([SDir, "key.pem"]), - SKC = filename:join([SDir, "keycert.pem"]), - append_files([SK, SC], SKC), - CDir = filename:join([NodeDir, "client"]), - CC = filename:join([CDir, "cert.pem"]), - CK = filename:join([CDir, "key.pem"]), - CKC = filename:join([CDir, "keycert.pem"]), - append_files([CK, CC], CKC). - -setup_dist_opts(Config) -> + DerConfig = public_key:pkix_test_data(#{server_chain => #{root => rsa_root_key(1), + intermediates => [rsa_intermediate(2)], + peer => rsa_peer_key(3)}, + client_chain => #{root => rsa_root_key(1), + intermediates => [rsa_intermediate(5)], + peer => rsa_peer_key(6)}}), + ClientBase = filename:join([PrivDir, "rsa"]), + SeverBase = filename:join([PrivDir, "rsa"]), + + _ = x509_test:gen_pem_config_files(DerConfig, ClientBase, SeverBase). + +setup_tls_opts(Config) -> PrivDir = proplists:get_value(priv_dir, Config), - DataDir = proplists:get_value(data_dir, Config), - Dhfile = filename:join([DataDir, "dHParam.pem"]), - NodeDir = filename:join([PrivDir, "Certs"]), - SDir = filename:join([NodeDir, proplists:get_value(server_cert_dir, Config, "server")]), - CDir = filename:join([NodeDir, proplists:get_value(client_cert_dir, Config, "client")]), - SC = filename:join([SDir, "cert.pem"]), - SK = filename:join([SDir, "key.pem"]), - SKC = filename:join([SDir, "keycert.pem"]), - SCA = filename:join([CDir, "cacerts.pem"]), - CC = filename:join([CDir, "cert.pem"]), - CK = filename:join([CDir, "key.pem"]), - CKC = filename:join([CDir, "keycert.pem"]), - CCA = filename:join([SDir, "cacerts.pem"]), - - DistOpts = case proplists:get_value(many_verify_opts, Config, false) of - false -> - "-proto_dist inet_tls " - ++ "-ssl_dist_opt server_certfile " ++ SKC ++ " " - ++ "-ssl_dist_opt client_certfile " ++ CKC ++ " "; - true -> - case os:type() of - {win32, _} -> - "-proto_dist inet_tls " - ++ "-ssl_dist_opt server_certfile " ++ SKC ++ " " - ++ "-ssl_dist_opt server_cacertfile " ++ SCA ++ " " - ++ "-ssl_dist_opt server_verify verify_peer " - ++ "-ssl_dist_opt server_fail_if_no_peer_cert true " - ++ "-ssl_dist_opt server_ciphers DHE-RSA-AES256-SHA\:DHE-RSA-AES128-SHA " - ++ "-ssl_dist_opt server_dhfile " ++ Dhfile ++ " " - ++ "-ssl_dist_opt client_certfile " ++ CKC ++ " " - ++ "-ssl_dist_opt client_cacertfile " ++ CCA ++ " " - ++ "-ssl_dist_opt client_verify verify_peer " - ++ "-ssl_dist_opt client_ciphers DHE-RSA-AES256-SHA\:DHE-RSA-AES128-SHA "; - _ -> - "-proto_dist inet_tls " - ++ "-ssl_dist_opt server_certfile " ++ SC ++ " " - ++ "-ssl_dist_opt server_keyfile " ++ SK ++ " " - ++ "-ssl_dist_opt server_cacertfile " ++ SCA ++ " " - ++ "-ssl_dist_opt server_verify verify_peer " - ++ "-ssl_dist_opt server_fail_if_no_peer_cert true " - ++ "-ssl_dist_opt server_ciphers DHE-RSA-AES256-SHA\:DHE-RSA-AES128-SHA " - ++ "-ssl_dist_opt server_dhfile " ++ Dhfile ++ " " - ++ "-ssl_dist_opt client_certfile " ++ CC ++ " " - ++ "-ssl_dist_opt client_keyfile " ++ CK ++ " " - ++ "-ssl_dist_opt client_cacertfile " ++ CCA ++ " " - ++ "-ssl_dist_opt client_verify verify_peer " - ++ "-ssl_dist_opt client_ciphers DHE-RSA-AES256-SHA\:DHE-RSA-AES128-SHA " - end - end, - MoreOpts = proplists:get_value(additional_dist_opts, Config, []), - DistOpts ++ MoreOpts. + SC = filename:join([PrivDir, "rsa_server_cert.pem"]), + SK = filename:join([PrivDir, "rsa_server_key.pem"]), + SCA = filename:join([PrivDir, "rsa_server_cacerts.pem"]), + CC = filename:join([PrivDir, "rsa_client_cert.pem"]), + CK = filename:join([PrivDir, "rsa_client_key.pem"]), + CCA = filename:join([PrivDir, "rsa_client_cacerts.pem"]), + + case proplists:get_value(tls_only_basic_opts, Config, []) of + [_|_] = BasicOpts -> %% No verify but server still need to have cert + "-proto_dist inet_tls " ++ "-ssl_dist_opt server_certfile " ++ SC ++ " " + ++ "-ssl_dist_opt server_keyfile " ++ SK ++ " " ++ BasicOpts; + [] -> %% Verify + case proplists:get_value(tls_verify_opts, Config, []) of + [_|_] -> + BasicVerifyOpts = "-proto_dist inet_tls " + ++ "-ssl_dist_opt server_certfile " ++ SC ++ " " + ++ "-ssl_dist_opt server_keyfile " ++ SK ++ " " + ++ "-ssl_dist_opt server_cacertfile " ++ SCA ++ " " + ++ "-ssl_dist_opt server_verify verify_peer " + ++ "-ssl_dist_opt server_fail_if_no_peer_cert true " + ++ "-ssl_dist_opt client_certfile " ++ CC ++ " " + ++ "-ssl_dist_opt client_keyfile " ++ CK ++ " " + ++ "-ssl_dist_opt client_cacertfile " ++ CCA ++ " " + ++ "-ssl_dist_opt client_verify verify_peer ", + BasicVerifyOpts ++ proplists:get_value(tls_verify_opts, Config, []); + _ -> %% No verify, no extra opts + "-proto_dist inet_tls " ++ "-ssl_dist_opt server_certfile " ++ SC ++ " " + ++ "-ssl_dist_opt server_keyfile " ++ SK ++ " " + end + end. %% %% Start scripts etc... @@ -914,20 +696,16 @@ add_ssl_opts_config(Config) -> SSL_VSN]), ok = file:close(RelFile), ok = systools:make_script(Script, []), - [{ssl_opts, "-boot " ++ Script} | Config] + [{app_opts, "-boot " ++ Script} | Config] catch _:_ -> - [{ssl_opts, "-pa \"" ++ filename:dirname(code:which(ssl))++"\""} + [{app_opts, "-pa \"" ++ filename:dirname(code:which(ssl))++"\""} | add_comment_config( "Bootscript wasn't used since the test wasn't run on an " "installed OTP system.", Config)] end. -%% -%% Add common comments to config -%% - add_comment_config(Comment, []) -> [{comment, Comment}]; add_comment_config(Comment, [{comment, OldComment} | Cs]) -> @@ -935,9 +713,6 @@ add_comment_config(Comment, [{comment, OldComment} | Cs]) -> add_comment_config(Comment, [C|Cs]) -> [C|add_comment_config(Comment, Cs)]. -%% -%% Call when test case success -%% success(Config) -> case lists:keysearch(comment, 1, Config) of @@ -965,6 +740,7 @@ verify_fail_always(_Certificate, _Event, _State) -> Parent = self(), spawn( fun() -> + catch ets:delete(verify_fun_ran), ets:new(verify_fun_ran, [public, named_table]), ets:insert(verify_fun_ran, {verify_fail_always_ran, true}), Parent ! go_ahead, @@ -979,6 +755,7 @@ verify_pass_always(_Certificate, _Event, State) -> Parent = self(), spawn( fun() -> + catch ets:delete(verify_fun_ran), ets:new(verify_fun_ran, [public, named_table]), ets:insert(verify_fun_ran, {verify_pass_always_ran, true}), Parent ! go_ahead, @@ -987,28 +764,6 @@ verify_pass_always(_Certificate, _Event, State) -> receive go_ahead -> ok end, {valid, State}. -%% ssl_crl_cache_api callbacks -lookup(_DistributionPoint, _DbHandle) -> - not_available. - -select({rdnSequence, NameParts}, {NodeDir, _}) -> - %% Extract the CN from the issuer name... - [CN] = [CN || - [#'AttributeTypeAndValue'{ - type = ?'id-at-commonName', - value = <<_, _, CN/binary>>}] <- NameParts], - %% ...and use that as the directory name to find the CRL. - error_logger:info_report([{found_cn, CN}]), - CRLFile = filename:join([NodeDir, CN, "crl.pem"]), - {ok, PemBin} = file:read_file(CRLFile), - PemEntries = public_key:pem_decode(PemBin), - CRLs = [ CRL || {'CertificateList', CRL, not_encrypted} - <- PemEntries], - CRLs. - -fresh_crl(_DistributionPoint, CRL) -> - CRL. - localhost_ip(InetVer) -> {ok, Addr} = inet:getaddr(net_adm:localhost(), InetVer), Addr. @@ -1026,3 +781,16 @@ localhost_ipstr(InetVer) -> inet_ver() -> inet. + +rsa_root_key(N) -> + %% As rsa keygen is not guaranteed to be fast + [{key, ssl_test_lib:hardcode_rsa_key(N)}]. + +rsa_peer_key(N) -> + %% As rsa keygen is not guaranteed to be fast + [{key, ssl_test_lib:hardcode_rsa_key(N)}]. + +rsa_intermediate(N) -> + [{key, ssl_test_lib:hardcode_rsa_key(N)}]. + + diff --git a/lib/ssl/test/ssl_dist_test_lib.erl b/lib/ssl/test/ssl_dist_test_lib.erl index bf010e6ad4..fc6646690d 100644 --- a/lib/ssl/test/ssl_dist_test_lib.erl +++ b/lib/ssl/test/ssl_dist_test_lib.erl @@ -104,7 +104,7 @@ start_ssl_node(Name, Args) -> case open_port({spawn, CmdLine}, []) of Port when is_port(Port) -> unlink(Port), - erlang:port_close(Port), + catch erlang:port_close(Port), case await_ssl_node_up(Name, LSock) of #node_handle{} = NodeHandle -> ?t:format("Ssl node ~s started.~n", [Name]), diff --git a/lib/ssl/test/ssl_eqc_SUITE.erl b/lib/ssl/test/ssl_eqc_SUITE.erl index 3c9a1d0ab0..4bfff1585e 100644 --- a/lib/ssl/test/ssl_eqc_SUITE.erl +++ b/lib/ssl/test/ssl_eqc_SUITE.erl @@ -39,7 +39,8 @@ tls_unorded_chains/1, tls_extraneous_chain/1, tls_extraneous_chains/1, - tls_extraneous_and_unorder_chains/1 + tls_extraneous_and_unorder_chains/1, + tls_client_cert_auth/1 ]). %%-------------------------------------------------------------------- @@ -56,7 +57,8 @@ all() -> tls_unorded_chains, tls_extraneous_chain, tls_extraneous_chains, - tls_extraneous_and_unorder_chains + tls_extraneous_and_unorder_chains, + tls_client_cert_auth ]. %%-------------------------------------------------------------------- @@ -123,3 +125,9 @@ tls_extraneous_and_unorder_chains(Config) when is_list(Config) -> ssl:start(), true = ct_property_test:quickcheck(ssl_eqc_chain:prop_tls_extraneous_and_unordered_path(), Config). + +tls_client_cert_auth(Config) when is_list(Config) -> + %% manual test: proper:quickcheck(ssl_eqc_chain:prop_client_cert_auth() + ssl:start(), + true = ct_property_test:quickcheck(ssl_eqc_chain:prop_client_cert_auth(), + Config). diff --git a/lib/ssl/test/ssl_session_SUITE.erl b/lib/ssl/test/ssl_session_SUITE.erl index b11e49ad89..552be7ddd6 100644 --- a/lib/ssl/test/ssl_session_SUITE.erl +++ b/lib/ssl/test/ssl_session_SUITE.erl @@ -216,7 +216,7 @@ make_sure_expired(Host, Port, Id) -> State = ssl_test_lib:state(Prop), ClientCache = element(2, State), - case ssl_session_cache:lookup(ClientCache, {{Host, Port}, Id}) of + case ssl_client_session_cache_db:lookup(ClientCache, {{Host, Port}, Id}) of undefined -> ok; #session{is_resumable = false} -> diff --git a/lib/ssl/test/ssl_session_cache_SUITE.erl b/lib/ssl/test/ssl_session_cache_SUITE.erl index 7c182ee063..8f67908ad7 100644 --- a/lib/ssl/test/ssl_session_cache_SUITE.erl +++ b/lib/ssl/test/ssl_session_cache_SUITE.erl @@ -63,6 +63,7 @@ -define(SLEEP, 1000). -define(TIMEOUT, {seconds, 20}). -define(MAX_TABLE_SIZE, 5). +-define(CLIENT_CB, ssl_client_session_cache_db). %%-------------------------------------------------------------------- %% Common Test interface functions ----------------------------------- @@ -225,7 +226,7 @@ client_unique_session(Config) when is_list(Config) -> State = ssl_test_lib:state(Prop), ClientCache = element(2, State), - 1 = ssl_session_cache:size(ClientCache), + 1 = ?CLIENT_CB:size(ClientCache), ssl_test_lib:close(Server, 500), ssl_test_lib:close(LastClient). @@ -263,7 +264,7 @@ session_cleanup(Config) when is_list(Config) -> SessionTimer = element(6, State), Id = proplists:get_value(session_id, SessionInfo), - CSession = ssl_session_cache:lookup(ClientCache, {{Hostname, Port}, Id}), + CSession = ?CLIENT_CB:lookup(ClientCache, {{Hostname, Port}, Id}), true = CSession =/= undefined, @@ -272,7 +273,7 @@ session_cleanup(Config) when is_list(Config) -> ct:sleep(?SLEEP), %% Make sure clean has had time to run - undefined = ssl_session_cache:lookup(ClientCache, {{Hostname, Port}, Id}), + undefined = ?CLIENT_CB:lookup(ClientCache, {{Hostname, Port}, Id}), process_flag(trap_exit, false), ssl_test_lib:close(Server), @@ -335,7 +336,7 @@ save_specific_session(Config) when is_list(Config) -> [_, _,_, _, Prop] = StatusInfo, State = ssl_test_lib:state(Prop), ClientCache = element(2, State), - 2 = ssl_session_cache:size(ClientCache), + 2 = ?CLIENT_CB:size(ClientCache), Server ! listen, @@ -379,7 +380,7 @@ max_table_size(Config) when is_list(Config) -> [_, _,_, _, Prop] = StatusInfo, State = ssl_test_lib:state(Prop), ClientCache = element(2, State), - M = ssl_session_cache:size(ClientCache), + M = ?CLIENT_CB:size(ClientCache), ct:pal("~p",[M]), ssl_test_lib:close(Server, 500), ssl_test_lib:close(LastClient), diff --git a/lib/ssl/test/ssl_session_cache_api_SUITE.erl b/lib/ssl/test/ssl_session_cache_api_SUITE.erl new file mode 100644 index 0000000000..03f88f755f --- /dev/null +++ b/lib/ssl/test/ssl_session_cache_api_SUITE.erl @@ -0,0 +1,105 @@ +%% +%% %CopyrightBegin% +%% +%% Copyright Ericsson AB 2010-2020. All Rights Reserved. +%% +%% Licensed under the Apache License, Version 2.0 (the "License"); +%% you may not use this file except in compliance with the License. +%% You may obtain a copy of the License at +%% +%% http://www.apache.org/licenses/LICENSE-2.0 +%% +%% Unless required by applicable law or agreed to in writing, software +%% distributed under the License is distributed on an "AS IS" BASIS, +%% WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +%% See the License for the specific language governing permissions and +%% limitations under the License. +%% +%% %CopyrightEnd% +%% + +%% + +-module(ssl_session_cache_api_SUITE). + +-behaviour(ct_suite). + +-include_lib("common_test/include/ct.hrl"). +-include("tls_handshake.hrl"). + +%% Callback functions +-export([all/0]). + +%% Testcases +-export([server_cb/0, + server_cb/1, + client_cb/0, + client_cb/1 + ]). + +%%-------------------------------------------------------------------- +%% Common Test interface functions ----------------------------------- +%%-------------------------------------------------------------------- +all() -> + [server_cb, + client_cb]. +%%-------------------------------------------------------------------- +%% Test Cases -------------------------------------------------------- +%%-------------------------------------------------------------------- +server_cb() -> + [{doc, "Test ssl_session_cache_api server callback implementation" + "Note that default implementation is treated special to avoid" + "an extra process to handle the functional db structure used" + "The callback is provided for the main purpose of having a" + "session table that may survive node restart an assumes + a db reference that is not updated" + }]. + +server_cb(Config) when is_list(Config) -> + Cb = ssl_server_session_cache_db, + Db0 = Cb:init([]), + Id0 = crypto:strong_rand_bytes(32), + DummySession0 = #session{session_id = Id0}, + 0 = Cb:size(Db0), + + Db1 = Cb:update(Db0, Id0, DummySession0), + 1 = Cb:size(Db1), + + Id1 = crypto:strong_rand_bytes(32), + DummySession1 = #session{session_id = Id1}, + Db2 = Cb:update(Db1, Id1, DummySession1), + 2 = Cb:size(Db2), + + DummySession0 = Cb:lookup(Db2, Id0), + DummySession1 = Cb:lookup(Db2, Id1), + + Db3 = Cb:delete(Db2, Id1), + 1 = Cb:size(Db3), + undefined = Cb:lookup(Db3, crypto:strong_rand_bytes(32)), + Cb:terminate(Db3). + +client_cb() -> + [{doc, "Test ssl_session_cache_api client callback implementation"}]. + +client_cb(Config) when is_list(Config) -> + Cb = ssl_client_session_cache_db, + Db = Cb:init([]), + Id0 = crypto:strong_rand_bytes(32), + DummySession0 = #session{session_id = Id0}, + 0 = Cb:size(Db), + + Cb:update(Db, Id0, DummySession0), + 1 = Cb:size(Db), + + Id1 = crypto:strong_rand_bytes(32), + DummySession1 = #session{session_id = Id1}, + Cb:update(Db, Id1, DummySession1), + 2 = Cb:size(Db), + + DummySession0 = Cb:lookup(Db, Id0), + DummySession1 = Cb:lookup(Db, Id1), + undefined = Cb:lookup(Db, crypto:strong_rand_bytes(32)), + + Cb:delete(Db, Id1), + 1 = Cb:size(Db), + Cb:terminate(Db). diff --git a/lib/ssl/test/ssl_session_ticket_SUITE.erl b/lib/ssl/test/ssl_session_ticket_SUITE.erl index 16791e2c36..d39afaa5b7 100644 --- a/lib/ssl/test/ssl_session_ticket_SUITE.erl +++ b/lib/ssl/test/ssl_session_ticket_SUITE.erl @@ -54,7 +54,23 @@ multiple_tickets/0, multiple_tickets/1, multiple_tickets_2hash/0, - multiple_tickets_2hash/1]). + multiple_tickets_2hash/1, + early_data_client_too_much_data/0, + early_data_client_too_much_data/1, + early_data_trial_decryption/0, + early_data_trial_decryption/1, + early_data_trial_decryption_failure/0, + early_data_trial_decryption_failure/1, + early_data_decryption_failure/0, + early_data_decryption_failure/1, + early_data_disabled_small_limit/0, + early_data_disabled_small_limit/1, + early_data_enabled_small_limit/0, + early_data_enabled_small_limit/1, + early_data_basic/0, + early_data_basic/1, + early_data_basic_auth/0, + early_data_basic_auth/1]). -include("tls_handshake.hrl"). @@ -83,7 +99,15 @@ session_tests() -> [basic, hello_retry_request, multiple_tickets, - multiple_tickets_2hash]. + multiple_tickets_2hash, + early_data_client_too_much_data, + early_data_trial_decryption, + early_data_trial_decryption_failure, + early_data_decryption_failure, + early_data_disabled_small_limit, + early_data_enabled_small_limit, + early_data_basic, + early_data_basic_auth]. mixed_tests() -> [ @@ -127,6 +151,7 @@ init_per_testcase(_, Config) -> Config. end_per_testcase(_TestCase, Config) -> + application:unset_env(ssl, server_session_ticket_max_early_data), Config. %%-------------------------------------------------------------------- @@ -583,6 +608,516 @@ multiple_tickets_2hash(Config) when is_list(Config) -> process_flag(trap_exit, false), ssl_test_lib:close(Server0). +early_data_trial_decryption() -> + [{doc,"Test trial decryption when server rejects early data (erlang client - erlang server)"}]. +early_data_trial_decryption(Config) when is_list(Config) -> + ClientOpts0 = ssl_test_lib:ssl_options(client_rsa_verify_opts, Config), + ServerOpts0 = ssl_test_lib:ssl_options(server_rsa_verify_opts, Config), + {ClientNode, ServerNode, Hostname} = ssl_test_lib:run_where(Config), + ServerTicketMode = proplists:get_value(server_ticket_mode, Config), + + %% Configure session tickets + ClientOpts1 = [{session_tickets, auto}, {log_level, debug}, + {versions, ['tlsv1.2','tlsv1.3']}|ClientOpts0], + %% Send maximum sized early data to verify calculation of plain text size + %% in the server. + ClientOpts2 = [{early_data, binary:copy(<<"F">>, 16384)}|ClientOpts1], + + %% Disabled early data triggers trial decryption upon receiving early data + ServerOpts = [{session_tickets, ServerTicketMode}, {early_data, disabled}, + {log_level, debug}, + {versions, ['tlsv1.2','tlsv1.3']}|ServerOpts0], + + Server0 = + ssl_test_lib:start_server([{node, ServerNode}, {port, 0}, + {from, self()}, + {mfa, {ssl_test_lib, + verify_active_session_resumption, + [false]}}, + {options, ServerOpts}]), + Port0 = ssl_test_lib:inet_port(Server0), + + %% Store ticket from first connection + Client0 = ssl_test_lib:start_client([{node, ClientNode}, + {port, Port0}, {host, Hostname}, + {mfa, {ssl_test_lib, %% Full handshake + verify_active_session_resumption, + [false]}}, + {from, self()}, {options, ClientOpts1}]), + ssl_test_lib:check_result(Server0, ok, Client0, ok), + + Server0 ! {listen, {mfa, {ssl_test_lib, + verify_active_session_resumption, + [true]}}}, + + %% Wait for session ticket + ct:sleep(100), + + ssl_test_lib:close(Client0), + + %% Use ticket + Client1 = ssl_test_lib:start_client([{node, ClientNode}, + {port, Port0}, {host, Hostname}, + {mfa, {ssl_test_lib, %% Short handshake + verify_active_session_resumption, + [true]}}, + {from, self()}, {options, ClientOpts2}]), + ssl_test_lib:check_result(Server0, ok, Client1, ok), + + process_flag(trap_exit, false), + ssl_test_lib:close(Server0), + ssl_test_lib:close(Client1). + +early_data_client_too_much_data() -> + [{doc,"Client sending too much early data (erlang client - erlang server)"}]. +early_data_client_too_much_data(Config) when is_list(Config) -> + ClientOpts0 = ssl_test_lib:ssl_options(client_rsa_verify_opts, Config), + ServerOpts0 = ssl_test_lib:ssl_options(server_rsa_verify_opts, Config), + {ClientNode, ServerNode, Hostname} = ssl_test_lib:run_where(Config), + ServerTicketMode = proplists:get_value(server_ticket_mode, Config), + + %% Configure session tickets + ClientOpts1 = [{session_tickets, manual}, {log_level, debug}, + {versions, ['tlsv1.2','tlsv1.3']}|ClientOpts0], + %% Send more early data than max_early_data_size to verify calculation + %% of plain text size in the server. + MaxEarlyDataSize = 10000, + ClientOpts2 = [{early_data, binary:copy(<<"F">>, 16384)}|ClientOpts1], + + ServerOpts = [{session_tickets, ServerTicketMode}, {early_data, disabled}, + {log_level, debug}, + {versions, ['tlsv1.2','tlsv1.3']}|ServerOpts0], + + application:set_env(ssl, server_session_ticket_max_early_data, MaxEarlyDataSize), + Server0 = + ssl_test_lib:start_server([{node, ServerNode}, {port, 0}, + {from, self()}, + {mfa, {ssl_test_lib, + verify_active_session_resumption, + [false]}}, + {options, ServerOpts}]), + application:unset_env(ssl, server_session_ticket_max_early_data), + Port0 = ssl_test_lib:inet_port(Server0), + + %% Store ticket from first connection + Client0 = ssl_test_lib:start_client([{node, ClientNode}, + {port, Port0}, {host, Hostname}, + {mfa, {ssl_test_lib, %% Full handshake + verify_active_session_resumption, + [false, no_reply, {tickets, 1}]}}, + {from, self()}, {options, ClientOpts1}]), + Tickets0 = ssl_test_lib:check_tickets(Client0), + ssl_test_lib:verify_session_ticket_extension(Tickets0, MaxEarlyDataSize), + %% ssl_test_lib:check_result(Server0, ok, Client0, ok), + + Server0 ! {listen, {mfa, {ssl_test_lib, + verify_active_session_resumption, + [false, no_reply]}}}, + + %% Wait for session ticket + ct:sleep(100), + + ssl_test_lib:close(Client0), + + %% Use ticket + Client1 = ssl_test_lib:start_client_error([{node, ClientNode}, + {port, Port0}, {host, Hostname}, + {mfa, {ssl_test_lib, %% Short handshake + verify_active_session_resumption, + [false, no_reply, no_tickets]}}, + {from, self()}, {options, [{use_ticket, Tickets0}|ClientOpts2]}]), + ssl_test_lib:check_client_alert(Client1, illegal_parameter), + process_flag(trap_exit, false), + ssl_test_lib:close(Server0). + +early_data_trial_decryption_failure() -> + [{doc,"Emulate faulty client that sends too much early data (erlang client - erlang server)"}]. +early_data_trial_decryption_failure(Config) when is_list(Config) -> + ClientOpts0 = ssl_test_lib:ssl_options(client_rsa_verify_opts, Config), + ServerOpts0 = ssl_test_lib:ssl_options(server_rsa_verify_opts, Config), + {ClientNode, ServerNode, Hostname} = ssl_test_lib:run_where(Config), + ServerTicketMode = proplists:get_value(server_ticket_mode, Config), + + %% Configure session tickets + ClientOpts1 = [{session_tickets, manual}, {log_level, debug}, + {versions, ['tlsv1.2','tlsv1.3']}|ClientOpts0], + %% Send more early data than max_early_data_size to verify calculation + %% of plain text size in the server. + MaxEarlyDataSize = 10000, + ClientOpts2 = [{early_data, binary:copy(<<"F">>, 16385)}|ClientOpts1], + + %% Disabled early data triggers trial decryption upon receiving early data + %% up to the configured amount. If more data is received the server triggers + %% a bad_record_mac alert. + %% It is not possible to trigger this condition in normal use cases: + %% - The ssl client in auto mode has a built in protection against sending + %% too much early data. It will not send any early data. + %% - The ssl client in manual mode can only send the mount that is received + %% in the ticket used for the 0-RTT handshake. If more data is sent the + %% client will trigger an illegal_parameter alert (too_much_early_data). + ServerOpts = [{session_tickets, ServerTicketMode}, {early_data, disabled}, + {log_level, debug}, + {versions, ['tlsv1.2','tlsv1.3']}|ServerOpts0], + + application:set_env(ssl, server_session_ticket_max_early_data, MaxEarlyDataSize), + Server0 = + ssl_test_lib:start_server([{node, ServerNode}, {port, 0}, + {from, self()}, + {mfa, {ssl_test_lib, + verify_active_session_resumption, + [false]}}, + {options, ServerOpts}]), + Port0 = ssl_test_lib:inet_port(Server0), + + %% Store ticket from first connection + Client0 = ssl_test_lib:start_client([{node, ClientNode}, + {port, Port0}, {host, Hostname}, + {mfa, {ssl_test_lib, %% Full handshake + verify_active_session_resumption, + [false, no_reply, {tickets, 1}]}}, + {from, self()}, {options, ClientOpts1}]), + Tickets0 = ssl_test_lib:check_tickets(Client0), + %% Simulate a faulty client by updating the max_early_data_size extension in + %% the received session ticket + Tickets1 = ssl_test_lib:update_session_ticket_extension(Tickets0, 16385), + %% ssl_test_lib:check_result(Server0, ok, Client0, ok), + + Server0 ! {listen, {mfa, {ssl_test_lib, + verify_active_session_resumption, + [false, no_reply]}}}, + + %% Wait for session ticket + ct:sleep(100), + + ssl_test_lib:close(Client0), + + %% Use ticket + Client1 = ssl_test_lib:start_client_error([{node, ClientNode}, + {port, Port0}, {host, Hostname}, + {mfa, {ssl_test_lib, %% Short handshake + verify_active_session_resumption, + [false, no_reply, no_tickets]}}, + {from, self()}, {options, [{use_ticket, Tickets1}|ClientOpts2]}]), + ssl_test_lib:check_server_alert(Server0, bad_record_mac), + process_flag(trap_exit, false), + application:unset_env(ssl, server_session_ticket_max_early_data), + ssl_test_lib:close(Server0). + +early_data_decryption_failure() -> + [{doc,"Emulate faulty client that sends too much early data - server early_data enabled (erlang client - erlang server)"}]. +early_data_decryption_failure(Config) when is_list(Config) -> + ClientOpts0 = ssl_test_lib:ssl_options(client_rsa_verify_opts, Config), + ServerOpts0 = ssl_test_lib:ssl_options(server_rsa_verify_opts, Config), + {ClientNode, ServerNode, Hostname} = ssl_test_lib:run_where(Config), + ServerTicketMode = proplists:get_value(server_ticket_mode, Config), + + %% Configure session tickets + ClientOpts1 = [{session_tickets, manual}, {log_level, debug}, + {versions, ['tlsv1.2','tlsv1.3']}|ClientOpts0], + %% Send more early data than max_early_data_size to verify calculation + %% of plain text size in the server. + MaxEarlyDataSize = 10000, + ClientOpts2 = [{early_data, binary:copy(<<"F">>, 16385)}|ClientOpts1], + + ServerOpts = [{session_tickets, ServerTicketMode}, {early_data, enabled}, + {log_level, debug}, + {versions, ['tlsv1.2','tlsv1.3']}|ServerOpts0], + + application:set_env(ssl, server_session_ticket_max_early_data, MaxEarlyDataSize), + Server0 = + ssl_test_lib:start_server([{node, ServerNode}, {port, 0}, + {from, self()}, + {mfa, {ssl_test_lib, + verify_active_session_resumption, + [false]}}, + {options, ServerOpts}]), + Port0 = ssl_test_lib:inet_port(Server0), + + %% Store ticket from first connection + Client0 = ssl_test_lib:start_client([{node, ClientNode}, + {port, Port0}, {host, Hostname}, + {mfa, {ssl_test_lib, %% Full handshake + verify_active_session_resumption, + [false, no_reply, {tickets, 1}]}}, + {from, self()}, {options, ClientOpts1}]), + Tickets0 = ssl_test_lib:check_tickets(Client0), + %% Simulate a faulty client by updating the max_early_data_size extension in + %% the received session ticket + Tickets1 = ssl_test_lib:update_session_ticket_extension(Tickets0, 16385), + %% ssl_test_lib:check_result(Server0, ok, Client0, ok), + + Server0 ! {listen, {mfa, {ssl_test_lib, + verify_active_session_resumption, + [false, no_reply]}}}, + + %% Wait for session ticket + ct:sleep(100), + + ssl_test_lib:close(Client0), + + %% Use ticket + Client1 = ssl_test_lib:start_client_error([{node, ClientNode}, + {port, Port0}, {host, Hostname}, + {mfa, {ssl_test_lib, %% Short handshake + verify_active_session_resumption, + [false, no_reply, no_tickets]}}, + {from, self()}, {options, [{use_ticket, Tickets1}|ClientOpts2]}]), + ssl_test_lib:check_server_alert(Server0, unexpected_message), + process_flag(trap_exit, false), + application:unset_env(ssl, server_session_ticket_max_early_data), + ssl_test_lib:close(Server0). + +early_data_disabled_small_limit() -> + [{doc,"Test trial decryption when server rejects early data (erlang client - erlang server)"}]. +early_data_disabled_small_limit(Config) when is_list(Config) -> + ClientOpts0 = ssl_test_lib:ssl_options(client_rsa_verify_opts, Config), + ServerOpts0 = ssl_test_lib:ssl_options(server_rsa_verify_opts, Config), + {ClientNode, ServerNode, Hostname} = ssl_test_lib:run_where(Config), + ServerTicketMode = proplists:get_value(server_ticket_mode, Config), + + %% Configure session tickets + ClientOpts1 = [{session_tickets, auto}, {log_level, debug}, + {versions, ['tlsv1.2','tlsv1.3']}|ClientOpts0], + %% Send maximum sized early data to verify calculation of plain text size + %% in the server. + MaxEarlyDataSize = 5, + ClientOpts2 = [{early_data, binary:copy(<<"F">>, 4)}|ClientOpts1], + + %% Disabled early data triggers trial decryption upon receiving early data + ServerOpts = [{session_tickets, ServerTicketMode}, {early_data, disabled}, + {log_level, debug}, + {versions, ['tlsv1.2','tlsv1.3']}|ServerOpts0], + application:set_env(ssl, server_session_ticket_max_early_data, MaxEarlyDataSize), + Server0 = + ssl_test_lib:start_server([{node, ServerNode}, {port, 0}, + {from, self()}, + {mfa, {ssl_test_lib, + verify_active_session_resumption, + [false]}}, + {options, ServerOpts}]), + Port0 = ssl_test_lib:inet_port(Server0), + + %% Store ticket from first connection + Client0 = ssl_test_lib:start_client([{node, ClientNode}, + {port, Port0}, {host, Hostname}, + {mfa, {ssl_test_lib, %% Full handshake + verify_active_session_resumption, + [false]}}, + {from, self()}, {options, ClientOpts1}]), + ssl_test_lib:check_result(Server0, ok, Client0, ok), + + Server0 ! {listen, {mfa, {ssl_test_lib, + verify_active_session_resumption, + [true]}}}, + + %% Wait for session ticket + ct:sleep(100), + + ssl_test_lib:close(Client0), + + %% Use ticket + Client1 = ssl_test_lib:start_client([{node, ClientNode}, + {port, Port0}, {host, Hostname}, + {mfa, {ssl_test_lib, %% Short handshake + verify_active_session_resumption, + [true]}}, + {from, self()}, {options, ClientOpts2}]), + ssl_test_lib:check_result(Server0, ok, Client1, ok), + + process_flag(trap_exit, false), + application:unset_env(ssl, server_session_ticket_max_early_data), + ssl_test_lib:close(Server0), + ssl_test_lib:close(Client1). + +early_data_enabled_small_limit() -> + [{doc,"Test decryption when server accepts early data (erlang client - erlang server)"}]. +early_data_enabled_small_limit(Config) when is_list(Config) -> + ClientOpts0 = ssl_test_lib:ssl_options(client_rsa_verify_opts, Config), + ServerOpts0 = ssl_test_lib:ssl_options(server_rsa_verify_opts, Config), + {ClientNode, ServerNode, Hostname} = ssl_test_lib:run_where(Config), + ServerTicketMode = proplists:get_value(server_ticket_mode, Config), + + %% Configure session tickets + ClientOpts1 = [{session_tickets, auto}, {log_level, debug}, + {versions, ['tlsv1.2','tlsv1.3']}|ClientOpts0], + %% Send maximum sized early data to verify calculation of plain text size + %% in the server. + MaxEarlyDataSize = 5, + ClientOpts2 = [{early_data, binary:copy(<<"F">>, 4)}|ClientOpts1], + + %% Disabled early data triggers trial decryption upon receiving early data + ServerOpts = [{session_tickets, ServerTicketMode}, {early_data, enabled}, + {log_level, debug}, + {versions, ['tlsv1.2','tlsv1.3']}|ServerOpts0], + application:set_env(ssl, server_session_ticket_max_early_data, MaxEarlyDataSize), + Server0 = + ssl_test_lib:start_server([{node, ServerNode}, {port, 0}, + {from, self()}, + {mfa, {ssl_test_lib, + verify_active_session_resumption, + [false]}}, + {options, ServerOpts}]), + Port0 = ssl_test_lib:inet_port(Server0), + + %% Store ticket from first connection + Client0 = ssl_test_lib:start_client([{node, ClientNode}, + {port, Port0}, {host, Hostname}, + {mfa, {ssl_test_lib, %% Full handshake + verify_active_session_resumption, + [false]}}, + {from, self()}, {options, ClientOpts1}]), + ssl_test_lib:check_result(Server0, ok, Client0, ok), + + Server0 ! {listen, {mfa, {ssl_test_lib, + verify_active_session_resumption, + [true]}}}, + + %% Wait for session ticket + ct:sleep(100), + + ssl_test_lib:close(Client0), + + %% Use ticket + Client1 = ssl_test_lib:start_client([{node, ClientNode}, + {port, Port0}, {host, Hostname}, + {mfa, {ssl_test_lib, %% Short handshake + verify_active_session_resumption, + [true]}}, + {from, self()}, {options, ClientOpts2}]), + ssl_test_lib:check_result(Server0, ok, Client1, ok), + + process_flag(trap_exit, false), + application:unset_env(ssl, server_session_ticket_max_early_data), + ssl_test_lib:close(Server0), + ssl_test_lib:close(Client1). + +early_data_basic() -> + [{doc,"Test early data when client is not authenticated (erlang client - erlang server)"}]. +early_data_basic(Config) when is_list(Config) -> + ClientOpts0 = ssl_test_lib:ssl_options(client_rsa_verify_opts, Config), + ServerOpts0 = ssl_test_lib:ssl_options(server_rsa_verify_opts, Config), + {ClientNode, ServerNode, Hostname} = ssl_test_lib:run_where(Config), + ServerTicketMode = proplists:get_value(server_ticket_mode, Config), + + %% Configure session tickets + ClientOpts1 = [{session_tickets, auto}, {log_level, debug}, + {versions, ['tlsv1.2','tlsv1.3']}|ClientOpts0], + %% Send maximum sized early data to verify calculation of plain text size + %% in the server. + EarlyData = binary:copy(<<"F">>, 16384), + ClientOpts2 = [{early_data, binary:copy(<<"F">>, 16384)}|ClientOpts1], + + %% Disabled early data triggers trial decryption upon receiving early data + ServerOpts = [{session_tickets, ServerTicketMode}, {early_data, enabled}, + {log_level, debug}, + {versions, ['tlsv1.2','tlsv1.3']}|ServerOpts0], + + Server0 = + ssl_test_lib:start_server([{node, ServerNode}, {port, 0}, + {from, self()}, + {mfa, {ssl_test_lib, + verify_active_session_resumption, + [false]}}, + {options, ServerOpts}]), + Port0 = ssl_test_lib:inet_port(Server0), + + %% Store ticket from first connection + Client0 = ssl_test_lib:start_client([{node, ClientNode}, + {port, Port0}, {host, Hostname}, + {mfa, {ssl_test_lib, %% Full handshake + verify_active_session_resumption, + [false]}}, + {from, self()}, {options, ClientOpts1}]), + ssl_test_lib:check_result(Server0, ok, Client0, ok), + + Server0 ! {listen, {mfa, {ssl_test_lib, + verify_server_early_data, + [wait_reply, EarlyData]}}}, + + %% Wait for session ticket + ct:sleep(100), + + ssl_test_lib:close(Client0), + + %% Use ticket + Client1 = ssl_test_lib:start_client([{node, ClientNode}, + {port, Port0}, {host, Hostname}, + {mfa, {ssl_test_lib, %% Short handshake + verify_active_session_resumption, + [true]}}, + {from, self()}, {options, ClientOpts2}]), + ssl_test_lib:check_result(Server0, ok, Client1, ok), + + process_flag(trap_exit, false), + ssl_test_lib:close(Server0), + ssl_test_lib:close(Client1). + +early_data_basic_auth() -> + [{doc,"Test early data when client is authenticated (erlang client - erlang server)"}]. +early_data_basic_auth(Config) when is_list(Config) -> + ClientOpts0 = ssl_test_lib:ssl_options(client_rsa_verify_opts, Config), + ServerOpts0 = ssl_test_lib:ssl_options(server_rsa_verify_opts, Config), + {ClientNode, ServerNode, Hostname} = ssl_test_lib:run_where(Config), + ServerTicketMode = proplists:get_value(server_ticket_mode, Config), + + %% Configure session tickets + ClientOpts1 = [{session_tickets, auto}, {log_level, debug}, + {versions, ['tlsv1.2','tlsv1.3']}|ClientOpts0], + %% Send maximum sized early data to verify calculation of plain text size + %% in the server. + EarlyData = binary:copy(<<"F">>, 16384), + ClientOpts2 = [{early_data, binary:copy(<<"F">>, 16384)}|ClientOpts1], + + %% Disabled early data triggers trial decryption upon receiving early data + ServerOpts = [{session_tickets, ServerTicketMode}, {early_data, enabled}, + {log_level, debug}, + {versions, ['tlsv1.2','tlsv1.3']}|ServerOpts0], + + Server0 = + ssl_test_lib:start_server([{node, ServerNode}, {port, 0}, + {from, self()}, + {mfa, {ssl_test_lib, + verify_active_session_resumption, + [false]}}, + {options, ServerOpts}]), + Port0 = ssl_test_lib:inet_port(Server0), + + %% Store ticket from first connection + Client0 = ssl_test_lib:start_client([{node, ClientNode}, + {port, Port0}, {host, Hostname}, + {mfa, {ssl_test_lib, %% Full handshake + verify_active_session_resumption, + [false]}}, + {from, self()}, {options, ClientOpts1}]), + ssl_test_lib:check_result(Server0, ok, Client0, ok), + + Server0 ! {listen, {mfa, {ssl_test_lib, + verify_server_early_data, + [wait_reply, EarlyData]}}}, + + %% Wait for session ticket + ct:sleep(100), + + ssl_test_lib:close(Client0), + + %% TODO This test should fail! + %% State transition is not implemented from wait_eoed to wait_cert! + %% Use ticket + Client1 = ssl_test_lib:start_client([{node, ClientNode}, + {port, Port0}, {host, Hostname}, + {mfa, {ssl_test_lib, %% Short handshake + verify_active_session_resumption, + [true]}}, + {from, self()}, + {options, + proplists:delete(keyfile, + proplists:delete(certfile, ClientOpts2))}]), + ssl_test_lib:check_result(Server0, ok, Client1, ok), + + process_flag(trap_exit, false), + ssl_test_lib:close(Server0), + ssl_test_lib:close(Client1). + %%-------------------------------------------------------------------- %% Internal functions ------------------------------------------------ diff --git a/lib/ssl/test/ssl_test_lib.erl b/lib/ssl/test/ssl_test_lib.erl index ab09e4e67a..012eb9217e 100644 --- a/lib/ssl/test/ssl_test_lib.erl +++ b/lib/ssl/test/ssl_test_lib.erl @@ -25,6 +25,7 @@ -include_lib("common_test/include/ct.hrl"). -include_lib("public_key/include/public_key.hrl"). +-include_lib("ssl/src/tls_handshake_1_3.hrl"). -export([clean_start/0, clean_start/1, @@ -98,6 +99,10 @@ verify_active_session_resumption/2, verify_active_session_resumption/3, verify_active_session_resumption/4, + verify_active_session_resumption/5, + verify_server_early_data/3, + verify_session_ticket_extension/2, + update_session_ticket_extension/2, check_sane_openssl_version/1, check_ok/1, check_result/4, @@ -119,7 +124,8 @@ server_msg/2, hardcode_rsa_key/1, bigger_buffers/0, - stop/2 + stop/2, + working_openssl_client/0 ]). -export([basic_test/3, @@ -170,7 +176,9 @@ ecdsa_suites/1, der_to_pem/2, pem_to_der/1, - appropriate_sha/1 + appropriate_sha/1, + format_certs/1, + format_cert/1 ]). -export([maybe_force_ipv4/1, @@ -193,12 +201,14 @@ version_flag/1, portable_cmd/2, portable_open_port/2, - close_port/1 + close_port/1, + verify_early_data/1 ]). -record(sslsocket, { fd = nil, pid = nil}). -define(SLEEP, 1000). -define(DEFAULT_CURVE, secp256r1). +-define(PRINT_DEPTH, 100). %%==================================================================== %% API @@ -284,6 +294,20 @@ init_per_group(GroupName, Config0) -> end end. +working_openssl_client() -> + case portable_cmd("openssl", ["version"]) of + %% Theses versions of OpenSSL has a client that + %% can not handle hello extensions. And will + %% fail with bad packet length if they are present + %% in ServerHello + "OpenSSL 0.9.8h" ++ _ -> + false; + "OpenSSL 0.9.8k" ++ _ -> + false; + _ -> + true + end. + init_per_group_openssl(GroupName, Config0) -> case is_tls_version(GroupName) andalso sufficient_crypto_support(GroupName) of true -> @@ -397,7 +421,7 @@ run_server(Opts) -> Options = proplists:get_value(options, Opts), Pid = proplists:get_value(from, Opts), Transport = proplists:get_value(transport, Opts, ssl), - ct:log("~p:~p~nssl:listen(~p, ~p)~n", [?MODULE,?LINE, Port, Options]), + ct:log("~p:~p~nssl:listen(~p, ~p)~n", [?MODULE,?LINE, Port, format_options(Options)]), %% {ok, ListenSocket} = Transport:listen(Port, Options), case Transport:listen(Port, Options) of {ok, ListenSocket} -> @@ -521,9 +545,9 @@ connect(ListenSocket, _Opts) -> connect(_, _, 0, AcceptSocket, _, _, _) -> AcceptSocket; connect(ListenSocket, Node, _N, _, Timeout, SslOpts, cancel) -> - ct:log("ssl:transport_accept(~p)~n", [ListenSocket]), + ct:log("ssl:transport_accept(~P)~n", [ListenSocket, ?PRINT_DEPTH]), {ok, AcceptSocket} = ssl:transport_accept(ListenSocket), - ct:log("~p:~p~nssl:handshake(~p,~p,~p)~n", [?MODULE,?LINE, AcceptSocket, SslOpts,Timeout]), + ct:log("~p:~p~nssl:handshake(~p,~p,~p)~n", [?MODULE,?LINE, AcceptSocket, format_options(SslOpts),Timeout]), case ssl:handshake(AcceptSocket, SslOpts, Timeout) of {ok, Socket0, Ext} -> @@ -535,7 +559,7 @@ connect(ListenSocket, Node, _N, _, Timeout, SslOpts, cancel) -> Result end; connect(ListenSocket, Node, N, _, Timeout, SslOpts, [_|_] =ContOpts0) -> - ct:log("ssl:transport_accept(~p)~n", [ListenSocket]), + ct:log("ssl:transport_accept(~P)~n", [ListenSocket, ?PRINT_DEPTH]), {ok, AcceptSocket} = ssl:transport_accept(ListenSocket), ct:log("~p:~p~nssl:handshake(~p,~p,~p)~n", [?MODULE,?LINE, AcceptSocket, SslOpts,Timeout]), @@ -567,7 +591,7 @@ connect(ListenSocket, Node, N, _, Timeout, SslOpts, [_|_] =ContOpts0) -> Result end; connect(ListenSocket, Node, N, _, Timeout, [], ContOpts) -> - ct:log("ssl:transport_accept(~p)~n", [ListenSocket]), + ct:log("ssl:transport_accept(~P)~n", [ListenSocket, ?PRINT_DEPTH]), {ok, AcceptSocket} = ssl:transport_accept(ListenSocket), ct:log("~p:~p~nssl:ssl_accept(~p, ~p)~n", [?MODULE,?LINE, AcceptSocket, Timeout]), @@ -579,7 +603,7 @@ connect(ListenSocket, Node, N, _, Timeout, [], ContOpts) -> Result end; connect(ListenSocket, _Node, _, _, Timeout, Opts, _) -> - ct:log("ssl:transport_accept(~p)~n", [ListenSocket]), + ct:log("ssl:transport_accept(~P)~n", [ListenSocket, ?PRINT_DEPTH]), {ok, AcceptSocket} = ssl:transport_accept(ListenSocket), ct:log("ssl:handshake(~p,~p, ~p)~n", [AcceptSocket, Opts, Timeout]), ssl:handshake(AcceptSocket, Opts, Timeout), @@ -645,7 +669,8 @@ start_openssl_server(Mode, Args0, Config) -> Node = proplists:get_value(node, Args0, ServerNode), Port = proplists:get_value(port, Args0, 0), ResponderPort = proplists:get_value(responder_port, Config, 0), - Args = [{from, self()}, {port, Port}] ++ ServerOpts ++ Args0, + PrivDir = proplists:get_value(priv_dir, Config), + Args = [{from, self()}, {port, Port}] ++ ServerOpts ++ Args0 ++ [{priv_dir, PrivDir}], Result = spawn_link(Node, ?MODULE, init_openssl_server, [Mode, ResponderPort,lists:delete(return_port, Args)]), receive @@ -667,10 +692,12 @@ init_openssl_server(openssl, _, Options) -> Exe = "openssl", Ciphers = proplists:get_value(ciphers, Options, default_ciphers(Version)), Groups0 = proplists:get_value(groups, Options), + EarlyData = proplists:get_value(early_data, Options, undefined), + PrivDir = proplists:get_value(priv_dir, Options), CertArgs = openssl_cert_options(Options, server), AlpnArgs = openssl_alpn_options(proplists:get_value(alpn, Options, undefined)), NpnArgs = openssl_npn_options(proplists:get_value(np, Options, undefined)), - Debug = openssl_debug_options(), + Debug = openssl_debug_options(PrivDir), Args0 = case Groups0 of undefined -> @@ -682,7 +709,14 @@ init_openssl_server(openssl, _, Options) -> ciphers(Ciphers, Version), "-groups", Group, version_flag(Version)] ++ AlpnArgs ++ NpnArgs ++ CertArgs ++ Debug end, - Args = maybe_force_ipv4(Args0), + Args1 = case EarlyData of + undefined -> + Args0; + MaxSize -> + Args0 ++ ["-early_data", "-no_anti_replay", "-max_early_data", + integer_to_list(MaxSize)] + end, + Args = maybe_force_ipv4(Args1), SslPort = portable_open_port(Exe, Args), wait_for_openssl_server(Port, proplists:get_value(protocol, Options, tls)), Pid ! {started, SslPort}, @@ -884,7 +918,7 @@ run_client(Opts) -> Options = proplists:get_value(options, Opts), ContOpts = proplists:get_value(continue_options, Opts, []), ct:log("~p:~p~n~p:connect(~p, ~p)@~p~n", [?MODULE,?LINE, Transport, Host, Port, Node]), - ct:log("SSLOpts: ~p", [Options]), + ct:log("SSLOpts: ~p", [format_options(Options)]), case ContOpts of [] -> client_loop(Node, Host, Port, Pid, Transport, Options, Opts); @@ -910,16 +944,21 @@ client_loop(_Node, Host, Port, Pid, Transport, Options, Opts) -> end, client_loop_core(Socket, Pid, Transport); {error, econnrefused = Reason} -> - case get(retries) of - N when N < 5 -> - ct:log("~p:~p~neconnrefused retries=~p sleep ~p",[?MODULE,?LINE, N,?SLEEP]), - put(retries, N+1), - ct:sleep(?SLEEP), - run_client(Opts); - _ -> - ct:log("~p:~p~nClient faild several times: connection failed: ~p ~n", [?MODULE,?LINE, Reason]), - Pid ! {self(), {error, Reason}} - end; + case proplists:get_value(return_error, Opts, undefined) of + econnrefused -> + Pid ! {connect_failed, Reason}; + _ -> + case get(retries) of + N when N < 5 -> + ct:log("~p:~p~neconnrefused retries=~p sleep ~p",[?MODULE,?LINE, N,?SLEEP]), + put(retries, N+1), + ct:sleep(?SLEEP), + run_client(Opts); + _ -> + ct:log("~p:~p~nClient faild several times: connection failed: ~p ~n", [?MODULE,?LINE, Reason]), + Pid ! {self(), {error, Reason}} + end + end; {error, econnreset = Reason} -> case get(retries) of N when N < 5 -> @@ -1102,11 +1141,14 @@ check_server_alert(Pid, Alert) -> {Pid, {error, {tls_alert, {Alert, STxt}}}} -> check_server_txt(STxt), ok; + {Pid, {error, {tls_alert, {OtherAlert, STxt}}}} -> + ct:fail("Unexpected alert during negative test: ~p - ~p", [OtherAlert, STxt]); {Pid, {error, closed}} -> ok; {Pid, {ok, _}} -> ct:fail("Successful connection during negative test.") end. + check_server_alert(Server, Client, Alert) -> receive {Server, {error, {tls_alert, {Alert, STxt}}}} -> @@ -1115,14 +1157,19 @@ check_server_alert(Server, Client, Alert) -> {Server, {ok, _}} -> ct:fail("Successful connection during negative test.") end. + check_client_alert(Pid, Alert) -> receive {Pid, {error, {tls_alert, {Alert, CTxt}}}} -> check_client_txt(CTxt), ok; + {Pid, {error, {tls_alert, {OtherAlert, CTxt}}}} -> + ct:fail("Unexpected alert during negative test: ~p - ~p", [OtherAlert, CTxt]); {Pid, {ssl_error, _, {tls_alert, {Alert, CTxt}}}} -> check_client_txt(CTxt), ok; + {Pid, {ssl_error, _, {tls_alert, {OtherAlert, CTxt}}}} -> + ct:fail("Unexpected alert during negative test: ~p - ~p", [OtherAlert, CTxt]); {Pid, {error, closed}} -> ok; {Pid, {ok, _}} -> @@ -1199,6 +1246,44 @@ wait_for_result(Pid, Msg) -> %% Unexpected end. +format_options([{cacerts, Certs}|R]) -> + [{cacerts, format_certs(Certs)} | format_options(R)]; +format_options([{cert, Certs}|R]) -> + [{cert, format_certs(Certs)} | format_options(R)]; +format_options([{key, Key}|R]) -> + [{key, lists:flatten(io_lib:format("~W",[Key, ?PRINT_DEPTH]))} | format_options(R)]; +format_options([Opt|R]) -> + [Opt | format_options(R)]; +format_options([]) -> + []. + +format_certs(Certs) when is_list(Certs) -> + [lists:flatten(format_cert(C)) || C <- Certs]; +format_certs(Cert) when is_binary(Cert) -> + lists:flatten(format_cert(Cert)). + +format_cert(BinCert) when is_binary(BinCert) -> + OtpCert = #'OTPCertificate'{tbsCertificate = Cert} = public_key:pkix_decode_cert(BinCert, otp), + #'OTPTBSCertificate'{subject = Subject, serialNumber = Nr, issuer = Issuer} = Cert, + case public_key:pkix_is_self_signed(OtpCert) of + true -> + io_lib:format("~.3w: ~s -> selfsigned", [Nr, format_subject(Subject)]); + false -> + case public_key:pkix_issuer_id(OtpCert, other) of + {ok, {IsNr, Issuer0}} -> + io_lib:format("~.3w:~s -> ~.3w:~s", [Nr, format_subject(Subject), IsNr, format_subject(Issuer0)]); + {error, _} -> + io_lib:format("~.3w:~s -> :~s", [Nr, format_subject(Subject), format_subject(Issuer)]) + end + end. + +format_subject({rdnSequence, Seq}) -> + format_subject(Seq); +format_subject([[{'AttributeTypeAndValue', ?'id-at-commonName', {_, String}}]|_]) -> + String; +format_subject([_|R]) -> + format_subject(R). + cert_options(Config) -> ClientCaCertFile = filename:join([proplists:get_value(priv_dir, Config), "client", "cacerts.pem"]), @@ -1261,8 +1346,7 @@ cert_options(Config) -> {server_bad_key, [{ssl_imp, new},{cacertfile, ServerCaCertFile}, {certfile, ServerCertFile}, {keyfile, BadKeyFile}]} | Config]. - - + make_dsa_cert(Config) -> CryptoSupport = crypto:supports(), case proplists:get_bool(dss, proplists:get_value(public_keys, CryptoSupport)) of @@ -1866,8 +1950,8 @@ accepters(Acc, N) -> basic_test(COpts, SOpts, Config) -> - SType = proplists:get_value(server_type, Config), - CType = proplists:get_value(client_type, Config), + SType = proplists:get_value(server_type, Config, erlang), + CType = proplists:get_value(client_type, Config, erlang), {Server, Port} = start_server(SType, COpts, SOpts, Config), Client = start_client(CType, Port, COpts, Config), gen_check_result(Server, SType, Client, CType), @@ -2055,9 +2139,11 @@ cipher_flag('tlsv1.3') -> cipher_flag(_) -> "-cipher". -ciphers(Ciphers, Version) -> +ciphers([#{}| _] = Ciphers, Version) -> Strs = [ssl_cipher_format:suite_map_to_openssl_str(Cipher) || Cipher <- Ciphers], - ciphers_concat(Version, Strs, ""). + ciphers_concat(Version, Strs, ""); +ciphers(Ciphers, Version) -> + ciphers_concat(Version, Ciphers, ""). ciphers_concat(_, [], [":" | Acc]) -> lists:append(lists:reverse(Acc)); @@ -2092,6 +2178,23 @@ openssl_maxfag_option(Int) -> openssl_debug_options() -> ["-msg", "-debug"]. +%% +openssl_debug_options(PrivDir) -> + case is_keylogfile_supported() of + true -> + ["-msg", "-debug","-keylogfile", PrivDir ++ "keylog"]; + false -> + ["-msg", "-debug"] + end. + +is_keylogfile_supported() -> + [{_,_, Bin}] = crypto:info_lib(), + case binary_to_list(Bin) of + "OpenSSL 1.1.1" ++ _ -> + true; + _ -> + false + end. start_server_with_raw_key(erlang, ServerOpts, Config) -> {_, ServerNode, _} = run_where(Config), @@ -2542,13 +2645,19 @@ send_recv_result_active_once(Socket) -> ssl:send(Socket, Data), active_once_recv_list(Socket, length(Data)). +%% This function can verify the following functionalities in clients: +%% - session resumption, sending/receiving application data, receiving session tickets +%% - verifying if client early data is accepted/rejected verify_active_session_resumption(Socket, SessionResumption) -> - verify_active_session_resumption(Socket, SessionResumption, wait_reply, no_tickets). + verify_active_session_resumption(Socket, SessionResumption, wait_reply, no_tickets, no_early_data). %% verify_active_session_resumption(Socket, SessionResumption, WaitReply) -> - verify_active_session_resumption(Socket, SessionResumption, WaitReply, no_tickets). + verify_active_session_resumption(Socket, SessionResumption, WaitReply, no_tickets, no_early_data). %% -verify_active_session_resumption(Socket, SessionResumption, WaitForReply, TicketOption) -> +verify_active_session_resumption(Socket, SessionResumption, WaitReply, TicketOption) -> + verify_active_session_resumption(Socket, SessionResumption, WaitReply, TicketOption, no_early_data). +%% +verify_active_session_resumption(Socket, SessionResumption, WaitForReply, TicketOption, EarlyData) -> case ssl:connection_information(Socket, [session_resumption]) of {ok, [{session_resumption, SessionResumption}]} -> Msg = boolean_to_log_msg(SessionResumption), @@ -2574,15 +2683,93 @@ verify_active_session_resumption(Socket, SessionResumption, WaitForReply, Ticket Else1 -> ct:fail("~p:~p~nFaulty parameter: ~p", [?MODULE, ?LINE, Else1]) end, - case TicketOption of - {tickets, N} -> - receive_tickets(N); - no_tickets -> - ok; - Else2 -> - ct:fail("~p:~p~nFaulty parameter: ~p", [?MODULE, ?LINE, Else2]) + Tickets = + case TicketOption of + {tickets, N} -> + receive_tickets(N); + no_tickets -> + ok; + Else2 -> + ct:fail("~p:~p~nFaulty parameter: ~p", [?MODULE, ?LINE, Else2]) + end, + case EarlyData of + {verify_early_data, Atom} -> + case verify_early_data(Atom) of + ok -> + Tickets; + Else -> + ct:fail("~p:~p~nFailed to verify early_data! (expected ~p, got ~p)", + [?MODULE, ?LINE, Atom, Else]) + end; + no_early_data -> + Tickets; + Else3 -> + ct:fail("~p:~p~nFaulty parameter: ~p", [?MODULE, ?LINE, Else3]) end. +verify_server_early_data(Socket, WaitForReply, EarlyData) -> + case ssl:connection_information(Socket, [session_resumption]) of + {ok, [{session_resumption, true}]} -> + Msg = boolean_to_log_msg(true), + ct:log("~p:~p~nSession resumption verified! (expected ~p, got ~p)!", + [?MODULE, ?LINE, Msg, Msg]); + {ok, [{session_resumption, Got0}]} -> + Expected = boolean_to_log_msg(true), + Got = boolean_to_log_msg(Got0), + ct:fail("~p:~p~nFailed to verify session resumption! (expected ~p, got ~p)", + [?MODULE, ?LINE, Expected, Got]); + {error, Reason} -> + ct:fail("~p:~p~nFailed to verify session resumption! Reason: ~p", + [?MODULE, ?LINE, Reason]) + end, + Data = "Hello world", + ssl:send(Socket, Data), + Reply = + case EarlyData of + no_early_data -> + Data; + _ -> + binary_to_list(EarlyData) ++ Data + end, + ct:log("Expected Reply: ~p~n", [Reply]), + case WaitForReply of + wait_reply -> + Reply = active_recv(Socket, length(Reply)); + no_reply -> + ok; + Else1 -> + ct:fail("~p:~p~nFaulty parameter: ~p", [?MODULE, ?LINE, Else1]) + end, + ok. + +verify_session_ticket_extension([Ticket0|_], MaxEarlyDataSize) -> + #{ticket := #new_session_ticket{ + extensions = #{early_data := + #early_data_indication_nst{ + indication = Size}}}} = Ticket0, + case Size of + MaxEarlyDataSize -> + ct:log("~p:~p~nmax_early_data_size verified! (expected ~p, got ~p)!", + [?MODULE, ?LINE, MaxEarlyDataSize, Size]); + Else -> + ct:log("~p:~p~nFailed to verify max_early_data_size! (expected ~p, got ~p)!", + [?MODULE, ?LINE, MaxEarlyDataSize, Else]) + end. + +update_session_ticket_extension([Ticket|_], MaxEarlyDataSize) -> + #{ticket := #new_session_ticket{ + extensions = #{early_data := + #early_data_indication_nst{ + indication = Size}}}} = Ticket, + ct:log("~p:~p~nOverwrite max_early_data_size (from ~p to ~p)!", + [?MODULE, ?LINE, Size, MaxEarlyDataSize]), + #{ticket := #new_session_ticket{ + extensions = #{early_data := Extensions0}} = NST0} = Ticket, + Extensions = #{early_data => #early_data_indication_nst{ + indication = MaxEarlyDataSize}}, + NST = NST0#new_session_ticket{extensions = Extensions}, + [Ticket#{ticket => NST}]. + boolean_to_log_msg(true) -> "OK"; boolean_to_log_msg(false) -> @@ -2595,7 +2782,7 @@ receive_tickets(0, Acc) -> Acc; receive_tickets(N, Acc) -> receive - {ssl, session_ticket, {_, Ticket}} -> + {ssl, session_ticket, Ticket} -> receive_tickets(N - 1, [Ticket|Acc]) end. @@ -2629,7 +2816,8 @@ active_recv(_Socket, N, Acc) when N < 0 -> T; active_recv(Socket, N, Acc) -> receive - {ssl, Socket, Bytes} -> + %% Filter {ssl, Socket, {early_data, Atom}} messages + {ssl, Socket, Bytes} when not is_tuple(Bytes) -> active_recv(Socket, N-data_length(Bytes), Acc ++ Bytes); {Socket, {data, Bytes0}} -> Bytes = filter_openssl_debug_data(Bytes0), @@ -2831,7 +3019,7 @@ check_sane_openssl_renegotiate(Config) -> {skip, "Known renegotiation bug in OpenSSL"}; "LibreSSL 2." ++ _ -> {skip, "Known renegotiation bug in LibreSSL"}; - "LibreSSL 3.1" ++ _ -> + "LibreSSL 3." ++ _ -> {skip, "Known renegotiation bug in LibreSSL"}; _ -> Config @@ -2928,19 +3116,23 @@ close_loop(Port, Time, SentClose) -> end. portable_open_port("openssl" = Exe, Args0) -> - case os:getenv("WSLENV") of - false -> - AbsPath = os:find_executable(Exe), - ct:pal("open_port({spawn_executable, ~p}, [{args, ~p}, stderr_to_stdout]).", - [AbsPath, Args0]), - open_port({spawn_executable, AbsPath}, - [{args, Args0}, stderr_to_stdout]); + IsWindows = case os:type() of + {win32, _} -> true; + _ -> false + end, + case IsWindows andalso os:getenv("WSLENV") of + false -> + AbsPath = os:find_executable(Exe), + ct:pal("open_port({spawn_executable, ~p}, [{args, ~p}, stderr_to_stdout]).", + [AbsPath, Args0]), + open_port({spawn_executable, AbsPath}, + [{args, Args0}, stderr_to_stdout]); _ -> %% I can't get the new windows version of openssl.exe to be stable %% certain server tests are failing for no reason. %% This is using "linux" openssl via wslenv - Translate = fun("c:/" ++ _ = Path) -> + Translate = fun([_Drive|":/" ++ _ ]= Path) -> string:trim(os:cmd("wsl wslpath -u " ++ Path)); (Arg) -> Arg @@ -2955,8 +3147,8 @@ portable_open_port("openssl" = Exe, Args0) -> portable_open_port(Exe, Args) -> AbsPath = os:find_executable(Exe), ct:pal("open_port({spawn_executable, ~p}, [{args, ~p}, stderr_to_stdout]).", [AbsPath, Args]), - open_port({spawn_executable, AbsPath}, - [{args, Args}, stderr_to_stdout]). + open_port({spawn_executable, AbsPath}, + [{args, Args}, stderr_to_stdout]). portable_cmd(Exe, Args) -> Port = portable_open_port(Exe, Args), @@ -3064,14 +3256,6 @@ protocol_version(Config, atom) -> tls_record:protocol_version(protocol_version(Config, tuple)) end. -protocol_versions(Config) -> - Version = protocol_version(Config), - case Version of - 'tlsv1.3' -> %% TLS-1.3 servers shall also support 1.2 - ['tlsv1.3', 'tlsv1.2']; - _ -> - [Version] - end. protocol_options(Config, Options) -> Protocol = proplists:get_value(protocol, Config, tls), {Protocol, Opts} = lists:keyfind(Protocol, 1, Options), @@ -3121,21 +3305,6 @@ clean_start(keep_version) -> clean_env(keep_version), ssl:start(). -is_psk_anon_suite({psk, _,_}) -> - true; -is_psk_anon_suite({dhe_psk,_,_}) -> - true; -is_psk_anon_suite({ecdhe_psk,_,_}) -> - true; -is_psk_anon_suite({psk, _,_,_}) -> - true; -is_psk_anon_suite({dhe_psk, _,_,_}) -> - true; -is_psk_anon_suite({ecdhe_psk, _,_,_}) -> - true; -is_psk_anon_suite(_) -> - false. - tls_version('dtlsv1' = Atom) -> dtls_v1:corresponding_tls_version(dtls_record:protocol_version(Atom)); @@ -3625,4 +3794,12 @@ default_ciphers(Version) -> ssl:cipher_suites(default, Version) end, [Cipher || Cipher <- Ciphers, lists:member(ssl:suite_to_openssl_str(Cipher), OpenSSLCiphers)]. - + +verify_early_data(Atom) -> + receive + {ssl, _Socket, {early_data, Atom}} -> + ok; + {ssl, _Socket, {early_data, Other}} -> + Other + end. + diff --git a/lib/ssl/test/tls_1_3_record_SUITE.erl b/lib/ssl/test/tls_1_3_record_SUITE.erl index 7a61cf9411..f5f57b534b 100644 --- a/lib/ssl/test/tls_1_3_record_SUITE.erl +++ b/lib/ssl/test/tls_1_3_record_SUITE.erl @@ -90,20 +90,25 @@ encode_decode(_Config) -> client_verify_data => undefined,compression_state => undefined, mac_secret => undefined,secure_renegotiation => undefined, security_parameters => - {security_parameters, - <<19,2>>, - 0,8,2,undefined,undefined,undefined,undefined,undefined, - sha384,undefined,undefined, - {handshake_secret, - <<128,229,186,211,62,127,182,20,62,166,233,23,135,64,121, - 3,104,251,214,161,253,31,3,2,232,37,8,221,189,72,64,218, - 121,41,112,148,254,34,68,164,228,60,161,201,132,55,56, - 157>>}, undefined, undefined, - undefined, - <<92,24,205,75,244,60,136,212,250,32,214,20,37,3,213,87,61,207, - 147,61,168,145,177,118,160,153,33,53,48,108,191,174>>, - undefined}, - sequence_number => 0,server_verify_data => undefined}, + #security_parameters{ + cipher_suite = <<19,2>>, + connection_end = 0, + bulk_cipher_algorithm = 8, + cipher_type = 2, + prf_algorithm = sha384, + master_secret = + {handshake_secret, + <<128,229,186,211,62,127,182,20,62,166,233,23,135,64,121, + 3,104,251,214,161,253,31,3,2,232,37,8,221,189,72,64,218, + 121,41,112,148,254,34,68,164,228,60,161,201,132,55,56, + 157>>}, + server_random = + <<92,24,205,75,244,60,136,212,250,32,214,20,37,3,213,87,61,207, + 147,61,168,145,177,118,160,153,33,53,48,108,191,174>>}, + sequence_number => 0,server_verify_data => undefined, + max_early_data_size => 0, + trial_decryption => false, + early_data_limit => false}, current_write => #{beast_mitigation => one_n_minus_one, cipher_state => @@ -116,19 +121,21 @@ encode_decode(_Config) -> client_verify_data => undefined,compression_state => undefined, mac_secret => undefined,secure_renegotiation => undefined, security_parameters => - {security_parameters, - <<19,2>>, - 0,8,2,undefined,undefined,undefined,undefined,undefined, - sha384,undefined,undefined, - {handshake_secret, - <<128,229,186,211,62,127,182,20,62,166,233,23,135,64,121, - 3,104,251,214,161,253,31,3,2,232,37,8,221,189,72,64,218, - 121,41,112,148,254,34,68,164,228,60,161,201,132,55,56, - 157>>}, undefined, undefined, - undefined, - <<92,24,205,75,244,60,136,212,250,32,214,20,37,3,213,87,61,207, - 147,61,168,145,177,118,160,153,33,53,48,108,191,174>>, - undefined}, + #security_parameters{ + cipher_suite = <<19,2>>, + connection_end = 0, + bulk_cipher_algorithm = 8, + cipher_type = 2, + prf_algorithm = sha384, + master_secret = + {handshake_secret, + <<128,229,186,211,62,127,182,20,62,166,233,23,135,64,121, + 3,104,251,214,161,253,31,3,2,232,37,8,221,189,72,64,218, + 121,41,112,148,254,34,68,164,228,60,161,201,132,55,56, + 157>>}, + server_random = + <<92,24,205,75,244,60,136,212,250,32,214,20,37,3,213,87,61,207, + 147,61,168,145,177,118,160,153,33,53,48,108,191,174>>}, sequence_number => 0,server_verify_data => undefined},max_fragment_length => undefined}, PlainText = [11, @@ -544,7 +551,8 @@ encode_decode(_Config) -> %% TODO: remove hardcoded IV size WriteIVInfo = tls_v1:create_info(<<"iv">>, <<>>, 12), - {WriteKey, WriteIV} = tls_v1:calculate_traffic_keys(HKDFAlgo, Cipher, SHSTrafficSecret), + KeyLength = ssl_cipher:key_material(Cipher), + {WriteKey, WriteIV} = tls_v1:calculate_traffic_keys(HKDFAlgo, KeyLength, SHSTrafficSecret), %% {server} construct an EncryptedExtensions handshake message: %% @@ -824,7 +832,7 @@ encode_decode(_Config) -> SWIV = hexstr2bin("cf 78 2b 88 dd 83 54 9a ad f1 e9 84"), - {SWKey, SWIV} = tls_v1:calculate_traffic_keys(HKDFAlgo, Cipher, SAPTrafficSecret), + {SWKey, SWIV} = tls_v1:calculate_traffic_keys(HKDFAlgo, KeyLength, SAPTrafficSecret), %% {server} derive read traffic keys for handshake data: %% @@ -849,7 +857,7 @@ encode_decode(_Config) -> SRIV = hexstr2bin("5b d3 c7 1b 83 6e 0b 76 bb 73 26 5f"), - {SRKey, SRIV} = tls_v1:calculate_traffic_keys(HKDFAlgo, Cipher, CHSTrafficSecret), + {SRKey, SRIV} = tls_v1:calculate_traffic_keys(HKDFAlgo, KeyLength, CHSTrafficSecret), %% {client} calculate finished "tls13 finished": %% @@ -926,7 +934,7 @@ encode_decode(_Config) -> CWIV = hexstr2bin("5b 78 92 3d ee 08 57 90 33 e5 23 d9"), - {CWKey, CWIV} = tls_v1:calculate_traffic_keys(HKDFAlgo, Cipher, CAPTrafficSecret), + {CWKey, CWIV} = tls_v1:calculate_traffic_keys(HKDFAlgo, KeyLength, CAPTrafficSecret), %% {client} derive secret "tls13 res master": %% diff --git a/lib/ssl/vsn.mk b/lib/ssl/vsn.mk index 8784c37881..ecb5111403 100644 --- a/lib/ssl/vsn.mk +++ b/lib/ssl/vsn.mk @@ -1 +1 @@ -SSL_VSN = 10.2.4.1 +SSL_VSN = 10.3 diff --git a/lib/stdlib/doc/src/ets.xml b/lib/stdlib/doc/src/ets.xml index 690e3558be..4f43ae28d0 100644 --- a/lib/stdlib/doc/src/ets.xml +++ b/lib/stdlib/doc/src/ets.xml @@ -60,7 +60,7 @@ <seecom marker="erts:erl#+e"><c>+e</c></seecom> before starting the Erlang runtime system. This hard limit has been removed, but it is currently useful to set the <c>ERL_MAX_ETS_TABLES</c> anyway. It should be - set to an approximate of the maximum amount of tables used. This since + set to an approximate of the maximum amount of tables used since an internal table for named tables is sized using this value. If large amounts of named tables are used and <c>ERL_MAX_ETS_TABLES</c> hasn't been increased, the performance of named table lookup will diff --git a/lib/stdlib/doc/src/notes.xml b/lib/stdlib/doc/src/notes.xml index ad41d33d1d..9c60fb860f 100644 --- a/lib/stdlib/doc/src/notes.xml +++ b/lib/stdlib/doc/src/notes.xml @@ -31,6 +31,47 @@ </header> <p>This document describes the changes made to the STDLIB application.</p> +<section><title>STDLIB 3.14.1</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Handle maps in <c>erl_parse:tokens()</c>.</p> + <p> + Own Id: OTP-16978</p> + </item> + <item> + <p> + The erlang shell function <c>rr</c> has been fixed to be + able to read records from files within a code archive.</p> + <p> + Own Id: OTP-17182 Aux Id: PR-3002 </p> + </item> + <item> + <p>If <c>beam_lib</c> is asked to return abstract code + for a BEAM file produced by Elixir and Elixir is not + installed on the computer, <c>beam_lib</c> will no longer + crash, but will return an error tuple. The + <c>cover:compile_beam()</c> and + <c>cover:compile_beam_directory()</c> functions have been + updated to also return an error tuple in that + situation.</p> + <p> + Own Id: OTP-17194 Aux Id: GH-4353 </p> + </item> + <item> + <p> + Correct example module <c>erl_id_trans</c> regarding the + <c>{char, C}</c> type.</p> + <p> + Own Id: OTP-17273</p> + </item> + </list> + </section> + +</section> + <section><title>STDLIB 3.14</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/stdlib/doc/src/timer.xml b/lib/stdlib/doc/src/timer.xml index f44caf9ce2..56342b94be 100644 --- a/lib/stdlib/doc/src/timer.xml +++ b/lib/stdlib/doc/src/timer.xml @@ -47,6 +47,12 @@ must not be changed.</p> <p>The time-outs are not exact, but are <em>at least</em> as long as requested.</p> + <p>Creating timers using + <seemfa marker="erts:erlang#send_after/3">erlang:send_after/3</seemfa> and + <seemfa marker="erts:erlang#start_timer/3">erlang:start_timer/3</seemfa> + is much more efficient than using the timers provided by this module. See + <seeguide marker="system/efficiency_guide:commoncaveats#timer-module">the + Timer Module section in the Efficiency Guide</seeguide>.</p> </description> <datatypes> @@ -195,6 +201,9 @@ can also be an atom of a registered name.)</p> <p>Returns <c>{ok, <anno>TRef</anno>}</c> or <c>{error, <anno>Reason</anno>}</c>.</p> + <p>See also + <seeguide marker="system/efficiency_guide:commoncaveats#timer-module"> + the Timer Module section in the Efficiency Guide</seeguide>.</p> </item> <tag><c>send_after/2</c></tag> <item> diff --git a/lib/stdlib/examples/erl_id_trans.erl b/lib/stdlib/examples/erl_id_trans.erl index a707c45eb9..b1562e0207 100644 --- a/lib/stdlib/examples/erl_id_trans.erl +++ b/lib/stdlib/examples/erl_id_trans.erl @@ -613,6 +613,8 @@ type({atom,Line,A}) -> {atom,Line,A}; type({integer,Line,I}) -> {integer,Line,I}; +type({char,Line,C}) -> + {char,Line,C}; type({op,Line,Op,T}) -> T1 = type(T), {op,Line,Op,T1}; diff --git a/lib/stdlib/src/beam_lib.erl b/lib/stdlib/src/beam_lib.erl index 0a6dc74bd0..23413f43b1 100644 --- a/lib/stdlib/src/beam_lib.erl +++ b/lib/stdlib/src/beam_lib.erl @@ -107,6 +107,7 @@ -type chnk_rsn() :: {'unknown_chunk', file:filename(), atom()} | {'key_missing_or_invalid', file:filename(), 'abstract_code' | 'debug_info'} + | {'missing_backend', file:filename(), module()} | info_rsn(). -type cmp_rsn() :: {'modules_different', module(), module()} | {'chunks_different', chunkid()} @@ -310,6 +311,9 @@ format_error(badfun) -> "not a fun or the fun has the wrong arity"; format_error(exists) -> "a fun has already been installed"; +format_error({missing_backend, File, Backend}) -> + io_lib:format("~tp: Cannot retrieve abstract code because the backend ~p is missing", + [File, Backend]); format_error(E) -> io_lib:format("~tp~n", [E]). @@ -682,10 +686,13 @@ chunks_to_data([{abst_chunk, Name} | CNs], Chunks, File, Cs, Module, Atoms, L) - {NewAtoms, Ret} = case catch chunk_to_data(debug_info, DbgiChunk, File, Cs, Atoms, Module) of {DbgiAtoms, {debug_info, {debug_info_v1, Backend, Metadata}}} -> - case Backend:debug_info(erlang_v1, Module, Metadata, []) of + try Backend:debug_info(erlang_v1, Module, Metadata, []) of {ok, Code} -> {DbgiAtoms, {abstract_code, {raw_abstract_v1, Code}}}; {error, _} -> {DbgiAtoms, {abstract_code, no_abstract_code}} - end; + catch + error:undef -> + error({missing_backend,File,Backend}) + end; {error,beam_lib,{key_missing_or_invalid,Path,debug_info}} -> error({key_missing_or_invalid,Path,abstract_code}); _ -> diff --git a/lib/stdlib/src/erl_parse.yrl b/lib/stdlib/src/erl_parse.yrl index 8c7e27fc5b..dd7a2c2cc1 100644 --- a/lib/stdlib/src/erl_parse.yrl +++ b/lib/stdlib/src/erl_parse.yrl @@ -1535,7 +1535,13 @@ tokens({cons,A,Head,Tail}, More) -> tokens({tuple,A,[]}, More) -> [{'{',A},{'}',A}|More]; tokens({tuple,A,[E|Es]}, More) -> - [{'{',A}|tokens(E, tokens_tuple(Es, ?anno(E), More))]. + [{'{',A}|tokens(E, tokens_tuple(Es, ?anno(E), More))]; +tokens({map,A,[]}, More) -> + [{'#',A},{'{',A},{'}',A}|More]; +tokens({map,A,[P|Ps]}, More) -> + [{'#',A},{'{',A}|tokens(P, tokens_tuple(Ps, ?anno(P), More))]; +tokens({map_field_assoc,A,K,V}, More) -> + tokens(K, [{'=>',A}|tokens(V, More)]). tokens_tail({cons,A,Head,Tail}, More) -> [{',',A}|tokens(Head, tokens_tail(Tail, More))]; diff --git a/lib/stdlib/src/otp_internal.erl b/lib/stdlib/src/otp_internal.erl index 1eca1c29c3..1cc11d9093 100644 --- a/lib/stdlib/src/otp_internal.erl +++ b/lib/stdlib/src/otp_internal.erl @@ -65,16 +65,6 @@ obsolete(crypto, stream_decrypt, 2) -> {deprecated, "use crypto:crypto_update/2 instead", "OTP 24"}; obsolete(crypto, stream_encrypt, 2) -> {deprecated, "use crypto:crypto_update/2 instead", "OTP 24"}; -obsolete(erl_tidy, dir, 0) -> - {deprecated, "use https://github.com/richcarl/erl_tidy", "OTP 24"}; -obsolete(erl_tidy, dir, 1) -> - {deprecated, "use https://github.com/richcarl/erl_tidy", "OTP 24"}; -obsolete(erl_tidy, file, 1) -> - {deprecated, "use https://github.com/richcarl/erl_tidy", "OTP 24"}; -obsolete(erl_tidy, module, 1) -> - {deprecated, "use https://github.com/richcarl/erl_tidy", "OTP 24"}; -obsolete(erl_tidy, module, 2) -> - {deprecated, "use https://github.com/richcarl/erl_tidy", "OTP 24"}; obsolete(erlang, get_stacktrace, 0) -> {deprecated, "use the new try/catch syntax for retrieving the stack backtrace", "OTP 24"}; obsolete(erlang, now, 0) -> @@ -497,6 +487,8 @@ obsolete(erl_scan, attributes_info, _) -> {removed, "erl_anno:{column,line,location,text}/1 instead"}; obsolete(erl_scan, token_info, _) -> {removed, "erl_scan:{category,column,line,location,symbol,text}/1 instead"}; +obsolete(erl_tidy, _, _) -> + {deprecated, "use https://github.com/richcarl/erl_tidy", "OTP 24"}; obsolete(gen_fsm, _, _) -> {deprecated, "use the 'gen_statem' module instead"}; obsolete(igor, _, _) -> diff --git a/lib/stdlib/src/shell.erl b/lib/stdlib/src/shell.erl index 041a89f909..b397b2fc36 100644 --- a/lib/stdlib/src/shell.erl +++ b/lib/stdlib/src/shell.erl @@ -1186,6 +1186,8 @@ record_bindings(Recs0, Bs0) -> read_records(FileOrModule, Opts0) -> Opts = lists:delete(report_warnings, Opts0), case find_file(FileOrModule) of + {beam, Beam, File} -> + read_records_from_beam(Beam, File); {files,[File]} -> read_file_records(File, Opts); {files,Files} -> @@ -1204,10 +1206,22 @@ read_records(FileOrModule, Opts0) -> find_file(Mod) when is_atom(Mod) -> case code:which(Mod) of File when is_list(File) -> - {files,[File]}; - preloaded -> - {_M,_Bin,File} = code:get_object_code(Mod), - {files,[File]}; + %% Special cases: + %% - Modules not in the code path (loaded with code:load_abs/1): + %% code:get_object_code/1 only searches in the code path + %% but code:which/1 finds all loaded modules + %% - File can also be a file in an archive, + %% beam_lib:chunks/2 cannot handle such paths but + %% erl_prim_loader:get_file/1 can + case erl_prim_loader:get_file(File) of + {ok, Beam, _} -> + {beam, Beam, File}; + error -> + {error, nofile} + end; + preloaded -> + {_M, Beam, File} = code:get_object_code(Mod), + {beam, Beam, File}; _Else -> % non_existing, interpreted, cover_compiled {error,nofile} end; @@ -1222,28 +1236,31 @@ find_file(File) -> read_file_records(File, Opts) -> case filename:extension(File) of ".beam" -> - case beam_lib:chunks(File, [abstract_code,"CInf"]) of - {ok,{_Mod,[{abstract_code,{Version,Forms}},{"CInf",CB}]}} -> - case record_attrs(Forms) of - [] when Version =:= raw_abstract_v1 -> - []; - [] -> - %% If the version is raw_X, then this test - %% is unnecessary. - try_source(File, CB); - Records -> - Records - end; - {ok,{_Mod,[{abstract_code,no_abstract_code},{"CInf",CB}]}} -> - try_source(File, CB); - Error -> - %% Could be that the "Abst" chunk is missing (pre R6). - Error - end; + read_records_from_beam(File, File); _ -> parse_file(File, Opts) end. +read_records_from_beam(Beam, File) -> + case beam_lib:chunks(Beam, [abstract_code,"CInf"]) of + {ok,{_Mod,[{abstract_code,{Version,Forms}},{"CInf",CB}]}} -> + case record_attrs(Forms) of + [] when Version =:= raw_abstract_v1 -> + []; + [] -> + %% If the version is raw_X, then this test + %% is unnecessary. + try_source(File, CB); + Records -> + Records + end; + {ok,{_Mod,[{abstract_code,no_abstract_code},{"CInf",CB}]}} -> + try_source(File, CB); + Error -> + %% Could be that the "Abst" chunk is missing (pre R6). + Error + end. + %% This is how the debugger searches for source files. See int.erl. try_source(Beam, RawCB) -> EbinDir = filename:dirname(Beam), diff --git a/lib/stdlib/src/stdlib.appup.src b/lib/stdlib/src/stdlib.appup.src index 2f44c251e0..bba0d1ceee 100644 --- a/lib/stdlib/src/stdlib.appup.src +++ b/lib/stdlib/src/stdlib.appup.src @@ -41,6 +41,8 @@ {<<"^3\\.13\\.0(?:\\.[0-9]+)+$">>,[restart_new_emulator]}, {<<"^3\\.13\\.1(?:\\.[0-9]+)*$">>,[restart_new_emulator]}, {<<"^3\\.13\\.2(?:\\.[0-9]+)*$">>,[restart_new_emulator]}, + {<<"^3\\.14$">>,[restart_new_emulator]}, + {<<"^3\\.14\\.0(?:\\.[0-9]+)+$">>,[restart_new_emulator]}, {<<"^3\\.5$">>,[restart_new_emulator]}, {<<"^3\\.5\\.0(?:\\.[0-9]+)+$">>,[restart_new_emulator]}, {<<"^3\\.5\\.1(?:\\.[0-9]+)*$">>,[restart_new_emulator]}, @@ -70,6 +72,8 @@ {<<"^3\\.13\\.0(?:\\.[0-9]+)+$">>,[restart_new_emulator]}, {<<"^3\\.13\\.1(?:\\.[0-9]+)*$">>,[restart_new_emulator]}, {<<"^3\\.13\\.2(?:\\.[0-9]+)*$">>,[restart_new_emulator]}, + {<<"^3\\.14$">>,[restart_new_emulator]}, + {<<"^3\\.14\\.0(?:\\.[0-9]+)+$">>,[restart_new_emulator]}, {<<"^3\\.5$">>,[restart_new_emulator]}, {<<"^3\\.5\\.0(?:\\.[0-9]+)+$">>,[restart_new_emulator]}, {<<"^3\\.5\\.1(?:\\.[0-9]+)*$">>,[restart_new_emulator]}, diff --git a/lib/stdlib/test/beam_lib_SUITE.erl b/lib/stdlib/test/beam_lib_SUITE.erl index 9ef85a15e0..e28301ec9e 100644 --- a/lib/stdlib/test/beam_lib_SUITE.erl +++ b/lib/stdlib/test/beam_lib_SUITE.erl @@ -36,7 +36,8 @@ -export([all/0, suite/0,groups/0,init_per_suite/1, end_per_suite/1, init_per_group/2,end_per_group/2, normal/1, error/1, cmp/1, cmp_literals/1, strip/1, strip_add_chunks/1, otp_6711/1, - building/1, md5/1, encrypted_abstr/1, encrypted_abstr_file/1]). + building/1, md5/1, encrypted_abstr/1, encrypted_abstr_file/1, + missing_debug_info_backend/1]). -export([init_per_testcase/2, end_per_testcase/2]). @@ -46,7 +47,7 @@ suite() -> all() -> [error, normal, cmp, cmp_literals, strip, strip_add_chunks, otp_6711, - building, md5, encrypted_abstr, encrypted_abstr_file]. + building, md5, encrypted_abstr, encrypted_abstr_file, missing_debug_info_backend]. groups() -> []. @@ -775,6 +776,29 @@ write_crypt_file(Contents0) -> io:format("~s\n", [binary_to_list(Contents)]), ok = file:write_file(".erlang.crypt", Contents). +%% GH-4353: Don't crash when the backend for generating the abstract code +%% is missing. +missing_debug_info_backend(Conf) -> + PrivDir = ?privdir, + Simple = filename:join(PrivDir, "simple"), + Source = Simple ++ ".erl", + BeamFile = Simple ++ ".beam", + simple_file(Source), + + %% Create a debug_info chunk with a non-existing backend. + {ok,simple} = compile:file(Source, [{outdir,PrivDir}]), + {ok,simple,All0} = beam_lib:all_chunks(BeamFile), + FakeBackend = definitely__not__an__existing__backend, + FakeDebugInfo = {debug_info_v1, FakeBackend, nothing_here}, + All = lists:keyreplace("Dbgi", 1, All0, {"Dbgi", term_to_binary(FakeDebugInfo)}), + {ok,NewBeam} = beam_lib:build_module(All), + ok = file:write_file(BeamFile, NewBeam), + + %% beam_lib should not crash, but return an error. + verify(missing_backend, beam_lib:chunks(BeamFile, [abstract_code])), + + ok. + compare_chunks(File1, File2, ChunkIds) -> {ok, {_, Chunks1}} = beam_lib:chunks(File1, ChunkIds), {ok, {_, Chunks2}} = beam_lib:chunks(File2, ChunkIds), diff --git a/lib/stdlib/test/epp_SUITE.erl b/lib/stdlib/test/epp_SUITE.erl index cb1638670c..a607598136 100644 --- a/lib/stdlib/test/epp_SUITE.erl +++ b/lib/stdlib/test/epp_SUITE.erl @@ -29,7 +29,7 @@ otp_8562/1, otp_8665/1, otp_8911/1, otp_10302/1, otp_10820/1, otp_11728/1, encoding/1, extends/1, function_macro/1, test_error/1, test_warning/1, otp_14285/1, - test_if/1,source_name/1]). + test_if/1,source_name/1,otp_16978/1]). -export([epp_parse_erl_form/2]). @@ -70,7 +70,7 @@ all() -> overload_mac, otp_8388, otp_8470, otp_8562, otp_8665, otp_8911, otp_10302, otp_10820, otp_11728, encoding, extends, function_macro, test_error, test_warning, - otp_14285, test_if, source_name]. + otp_14285, test_if, source_name, otp_16978]. groups() -> [{upcase_mac, [], [upcase_mac_1, upcase_mac_2]}, @@ -1720,19 +1720,40 @@ source_name_1(File, Expected) -> Res = epp:parse_file(File, [{source_name, Expected}]), {ok, [{attribute,_,file,{Expected,_}} | _Forms]} = Res. +otp_16978(Config) when is_list(Config) -> + %% A test of erl_parse:tokens(). + P = <<"t() -> ?a.">>, + Vs = [#{}, + #{k => 1,[[a],[{}]] => "str"}, + #{#{} => [{#{x=>#{3=>$3}}},{3.14,#{}}]}], + Ts = [{erl_parse_tokens, + P, + [{d,{a,V}}], + V} || V <- Vs], + [] = run(Config, Ts), + + ok. + check(Config, Tests) -> - eval_tests(Config, fun check_test/2, Tests). + eval_tests(Config, fun check_test/3, Tests). compile(Config, Tests) -> - eval_tests(Config, fun compile_test/2, Tests). + eval_tests(Config, fun compile_test/3, Tests). run(Config, Tests) -> - eval_tests(Config, fun run_test/2, Tests). + eval_tests(Config, fun run_test/3, Tests). eval_tests(Config, Fun, Tests) -> - F = fun({N,P,E}, BadL) -> + TestsWithOpts = + [case Test of + {N,P,E} -> + {N,P,[],E}; + {_,_,_,_} -> + Test + end || Test <- Tests], + F = fun({N,P,Opts,E}, BadL) -> %% io:format("Testing ~p~n", [P]), - Return = Fun(Config, P), + Return = Fun(Config, P, Opts), case message_compare(E, Return) of true -> case E of @@ -1748,14 +1769,14 @@ eval_tests(Config, Fun, Tests) -> fail() end end, - lists:foldl(F, [], Tests). + lists:foldl(F, [], TestsWithOpts). -check_test(Config, Test) -> +check_test(Config, Test, Opts) -> Filename = "epp_test.erl", PrivDir = proplists:get_value(priv_dir, Config), File = filename:join(PrivDir, Filename), ok = file:write_file(File, Test), - case epp:parse_file(File, [PrivDir], []) of + case epp:parse_file(File, [PrivDir], Opts) of {ok,Forms} -> Errors = [E || E={error,_} <- Forms], call_format_error([E || {error,E} <- Errors]), @@ -1764,13 +1785,13 @@ check_test(Config, Test) -> Error end. -compile_test(Config, Test0) -> +compile_test(Config, Test0, Opts0) -> Test = [<<"-module(epp_test). ">>, Test0], Filename = "epp_test.erl", PrivDir = proplists:get_value(priv_dir, Config), File = filename:join(PrivDir, Filename), ok = file:write_file(File, Test), - Opts = [export_all,nowarn_export_all,return,nowarn_unused_record,{outdir,PrivDir}], + Opts = [export_all,nowarn_export_all,return,nowarn_unused_record,{outdir,PrivDir}] ++ Opts0, case compile_file(File, Opts) of {ok, Ws} -> warnings(File, Ws); {errors, Errors}=Else -> @@ -1821,13 +1842,13 @@ epp_parse_file(File, Opts) -> unopaque_forms(Forms) -> [erl_parse:anno_to_term(Form) || Form <- Forms]. -run_test(Config, Test0) -> +run_test(Config, Test0, Opts0) -> Test = [<<"-module(epp_test). -export([t/0]). ">>, Test0], Filename = "epp_test.erl", PrivDir = proplists:get_value(priv_dir, Config), File = filename:join(PrivDir, Filename), ok = file:write_file(File, Test), - Opts = [return, {i,PrivDir},{outdir,PrivDir}], + Opts = [return, {i,PrivDir},{outdir,PrivDir}] ++ Opts0, {ok, epp_test, []} = compile:file(File, Opts), AbsFile = filename:rootname(File, ".erl"), {module, epp_test} = code:load_abs(AbsFile, epp_test), diff --git a/lib/stdlib/test/shell_SUITE.erl b/lib/stdlib/test/shell_SUITE.erl index 4d7a2ea078..4df0a2238a 100644 --- a/lib/stdlib/test/shell_SUITE.erl +++ b/lib/stdlib/test/shell_SUITE.erl @@ -64,8 +64,9 @@ end_per_testcase(_Case, Config) -> OrigPath = proplists:get_value(orig_path,Config), code:set_path(OrigPath), application:unset_env(stdlib, restricted_shell), - (catch code:purge(user_default)), - (catch code:delete(user_default)), + purge_and_delete(user_default), + %% used by `records' test case + purge_and_delete(test), ok. -endif. @@ -298,8 +299,7 @@ restricted_local(Config) when is_list(Config) -> comm_err(<<"begin shell:stop_restricted() end.">>), undefined = application:get_env(stdlib, restricted_shell), - (catch code:purge(user_default)), - true = (catch code:delete(user_default)), + true = purge_and_delete(user_default), ok. @@ -428,6 +428,30 @@ records(Config) when is_list(Config) -> [{error,invalid_filename}] = scan(<<"rr({foo}).">>), [[]] = scan(<<"rr(\"not_a_file\").">>), + %% load record from archive + true = purge_and_delete(test), + + PrivDir = proplists:get_value(priv_dir, Config), + AppDir = filename:join(PrivDir, "test_app"), + ok = file:make_dir(AppDir), + AppEbinDir = filename:join(AppDir, "ebin"), + ok = file:make_dir(AppEbinDir), + + ok = file:write_file(Test, Contents), + {ok, test} = compile:file(Test, [{outdir, AppEbinDir}]), + + Ext = init:archive_extension(), + Archive = filename:join(PrivDir, "test_app" ++ Ext), + {ok, _} = zip:create(Archive, ["test_app"], [{compress, []}, {cwd, PrivDir}]), + + ArchiveEbinDir = filename:join(Archive, "test_app/ebin"), + true = code:add_path(ArchiveEbinDir), + {module, test} = code:load_file(test), + BeamInArchive = filename:join(ArchiveEbinDir, "test.beam"), + BeamInArchive = code:which(test), + + [[state]] = scan(<<"rr(test).">>), + %% using records [2] = scan(<<"rd(foo,{bar}), record_info(size, foo).">>), [true] = scan(<<"rd(foo,{bar}), is_record(#foo{}, foo).">>), @@ -3218,3 +3242,6 @@ start_node(Name, Xargs) -> global:sync(), N. +purge_and_delete(Module) -> + (catch code:purge(Module)), + (catch code:delete(Module)). diff --git a/lib/stdlib/vsn.mk b/lib/stdlib/vsn.mk index 5f8e568387..e234b0cd58 100644 --- a/lib/stdlib/vsn.mk +++ b/lib/stdlib/vsn.mk @@ -1 +1 @@ -STDLIB_VSN = 3.14 +STDLIB_VSN = 3.14.1 diff --git a/lib/syntax_tools/doc/src/notes.xml b/lib/syntax_tools/doc/src/notes.xml index b7d304aabe..a1f280e594 100644 --- a/lib/syntax_tools/doc/src/notes.xml +++ b/lib/syntax_tools/doc/src/notes.xml @@ -32,6 +32,35 @@ <p>This document describes the changes made to the Syntax_Tools application.</p> +<section><title>Syntax_Tools 2.5</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p><c>epp_dodger</c> was unable to handle a parameterized + macro in a function head.</p> + <p> + Own Id: OTP-17064 Aux Id: GH-4445, PR-2964 </p> + </item> + </list> + </section> + + + <section><title>Improvements and New Features</title> + <list> + <item> + <p>All functions in <c>erl_tidy</c> in syntax_tools have + now been deprecated and are scheduled for removal in OTP + 24. Users who still need it can find it at <url + href="https://github.com/richcarl/erl_tidy">https://github.com/richcarl/erl_tidy</url>.</p> + <p> + Own Id: OTP-17167 Aux Id: OTP-17046 </p> + </item> + </list> + </section> + +</section> + <section><title>Syntax_Tools 2.4</title> <section><title>Improvements and New Features</title> diff --git a/lib/syntax_tools/src/epp_dodger.erl b/lib/syntax_tools/src/epp_dodger.erl index da22a91de0..8ee6368868 100644 --- a/lib/syntax_tools/src/epp_dodger.erl +++ b/lib/syntax_tools/src/epp_dodger.erl @@ -544,6 +544,7 @@ quickscan_macros([{'?',_}, {Type, _, _}=N | [{'(',_}|_]=Ts], Ts1 = case skip_macro_args(Ts) of {_, [{'->',_} | _] = Ts2} -> Ts2; {_, [{'when',_} | _] = Ts2} -> Ts2; + {_, [{':',_} | _] = Ts2} -> Ts2; _ -> Ts %% assume macro without arguments end, quickscan_macros_1(N, Ts1, As); @@ -705,6 +706,8 @@ scan_macros([{'?', L}, {Type, _, _}=N | [{'(',_}|_]=Ts], macro_call(Args, L, N, Rest, As, Opt); [{'when',_} | _] -> macro_call(Args, L, N, Rest, As, Opt); + [{':',_} | _] -> + macro_call(Args, L, N, Rest, As, Opt); _ -> macro(L, N, Ts, As, Opt) end; @@ -722,7 +725,7 @@ scan_macros([T | Ts], As, Opt) -> scan_macros([], As, _Opt) -> lists:reverse(As). -%% Rewriting to a call which will be recognized by the post-parse pass +%% Rewriting to a tuple which will be recognized by the post-parse pass %% (we insert parentheses to preserve the precedences when parsing). macro(L, {Type, _, A}, Rest, As, Opt) -> @@ -731,17 +734,28 @@ macro(L, {Type, _, A}, Rest, As, Opt) -> macro_call([{'(',_}, {')',_}], L, {_, Ln, _}=N, Rest, As, Opt) -> {Open, Close} = parentheses(As), scan_macros_1([], Rest, - lists:reverse(Open ++ [{atom,L,?macro_call}, - {'(',L}, N, {')',Ln}] ++ Close, - As), Opt); + %% {'?macro_call', N } + lists:reverse(Open ++ [{'{', L}, + {atom, L, ?macro_call}, + {',', L}, + N, + {'}', Ln}] ++ Close, + As), Opt); macro_call([{'(',_} | Args], L, {_, Ln, _}=N, Rest, As, Opt) -> {Open, Close} = parentheses(As), + %% drop closing parenthesis + {')', _} = lists:last(Args), %% assert + Args1 = lists:droplast(Args), %% note that we must scan the argument list; it may not be skipped - scan_macros_1(Args ++ Close, - Rest, - lists:reverse(Open ++ [{atom,L,?macro_call}, - {'(',L}, N, {',',Ln}], - As), Opt). + scan_macros_1(Args1 ++ [{'}', Ln} | Close], + Rest, + %% {'?macro_call', N, Arg1, ... } + lists:reverse(Open ++ [{'{', L}, + {atom, L, ?macro_call}, + {',', L}, + N, + {',', Ln}], + As), Opt). macro_atom(atom, A) -> list_to_atom(?atom_prefix ++ atom_to_list(A)); @@ -798,21 +812,24 @@ rewrite(Node) -> _ -> Node end; - application -> - F = erl_syntax:application_operator(Node), - case erl_syntax:type(F) of - atom -> - case erl_syntax:atom_value(F) of - ?macro_call -> - [A | As] = erl_syntax:application_arguments(Node), - M = erl_syntax:macro(A, rewrite_list(As)), - erl_syntax:copy_pos(Node, M); - _ -> - rewrite_1(Node) - end; - _ -> - rewrite_1(Node) - end; + tuple -> + case erl_syntax:tuple_elements(Node) of + [MagicWord, A | As] -> + case erl_syntax:type(MagicWord) of + atom -> + case erl_syntax:atom_value(MagicWord) of + ?macro_call -> + M = erl_syntax:macro(A, rewrite_list(As)), + erl_syntax:copy_pos(Node, M); + _ -> + rewrite_1(Node) + end; + _ -> + rewrite_1(Node) + end; + _ -> + rewrite_1(Node) + end; _ -> rewrite_1(Node) end. diff --git a/lib/syntax_tools/src/erl_tidy.erl b/lib/syntax_tools/src/erl_tidy.erl index fb312bb9d4..f6ae2cb960 100644 --- a/lib/syntax_tools/src/erl_tidy.erl +++ b/lib/syntax_tools/src/erl_tidy.erl @@ -48,11 +48,7 @@ %% @type filename() = file:filename(). -module(erl_tidy). --deprecated([{dir,0,"use https://github.com/richcarl/erl_tidy"}]). --deprecated([{dir,1,"use https://github.com/richcarl/erl_tidy"}]). --deprecated([{file,1,"use https://github.com/richcarl/erl_tidy"}]). --deprecated([{module,1,"use https://github.com/richcarl/erl_tidy"}]). --deprecated([{module,2,"use https://github.com/richcarl/erl_tidy"}]). +-deprecated([{'_','_',"use https://github.com/richcarl/erl_tidy"}]). -export([dir/0, dir/1, dir/2, file/1, file/2, module/1, module/2]). diff --git a/lib/syntax_tools/test/syntax_tools_SUITE.erl b/lib/syntax_tools/test/syntax_tools_SUITE.erl index 9baf36ce11..14a7912642 100644 --- a/lib/syntax_tools/test/syntax_tools_SUITE.erl +++ b/lib/syntax_tools/test/syntax_tools_SUITE.erl @@ -25,7 +25,8 @@ %% Test cases -export([app_test/1,appup_test/1,smoke_test/1,revert/1,revert_map/1, revert_map_type/1,wrapped_subtrees/1, - t_abstract_type/1,t_erl_parse_type/1,t_type/1, t_epp_dodger/1, + t_abstract_type/1,t_erl_parse_type/1,t_type/1, + t_epp_dodger/1,t_epp_dodger_clever/1, t_comment_scan/1,t_igor/1,t_erl_tidy/1,t_prettypr/1]). suite() -> [{ct_hooks,[ts_install_cth]}]. @@ -33,7 +34,8 @@ suite() -> [{ct_hooks,[ts_install_cth]}]. all() -> [app_test,appup_test,smoke_test,revert,revert_map,revert_map_type, wrapped_subtrees, - t_abstract_type,t_erl_parse_type,t_type,t_epp_dodger, + t_abstract_type,t_erl_parse_type,t_type, + t_epp_dodger,t_epp_dodger_clever, t_comment_scan,t_igor,t_erl_tidy,t_prettypr]. groups() -> @@ -330,6 +332,13 @@ t_epp_dodger(Config) when is_list(Config) -> ok = test_epp_dodger(Filenames,DataDir,PrivDir), ok. +t_epp_dodger_clever(Config) when is_list(Config) -> + DataDir = ?config(data_dir, Config), + PrivDir = ?config(priv_dir, Config), + Filenames = ["epp_dodger_clever.erl"], + ok = test_epp_dodger_clever(Filenames,DataDir,PrivDir), + ok. + t_comment_scan(Config) when is_list(Config) -> DataDir = ?config(data_dir, Config), Filenames = test_files(), @@ -447,9 +456,30 @@ test_epp_dodger([Filename|Files],DataDir,PrivDir) -> ok = pretty_print_parse_forms(FsForms,PrivDir,Filename), test_epp_dodger(Files,DataDir,PrivDir). +test_epp_dodger_clever([], _, _) -> ok; +test_epp_dodger_clever([Filename|Files],DataDir,PrivDir) -> + io:format("Parsing ~p~n", [Filename]), + InFile = filename:join(DataDir, Filename), + Parsers = [{fun(File) -> + epp_dodger:parse_file(File, [clever]) + end, parse_file}, + {fun(File) -> + epp_dodger:quick_parse_file(File, [clever]) + end, quick_parse_file}], + FsForms = parse_with(Parsers, InFile), + ok = pretty_print_parse_forms(FsForms,PrivDir,Filename), + test_epp_dodger_clever(Files,DataDir,PrivDir). + parse_with([],_) -> []; parse_with([{Fun,ParserType}|Funs],File) -> {ok, Fs} = Fun(File), + ErrorMarkers = [begin + print_error_markers(F, File), + F + end + || F <- Fs, + erl_syntax:type(F) =:= error_marker], + [] = ErrorMarkers, [{Fs,ParserType}|parse_with(Funs,File)]. pretty_print_parse_forms([],_,_) -> ok; diff --git a/lib/syntax_tools/test/syntax_tools_SUITE_data/epp_dodger_clever.erl b/lib/syntax_tools/test/syntax_tools_SUITE_data/epp_dodger_clever.erl new file mode 100644 index 0000000000..4a3be06d98 --- /dev/null +++ b/lib/syntax_tools/test/syntax_tools_SUITE_data/epp_dodger_clever.erl @@ -0,0 +1,12 @@ +-module(epp_dodger_clever). + +-export([foo1/0]). + +-define(macro_string, "hello world"). + +foo1() -> + % string combining ? + [?macro_string + "hello world ", + "more hello" + ?macro_string]. diff --git a/lib/syntax_tools/test/syntax_tools_SUITE_data/syntax_tools_test.erl b/lib/syntax_tools/test/syntax_tools_SUITE_data/syntax_tools_test.erl index dd3f88d7a8..b8a21ef0ab 100644 --- a/lib/syntax_tools/test/syntax_tools_SUITE_data/syntax_tools_test.erl +++ b/lib/syntax_tools/test/syntax_tools_SUITE_data/syntax_tools_test.erl @@ -6,7 +6,7 @@ -module(syntax_tools_test). --export([foo1/0,foo2/2,foo3/0,foo4/3,foo5/1]). +-export([foo1/0,foo2/2,foo3/0,foo4/3,foo5/1,foo6/2]). -include_lib("kernel/include/file.hrl"). -record(state, { a, b, c, d}). @@ -24,6 +24,7 @@ -define(macro_string, "hello world"). -define(macro_argument1(X), (X + 3)). -define(macro_argument2(X,Y), (X + 3 * Y)). +-define(macro_argument3(X), {error, X}). -define(macro_block(X), begin X end). -define(macro_if(X1,X2), if X1 -> X2; true -> none end). @@ -48,8 +49,7 @@ foo1() -> %% macro test foo2(A,B) -> % string combining ? - [?macro_string, ?macro_string - ?macro_string, + [?macro_string, ?macro_string ++ "hello world " "more hello", [?macro_simple1, @@ -113,3 +113,16 @@ foo5(A) -> error:?macro_simple5 -> nope end. + +%% macros in patterns +foo6(?MACRO_SIMPLE2, ?macro_argument3(A)) -> + try foo2(A,A) of + R -> R + catch + ?macro_argument3(B) -> + B; + error:?macro_argument3(B) -> + B; + error:?macro_argument3(B):_ -> + B + end. diff --git a/lib/syntax_tools/vsn.mk b/lib/syntax_tools/vsn.mk index 7f0b8f40dd..7a93e81b94 100644 --- a/lib/syntax_tools/vsn.mk +++ b/lib/syntax_tools/vsn.mk @@ -1 +1 @@ -SYNTAX_TOOLS_VSN = 2.4 +SYNTAX_TOOLS_VSN = 2.5 diff --git a/lib/tools/doc/src/cover.xml b/lib/tools/doc/src/cover.xml index 8073bfc528..0889a16f65 100644 --- a/lib/tools/doc/src/cover.xml +++ b/lib/tools/doc/src/cover.xml @@ -217,9 +217,10 @@ <v>ModFiles = ModFile | [ModFile]</v> <v>ModFile = Module | BeamFile</v> <v> Module = atom()</v> + <v> BackendModule = atom()</v> <v> BeamFile = string()</v> <v>Result = {ok,Module} | {error,BeamFile} | {error,Reason}</v> - <v> Reason = non_existing | {no_abstract_code,BeamFile} | {encrypted_abstract_code,BeamFile} | {already_cover_compiled,no_beam_found,Module} | not_main_node</v> + <v> Reason = non_existing | {no_abstract_code,BeamFile} | {{missing_backend,BackendModule},BeamFile} | {encrypted_abstract_code,BeamFile} | {already_cover_compiled,no_beam_found,Module} | not_main_node</v> </type> <desc> <p>Does the same as <c>compile/1,2</c>, but uses an existing diff --git a/lib/tools/doc/src/fprof.xml b/lib/tools/doc/src/fprof.xml index 5d2683846f..b3ba4a200c 100644 --- a/lib/tools/doc/src/fprof.xml +++ b/lib/tools/doc/src/fprof.xml @@ -598,7 +598,7 @@ -module(foo). -export([create_file_slow/2]). -create_file_slow(Name, N) when integer(N), N >= 0 -> +create_file_slow(Name, N) when is_integer(N), N >= 0 -> {ok, FD} = file:open(Name, [raw, write, delayed_write, binary]), if N > 256 -> diff --git a/lib/tools/doc/src/notes.xml b/lib/tools/doc/src/notes.xml index 5b98d338b0..358bcf5043 100644 --- a/lib/tools/doc/src/notes.xml +++ b/lib/tools/doc/src/notes.xml @@ -31,6 +31,40 @@ </header> <p>This document describes the changes made to the Tools application.</p> +<section><title>Tools 3.4.4</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p><c>cover</c> would crash when compiling a module + having an exported function named <c>clauses</c>.</p> + <p> + Own Id: OTP-17162 Aux Id: GH-4549, PR-2997, PR-4555, + elixir-lang/elixir#10666 </p> + </item> + <item> + <p>If <c>beam_lib</c> is asked to return abstract code + for a BEAM file produced by Elixir and Elixir is not + installed on the computer, <c>beam_lib</c> will no longer + crash, but will return an error tuple. The + <c>cover:compile_beam()</c> and + <c>cover:compile_beam_directory()</c> functions have been + updated to also return an error tuple in that + situation.</p> + <p> + Own Id: OTP-17194 Aux Id: GH-4353 </p> + </item> + <item> + <p> + Make emacs mode work on emacs-27.</p> + <p> + Own Id: OTP-17225 Aux Id: PR-4542, GH-4451 </p> + </item> + </list> + </section> + +</section> + <section><title>Tools 3.4.3</title> <section><title>Fixed Bugs and Malfunctions</title> @@ -107,6 +141,22 @@ </section> +<section><title>Tools 3.3.1.1</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p><c>cover</c> would crash when compiling a module + having an exported function named <c>clauses</c>.</p> + <p> + Own Id: OTP-17162 Aux Id: GH-4549, PR-2997, PR-4555, + elixir-lang/elixir#10666 </p> + </item> + </list> + </section> + +</section> + <section><title>Tools 3.3.1</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/tools/emacs/erlang-eunit.el b/lib/tools/emacs/erlang-eunit.el index 73c25246df..6212553611 100644 --- a/lib/tools/emacs/erlang-eunit.el +++ b/lib/tools/emacs/erlang-eunit.el @@ -216,7 +216,7 @@ buffer and vice versa" With prefix arg, compiles for debug and runs tests with the verbose flag set." (interactive) - (case (erlang-eunit-recent 'mode) + (cl-case (erlang-eunit-recent 'mode) ('test-mode (erlang-eunit-compile-and-test 'erlang-eunit-run-test (list (erlang-eunit-recent 'module) diff --git a/lib/tools/emacs/erlang-test.el b/lib/tools/emacs/erlang-test.el index c1c0cd7222..3dab5e5525 100644 --- a/lib/tools/emacs/erlang-test.el +++ b/lib/tools/emacs/erlang-test.el @@ -62,8 +62,6 @@ ;;; Code: -(eval-when-compile - (require 'cl)) (require 'ert) (require 'erlang) @@ -127,8 +125,8 @@ concatenated to form an erlang file to test on.") (defun erlang-test-create-erlang-file (erlang-file) (with-temp-file erlang-file - (loop for (_ . code) in erlang-test-code - do (insert code "\n")))) + (cl-loop for (_ . code) in erlang-test-code + do (insert code "\n")))) (defun erlang-test-compile-tags (erlang-file tags-file) (should (zerop (call-process "etags" nil nil nil @@ -143,20 +141,20 @@ concatenated to form an erlang file to test on.") (sort (erlang-expected-completion-table) #'string-lessp)))) (defun erlang-expected-completion-table () - (append (loop for (symbol . _) in erlang-test-code - when (stringp symbol) - append (list symbol (concat "erlang_test:" symbol))) + (append (cl-loop for (symbol . _) in erlang-test-code + when (stringp symbol) + append (list symbol (concat "erlang_test:" symbol))) (list "erlang_test:" "erlang_test:module_info"))) (defun erlang-test-xref-find-definitions (erlang-file erlang-buffer) - (loop for (tagname . code) in erlang-test-code - for line = 1 then (1+ line) - do (when tagname - (switch-to-buffer erlang-buffer) - (erlang-test-xref-jump tagname erlang-file line) - (when (string-equal tagname "function") - (erlang-test-xref-jump (concat "erlang_test:" tagname) - erlang-file line)))) + (cl-loop for (tagname . code) in erlang-test-code + for line = 1 then (1+ line) + do (when tagname + (switch-to-buffer erlang-buffer) + (erlang-test-xref-jump tagname erlang-file line) + (when (string-equal tagname "function") + (erlang-test-xref-jump (concat "erlang_test:" tagname) + erlang-file line)))) (erlang-test-xref-jump "erlang_test:" erlang-file 1)) (defun erlang-test-xref-jump (id expected-file expected-line) @@ -225,27 +223,27 @@ concatenated to form an erlang file to test on.") (ert-deftest erlang-test-parse-id () - (loop for id-string in '("fun/10" - "qualified-function module:fun/10" - "record reko" - "macro _SYMBOL" - "macro MACRO/10" - "module modula" - "macro" - nil) - for id-list in '((nil nil "fun" 10) - (qualified-function "module" "fun" 10) - (record nil "reko" nil) - (macro nil "_SYMBOL" nil) - (macro nil "MACRO" 10) - (module nil "modula" nil) - (nil nil "macro" nil) - nil) - for id-list2 = (erlang-id-to-list id-string) - do (should (equal id-list id-list2)) - for id-string2 = (erlang-id-to-string id-list) - do (should (equal id-string id-string2)) - collect id-list2)) + (cl-loop for id-string in '("fun/10" + "qualified-function module:fun/10" + "record reko" + "macro _SYMBOL" + "macro MACRO/10" + "module modula" + "macro" + nil) + for id-list in '((nil nil "fun" 10) + (qualified-function "module" "fun" 10) + (record nil "reko" nil) + (macro nil "_SYMBOL" nil) + (macro nil "MACRO" 10) + (module nil "modula" nil) + (nil nil "macro" nil) + nil) + for id-list2 = (erlang-id-to-list id-string) + do (should (equal id-list id-list2)) + for id-string2 = (erlang-id-to-string id-list) + do (should (equal id-string id-string2)) + collect id-list2)) (provide 'erlang-test) diff --git a/lib/tools/emacs/erlang.el b/lib/tools/emacs/erlang.el index a1d14dfcaf..de5dd4955f 100644 --- a/lib/tools/emacs/erlang.el +++ b/lib/tools/emacs/erlang.el @@ -76,11 +76,15 @@ ;; M-x toggle-debug-on-error RET ;;; Code: -(eval-when-compile (require 'cl)) (require 'align) (require 'comint) (require 'tempo) +;;; `caddr' is builtin since Emacs 26. +(eval-and-compile + (or (fboundp 'caddr) + (defun caddr (x) (car (cdr (cdr x)))))) + ;; Variables: (defgroup erlang nil @@ -1420,7 +1424,7 @@ Other commands: (erlang-electric-init) (erlang-menu-init) (erlang-mode-variables) - (erlang-check-module-name-init) + (add-hook 'before-save-hook 'erlang-check-module-name nil t) (erlang-man-init) (erlang-tags-init) (erlang-font-lock-init) @@ -1431,7 +1435,6 @@ Other commands: (setq-local eldoc-documentation-function #'ignore)) (add-function :before-until (local 'eldoc-documentation-function) #'erldoc-eldoc-function)) - (run-hooks 'erlang-mode-hook) ;; Align maps. (add-to-list 'align-rules-list @@ -2674,8 +2677,6 @@ This is automagically called by the user level function `indent-region'." (defmacro erlang-push (x stack) (list 'setq stack (list 'cons x stack))) (defmacro erlang-pop (stack) (list 'setq stack (list 'cdr stack))) -;; Would much prefer to make caddr a macro but this clashes. -(defun erlang-caddr (x) (car (cdr (cdr x)))) (defun erlang-calculate-indent (&optional parse-start) @@ -3076,8 +3077,8 @@ Return nil if inside string, t if in a comment." (if (eq (car stack-top) '->) (erlang-pop stack)) (cond ((and stack (looking-at ";")) - (+ (erlang-caddr (car stack)) (- erlang-indent-level 2))) - (stack (erlang-caddr (car stack))) + (+ (caddr (car stack)) (- erlang-indent-level 2))) + (stack (caddr (car stack))) (t off))) ((looking-at "catch\\b\\($\\|[^_a-zA-Z0-9]\\)") ;; Are we in a try @@ -3091,12 +3092,12 @@ Return nil if inside string, t if in a comment." (if (eq (car stack-top) '->) (erlang-pop stack)) (if stack - (erlang-caddr (car stack)) + (caddr (car stack)) 0))) (t (erlang-indent-standard indent-point token base 'nil))))) ;; old catch ;; Indent result types ((eq (car (car (cdr stack))) 'spec_arg) - (setq base (+ (erlang-caddr (car (last stack))) erlang-indent-level)) + (setq base (+ (caddr (car (last stack))) erlang-indent-level)) (erlang-indent-standard indent-point token base 'nil)) (t (erlang-indent-standard indent-point token base 'nil) @@ -3224,7 +3225,7 @@ Return nil if inside string, t if in a comment." ;; Take parent identation + offset, ;; else just erlang-indent-level if no parent (if stack - (+ (erlang-caddr (car stack)) + (+ (caddr (car stack)) offset) erlang-indent-level)) (erlang-skip-blank indent-point) @@ -4084,11 +4085,11 @@ of arguments could be found, otherwise nil." (defun erlang-match-next-exported-function (max) "Returns non-nil if there is an exported function in the current buffer between point and MAX." - (block nil - (while (and (not erlang-inhibit-exported-function-name-face) - (erlang-match-next-function max)) - (when (erlang-last-match-exported-p) - (return (match-data)))))) + (catch 'return + (while (and (not erlang-inhibit-exported-function-name-face) + (erlang-match-next-function max)) + (when (erlang-last-match-exported-p) + (throw 'return (match-data)))))) (defun erlang-match-next-function (max) "Searches forward in current buffer for the next erlang function, @@ -4116,28 +4117,6 @@ exported function." ;;; Check module name -;; The function `write-file', bound to C-x C-w, calls -;; `set-visited-file-name' which clears the hook. :-( -;; To make sure that the hook always is present, we advise -;; `set-visited-file-name'. -(defun erlang-check-module-name-init () - "Initialize the functionality to compare file and module names. - -Unless we have `before-save-hook', we advice the function -`set-visited-file-name' since it clears the variable -`local-write-file-hooks'." - (if (boundp 'before-save-hook) - (add-hook 'before-save-hook 'erlang-check-module-name nil t) - (require 'advice) - (when (fboundp 'ad-advised-definition-p) - (unless (ad-advised-definition-p 'set-visited-file-name) - (defadvice set-visited-file-name (after erlang-set-visited-file-name - activate) - (if (eq major-mode 'erlang-mode) - (add-hook 'local-write-file-hooks 'erlang-check-module-name)))) - (add-hook 'local-write-file-hooks 'erlang-check-module-name)))) - - (defun erlang-check-module-name () "If the module name doesn't match file name, ask for permission to change. @@ -4146,7 +4125,7 @@ function. It it is nil, this function does nothing. If it is t, the source is silently changed. If it is set to the atom `ask', the user is prompted. -This function is normally placed in the hook `local-write-file-hooks'." +This function is normally placed in the hook `before-save-hook'." (if erlang-check-module-name (let ((mn (erlang-add-quotes-if-needed (erlang-get-module))) @@ -5358,7 +5337,7 @@ is non-nil then TAG is a regexp." (cl-loop for xref in xrefs for loc = (xref-item-location xref) for file = (xref-location-group loc) - do (pushnew file files :test 'string-equal)) + do (cl-pushnew file files :test 'string-equal)) (or (cl-loop for file in files append (erlang-xrefs-in-file file kind tag is-regexp)) ;; Failed for some reason. Pretend like it is raining and diff --git a/lib/tools/emacs/erldoc.el b/lib/tools/emacs/erldoc.el index a8ac81ecb3..51fc67c513 100644 --- a/lib/tools/emacs/erldoc.el +++ b/lib/tools/emacs/erldoc.el @@ -60,9 +60,10 @@ ;;; Code: -(eval-when-compile (require 'url-parse)) (require 'cl-lib) +(require 'json) (require 'erlang) +(eval-when-compile (require 'url-parse)) (eval-and-compile ;for emacs < 24.3 (or (fboundp 'user-error) (defalias 'user-error 'error))) @@ -268,7 +269,6 @@ up the indexing." (with-temp-buffer (if (not json) (pp table (current-buffer)) - (eval-and-compile (require 'json)) (let ((json-encoding-pretty-print t)) (insert (json-encode table)))) (unless (file-directory-p (file-name-directory output)) diff --git a/lib/tools/src/cover.erl b/lib/tools/src/cover.erl index 9f86a68942..28cbedddce 100644 --- a/lib/tools/src/cover.erl +++ b/lib/tools/src/cover.erl @@ -1552,14 +1552,12 @@ do_compile2(File, UserOptions, LocalOnly) -> do_compile_beam1(Module,Beam,UserOptions,LocalOnly) -> %% Clear database do_clear(Module), - + %% Extract the abstract format. case get_abstract_code(Module, Beam) of - no_abstract_code=E -> - {error,E}; - encrypted_abstract_code=E -> - {error,E}; - {raw_abstract_v1,Code} -> + {error,_}=Error -> + Error; + {ok,{raw_abstract_v1,Code}} -> Forms0 = epp:interpret_file_attribute(Code), case find_main_filename(Forms0) of {ok,MainFile} -> @@ -1568,7 +1566,7 @@ do_compile_beam1(Module,Beam,UserOptions,LocalOnly) -> Error -> Error end; - {_VSN,_Code} -> + {ok,{_VSN,_Code}} -> %% Wrong version of abstract code. Just report that there %% is no abstract code. {error,no_abstract_code} @@ -1577,10 +1575,14 @@ do_compile_beam1(Module,Beam,UserOptions,LocalOnly) -> get_abstract_code(Module, Beam) -> case beam_lib:chunks(Beam, [abstract_code]) of {ok, {Module, [{abstract_code, AbstractCode}]}} -> - AbstractCode; + case AbstractCode of + no_abstract_code=E -> {error, E}; + _ -> {ok,AbstractCode} + end; {error,beam_lib,{key_missing_or_invalid,_,_}} -> - encrypted_abstract_code; - Error -> Error + {error,encrypted_abstract_code}; + {error,beam_lib,{missing_backend,_,Backend}} -> + {error,{missing_backend,Backend}} end. do_compile_beam2(Module,Beam,UserOptions,Forms0,MainFile,LocalOnly) -> @@ -2179,6 +2181,8 @@ patch_code1({'BUMP',_Line,Index}, {local_only,AbstrCref}) -> [AbstrCref,{integer,A,Index},{integer,A,1}]}; patch_code1({clauses,Cs}, Key) -> {clauses,[patch_code1(El, Key) || El <- Cs]}; +patch_code1({attribute, _, _, _} = Attribute, _Key) -> + Attribute; patch_code1([_|_]=List, Key) -> [patch_code1(El, Key) || El <- List]; patch_code1(Tuple, Key) when tuple_size(Tuple) >= 3 -> diff --git a/lib/tools/test/cover_SUITE.erl b/lib/tools/test/cover_SUITE.erl index f4c3a0e3b6..771f527e43 100644 --- a/lib/tools/test/cover_SUITE.erl +++ b/lib/tools/test/cover_SUITE.erl @@ -31,13 +31,14 @@ all() -> NoStartStop = [eif,otp_5305,otp_5418,otp_7095,otp_8273, otp_8340,otp_8188,compile_beam_opts,eep37, analyse_no_beam, line_0, compile_beam_no_file, + compile_beam_missing_backend, otp_13277, otp_13289], StartStop = [start, compile, analyse, misc, stop, distribution, reconnect, die_and_reconnect, dont_reconnect_after_stop, stop_node_after_disconnect, export_import, otp_5031, otp_6115, otp_8270, otp_10979_hanging_node, otp_14817, - local_only, startup_race, otp_16476], + local_only, startup_race, otp_16476, cover_clauses], case whereis(cover_server) of undefined -> [coverage,StartStop ++ NoStartStop]; @@ -1600,6 +1601,15 @@ otp_14817(Config) when is_list(Config) -> ok = file:delete(CovOut), ok. +%% Tests a bug where cover failed for an export named clauses +cover_clauses(Config) when is_list(Config) -> + Test = <<"-module(cover_clauses). + -export([clauses/0]). + clauses() -> ok. + ">>, + File = cc_mod(cover_clauses, Test, Config), + ok. + %% Take compiler options from beam in cover:compile_beam compile_beam_opts(Config) when is_list(Config) -> {ok, Cwd} = file:get_cwd(), @@ -1706,12 +1716,42 @@ compile_beam_no_file(Config) -> [{error,{no_file_attribute,BeamFile}}] = cover:compile_beam_directory(Dir), ok. +%% GH-4353: Don't crash when the backend for generating the abstract code +%% is missing. +compile_beam_missing_backend(Config) -> + PrivDir = proplists:get_value(priv_dir, Config), + Dir = filename:join(PrivDir, ?FUNCTION_NAME), + ok = filelib:ensure_dir(filename:join(Dir, "*")), + code:add_patha(Dir), + Str = lists:append( + ["-module(no_backend).\n" + "-compile(export_all).\n" + "foo() -> ok.\n"]), + TT = do_scan(Str), + Forms = [ begin {ok,Y} = erl_parse:parse_form(X),Y end || X <- TT ], + {ok,_,Bin} = compile:forms(Forms, [debug_info]), + + %% Create a debug_info chunk with a non-existing backend. + {ok,no_backend,All0} = beam_lib:all_chunks(Bin), + FakeBackend = definitely__not__an__existing__backend, + FakeDebugInfo = {debug_info_v1,FakeBackend,nothing_here}, + All = lists:keyreplace("Dbgi", 1, All0, {"Dbgi", term_to_binary(FakeDebugInfo)}), + {ok,NewBeam} = beam_lib:build_module(All), + BeamFile = filename:join(Dir, "no_backend.beam"), + ok = file:write_file(BeamFile, NewBeam), + + {error,{{missing_backend,FakeBackend},BeamFile}} = cover:compile_beam(no_backend), + [{error,{{missing_backend,FakeBackend},BeamFile}}] = cover:compile_beam_directory(Dir), + + ok. + do_scan([]) -> []; do_scan(Str) -> {done,{ok,T,_},C} = erl_scan:tokens([],Str,0), [ T | do_scan(C) ]. + %% PR 856. Fix a bc bug. otp_13277(Config) -> Test = <<"-module(t). diff --git a/lib/tools/test/emacs_SUITE.erl b/lib/tools/test/emacs_SUITE.erl index e5587b35b1..f74a159cd2 100644 --- a/lib/tools/test/emacs_SUITE.erl +++ b/lib/tools/test/emacs_SUITE.erl @@ -107,7 +107,7 @@ compile_and_load(_Config) -> %% Workaround byte-compile-error-on-warn which seem broken in %% Emacs 25. "\"(advice-add #'display-warning :after " - "(lambda (_ f _ _) (error \"%s\" f)))\""; + "(lambda (_ f &optional _ _) (error \\\"%s\\\" f)))\""; _ -> "\"(setq byte-compile-error-on-warn t)\"" end, diff --git a/lib/tools/test/instrument_SUITE.erl b/lib/tools/test/instrument_SUITE.erl index 0708251f10..41b7b11cd2 100644 --- a/lib/tools/test/instrument_SUITE.erl +++ b/lib/tools/test/instrument_SUITE.erl @@ -19,7 +19,7 @@ %% -module(instrument_SUITE). --export([all/0, suite/0]). +-export([all/0, suite/0, init_per_suite/1, end_per_suite/1]). -export([allocations_enabled/1, allocations_disabled/1, allocations_ramv/1, carriers_enabled/1, carriers_disabled/1]). @@ -37,6 +37,19 @@ all() -> [allocations_enabled, allocations_disabled, allocations_ramv, carriers_enabled, carriers_disabled]. +init_per_suite(Config) -> + case test_server:is_asan() of + true -> + %% No point testing own allocators under address sanitizer. + {skip, "Address sanitizer"}; + false -> + Config + end. + +end_per_suite(_Config) -> + ok. + + -define(GENERATED_SBC_BLOCK_COUNT, 1000). -define(GENERATED_MBC_BLOCK_COUNT, ?GENERATED_SBC_BLOCK_COUNT). diff --git a/lib/tools/test/xref_SUITE_data/lib_test/bi.erl b/lib/tools/test/xref_SUITE_data/lib_test/bi.erl deleted file mode 100644 index e083fa0f3c..0000000000 --- a/lib/tools/test/xref_SUITE_data/lib_test/bi.erl +++ /dev/null @@ -1,3 +0,0 @@ --module(bi). - --callback a() -> ok. diff --git a/lib/tools/test/xref_SUITE_data/lib_test/no_bi.erl b/lib/tools/test/xref_SUITE_data/lib_test/no_bi.erl deleted file mode 100644 index 5aac4a193e..0000000000 --- a/lib/tools/test/xref_SUITE_data/lib_test/no_bi.erl +++ /dev/null @@ -1,6 +0,0 @@ --module(no_bi). - --export([behaviour_info/1]). - -behaviour_info(_) -> - ok. diff --git a/lib/tools/vsn.mk b/lib/tools/vsn.mk index 6e4ee2fd14..33ff4b4a7c 100644 --- a/lib/tools/vsn.mk +++ b/lib/tools/vsn.mk @@ -1 +1 @@ -TOOLS_VSN = 3.4.3 +TOOLS_VSN = 3.4.4 diff --git a/lib/wx/c_src/wxe_ps_init.c b/lib/wx/c_src/wxe_ps_init.c index d82d142967..5d19502fbe 100644 --- a/lib/wx/c_src/wxe_ps_init.c +++ b/lib/wx/c_src/wxe_ps_init.c @@ -29,19 +29,8 @@ extern OSErr CPSSetProcessName (ProcessSerialNumber *psn, char *processname); -void * wxe_ps_init() +void * wxe_ps_init() { - ProcessSerialNumber psn; - // Enable GUI - if(!GetCurrentProcess(&psn)) { - TransformProcessType(&psn, kProcessTransformToForegroundApplication); -#ifdef MAC_OS_X_VERSION_10_6 - [[NSRunningApplication currentApplication] activateWithOptions: - (NSApplicationActivateAllWindows | NSApplicationActivateIgnoringOtherApps)]; -#else - SetFrontProcess(&psn); -#endif - } return (void *) 0; } @@ -66,7 +55,6 @@ void * wxe_ps_init2() { char * app_title; size_t app_icon_len = 1023; char app_icon_buf[1024]; - char * app_icon; // Setup and enable gui pool = [[NSAutoreleasePool alloc] init]; diff --git a/lib/wx/doc/src/notes.xml b/lib/wx/doc/src/notes.xml index fe5a9966f8..62f333f0c1 100644 --- a/lib/wx/doc/src/notes.xml +++ b/lib/wx/doc/src/notes.xml @@ -32,6 +32,24 @@ <p>This document describes the changes made to the wxErlang application.</p> +<section><title>Wx 1.9.3</title> + + <section><title>Fixed Bugs and Malfunctions</title> + <list> + <item> + <p> + Fixed wx initialization on mac, top level menus did not + always work on newer MacOS versions. The menues will not + work until wxWidgets-3.1.5 is released and used on these + MacOS versions.</p> + <p> + Own Id: OTP-17187</p> + </item> + </list> + </section> + +</section> + <section><title>Wx 1.9.2</title> <section><title>Fixed Bugs and Malfunctions</title> diff --git a/lib/wx/vsn.mk b/lib/wx/vsn.mk index e2ff4812e9..42776cc17a 100644 --- a/lib/wx/vsn.mk +++ b/lib/wx/vsn.mk @@ -1 +1 @@ -WX_VSN = 1.9.2 +WX_VSN = 1.9.3 diff --git a/make/otp_subdir.mk b/make/otp_subdir.mk index 19c744955c..f9b993e048 100644 --- a/make/otp_subdir.mk +++ b/make/otp_subdir.mk @@ -20,12 +20,12 @@ # Make include file for otp .PHONY: debug opt lcnt release docs release_docs tests release_tests \ - clean depend valgrind static_lib + clean depend valgrind asan static_lib # # Targets that don't affect documentation directories # -opt debug lcnt release docs release_docs tests release_tests clean depend valgrind static_lib xmllint: +opt debug lcnt release docs release_docs tests release_tests clean depend valgrind asan static_lib xmllint: @set -e ; \ app_pwd=`pwd` ; \ if test -f vsn.mk; then \ diff --git a/make/otp_version_tickets b/make/otp_version_tickets index 9dc3c0a550..b8220e1a87 100644 --- a/make/otp_version_tickets +++ b/make/otp_version_tickets @@ -1 +1 @@ -OTP-17289 +DEVELOPMENT diff --git a/make/run_make.mk b/make/run_make.mk index bcbbf53f7d..087129866d 100644 --- a/make/run_make.mk +++ b/make/run_make.mk @@ -29,9 +29,9 @@ include $(ERL_TOP)/make/output.mk include $(ERL_TOP)/make/target.mk -.PHONY: valgrind +.PHONY: valgrind asan -opt debug purify quantify purecov valgrind gcov gprof lcnt frmptr icount: +opt debug purify quantify purecov valgrind asan gcov gprof lcnt frmptr icount: $(make_verbose)$(MAKE) -f $(TARGET)/Makefile TYPE=$@ plain smp frag smp_frag: diff --git a/otp_versions.table b/otp_versions.table index 8c97c7f14f..f6ccfd0612 100644 --- a/otp_versions.table +++ b/otp_versions.table @@ -1,3 +1,5 @@ +OTP-23.3.1 : ssh-4.11.1 # asn1-5.0.14 common_test-1.20 compiler-7.6.7 crypto-4.9 debugger-5.0 dialyzer-4.3.1 diameter-2.2.3 edoc-0.12 eldap-1.2.9 erl_docgen-1.0.2 erl_interface-4.0.2 erts-11.2 et-1.6.4 eunit-2.6 ftp-1.0.5 hipe-4.0.1 inets-7.3.2 jinterface-1.11.1 kernel-7.3 megaco-3.19.5 mnesia-4.19 observer-2.9.5 odbc-2.13.3 os_mon-2.6.1 parsetools-2.2 public_key-1.10 reltool-0.8 runtime_tools-1.16 sasl-4.0.2 snmp-5.8 ssl-10.3 stdlib-3.14.1 syntax_tools-2.5 tftp-1.0.2 tools-3.4.4 wx-1.9.3 xmerl-1.3.26 : +OTP-23.3 : common_test-1.20 compiler-7.6.7 crypto-4.9 dialyzer-4.3.1 eldap-1.2.9 erts-11.2 jinterface-1.11.1 kernel-7.3 mnesia-4.19 odbc-2.13.3 public_key-1.10 runtime_tools-1.16 sasl-4.0.2 snmp-5.8 ssh-4.11 ssl-10.3 stdlib-3.14.1 syntax_tools-2.5 tools-3.4.4 wx-1.9.3 # asn1-5.0.14 debugger-5.0 diameter-2.2.3 edoc-0.12 erl_docgen-1.0.2 erl_interface-4.0.2 et-1.6.4 eunit-2.6 ftp-1.0.5 hipe-4.0.1 inets-7.3.2 megaco-3.19.5 observer-2.9.5 os_mon-2.6.1 parsetools-2.2 reltool-0.8 tftp-1.0.2 xmerl-1.3.26 : OTP-23.2.7.1 : ssl-10.2.4.1 # asn1-5.0.14 common_test-1.19.1 compiler-7.6.6 crypto-4.8.3 debugger-5.0 dialyzer-4.3 diameter-2.2.3 edoc-0.12 eldap-1.2.8 erl_docgen-1.0.2 erl_interface-4.0.2 erts-11.1.8 et-1.6.4 eunit-2.6 ftp-1.0.5 hipe-4.0.1 inets-7.3.2 jinterface-1.11 kernel-7.2.1 megaco-3.19.5 mnesia-4.18.1 observer-2.9.5 odbc-2.13.2 os_mon-2.6.1 parsetools-2.2 public_key-1.9.2 reltool-0.8 runtime_tools-1.15.1 sasl-4.0.1 snmp-5.7.3 ssh-4.10.8 stdlib-3.14 syntax_tools-2.4 tftp-1.0.2 tools-3.4.3 wx-1.9.2 xmerl-1.3.26 : OTP-23.2.7 : kernel-7.2.1 ssl-10.2.4 # asn1-5.0.14 common_test-1.19.1 compiler-7.6.6 crypto-4.8.3 debugger-5.0 dialyzer-4.3 diameter-2.2.3 edoc-0.12 eldap-1.2.8 erl_docgen-1.0.2 erl_interface-4.0.2 erts-11.1.8 et-1.6.4 eunit-2.6 ftp-1.0.5 hipe-4.0.1 inets-7.3.2 jinterface-1.11 megaco-3.19.5 mnesia-4.18.1 observer-2.9.5 odbc-2.13.2 os_mon-2.6.1 parsetools-2.2 public_key-1.9.2 reltool-0.8 runtime_tools-1.15.1 sasl-4.0.1 snmp-5.7.3 ssh-4.10.8 stdlib-3.14 syntax_tools-2.4 tftp-1.0.2 tools-3.4.3 wx-1.9.2 xmerl-1.3.26 : OTP-23.2.6 : inets-7.3.2 ssh-4.10.8 # asn1-5.0.14 common_test-1.19.1 compiler-7.6.6 crypto-4.8.3 debugger-5.0 dialyzer-4.3 diameter-2.2.3 edoc-0.12 eldap-1.2.8 erl_docgen-1.0.2 erl_interface-4.0.2 erts-11.1.8 et-1.6.4 eunit-2.6 ftp-1.0.5 hipe-4.0.1 jinterface-1.11 kernel-7.2 megaco-3.19.5 mnesia-4.18.1 observer-2.9.5 odbc-2.13.2 os_mon-2.6.1 parsetools-2.2 public_key-1.9.2 reltool-0.8 runtime_tools-1.15.1 sasl-4.0.1 snmp-5.7.3 ssl-10.2.3 stdlib-3.14 syntax_tools-2.4 tftp-1.0.2 tools-3.4.3 wx-1.9.2 xmerl-1.3.26 : @@ -18,6 +20,10 @@ OTP-23.0.3 : compiler-7.6.2 erts-11.0.3 # asn1-5.0.13 common_test-1.19 crypto-4. OTP-23.0.2 : erts-11.0.2 megaco-3.19.1 # asn1-5.0.13 common_test-1.19 compiler-7.6.1 crypto-4.7 debugger-5.0 dialyzer-4.2 diameter-2.2.3 edoc-0.12 eldap-1.2.8 erl_docgen-1.0 erl_interface-4.0 et-1.6.4 eunit-2.5 ftp-1.0.4 hipe-4.0 inets-7.2 jinterface-1.11 kernel-7.0 mnesia-4.17 observer-2.9.4 odbc-2.13 os_mon-2.5.2 parsetools-2.2 public_key-1.8 reltool-0.8 runtime_tools-1.15 sasl-4.0 snmp-5.6 ssh-4.10 ssl-10.0 stdlib-3.13 syntax_tools-2.3 tftp-1.0.2 tools-3.4 wx-1.9.1 xmerl-1.3.25 : OTP-23.0.1 : compiler-7.6.1 erts-11.0.1 # asn1-5.0.13 common_test-1.19 crypto-4.7 debugger-5.0 dialyzer-4.2 diameter-2.2.3 edoc-0.12 eldap-1.2.8 erl_docgen-1.0 erl_interface-4.0 et-1.6.4 eunit-2.5 ftp-1.0.4 hipe-4.0 inets-7.2 jinterface-1.11 kernel-7.0 megaco-3.19 mnesia-4.17 observer-2.9.4 odbc-2.13 os_mon-2.5.2 parsetools-2.2 public_key-1.8 reltool-0.8 runtime_tools-1.15 sasl-4.0 snmp-5.6 ssh-4.10 ssl-10.0 stdlib-3.13 syntax_tools-2.3 tftp-1.0.2 tools-3.4 wx-1.9.1 xmerl-1.3.25 : OTP-23.0 : asn1-5.0.13 common_test-1.19 compiler-7.6 crypto-4.7 debugger-5.0 dialyzer-4.2 edoc-0.12 erl_docgen-1.0 erl_interface-4.0 erts-11.0 eunit-2.5 hipe-4.0 inets-7.2 jinterface-1.11 kernel-7.0 megaco-3.19 mnesia-4.17 observer-2.9.4 odbc-2.13 os_mon-2.5.2 parsetools-2.2 public_key-1.8 runtime_tools-1.15 sasl-4.0 snmp-5.6 ssh-4.10 ssl-10.0 stdlib-3.13 syntax_tools-2.3 tools-3.4 wx-1.9.1 xmerl-1.3.25 # diameter-2.2.3 eldap-1.2.8 et-1.6.4 ftp-1.0.4 reltool-0.8 tftp-1.0.2 : +OTP-22.3.4.17 : erts-10.7.2.9 kernel-6.5.2.2 ssh-4.9.1.3 tools-3.3.1.1 # asn1-5.0.12 common_test-1.18.2 compiler-7.5.4.3 crypto-4.6.5.2 debugger-4.2.8 dialyzer-4.1.1 diameter-2.2.3 edoc-0.11 eldap-1.2.8 erl_docgen-0.11 erl_interface-3.13.2 et-1.6.4 eunit-2.4.1 ftp-1.0.4.1 hipe-3.19.3 inets-7.1.3.3 jinterface-1.10.1 megaco-3.18.8.3 mnesia-4.16.3.1 observer-2.9.3 odbc-2.12.4 os_mon-2.5.1.1 parsetools-2.1.8 public_key-1.7.2 reltool-0.8 runtime_tools-1.14 sasl-3.4.2 snmp-5.5.0.4 ssl-9.6.2.3 stdlib-3.12.1 syntax_tools-2.2.1 tftp-1.0.2 wx-1.9 xmerl-1.3.24 : +OTP-22.3.4.16 : erts-10.7.2.8 # asn1-5.0.12 common_test-1.18.2 compiler-7.5.4.3 crypto-4.6.5.2 debugger-4.2.8 dialyzer-4.1.1 diameter-2.2.3 edoc-0.11 eldap-1.2.8 erl_docgen-0.11 erl_interface-3.13.2 et-1.6.4 eunit-2.4.1 ftp-1.0.4.1 hipe-3.19.3 inets-7.1.3.3 jinterface-1.10.1 kernel-6.5.2.1 megaco-3.18.8.3 mnesia-4.16.3.1 observer-2.9.3 odbc-2.12.4 os_mon-2.5.1.1 parsetools-2.1.8 public_key-1.7.2 reltool-0.8 runtime_tools-1.14 sasl-3.4.2 snmp-5.5.0.4 ssh-4.9.1.2 ssl-9.6.2.3 stdlib-3.12.1 syntax_tools-2.2.1 tftp-1.0.2 tools-3.3.1 wx-1.9 xmerl-1.3.24 : +OTP-22.3.4.15 : crypto-4.6.5.2 # asn1-5.0.12 common_test-1.18.2 compiler-7.5.4.3 debugger-4.2.8 dialyzer-4.1.1 diameter-2.2.3 edoc-0.11 eldap-1.2.8 erl_docgen-0.11 erl_interface-3.13.2 erts-10.7.2.7 et-1.6.4 eunit-2.4.1 ftp-1.0.4.1 hipe-3.19.3 inets-7.1.3.3 jinterface-1.10.1 kernel-6.5.2.1 megaco-3.18.8.3 mnesia-4.16.3.1 observer-2.9.3 odbc-2.12.4 os_mon-2.5.1.1 parsetools-2.1.8 public_key-1.7.2 reltool-0.8 runtime_tools-1.14 sasl-3.4.2 snmp-5.5.0.4 ssh-4.9.1.2 ssl-9.6.2.3 stdlib-3.12.1 syntax_tools-2.2.1 tftp-1.0.2 tools-3.3.1 wx-1.9 xmerl-1.3.24 : +OTP-22.3.4.14 : compiler-7.5.4.3 erts-10.7.2.7 # asn1-5.0.12 common_test-1.18.2 crypto-4.6.5.1 debugger-4.2.8 dialyzer-4.1.1 diameter-2.2.3 edoc-0.11 eldap-1.2.8 erl_docgen-0.11 erl_interface-3.13.2 et-1.6.4 eunit-2.4.1 ftp-1.0.4.1 hipe-3.19.3 inets-7.1.3.3 jinterface-1.10.1 kernel-6.5.2.1 megaco-3.18.8.3 mnesia-4.16.3.1 observer-2.9.3 odbc-2.12.4 os_mon-2.5.1.1 parsetools-2.1.8 public_key-1.7.2 reltool-0.8 runtime_tools-1.14 sasl-3.4.2 snmp-5.5.0.4 ssh-4.9.1.2 ssl-9.6.2.3 stdlib-3.12.1 syntax_tools-2.2.1 tftp-1.0.2 tools-3.3.1 wx-1.9 xmerl-1.3.24 : OTP-22.3.4.13 : compiler-7.5.4.2 erts-10.7.2.6 megaco-3.18.8.3 snmp-5.5.0.4 # asn1-5.0.12 common_test-1.18.2 crypto-4.6.5.1 debugger-4.2.8 dialyzer-4.1.1 diameter-2.2.3 edoc-0.11 eldap-1.2.8 erl_docgen-0.11 erl_interface-3.13.2 et-1.6.4 eunit-2.4.1 ftp-1.0.4.1 hipe-3.19.3 inets-7.1.3.3 jinterface-1.10.1 kernel-6.5.2.1 mnesia-4.16.3.1 observer-2.9.3 odbc-2.12.4 os_mon-2.5.1.1 parsetools-2.1.8 public_key-1.7.2 reltool-0.8 runtime_tools-1.14 sasl-3.4.2 ssh-4.9.1.2 ssl-9.6.2.3 stdlib-3.12.1 syntax_tools-2.2.1 tftp-1.0.2 tools-3.3.1 wx-1.9 xmerl-1.3.24 : OTP-22.3.4.12 : erts-10.7.2.5 ssl-9.6.2.3 # asn1-5.0.12 common_test-1.18.2 compiler-7.5.4.1 crypto-4.6.5.1 debugger-4.2.8 dialyzer-4.1.1 diameter-2.2.3 edoc-0.11 eldap-1.2.8 erl_docgen-0.11 erl_interface-3.13.2 et-1.6.4 eunit-2.4.1 ftp-1.0.4.1 hipe-3.19.3 inets-7.1.3.3 jinterface-1.10.1 kernel-6.5.2.1 megaco-3.18.8.2 mnesia-4.16.3.1 observer-2.9.3 odbc-2.12.4 os_mon-2.5.1.1 parsetools-2.1.8 public_key-1.7.2 reltool-0.8 runtime_tools-1.14 sasl-3.4.2 snmp-5.5.0.3 ssh-4.9.1.2 stdlib-3.12.1 syntax_tools-2.2.1 tftp-1.0.2 tools-3.3.1 wx-1.9 xmerl-1.3.24 : OTP-22.3.4.11 : erts-10.7.2.4 mnesia-4.16.3.1 os_mon-2.5.1.1 ssh-4.9.1.2 # asn1-5.0.12 common_test-1.18.2 compiler-7.5.4.1 crypto-4.6.5.1 debugger-4.2.8 dialyzer-4.1.1 diameter-2.2.3 edoc-0.11 eldap-1.2.8 erl_docgen-0.11 erl_interface-3.13.2 et-1.6.4 eunit-2.4.1 ftp-1.0.4.1 hipe-3.19.3 inets-7.1.3.3 jinterface-1.10.1 kernel-6.5.2.1 megaco-3.18.8.2 observer-2.9.3 odbc-2.12.4 parsetools-2.1.8 public_key-1.7.2 reltool-0.8 runtime_tools-1.14 sasl-3.4.2 snmp-5.5.0.3 ssl-9.6.2.2 stdlib-3.12.1 syntax_tools-2.2.1 tftp-1.0.2 tools-3.3.1 wx-1.9 xmerl-1.3.24 : @@ -63,6 +69,10 @@ OTP-22.0.3 : compiler-7.4.2 dialyzer-4.0.1 erts-10.4.2 ssl-9.3.2 stdlib-3.9.2 # OTP-22.0.2 : compiler-7.4.1 crypto-4.5.1 erts-10.4.1 stdlib-3.9.1 # asn1-5.0.9 common_test-1.17.3 debugger-4.2.7 dialyzer-4.0 diameter-2.2.1 edoc-0.11 eldap-1.2.8 erl_docgen-0.9.1 erl_interface-3.12 et-1.6.4 eunit-2.3.7 ftp-1.0.2 hipe-3.19 inets-7.0.8 jinterface-1.10 kernel-6.4 megaco-3.18.5 mnesia-4.16 observer-2.9.1 odbc-2.12.4 os_mon-2.5 parsetools-2.1.8 public_key-1.6.7 reltool-0.8 runtime_tools-1.13.3 sasl-3.4 snmp-5.3 ssh-4.7.7 ssl-9.3.1 syntax_tools-2.2 tftp-1.0.1 tools-3.2 wx-1.8.8 xmerl-1.3.21 : OTP-22.0.1 : ssl-9.3.1 # asn1-5.0.9 common_test-1.17.3 compiler-7.4 crypto-4.5 debugger-4.2.7 dialyzer-4.0 diameter-2.2.1 edoc-0.11 eldap-1.2.8 erl_docgen-0.9.1 erl_interface-3.12 erts-10.4 et-1.6.4 eunit-2.3.7 ftp-1.0.2 hipe-3.19 inets-7.0.8 jinterface-1.10 kernel-6.4 megaco-3.18.5 mnesia-4.16 observer-2.9.1 odbc-2.12.4 os_mon-2.5 parsetools-2.1.8 public_key-1.6.7 reltool-0.8 runtime_tools-1.13.3 sasl-3.4 snmp-5.3 ssh-4.7.7 stdlib-3.9 syntax_tools-2.2 tftp-1.0.1 tools-3.2 wx-1.8.8 xmerl-1.3.21 : OTP-22.0 : asn1-5.0.9 common_test-1.17.3 compiler-7.4 crypto-4.5 debugger-4.2.7 dialyzer-4.0 edoc-0.11 eldap-1.2.8 erl_docgen-0.9.1 erl_interface-3.12 erts-10.4 hipe-3.19 inets-7.0.8 jinterface-1.10 kernel-6.4 megaco-3.18.5 mnesia-4.16 observer-2.9.1 odbc-2.12.4 os_mon-2.5 public_key-1.6.7 reltool-0.8 runtime_tools-1.13.3 sasl-3.4 snmp-5.3 ssh-4.7.7 ssl-9.3 stdlib-3.9 syntax_tools-2.2 tools-3.2 wx-1.8.8 xmerl-1.3.21 # diameter-2.2.1 et-1.6.4 eunit-2.3.7 ftp-1.0.2 parsetools-2.1.8 tftp-1.0.1 : +OTP-21.3.8.22 : erts-10.3.5.17 ssh-4.7.6.6 # asn1-5.0.8 common_test-1.17.2.1 compiler-7.3.2 crypto-4.4.2.3 debugger-4.2.6 dialyzer-3.3.2 diameter-2.2.1 edoc-0.10 eldap-1.2.7 erl_docgen-0.9 erl_interface-3.11.3.1 et-1.6.4 eunit-2.3.7 ftp-1.0.2.2 hipe-3.18.3 inets-7.0.7.2 jinterface-1.9.1 kernel-6.3.1.3 megaco-3.18.4 mnesia-4.15.6 observer-2.9 odbc-2.12.3 os_mon-2.4.7 otp_mibs-1.2.1 parsetools-2.1.8 public_key-1.6.6.1 reltool-0.7.8 runtime_tools-1.13.2 sasl-3.3 snmp-5.2.12 ssl-9.2.3.7 stdlib-3.8.2.4 syntax_tools-2.1.7.1 tftp-1.0.1 tools-3.1.0.1 wx-1.8.7 xmerl-1.3.20.1 : +OTP-21.3.8.21 : erts-10.3.5.16 # asn1-5.0.8 common_test-1.17.2.1 compiler-7.3.2 crypto-4.4.2.3 debugger-4.2.6 dialyzer-3.3.2 diameter-2.2.1 edoc-0.10 eldap-1.2.7 erl_docgen-0.9 erl_interface-3.11.3.1 et-1.6.4 eunit-2.3.7 ftp-1.0.2.2 hipe-3.18.3 inets-7.0.7.2 jinterface-1.9.1 kernel-6.3.1.3 megaco-3.18.4 mnesia-4.15.6 observer-2.9 odbc-2.12.3 os_mon-2.4.7 otp_mibs-1.2.1 parsetools-2.1.8 public_key-1.6.6.1 reltool-0.7.8 runtime_tools-1.13.2 sasl-3.3 snmp-5.2.12 ssh-4.7.6.5 ssl-9.2.3.7 stdlib-3.8.2.4 syntax_tools-2.1.7.1 tftp-1.0.1 tools-3.1.0.1 wx-1.8.7 xmerl-1.3.20.1 : +OTP-21.3.8.20 : erl_interface-3.11.3.1 # asn1-5.0.8 common_test-1.17.2.1 compiler-7.3.2 crypto-4.4.2.3 debugger-4.2.6 dialyzer-3.3.2 diameter-2.2.1 edoc-0.10 eldap-1.2.7 erl_docgen-0.9 erts-10.3.5.15 et-1.6.4 eunit-2.3.7 ftp-1.0.2.2 hipe-3.18.3 inets-7.0.7.2 jinterface-1.9.1 kernel-6.3.1.3 megaco-3.18.4 mnesia-4.15.6 observer-2.9 odbc-2.12.3 os_mon-2.4.7 otp_mibs-1.2.1 parsetools-2.1.8 public_key-1.6.6.1 reltool-0.7.8 runtime_tools-1.13.2 sasl-3.3 snmp-5.2.12 ssh-4.7.6.5 ssl-9.2.3.7 stdlib-3.8.2.4 syntax_tools-2.1.7.1 tftp-1.0.1 tools-3.1.0.1 wx-1.8.7 xmerl-1.3.20.1 : +OTP-21.3.8.19 : crypto-4.4.2.3 erts-10.3.5.15 # asn1-5.0.8 common_test-1.17.2.1 compiler-7.3.2 debugger-4.2.6 dialyzer-3.3.2 diameter-2.2.1 edoc-0.10 eldap-1.2.7 erl_docgen-0.9 erl_interface-3.11.3 et-1.6.4 eunit-2.3.7 ftp-1.0.2.2 hipe-3.18.3 inets-7.0.7.2 jinterface-1.9.1 kernel-6.3.1.3 megaco-3.18.4 mnesia-4.15.6 observer-2.9 odbc-2.12.3 os_mon-2.4.7 otp_mibs-1.2.1 parsetools-2.1.8 public_key-1.6.6.1 reltool-0.7.8 runtime_tools-1.13.2 sasl-3.3 snmp-5.2.12 ssh-4.7.6.5 ssl-9.2.3.7 stdlib-3.8.2.4 syntax_tools-2.1.7.1 tftp-1.0.1 tools-3.1.0.1 wx-1.8.7 xmerl-1.3.20.1 : OTP-21.3.8.18 : erts-10.3.5.14 ssh-4.7.6.5 # asn1-5.0.8 common_test-1.17.2.1 compiler-7.3.2 crypto-4.4.2.2 debugger-4.2.6 dialyzer-3.3.2 diameter-2.2.1 edoc-0.10 eldap-1.2.7 erl_docgen-0.9 erl_interface-3.11.3 et-1.6.4 eunit-2.3.7 ftp-1.0.2.2 hipe-3.18.3 inets-7.0.7.2 jinterface-1.9.1 kernel-6.3.1.3 megaco-3.18.4 mnesia-4.15.6 observer-2.9 odbc-2.12.3 os_mon-2.4.7 otp_mibs-1.2.1 parsetools-2.1.8 public_key-1.6.6.1 reltool-0.7.8 runtime_tools-1.13.2 sasl-3.3 snmp-5.2.12 ssl-9.2.3.7 stdlib-3.8.2.4 syntax_tools-2.1.7.1 tftp-1.0.1 tools-3.1.0.1 wx-1.8.7 xmerl-1.3.20.1 : OTP-21.3.8.17 : erts-10.3.5.13 ssl-9.2.3.7 # asn1-5.0.8 common_test-1.17.2.1 compiler-7.3.2 crypto-4.4.2.2 debugger-4.2.6 dialyzer-3.3.2 diameter-2.2.1 edoc-0.10 eldap-1.2.7 erl_docgen-0.9 erl_interface-3.11.3 et-1.6.4 eunit-2.3.7 ftp-1.0.2.2 hipe-3.18.3 inets-7.0.7.2 jinterface-1.9.1 kernel-6.3.1.3 megaco-3.18.4 mnesia-4.15.6 observer-2.9 odbc-2.12.3 os_mon-2.4.7 otp_mibs-1.2.1 parsetools-2.1.8 public_key-1.6.6.1 reltool-0.7.8 runtime_tools-1.13.2 sasl-3.3 snmp-5.2.12 ssh-4.7.6.4 stdlib-3.8.2.4 syntax_tools-2.1.7.1 tftp-1.0.1 tools-3.1.0.1 wx-1.8.7 xmerl-1.3.20.1 : OTP-21.3.8.16 : erts-10.3.5.12 # asn1-5.0.8 common_test-1.17.2.1 compiler-7.3.2 crypto-4.4.2.2 debugger-4.2.6 dialyzer-3.3.2 diameter-2.2.1 edoc-0.10 eldap-1.2.7 erl_docgen-0.9 erl_interface-3.11.3 et-1.6.4 eunit-2.3.7 ftp-1.0.2.2 hipe-3.18.3 inets-7.0.7.2 jinterface-1.9.1 kernel-6.3.1.3 megaco-3.18.4 mnesia-4.15.6 observer-2.9 odbc-2.12.3 os_mon-2.4.7 otp_mibs-1.2.1 parsetools-2.1.8 public_key-1.6.6.1 reltool-0.7.8 runtime_tools-1.13.2 sasl-3.3 snmp-5.2.12 ssh-4.7.6.4 ssl-9.2.3.6 stdlib-3.8.2.4 syntax_tools-2.1.7.1 tftp-1.0.1 tools-3.1.0.1 wx-1.8.7 xmerl-1.3.20.1 : diff --git a/system/doc/general_info/DEPRECATIONS b/system/doc/general_info/DEPRECATIONS index 0f212348cf..358bc319af 100644 --- a/system/doc/general_info/DEPRECATIONS +++ b/system/doc/general_info/DEPRECATIONS @@ -21,11 +21,7 @@ # Added in OTP 23.2. # igor:_/_ since=23 remove=24 -erl_tidy:dir/0 since=23 remove=24 -erl_tidy:dir/1 since=23 remove=24 -erl_tidy:file/1 since=23 remove=24 -erl_tidy:module/1 since=23 remove=24 -erl_tidy:module/2 since=23 remove=24 +erl_tidy:_/_ since=23 remove=24 # # Added in OTP 23. diff --git a/system/doc/programming_examples/list_comprehensions.xml b/system/doc/programming_examples/list_comprehensions.xml index 706cb337ad..f9ce57f478 100644 --- a/system/doc/programming_examples/list_comprehensions.xml +++ b/system/doc/programming_examples/list_comprehensions.xml @@ -40,10 +40,10 @@ <c>[1,2,a,...]</c> and X is greater than 3.</p> <p>The notation <c><![CDATA[X <- [1,2,a,...]]]></c> is a generator and the expression <c>X > 3</c> is a filter.</p> - <p>An additional filter, <c>integer(X)</c>, can be added to restrict + <p>An additional filter, <c>is_integer(X)</c>, can be added to restrict the result to integers:</p> <pre> -> <input>[X || X <- [1,2,a,3,4,b,5,6], integer(X), X > 3].</input> +> <input>[X || X <- [1,2,a,3,4,b,5,6], is_integer(X), X > 3].</input> [4,5,6]</pre> <p>Generators can be combined. For example, the Cartesian product of two lists can be written as follows:</p> diff --git a/system/doc/programming_examples/records.xml b/system/doc/programming_examples/records.xml index d74ce22e4e..04c06816c5 100644 --- a/system/doc/programming_examples/records.xml +++ b/system/doc/programming_examples/records.xml @@ -222,7 +222,7 @@ print(#person{name = Name, age = Age, %% Demonstrates type testing, selector, updating. -birthday(P) when record(P, person) -> +birthday(P) when is_record(P, person) -> P#person{age = P#person.age + 1}. register_two_hackers() -> diff --git a/system/doc/reference_manual/processes.xml b/system/doc/reference_manual/processes.xml index 2b68be4b31..597258b9fd 100644 --- a/system/doc/reference_manual/processes.xml +++ b/system/doc/reference_manual/processes.xml @@ -122,14 +122,39 @@ spawn(Module, Name, Args) -> pid() <section> <title>Links</title> - <p>Two processes can be <em>linked</em> to each other. A link - between two processes <c>Pid1</c> and <c>Pid2</c> is created - by <c>Pid1</c> calling the BIF <c>link(Pid2)</c> (or conversely). - There also exist a number of <c>spawn_link</c> BIFs, which spawn - and link to a process in one operation.</p> - <p>Links are bidirectional and there can only be one link between - two processes. Repeated calls to <c>link(Pid)</c> have no effect.</p> - <p>A link can be removed by calling the BIF <c>unlink(Pid)</c>.</p> + <p> + Two processes can be <em>linked</em> to each other. Also a + process and a port that reside on the same node can be linked + to each other. A link beteen two processes can be created + if one of them calls the + <seemfa marker="erts:erlang#link/1"><c>link/1</c></seemfa> BIF + with the process identifier of the other process as argument. + Links can also be created using one the following spawn BIFs + <seemfa marker="erts:erlang#spawn_link/4"><c>spawn_link()</c></seemfa>, + <seemfa marker="erts:erlang#spawn_opt/5"><c>spawn_opt()</c></seemfa>, or + <seemfa marker="erts:erlang#spawn_request/5"><c>spawn_request()</c></seemfa>. + The spawn operation and the link operation will + be performed atomically, in these cases. + </p> + <p> + If one of the participants of a link terminates, it will + <seeguide marker="system/reference_manual:processes#sending_exit_signals">send + an exit signal</seeguide> to the other participant. The exit + signal will contain the + <seeguide marker="system/reference_manual:processes#link_exit_signal_reason">exit + reason</seeguide> of the terminated participant. + </p> + <p> + A link can be removed by calling the + <seemfa marker="erts:erlang#unlink/1"><c>unlink/1</c></seemfa> + BIF. + </p> + <p> + Links are bidirectional and there can only be one link between + two processes. Repeated calls to <c>link()</c> have no effect. + Either one of the involved processes may create or remove a + link. + </p> <p>Links are used to monitor the behaviour of other processes, see <seeguide marker="#errors">Error Handling</seeguide>.</p> </section> @@ -149,35 +174,198 @@ spawn(Module, Name, Args) -> pid() OTP supervision trees, which use this feature.</p> <section> - <title>Emitting Exit Signals</title> - <p>When a process terminates, it terminates with an - <em>exit reason</em> as explained in <seeguide marker="#term"> - Process Termination</seeguide>. This exit reason is emitted in - an <em>exit signal</em> to all linked processes.</p> - <p>A process can also call the function <c>exit(Pid,Reason)</c>. - This results in an exit signal with exit reason - <c>Reason</c> being emitted to <c>Pid</c>, but does not affect - the calling process.</p> + <marker id="sending_exit_signals"/> + <title>Sending Exit Signals</title> + <p> + When a process or port + <seeguide marker="#term">terminates</seeguide> it will + send exit signals to all processes and ports that it + is <seeguide marker="#links">linked</seeguide> to. + The exit signal will contain the following information: + </p> + <taglist> + <tag>Sender identifier</tag> + <item><p> + The process or port identifier of the process or port + that terminated. + </p></item> + <tag>Receiver identifier</tag> + <item><p> + The process or port identifier of the process or port + which the exit signal is sent to. + </p></item> + <tag>The <c>link</c> flag</tag> + <item><p> + This flag will be set indicating that the exit signal + was sent due to a link. + </p></item> + <tag><marker id="link_exit_signal_reason"/>Exit reason</tag> + <item><p> + The exit reason of the process or port that + terminated or the atom:</p> + <list> + <item><p> + <c>noproc</c> in case no process or port was + found when setting up a link in a preceeding + call to the + <seemfa marker="erts:erlang#link/1"><c>link(PidOrPort)</c></seemfa> + BIF. The process or port identified as sender + of the exit signal will equal the <c>PidOrPort</c> + argument passed to <c>link/1</c>. + </p></item> + <item><p> + <c>noconnection</c> in case the linked + processes resides on different nodes and + the connection between the nodes was lost or + could not be established. The process or port + identified as sender of the exit signal might + in this case still be alive. + </p></item> + </list> + </item> + </taglist> + + <p> + Exit signals can also be sent explicitly by calling the + <seemfa marker="erts:erlang#exit/2"><c>exit(PidOrPort, + Reason)</c></seemfa> BIF. The exit signal is sent to the + process or port identified by the <c>PidOrPort</c> argument. + The exit signal sent will contain the following information: + </p> + <taglist> + <tag>Sender identifier</tag> + <item><p> + The process identifier of the process that called + <c>exit/2</c>. + </p></item> + <tag>Receiver identifier</tag> + <item><p> + The process or port identifier of the process or port + which the exit signal is sent to. + </p></item> + <tag>The <c>link</c> flag</tag> + <item><p> + This flag will not be set, indicating that this exit + signal was not sent due to a link. + </p></item> + <tag>Exit reason</tag> + <item><p> + The term passed as <c>Reason</c> in the call to + <c>exit/2</c>. If <c>Reason</c> is the atom <c>kill</c>, + the receiver cannot + <seeerl marker="erts:erlang#process_flag_trap_exit">trap + the exit</seeerl> signal and will unconditionally + terminate when it receives the signal. + </p></item> + </taglist> </section> <section> + <marker id="receiving_exit_signals"/> <title>Receiving Exit Signals</title> - <p>The default behaviour when a process receives an exit signal - with an exit reason other than <c>normal</c>, is to terminate - and in turn emit exit signals with the same exit reason to its - linked processes. An exit signal with reason <c>normal</c> is - ignored.</p> - <p>A process can be set to trap exit signals by calling:</p> - <pre> -process_flag(trap_exit, true)</pre> - <p>When a process is trapping exits, it does not terminate when - an exit signal is received. Instead, the signal is transformed - into a message <c>{'EXIT',FromPid,Reason}</c>, which is put into - the mailbox of the process, just like a regular message.</p> - <p>An exception to the above is if the exit reason is <c>kill</c>, - that is if <c>exit(Pid,kill)</c> has been called. This - unconditionally terminates the process, regardless of if it is - trapping exit signals.</p> + + <p>What happens when a process receives an exit signal depends on:</p> + <list> + <item><p> + The <seeerl marker="erts:erlang#process_flag_trap_exit">trap exit</seeerl> + state of the receiver at the time when the exit signal is received. + </p></item> + <item><p> + The exit reason of the exit signal. + </p></item> + <item><p> + The sender of the exit signal. + </p></item> + <item><p> + The state of the <c>link</c> flag of the exit signal. If the + <c>link</c> flag is set, the exit signal was sent due to a + link; otherwise, the exit signal was sent by a call to the + <seemfa marker="erts:erlang#exit/2"><c>exit/2</c></seemfa> BIF. + </p></item> + <item><p> + If the <c>link</c> flag is set, what happens also depends on + whether the <seemfa marker="erts:erlang#unlink/1">link is still active + or not</seemfa> when the exit signal is received. + </p></item> + </list> + + <p> + Based on the above states, the following will happen when an + exit signal is received by a process: + </p> + <list> + <item> + <p>The exit signal is silently dropped if:</p> + <list> + <item><p> + the <c>link</c> flag of the exit signal is set and + the corresponding link has been deactivated. + </p></item> + <item><p> + the exit reason of the exit signal is the atom <c>normal</c>, + the receiver is not trapping exits, and the receiver and + sender are not the same process. + </p></item> + </list> + </item> + <item> + <p>The receiving process is terminated if:</p> + <list> + <item><p> + the <c>link</c> flag of the exit signal + is not set, and the exit reason of the exit signal + is the atom <c>kill</c>. The receiving process will + terminate with the atom <c>killed</c> as exit reason. + </p></item> + <item><p> + the receiver is not trapping exits, and the exit + reason is something other than the atom <c>normal</c>. + Also, if the <c>link</c> flag of the exit signal + is set, the link also needs to be active otherwise the + exit signal will be dropped. The exit reason of the + receiving process will equal the exit reason of the + exit signal. Note that if the <c>link</c> flag + is set, an exit reason of <c>kill</c> will <em>not</em> + be converted to <c>killed</c>. + </p></item> + <item><p> + the exit reason of the exit signal is the atom + <c>normal</c> and the sender of the exit signal is + the same process as the receiver. The <c>link</c> + flag cannot be set in this case. The exit reason + of the receiving process will be the atom <c>normal</c>. + </p></item> + </list> + </item> + <item> + <p> + The exit signal is converted to a message signal and + moved into the message queue of the receiver, if the + receiver is trapping exits, the <c>link</c> flag + of the exit signal is: + </p> + <list> + <item><p> + not set, and the exit reason of the signal is not + the atom <c>kill</c>. + </p></item> + <item><p> + set, and the corresponding link is active. + Note that an exit reason of <c>kill</c> will + <em>not</em> terminate the process in this + case and it will not be converted to + <c>killed</c>. + </p></item> + </list> + <p> + The converted message will be on the form + <c>{'EXIT', SenderID, Reason}</c> where <c>Reason</c> + equals the exit reason of the exit signal and + <c>SenderID</c> is the identifier of the process + or port that sent the exit signal. + </p> + </item> + </list> </section> </section> @@ -216,4 +404,3 @@ erase(Key) erase()</pre> </section> </chapter> - diff --git a/system/doc/reference_manual/typespec.xml b/system/doc/reference_manual/typespec.xml index a450f85742..b46cf0e8e8 100644 --- a/system/doc/reference_manual/typespec.xml +++ b/system/doc/reference_manual/typespec.xml @@ -476,22 +476,20 @@ <c>-spec</c> attribute. The general format is as follows: </p> <pre> - -spec Module:Function(ArgType1, ..., ArgTypeN) -> ReturnType.</pre> - <p> - The arity of the function must match the number of arguments, - else a compilation error occurs. - </p> + -spec Function(ArgType1, ..., ArgTypeN) -> ReturnType.</pre> <p> - This form can also be used in header files (.hrl) to declare type - information for exported functions. - Then these header files can be included in files that (implicitly or - explicitly) import these functions. + An implementation of the function with the same name + <c>Function</c> must exist in the current module, and the arity + of the function must match the number of arguments, else a + compilation error occurs. </p> <p> - Within a given module, the following shorthand suffices in most cases: + The following longer format with module name is also valid as + long as <c>Module</c> is the name of the current module. This + can be useful for documentation purposes. </p> <pre> - -spec Function(ArgType1, ..., ArgTypeN) -> ReturnType.</pre> + -spec Module:Function(ArgType1, ..., ArgTypeN) -> ReturnType.</pre> <p> Also, for documentation purposes, argument names can be given: </p> diff --git a/system/doc/system_architecture_intro/sys_arch_intro.xml b/system/doc/system_architecture_intro/sys_arch_intro.xml index e8ada6427b..f04e7a7879 100644 --- a/system/doc/system_architecture_intro/sys_arch_intro.xml +++ b/system/doc/system_architecture_intro/sys_arch_intro.xml @@ -93,7 +93,7 @@ <p>Database Management.</p> <list type="bulleted"> <item><em>QLC</em> Query language support for Mnesia DBMS.</item> - <item><em>Mnesia</em> A heavy duty real-time distributed database.</item> + <item><em>Mnesia</em> A heavy-duty real-time distributed database.</item> <item><em>ODBC</em> ODBC database interface.</item> </list> </item> |