Fix the RSA_SSLV23_PADDING padding type

This also fixes the public function RSA_padding_check_SSLv23.

Commit 6555a89 changed the padding check logic in RSA_padding_check_SSLv23
so that padding is rejected if the nul delimiter byte is not immediately
preceded by at least 8 bytes containing 0x03. Prior to that commit the
padding is rejected if it *is* preceded by at least 8 bytes containing 0x03.

Presumably this change was made to be consistent with what it says in
appendix E.3 of RFC 5246. Unfortunately that RFC is in error, and the
original behaviour was correct. This is fixed in later errata issued for
that RFC.

This has no impact on libssl for modern versions of OpenSSL because
there is no protocol support for SSLv2 in these versions. However
applications that call RSA_paddin_check_SSLv23 directly, or use the
RSA_SSLV23_PADDING mode may still be impacted. The effect of the original
error is that an RSA message encrypted by an SSLv2 only client will fail to
be decrypted properly by a TLS capable server, or a message encrypted by a
TLS capable client will fail to decrypt on an SSLv2 only server. Most
significantly an RSA message encrypted by a TLS capable client will be
successfully decrypted by a TLS capable server. This last case should fail
due to a rollback being detected.

Thanks to D. Katz and Joel Luellwitz (both from Trustwave) for reporting
this issue.

CVE-2021-23839

Reviewed-by: Paul Dale <pauli@openssl.org>

1 files changed, 8 insertions, 2 deletions

diff --git a/crypto/rsa/rsa_ssl.c b/crypto/rsa/rsa_ssl.c
index 1f155be175..ecdb3cee1f 100644
--- a/crypto/rsa/rsa_ssl.c
+++ b/crypto/rsa/rsa_ssl.c
@@ -55,7 +55,7 @@ int RSA_padding_add_SSLv23(unsigned char *to, int tlen,
 
 /*
  * Copy of RSA_padding_check_PKCS1_type_2 with a twist that rejects padding
- * if nul delimiter is not preceded by 8 consecutive 0x03 bytes. It also
+ * if nul delimiter is preceded by 8 consecutive 0x03 bytes. It also
  * preserves error code reporting for backward compatibility.
  */
 int RSA_padding_check_SSLv23(unsigned char *to, int tlen,
@@ -122,7 +122,13 @@ int RSA_padding_check_SSLv23(unsigned char *to, int tlen,
                                    RSA_R_NULL_BEFORE_BLOCK_MISSING);
     mask = ~good;
 
-    good &= constant_time_ge(threes_in_row, 8);
+    /*
+     * Reject if nul delimiter is preceded by 8 consecutive 0x03 bytes. Note
+     * that RFC5246 incorrectly states this the other way around, i.e. reject
+     * if it is not preceded by 8 consecutive 0x03 bytes. However this is
+     * corrected in subsequent errata for that RFC.
+     */
+    good &= constant_time_lt(threes_in_row, 8);
     err = constant_time_select_int(mask | good, err,
                                    RSA_R_SSLV3_ROLLBACK_ATTACK);
     mask = ~good;