diff options
| author | Kazuki Yamaguchi <k@rhe.jp> | 2022-09-30 20:33:08 +0900 |
|---|---|---|
| committer | Pauli <pauli@openssl.org> | 2022-10-04 19:05:26 +1100 |
| commit | 0b3867634f74f6cb7b60b3a0adde396421207214 (patch) | |
| tree | 7a7231dc9d195436e1b9cc4f79defaaa7dea4ee4 /crypto/bn | |
| parent | 47cd0e5b1f98fb88d6d8337f7ec0e16bb83cea32 (diff) | |
| download | openssl-new-0b3867634f74f6cb7b60b3a0adde396421207214.tar.gz | |
Fix error propagatation in BN_check_prime()
BN_check_prime() is supposed to return 0 for a composite number and -1
on error. Properly translate the return value of the internal function
ossl_bn_miller_rabin_is_prime(), where 0 means an error.
The confusion prevented BN_GENCB callbacks from aborting the primality
test or key generation routines utilizing this.
Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Nicola Tuveri <nic.tuv@gmail.com>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/19314)
Diffstat (limited to 'crypto/bn')
| -rw-r--r-- | crypto/bn/bn_prime.c | 5 |
1 files changed, 3 insertions, 2 deletions
diff --git a/crypto/bn/bn_prime.c b/crypto/bn/bn_prime.c index 9e2f6861a5..54f7ca611f 100644 --- a/crypto/bn/bn_prime.c +++ b/crypto/bn/bn_prime.c @@ -308,9 +308,10 @@ static int bn_is_prime_int(const BIGNUM *w, int checks, BN_CTX *ctx, goto err; #endif - ret = ossl_bn_miller_rabin_is_prime(w, checks, ctx, cb, 0, &status); - if (!ret) + if (!ossl_bn_miller_rabin_is_prime(w, checks, ctx, cb, 0, &status)) { + ret = -1; goto err; + } ret = (status == BN_PRIMETEST_PROBABLY_PRIME); err: #ifndef FIPS_MODULE |