* correct buffer overflows cause by integer overflow in openssl

frustratingly, there is no test for this -- that's because testing this
requires allocating more memory than is available in CI.

fixes #5615.

* backport CI fixes

* another CI backport

* Removed unused argument.

* Added support for RSA signature recovery.

* Syntatic corrections for passing pep8 tests.

* Corrected typo.

* Added test of invalid Prehashed parameter to RSA signature recover.

* Renamed recover to a more descriptive name.

* Extended RSA signature recovery with option to return full data (not
only the digest part).

* Added missing words to pass spell check.

closes #5510

* disallow p less than 512-bit on DH

OpenSSL 3.0.0 enforces this so we'll go ahead and enforce it everywhere
that's practical for us. (Note that we do not enforce on deserializing
PKCS1/PKCS8 keys in < 3.0.0, but this PR adds a test so that in the
3.0.0 support branch we can test an error path)

* missing test

* black

* _MIN_MODULUS_SIZE is now a thing

* skip on fips

* Reduce granularity of error msging when deserializing keys

In OpenSSL 3.0 it is no longer possible to determine whether the reason
a key failed to deserialize is because of an unsupported cipher. Since
we want to be more resilient to OpenSSL error code instability we'll
just remove these paths.

* black

* changelog and update docs

the error will be opensslv.h these days and we don't need to talk
specifically about macOS