RDRAND implementation does not appear to check CF as required in Intel software implementation guide about randomgen HOT 6 CLOSED

I've known about this for a long time but didn't really want to address it since it was a bit difficult in the structure of the NumPy random system.

I have not added retries which defaults to 10, but can be any non-negative integer (0 for no retries, something large like 1000 (this will delay for up to 40us which should be equivalent to infinite in practice). This was simple and should have been done.

The exception handling was more fragile. If you just want 64 bits, then it is very robust and will raise RuntimError. If you want to use it with a NumPy Generator, then it relies on a custom Lock to be used around all calls where the lock checks for an error on exit. The lock will raise RuntimError if RDRAND fails.

Finally, if you do end up in a failed state, the RDRAND instance should not be used again (and will in fact error).

from randomgen.

Our problem is that we need approximately 7 x 10^14 bits of randomness for our application. We are running on a cluster with approximately 160 individual CPUs (1920 cores). It will take 2 hours or so to generate the randomness that we need. We are using numpy. We need to have RDRAND check the carry flag.

How does one end up in a 'failed state' ?

It would be nice to have a fallback, such as 'use the 0th RDRAND value as a seed for a CSPRNG`

How should we proceed?

from randomgen.

You end up in the failed state if RDRAND fails to produce a random value after retries retries.

You can check for RuntimeError and then decide what you would like to do:

Here is an intention fail that uses a trick to make it always fail:

import randomgen as rg
import numpy as np

rd = rg.RDRAND()
st = rd.state
st["retries"] = -1  # always fail!
rd.state = st

gen = np.random.Generator(rd)
try:
    gen.integers(0,2**64, dtype=np.uint64, size=10)
except RuntimeError:
    print("RuntimeError hit, do something else or wait a bit and retry")

Once it passes CI I will put together a 1.19.2 release with these changes.

from randomgen.

I have also added a property success of a RDRAND instance that you can directly access to see if there has ever been a failure.

from randomgen.

We're running this in a batch environment. We basically want retries=∞. It looks like with your code we can do:

st["retries"] = 1E9

Thanks for attending to this.

from randomgen.

The wheels are now available on PyPI.

from randomgen.