Giter Club home page Giter Club logo

randomgen's Introduction

RandomGen

This package contains additional bit generators for NumPy's Generator and an ExtendedGenerator exposing methods not in Generator.

Continuous Integration

Azure Build Status Cirrus CI Build Status

Coverage

codecov

Latest Release

PyPI version Anacnoda Cloud

License

NCSA License BSD License DOI

This is a library and generic interface for alternative random generators in Python and NumPy.

New Features

The the development documentation for the latest features, or the stable documentation for the latest released features.

WARNINGS

Changes in v1.24

Generator and RandomState were removed in 1.23.0.

Changes from 1.18 to 1.19

Generator and RandomState have been officially deprecated in 1.19, and will warn with a FutureWarning about their removal. They will also receive virtually no maintenance. It is now time to move to NumPy's np.random.Generator which has features not in randomstate.Generator and is maintained more actively.

A few distributions that are not present in np.random.Generator have been moved to randomstate.ExtendedGenerator:

  • multivariate_normal: which supports broadcasting
  • uintegers: fast 32 and 64-bit uniform integers
  • complex_normal: scalar complex normals

There are no plans to remove any of the bit generators, e.g., AESCounter, ThreeFry, or PCG64.

Changes from 1.16 to 1.18

There are many changes between v1.16.x and v1.18.x. These reflect API decision taken in conjunction with NumPy in preparation of the core of randomgen being used as the preferred random number generator in NumPy. These all issue DeprecationWarnings except for BasicRNG.generator which raises NotImplementedError. The C-API has also changed to reflect the preferred naming the underlying Pseudo-RNGs, which are now known as bit generators (or BigGenerators).

Future Plans

  • Add some distributions that are not supported in NumPy. Ongoing
  • Add any interesting bit generators I come across. Recent additions include the DXSM and CM-DXSM variants of PCG64 and the LXM generator.

Included Pseudo Random Number Generators

This module includes a number of alternative random number generators in addition to the MT19937 that is included in NumPy. The RNGs include:

  • Cryptographic cipher-based random number generator based on AES, ChaCha20, HC128 and Speck128.
  • MT19937, the NumPy rng
  • dSFMT a SSE2-aware version of the MT19937 generator that is especially fast at generating doubles
  • xoroshiro128+, xorshift1024*φ, xoshiro256**, and xoshiro512**
  • PCG64
  • ThreeFry and Philox from Random123
  • Other cryptographic-based generators: AESCounter, SPECK128, ChaCha, and HC128.
  • Hardware (non-reproducible) random number generator on AMD64 using RDRAND.
  • Chaotic PRNGS: Small-Fast Chaotic (SFC64) and Jenkin's Small-Fast (JSF).

Status

  • Builds and passes all tests on:
    • Linux 32/64 bit, Python 3.7, 3.8, 3.9, 3.10
    • Linux (ARM/ARM64), Python 3.8
    • OSX 64-bit, Python 3.9
    • Windows 32/64 bit, Python 3.7, 3.8, 3.9, 3.10
    • FreeBSD 64-bit

Version

The package version matches the latest version of NumPy when the package is released.

Documentation

Documentation for the latest release is available on my GitHub pages. Documentation for the latest commit (unreleased) is available under devel.

Requirements

Building requires:

  • Python (3.6, 3.7, 3.8, 3.9, 3.10)
  • NumPy (1.17+)
  • Cython (0.29+)
  • tempita (0.5+), if not provided by Cython

Testing requires pytest (6+).

Note: it might work with other versions but only tested with these versions.

Development and Testing

All development has been on 64-bit Linux, and it is regularly tested on Azure (Linux-AMD64, Window, and OSX) and Cirrus (FreeBSD and Linux-ARM).

Tests are in place for all RNGs. The MT19937 is tested against NumPy's implementation for identical results. It also passes NumPy's test suite where still relevant.

Installing

Either install from PyPi using

python -m pip install randomgen

or, if you want the latest version,

python -m pip install git+https://github.com/bashtage/randomgen.git

or from a cloned repo,

python -m pip install .

If you use conda, you can install using conda forge

conda install -c conda-forge randomgen

SSE2

dSFTM makes use of SSE2 by default. If you have a very old computer or are building on non-x86, you can install using:

export RANDOMGEN_NO_SSE2=1
python -m pip install . 

Windows

Either use a binary installer, or if building from scratch, use Python 3.6/3.7 with Visual Studio 2015 Build Toolx.

License

Dual: BSD 3-Clause and NCSA, plus sub licenses for components.

randomgen's People

Contributors

bashtage avatar bduvenhage avatar dham avatar jorgectf avatar pdebuyl avatar snyk-bot avatar thomastno avatar zoj613 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

randomgen's Issues

Possible off by one error on high/range of random numbers.

Hi

There seems to be an off by one error in comments at

cdef object _rand_{{nptype}}(object low, object high, object size, brng_t *state, object lock):

The comments mention "interval [low, high)", but also "high : If provided, the largest (signed) integer to be drawn from the distribution (see above for behaviour if high=None)."

High seem to be used correctly further on as the largest number to be drawn.

Numpy 2.0 compatibility

I am getting the following error when using randomgen on Numpy 2.0 (which has just released today):

  File "randomgen/aes.pyx", line 1, in init randomgen.aes
  File "randomgen/common.pyx", line 1, in init randomgen.common
ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

No issue on Numpy 1.26.4, the max prior version. I think it's likely that Numpy's ABI has changed in 2.0, causing an incompatibility.

Thanks for maintaining this package!

DOC: COrrect ChaCha's period

Why is the period of ChaCha 2^{128} -1, not 2^{128} ?
What is the rationale for subtracting one?
Is there any literature?

Enquiries about EFIIX64

EFIIX64 (also known as efiix64x48)
The name in Pracrand is Efiix64x384
Why is x48 written here and x384 written in Pracrand?

Random_raw (size=None) Is the output period 2^64?

ImportWarning

In Python 3.6, randomgen 1.14.1 throws the following warning:

>>> import warnings
>>> warnings.simplefilter('always')                                                                                                                                                                             
>>> import randomgen
/usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
/usr/lib/python3.6/importlib/_bootstrap.py:219: ImportWarning: can't resolve package from __spec__ or __package__, falling back on __name__ and __path__
  return f(*args, **kwds)
...

RLS: Preparing for 1.18

I have decided to skip 1.17 and release when NumPy 1.18 is out.

Tasks

  • Redo seeding of bit generators
  • Adopt language common with NumPy
  • Test against NumPy 1.18

ENH: Allow all values to be set in AesCounter counter range on initialization

>>> AESCounter(1, counter = (2**128)-4, mode = "sequence") #is OK

but

>>> AESCounter(1, counter = (2**128)-3, mode = "sequence")
ValueError: counter must be positive and less than 2**128.
>>> AESCounter(1, counter = (2**128)-2, mode = "sequence")
ValueError: counter must be positive and less than 2**128.
>>> AESCounter(1, counter = (2**128)-1, mode = "sequence")
ValueError: counter must be positive and less than 2**128.

Get some feedback

@rkern - not really sure how to try and get feedback on this. It is far from being ready for a PR, but it would be very helpful to get some feedback if this looks like a reasonable approach.

I've basically gone with the full void-pointer method and exposed 3 functions the return 64 bits, 32 bits and a double.

Performance differences with Julia?

Thank you very much for randomgen. It saves Python for fast random number generation!

Random number generation in Julia (https://docs.julialang.org/en/v1/stdlib/Random/index.html) with dSFMT (http://www.math.sci.hiroshima-u.ac.jp/~m-mat/MT/SFMT/#dSFMT) seems to be super fast, i.e. more than 2 times faster than with randomgen.

Anyway, do you understand how Julia can be so fast with dSFMT?

With Julia 1.0.2:

julia> using BenchmarkTools

julia> @btime rand(UInt32, 1000000);
  820.706 μs (2 allocations: 3.81 MiB)

julia> @btime rand(1000000);
  1.400 ms (2 allocations: 7.63 MiB)

With randomgen (recompiled locally):

In [1]: from randomgen import RandomGenerator, DSFMT

In [2]: rnd = RandomGenerator()

In [3]: %timeit rnd.random_uintegers(1000000, bits=32)
2.71 ms ± 2.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [4]: %timeit rnd.rand(1000000)
3.88 ms ± 4.02 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [5]: rnd = RandomGenerator(DSFMT())

In [6]: %timeit rnd.random_uintegers(1000000, bits=32)
3.73 ms ± 6.52 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In [7]: %timeit rnd.rand(1000000)
3.84 ms ± 6.57 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Document dependency between streams in PCG

PCG streams obtained by modifying the increment constant are not independent. The topic has been discussed at length for the Rust implementation:

rust-random/rand#907

This should be something the user is made aware of, as having a parameter and talking about "streams" in the documentation gives the idea that the streams are somehow independent.

In Threefry's docstrings, it says "Philox"?

Notes
ThreeFry is a 32 or 64-bit PRNG that uses a counter-based design based on
weaker (and faster) versions of cryptographic functions [1]_. Instances
using different values of the key produce distinct sequences. Philox
has a period of :math:N*2^{N*W}

1.16.2 broke Python 2.7 support

This commit breaks Python 2.7 support:

27bd211

Now I am all in favor of moving to Python 3 only, but the README says 1.16 will still support Python 2.7 so this should probably be fixed. I only noticed because we have a library that has randomgen as a requirement and its tests for Python 2.7 started failing.

randomgen does not build under numpy 1.18.1

@bashtage , I am so sorry to bother you again.

I am unable to get randomgen to compile with numpy 1.18.1 and python 3.7.6 (Anaconda 2020.02 distribution).

My previous compile attempts gave a warning that randomgen was using a deprecated interface. Now I get these errors:

gcc -pthread -B /mnt/apps/anaconda3-2020.02/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -I./randomgen -I/mnt/apps5/anaconda3-2020.02/lib/python3.7/site-packages/numpy/core/include -I/mnt/apps5/anaconda3-2020.02/include/python3.7m -c randomgen/bounded_integers.c -o build/temp.linux-x86_64-3.7/randomgen/bounded_integers.o -std=c99 -U__GNUC_GNU_INLINE__ -msse2
randomgen/bounded_integers.c: In function ‘__pyx_pf_5numpy_7ndarray___getbuffer__’:
randomgen/bounded_integers.c:21332:50: error: ‘NPY_C_CONTIGUOUS’ undeclared (first use in this function); did you mean ‘PyBUF_C_CONTIGUOUS’?
   __pyx_t_2 = ((!(PyArray_CHKFLAGS(__pyx_v_self, NPY_C_CONTIGUOUS) != 0)) != 0);
                                                  ^~~~~~~~~~~~~~~~
                                                  PyBUF_C_CONTIGUOUS
randomgen/bounded_integers.c:21332:50: note: each undeclared identifier is reported only once for each function it appears in
randomgen/bounded_integers.c:21388:50: error: ‘NPY_F_CONTIGUOUS’ undeclared (first use in this function); did you mean ‘NPY_C_CONTIGUOUS’?
   __pyx_t_2 = ((!(PyArray_CHKFLAGS(__pyx_v_self, NPY_F_CONTIGUOUS) != 0)) != 0);
                                                  ^~~~~~~~~~~~~~~~
                                                  NPY_C_CONTIGUOUS
randomgen/bounded_integers.c:21574:40: error: ‘PyArrayObject {aka struct tagPyArrayObject}’ has no member named ‘descr’
   __pyx_t_3 = ((PyObject *)__pyx_v_self->descr);
                                        ^~
In file included from /mnt/apps5/anaconda3-2020.02/include/python3.7m/pytime.h:6:0,
                 from /mnt/apps5/anaconda3-2020.02/include/python3.7m/Python.h:87,
                 from randomgen/bounded_integers.c:41:
randomgen/bounded_integers.c: In function ‘__pyx_f_5numpy_set_array_base’:
randomgen/bounded_integers.c:23218:27: error: ‘PyArrayObject {aka struct tagPyArrayObject}’ has no member named ‘base’; did you mean ‘ob_base’?
   Py_XDECREF(__pyx_v_arr->base);
                           ^
/mnt/apps5/anaconda3-2020.02/include/python3.7m/object.h:861:50: note: in definition of macro ‘Py_XDECREF’
         PyObject *_py_xdecref_tmp = (PyObject *)(op); \
                                                  ^~
randomgen/bounded_integers.c:23227:16: error: ‘PyArrayObject {aka struct tagPyArrayObject}’ has no member named ‘base’; did you mean ‘ob_base’?
   __pyx_v_arr->base = __pyx_v_baseptr;
                ^~~~
                ob_base
randomgen/bounded_integers.c: In function ‘__pyx_f_5numpy_get_array_base’:
randomgen/bounded_integers.c:23262:30: error: ‘PyArrayObject {aka struct tagPyArrayObject}’ has no member named ‘base’; did you mean ‘ob_base’?
   __pyx_t_1 = ((__pyx_v_arr->base == NULL) != 0);
                              ^~~~
                              ob_base
In file included from /mnt/apps5/anaconda3-2020.02/include/python3.7m/pytime.h:6:0,
                 from /mnt/apps5/anaconda3-2020.02/include/python3.7m/Python.h:87,
                 from randomgen/bounded_integers.c:41:
randomgen/bounded_integers.c:23294:44: error: ‘PyArrayObject {aka struct tagPyArrayObject}’ has no member named ‘base’; did you mean ‘ob_base’?
     __Pyx_INCREF(((PyObject *)__pyx_v_arr->base));
                                            ^
/mnt/apps5/anaconda3-2020.02/include/python3.7m/object.h:796:19: note: in definition of macro ‘Py_INCREF’
     ((PyObject *)(op))->ob_refcnt++)
                   ^~
randomgen/bounded_integers.c:23294:5: note: in expansion of macro ‘__Pyx_INCREF’
     __Pyx_INCREF(((PyObject *)__pyx_v_arr->base));
     ^~~~~~~~~~~~
randomgen/bounded_integers.c:23295:41: error: ‘PyArrayObject {aka struct tagPyArrayObject}’ has no member named ‘base’; did you mean ‘ob_base’?
     __pyx_r = ((PyObject *)__pyx_v_arr->base);
                                         ^~~~
                                         ob_base
error: command 'gcc' failed with exit status 1
$  python
Python 3.7.6 (default, Jan  8 2020, 19:59:22)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.__version__
'1.18.1'
>>>

I am also unable to install your wheel:

$ pip install randomgen-1.19.2-cp36-cp36m-manylinux1_x86_64.whl
ERROR: randomgen-1.19.2-cp36-cp36m-manylinux1_x86_64.whl is not a supported wheel on this platform.

random_uintegers?

I've installed the current pip version of randomgen (41.0.1), and it seems to have no random_uintegers:
AttributeError: 'randomgen.generator.RandomGenerator' object has no attribute 'random_uintegers'

I seem to recall we discussed this in the past, but I can't seem to find the discussion.

typo: psuedo/pseudo

The project description uses the word "psuedo" whereas I think the correct word is pseudo. I assume it is a typographical error.

Integrating randomgen components with Cython code

I've been able to use randomgen successfully with my Cython-based project, though I had a decent amount of trouble getting proper object/pointer cleanup working. Part of this is due to my lack of familiarity with PyCapsules, which are used in the examples, and working with the pointer it provides.

Before I get too far into the weeds, I was curious about any thoughts on creating *.pxd definition files for each of the RNG classes for the purpose of being able to cimport the extension types directly into external Cython code. Is there a good reason for avoiding this approach and using PyCapsules instead?

I should mention that I was drawn to randomgen in large part because it grants access to disparate, state-of-the-art RNGs with the possibility of using them in parallel applications (e.g. using nogil functions in parallel/prange blocks in Cython).

My notes on testing PRNGs

I see that random generator testing is currently underway in this repository.

On that note, I have written a page on testing PRNGs for high-quality randomness.

The following are some points on PRNG testing:

  • There are many kinds of PRNGs, including traditional, counter-based, and splittable PRNGs, as well as PRNGs that combine two or more other PRNGs, and there are different approaches to testing all these kinds.

  • The testing currently underway uses PractRand on each generator up to 4 TiB (2^42 bytes). It would be clearer if any results that don't fail at that level say "> 4 TiB" rather than "4 TiB". Also, I currently consider a PRNG statistically good if it doesn't fail PractRand at 1 TiB (2^40 bytes); is this threshold too low?

  • Different PRNGs have different ways to produce independent random number sequences ("streams"), such as by incrementing a seed, or discarding a huge number of outputs. A given stream strategy can be tested by interleaving the outputs of those streams. The number of streams can be two, three, even ten or more. Another strategy I have seen used to test T. Tyl's "shishua" PRNG is to interleave streams byte-by-byte, rather than output-by-output.

  • A relatively low-quality PRNG can be combined with another number sequence to produce a high-quality PRNG, enough to show no PractRand failure at 1 TiB (see "Combined PRNGs" for details). A notable example is L'Ecuyer's combined multiple recursive generators.

  • Counter-based PRNGs and splittable PRNGs (such as the PRNG used in JAX) can be based on hash functions and other mixing functions, so it's useful to test these functions to see if they are viable for use in these kinds of PRNGs. (I give some of these constructions in detail.) On the subject of mixing functions, a testing procedure by P. Evensen may be of interest.

  • Finally, see https://github.com/tweag/random-quality for a similar effort to PRNG testing for Haskell.

Shall we add ARS to randomgen?

It seems that ARS-5 might be the fastest counter based RNG out here with CPUs have AES instruction. Is there any reason it is not included? MKL use ARS-5 as one of its random generator, and it should be pretty mature. Since it's actually a simplified version of AES-CTR in this repo, would it be better that you open a branch with your code on AES round function, or what is your advice on reusing that part of code?

BTW, I apologize for using the phrase 'it's not vectorized' at discussion in NumPy; that may have come across as quite harsh. I was trying to defend my idea regarding qrand, whose core concept is ARS with modifications to reduce rounds and achieve vectorization similar to dSFMT with VAES and AVX. This library is awesome for experimenting with various random generators, and I would be delighted to help in adding a vectorized ARS-5 to it if you think it's worthwhile.

question

Hello,
Sorry. I`m learning and maybe is a nob question.
How can I use randomgen library to generate a random number like script below?
Thank you for your time.

import random
ran = random.randrange(3689,7378)
print (ran)

Please document source and meaning of constant 11400714819323198485ULL

We very much appreciate your rapid turn-around on the bugfix to rdrand.

However, in reviewing the code, we have discovered that the constant 11400714819323198485ULL is hardcoded. We cannot determine the source of this constant or its purpose.

state->buffer[i] = state->weyl_seq += 11400714819323198485ULL;

For our evaluation of this code, we need to be able to document the source of magic numbers such as this.

Also, it seems that you are not checking the carry flag, but are instead relying on the fact that RDRAND returns all 0s when it fails.

Finally, we believe that this should generate a Python exception when it is run on platforms that do not implement RDRAND, as is the case with other RDRAND implementations, rather than simply returning 0. That is your choice, but it is not a fail safe condition, so we will need to document that as well.

Thanks again for your maintenance of this program.

invalid bytes?

from numpy.random import Generator
from randomgen import AESCounter

 r546=AESCounter(seed=0, counter=0, mode="sequence").random_raw(size=None, output=True)
 r546= r546.to_bytes(16, byteorder='little').hex()

gives '3883cf555803f8ab0000000000000000'

but different form, starting from beginning

rg2 = Generator(AESCounter(seed=0, counter=0, mode="sequence"))
curr= rg2.bytes(8).hex()

gives '5ab7e3eeba6fcda0'

I don't see much similarity, why is it that different, why don't they match? Which one is valid?
Are there test vectors?

randomgen==1.23.1
numpy==1.19.5
Python 3.7.0 64bit

does this work on mac?

Hello,

I installed this this via anaconda on my macbook.

however when running import randomgen i get the following error:

<ipython-input-47-19d4274f594d> in <module>()
----> 1 import randomgen

/Users/yuliamahtani/anaconda/lib/python3.5/site-packages/randomgen/__init__.py in <module>()
      7 from randomgen.threefry import ThreeFry
      8 from randomgen.threefry32 import ThreeFry32
----> 9 from randomgen.xoroshiro128 import Xoroshiro128
     10 from randomgen.xorshift1024 import Xorshift1024
     11 

/Users/yuliamahtani/anaconda/lib/python3.5/site-packages/randomgen/xoroshiro128.pyx in init randomgen.xoroshiro128()
      7 cimport numpy as np
      8 
----> 9 from randomgen.common import interface
     10 from randomgen.common cimport *
     11 from randomgen.distributions cimport brng_t

ImportError: cannot import name interface

any thoughts on why it's not working?

Using RandomGenerator as a drop-in replacement for numpy's RandomState

This might be more of a feature request than an issue, per se. I was trying to use a randomgen.generator.RandomGenerator instance for repeated, stratified K-fold cross-validation using scikit-learn. I run several classification programs in parallel, and each uses cross-validation - I'd like each cross-validator to have a fixed seed, but to belong to an independent stream, which is why randomgen seemed to be a very attractive package.

The issue I ran into was that scikit-learn did not recognize a RandomGenerator object as a RandomState instance, and so it rejected it. Indeed, since RandomGenerator is a Cython class, I don't think it can inherit from RandomState.

I was looking for workarounds, and found that I could hack together something like this:

import numpy as np
import randomgen
from sklearn.model_selection import RepeatedStratifiedKFold 

# Create a new class which subclasses RandomState
class MyRandomGenerator(np.random.RandomState): 
    def __init__(self, source=None): 
        if source is not None: 
            # Override all methods coming from RandomState using a RandomGenerator instance
            for d in source.__dir__(): 
                if not d.startswith('_'): 
                    self.__dict__.update({d: getattr(source, d)}) 

rg = randomgen.generator.RandomGenerator(randomgen.Philox(42))
myrg = MyRandomGenerator(rg)

rskf = RepeatedStratifiedKFold(n_splits=2, n_repeats=2, random_state=myrg) 
X = np.arange(20).reshape((10, 2))
y = (np.random.rand(10) < 0.5).astype(int)
print([(train, test) for train, test in rskf.split(X, y)])
print(myrg.shuffle.__self__ is rg)

I'm sure this won't work in general, but it works here, since I was only intent on preserving the shuffle method. Is there a better way to do this, and do you plan to have a built-in mechanism for something like this?

RDRAND implementation does not appear to check CF as required in Intel software implementation guide

Intel's software implementation guide for the RDRAND instruction states that the Carry Flag must be checked after calling RDRAND to assure that the DRNG hasn't been saturated (it can only deliver 800Mbits/sec of randomness)

3.3.1 RDRAND

RDRAND retrieves a hardware-generated random value from the SP800-90A compliant DRGB and stores it in the destination register given as an argument to the instruction. The size of the random value (16-, 32-, or 64-bits) is determined by the size of the register given. The carry flag (CF) must be checked to determine whether a random value was available at the time of instruction execution.

Note that RDRAND is available to any system or application software running on the platform. That is, there are no hardware ring requirements that restrict access based on process privilege level. As such, RDRAND may be invoked as part of an operating system or hypervisor system library, a shared software library, or directly by an application.

To determine programmatically whether a given Intel platform supports RDRAND, developers can use the CPUID instruction to examine bit 30 of the ECX register. See Reference (7) for details.

In my review of the code, this doesn't seem to be happen.

randomgen/rdrand.pyx just calls rdrand_next64() and rdrand_next32():

return rdrand_next64(<rdrand_state*>st)

And rdrand_next64 just calls _rdrand64_step:

state->status &= _rdrand64_step((long long unsigned int *)&val);

This problem won't show up in unit testing, because if no other process is calling RDRAND, random bits will be available.
But if you are running on a 12-core CPU, all CPUs share the same RDRAND engine, and that poor engine can only generate 800 MB/sec of random bits:

3.4.1 RDRAND Performance†

In current-generation Intel processors the DRBG runs on a self-timed circuit clocked at 800 MHz and can service a RDRAND transaction (1 Tx) every 8 clocks for a maximum of 100 MTx per second. A transaction can be for a 16-, 32-, or 64-bit RDRAND, and the greatest throughput is achieved with 64-bit RDRANDs, capping the throughput ceiling at 800 MB/sec. These limits are an upper bound on all hardware threads across all cores on the CPU.

In an application that requires a lot of randomness on many cores, this will result in many threads getting ALL ZEROs.

The solution is for this code to test CF and go into a spin-loop. After 10 spins, you can throw an exception (as Intel would have you do), or you can keep spinning (as I would have you do).

Xoshiro256 jump is not deterministic

Possibly related to #321

When running the code below, the entropy of the jumped prng is different despite using a fixed initial seed.

from randomgen import Xoshiro256
bg = Xoshiro256(seed = 185147845955919253731470418070286414128)
bg = bg.jumped(1)
print(bg.seed_seq.entropy)
46920130038598399633968216498577965512

bg = Xoshiro256(seed = 185147845955919253731470418070286414128)
bg = bg.jumped(1)
print(bg.seed_seq.entropy)
339793229738386689970314384279387595865

Im using this prng in parallel where each mpi rank jumps by its rank number. I would like to have reproducible code based on the original instantiating seed but I cant seem to get deterministic entropies after jumping on each rank,

Future Warning

How do I turn off this warning? How do I set the mode to legacy?

'FutureWarning: mode is None which currently defaults to "legacy". After 1.19 this will change to "sequence". To silence this warning, set mode to "legacy" to use legacy seeding or "sequence" to defer seeding to NumPy's SeedSequence.'

New release

Follow-up of:

I'll try to get one out later this week. Would be good to get CI back to working on ARM and FreeBSD.

Originally posted by @bashtage in #310 (comment)

Would it be possible to also add wheels for Python3.10 while fixing CI?

permutation seems MUCH slower than shuffle?

IIUC, permutation(x) is the same as shuffle(x.copy()). But the timing is very different!

In a test using line_profiler, I'm getting:
44 539 946.0 1.8 1.7 shuffled = points.copy()
45 539 2451.0 4.5 4.4 rs.shuffle(shuffled)
46 539 44780.0 83.1 80.7 shuffled = rs.permutation(shuffled)

Trouble with examples and `distributions.pxd`

Hi there,

Just wanted to start off by saying that I'm really excited about this package and I hope to become engaged with the project eventually. I do a lot of physics simulations that involve stochastic equations and I've been dying to have more straightforward access to fast, robust, thread-safe RNGs via python/cython that can be swapped in/out (without so much recompiling or relying on Intel MKL).

Anyway, I'm currently interested in accessing the distribution generators using a cython cimport—e.g. from randomgen.distributions cimport random_gauss_zig. Ideally, I would like to generate single Gaussian RVs within a nogil block, so I need to avoid Python objects. However, I seem to get stuck with compilation unable proceed past this error from attempting to use the extending_distributions.pyx example:

fatal error: src/distributions/distributions.h: No such file or directory
 #include "src/distributions/distributions.h"

I've encountered this problem through the following approaches:

  1. cd to the examples/cython directory and run python setup.py install to directly compile the examples.
  2. Import random_gauss_zig in my own module using from randomgen.distributions cimport random_gauss_zig, then compile my extension module from my own setup.py. (Note: that in the Docs under Extending > Cython, import instead of cimport is written; I'm guessing this is a small typo given that the code in extending_distributions.pyx has the latter.)
  3. Do step 2, but explicitly add the randomgen/src/distributions directory to the list of included dirs
  4. Do step 3, and also explicitly add randomgen/src/distributions/distributions.c to the list of source files when compiling my extension module.

In particular, when I include the distributions directory during extension module compilation, I get the set of nasty errors below (so I suspect this isn't a kosher approach):

In file included from /usr/include/c++/7/bits/char_traits.h:39:0,
                 from /usr/include/c++/7/ios:40,
                 from /usr/include/c++/7/istream:38,
                 from /usr/include/c++/7/sstream:38,
                 from /usr/include/c++/7/complex:45,
                 from glrandom.c:869:
/usr/include/c++/7/bits/stl_algobase.h:243:56: error: macro "min" passed 3 arguments, but takes just 2
     min(const _Tp& __a, const _Tp& __b, _Compare __comp)
                                                        ^
/usr/include/c++/7/bits/stl_algobase.h:265:56: error: macro "max" passed 3 arguments, but takes just 2
     max(const _Tp& __a, const _Tp& __b, _Compare __comp)
                                                        ^
In file included from glrandom.c:653:0:
/usr/include/c++/7/bits/stl_algobase.h:195:5: error: expected unqualified-id before ‘const’
     min(const _Tp& __a, const _Tp& __b)
     ^
/usr/include/c++/7/bits/stl_algobase.h:195:5: error: expected ‘)’ before ‘const’
/usr/include/c++/7/bits/stl_algobase.h:195:5: error: expected ‘)’ before ‘const’
/usr/include/c++/7/bits/stl_algobase.h:195:5: error: expected initializer before ‘const’
/usr/include/c++/7/bits/stl_algobase.h:219:5: error: expected unqualified-id before ‘const’
     max(const _Tp& __a, const _Tp& __b)
     ^
/usr/include/c++/7/bits/stl_algobase.h:219:5: error: expected ‘)’ before ‘const’
/usr/include/c++/7/bits/stl_algobase.h:219:5: error: expected ‘)’ before ‘const’
/usr/include/c++/7/bits/stl_algobase.h:219:5: error: expected initializer before ‘const’
In file included from /usr/include/c++/7/bits/char_traits.h:39:0,
                 from /usr/include/c++/7/ios:40,
                 from /usr/include/c++/7/istream:38,
                 from /usr/include/c++/7/sstream:38,
                 from /usr/include/c++/7/complex:45,
                 from glrandom.c:869:
/usr/include/c++/7/bits/stl_algobase.h:246:7: error: expected primary-expression before ‘if’
       if (__comp(__b, __a))
       ^~
/usr/include/c++/7/bits/stl_algobase.h:246:7: error: expected ‘}’ before ‘if’
/usr/include/c++/7/bits/stl_algobase.h:246:7: error: expected ‘;’ before ‘if’
/usr/include/c++/7/bits/stl_algobase.h:248:7: error: expected unqualified-id before ‘return’
       return __a;
       ^~~~~~
/usr/include/c++/7/bits/stl_algobase.h:268:7: error: expected primary-expression before ‘if’
       if (__comp(__a, __b))
       ^~
/usr/include/c++/7/bits/stl_algobase.h:268:7: error: expected ‘}’ before ‘if’
/usr/include/c++/7/bits/stl_algobase.h:268:7: error: expected ‘;’ before ‘if’
/usr/include/c++/7/bits/stl_algobase.h:270:7: error: expected unqualified-id before ‘return’
       return __a;
       ^~~~~~
/usr/include/c++/7/bits/stl_algobase.h:271:5: error: expected declaration before ‘}’ token
     }
     ^

I'm using GCC 7.3.0 on Ubuntu 18.04 LTS, Python 2.7.15, Cython 0.28.3, Numpy 1.14.4, etc. I originally installed randomgen using python setup.py install in the toplevel directory, but I also re-attempted everything with a developer installation via pip as well as installing directly from your conda channel (conda install -c bashtage randomgen). There's a good chance I'm doing something stupid, so I do apologize in advance if I'm making a facile mistake. Any help would be very much appreciated and I'm happy to provide any useful information I might have left out.

Error while importing randomgen 1.16.0 on Numpy 1.15.4, Python 3.7.2

Firstly, thank you for working on this amazing package!

I just installed the package via pip. I believe the latest release (1.16.0) introduced a bug that throws a value error as soon as I try to import...

Python 3.7.2 (default, Dec 29 2018, 06:19:36) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.2.0 -- An enhanced Interactive Python. Type '?' for help.
>>> import numpy as np                                                                                                                                                                                                                        
>>> np.version.full_version                                                                                                                                                                                                                   
'1.15.4'
>>> import randomgen                                                                                                                                                                                                                          
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-3-5a0683341995> in <module>
----> 1 import randomgen

~/.anaconda3/lib/python3.7/site-packages/randomgen/__init__.py in <module>
----> 1 from randomgen.dsfmt import DSFMT
      2 from randomgen.generator import RandomGenerator
      3 from randomgen.mt19937 import MT19937
      4 from randomgen.pcg32 import PCG32
      5 from randomgen.pcg64 import PCG64

__init__.pxd in init randomgen.dsfmt()

ValueError: numpy.ufunc size changed, may indicate binary incompatibility. Expected 216 from C header, got 192 from PyObject

This does not happen with randomgen 1.15.1:

Python 3.7.2 (default, Dec 29 2018, 06:19:36) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.2.0 -- An enhanced Interactive Python. Type '?' for help.
>>> import randomgen                                                                                                                                                                                                                          
>>> randomgen.__version__                                                                                                                                                                                                                     
'1.15.1'
>>> import numpy as np                                                                                                                                                                                                                        
>>> np.version.full_version                                                                                                                                                                                                                   
'1.15.4'

Please let me know if you need any other info.

Partially-Sampled Random Numbers

Since you are dealing with random number generation here, I want to let you know about a page I have about so-called partially-sampled random numbers, which could be useful for accurately sampling from certain continuous distributions, including the beta and exponential distributions, to an arbitrary precision and without relying on floating-point arithmetic. Please comment on this page.

clarify relation between different parallel operations

https://bashtage.github.io/randomgen/parallel.html discusses three strategies for getting parallel random numbers, but it doesn't explicitly say what happens if they are combined. Or whether they should be combined. I've been adapting some old code, and it has led me to contemplating such mixing. Though I'm pretty sure I shouldn't 😃. It would be nice if the docs were clearer.

For example, what if one creates bg = PCG64(888888, 2) and then does bg.jumped()? If mixing jumped and advance, is

bg.advance(100)
bg.jumped()

equivalent to either bg.advance(100+JUMP_SIZE) or bg.jumped() alone (JUMP_SIZE is the number of steps jumped moves according to documents for the bit generator)?

In the following code, do the 2 bit generators end up in the same state, i.e., is using the stream index equivalent to jumping?

bg1 = PCG64(88, 2)
bg1.jumped()
bg2 = PCG64(88, 3)

unable to import "randomgen" on colab

I have been successfully using "randomgen" for a past couple of weeks. However, recently, I encountered this error and now, I am unable to import "randomgen" on colab. The commands are as follows:

!pip install randomgen
import randomgen

image

When I try to import it the second time, it displays the following error:

image

Any ideas please?

wheels for 3.11

Would be nice.

To build it on windows, you are forced to install microsoft spyware obliterating the target system's integrity.

I understand 3.11 is very new, but maybe it's as simple as incrementing a number in this case?

License file

Would be nice to have a license file included. Better yet if it could be listed in MANIFEST.in and setup.cfg for sdists and whls.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.