Giter Club home page Giter Club logo

rust-numpy's Introduction

rust-numpy

Actions Status Crate Minimum rustc 1.48 Documentation codecov

Rust bindings for the NumPy C-API.

API documentation

Requirements

  • Rust >= 1.48.0
    • Basically, our MSRV follows the one of PyO3
  • Python >= 3.7
    • Python 3.6 support was dropped from 0.16
  • Some Rust libraries
  • numpy installed in your Python environments (e.g., via pip install numpy)
    • We recommend numpy >= 1.16.0, though older versions may work

Example

Write a Python module in Rust

Please see the simple example for how to get started.

There are also examples using ndarray-linalg and rayon.

[lib]
name = "rust_ext"
crate-type = ["cdylib"]

[dependencies]
pyo3 = { version = "0.21", features = ["extension-module"] }
numpy = "0.21"
use numpy::ndarray::{ArrayD, ArrayViewD, ArrayViewMutD};
use numpy::{IntoPyArray, PyArrayDyn, PyReadonlyArrayDyn, PyArrayMethods};
use pyo3::{pymodule, types::PyModule, PyResult, Python, Bound};

#[pymodule]
fn rust_ext<'py>(_py: Python<'py>, m: &Bound<'py, PyModule>) -> PyResult<()> {
    // example using immutable borrows producing a new array
    fn axpy(a: f64, x: ArrayViewD<'_, f64>, y: ArrayViewD<'_, f64>) -> ArrayD<f64> {
        a * &x + &y
    }

    // example using a mutable borrow to modify an array in-place
    fn mult(a: f64, mut x: ArrayViewMutD<'_, f64>) {
        x *= a;
    }

    // wrapper of `axpy`
    #[pyfn(m)]
    #[pyo3(name = "axpy")]
    fn axpy_py<'py>(
        py: Python<'py>,
        a: f64,
        x: PyReadonlyArrayDyn<'py, f64>,
        y: PyReadonlyArrayDyn<'py, f64>,
    ) -> Bound<'py, PyArrayDyn<f64>> {
        let x = x.as_array();
        let y = y.as_array();
        let z = axpy(a, x, y);
        z.into_pyarray_bound(py)
    }

    // wrapper of `mult`
    #[pyfn(m)]
    #[pyo3(name = "mult")]
    fn mult_py<'py>(a: f64, x: &Bound<'py, PyArrayDyn<f64>>) {
        let x = unsafe { x.as_array_mut() };
        mult(a, x);
    }

    Ok(())
}

Execute a Python program from Rust and get results

[package]
name = "numpy-test"

[dependencies]
pyo3 = { version = "0.21", features = ["auto-initialize"] }
numpy = "0.21"
use numpy::{PyArray1, PyArrayMethods};
use pyo3::{types::{IntoPyDict, PyAnyMethods}, PyResult, Python};

fn main() -> PyResult<()> {
    Python::with_gil(|py| {
        let np = py.import_bound("numpy")?;
        let locals = [("np", np)].into_py_dict_bound(py);

        let pyarray = py
            .eval_bound("np.absolute(np.array([-1, -2, -3], dtype='int32'))", Some(&locals), None)?
            .downcast_into::<PyArray1<i32>>()?;

        let readonly = pyarray.readonly();
        let slice = readonly.as_slice()?;
        assert_eq!(slice, &[1, 2, 3]);

        Ok(())
    })
}

Dependency on ndarray

This crate uses types from ndarray in its public API. ndarray is re-exported in the crate root so that you do not need to specify it as a direct dependency.

Furthermore, this crate is compatible with multiple versions of ndarray and therefore depends on a range of semver-incompatible versions, currently >= 0.13, < 0.16. Cargo does not automatically choose a single version of ndarray by itself if you depend directly or indirectly on anything but that exact range. It can therefore be necessary to manually unify these dependencies.

For example, if you specify the following dependencies

numpy = "0.21"
ndarray = "0.13"

this will currently depend on both version 0.13.1 and 0.15.3 of ndarray by default even though 0.13.1 is within the range >= 0.13, < 0.16. To fix this, you can run

cargo update --package ndarray:0.15.3 --precise 0.13.1

to achieve a single dependency on version 0.13.1 of ndarray.

Contributing

We welcome issues and pull requests.

PyO3's Contributing.md is a nice guide for starting.

Also, we have a Gitter channel for communicating.

rust-numpy's People

Contributors

adamreichold avatar aldanor avatar askannz avatar atouchet avatar bmatthieu3 avatar clbarnes avatar davidhewitt avatar de-vri-es avatar dependabot[bot] avatar dsgibbons avatar fzyzcjy avatar g-bauer avatar icxolu avatar kngwyu avatar konstin avatar m-ou-se avatar masterchef365 avatar matthieubizien avatar messense avatar michaelcg8 avatar mtreinish avatar nihaals avatar ptnobel avatar rth avatar samster25 avatar sebasv avatar shatur avatar stefan-k avatar termoshtt avatar tesuji avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

rust-numpy's Issues

Cargo release for pyo3 ?

Hi, thanks for working on this library, it helps getting the FFI setup out of the way.

Are you planning on releasing the latest version (using pyo3) on crates.io anytime soon ?

Integer types not supported

Hi!

I've been using this awesomeness to extend python with Rust. However it really seems to me it is not possible to convert from an integer numpy array to a rust ndarray of the same element type. Interestingly when I supply an int32 array from python and I expect an i64 array in rust the conversion does not even fail but the array in rust will contain invalid values.

Compatibility with pyo3 v0.5.1

There might be some compatibility issue with the pyo3 v0.5.1 (that was released yesterday).

When building the example whith rust nightly on Linux, using,

pyo3 = "^0.5.0-alpha.2"

I get the following compilation error,

cargo rustc --lib --manifest-path Cargo.toml --features numpy/python3 pyo3/extension-module pyo3/python3 --verbose -- --crate-type cdylib --cfg=Py_3
       Fresh version_check v0.1.5
       Fresh cfg-if v0.1.6
       Fresh unicode-xid v0.1.0
       Fresh lazy_static v1.2.0
       Fresh ucd-util v0.1.3
       Fresh utf8-ranges v1.0.2
       Fresh proc-macro-hack-impl v0.4.1
       Fresh either v1.5.0
       Fresh rawpointer v0.1.0
       Fresh spin v0.4.10
       Fresh thread_local v0.3.6
       Fresh regex-syntax v0.6.3
       Fresh proc-macro-hack v0.4.1
       Fresh itertools v0.7.11
       Fresh libc v0.2.44
       Fresh proc-macro2 v0.4.24
       Fresh num-traits v0.2.6
       Fresh matrixmultiply v0.1.15
       Fresh memchr v2.1.1
       Fresh quote v0.6.10
       Fresh mashup-impl v0.1.9
       Fresh num-complex v0.2.1
       Fresh aho-corasick v0.6.9
       Fresh syn v0.15.22
       Fresh mashup v0.1.9
       Fresh ndarray v0.12.1
       Fresh regex v1.0.6
       Fresh pyo3-derive-backend v0.5.1
       Fresh pyo3cls v0.5.1
       Fresh pyo3 v0.5.1
   Compiling numpy v0.4.0-alpha.1 (https://github.com/rust-numpy/rust-numpy#baa3fe80)
     Running `rustc --crate-name numpy /usr/local/cargo/git/checkouts/rust-numpy-1fb78b81c057f15a/baa3fe8/src/lib.rs --color always --crate-type lib --emit=dep-info,link -C debuginfo=2 --cfg 'feature="default"' --cfg 'feature="pyo3"' --cfg 'feature="python3"' -C metadata=f174001306587673 -C extra-filename=-f174001306587673 --out-dir /src/target/debug/deps -L dependency=/src/target/debug/deps --extern cfg_if=/src/target/debug/deps/libcfg_if-32003ddcb6e568a3.rlib --extern libc=/src/target/debug/deps/liblibc-89b670f2733249b5.rlib --extern ndarray=/src/target/debug/deps/libndarray-237fe81e53a83645.rlib --extern num_complex=/src/target/debug/deps/libnum_complex-cc3986fda7fdfc2b.rlib --extern num_traits=/src/target/debug/deps/libnum_traits-a15758b954e4bbaa.rlib --extern pyo3=/src/target/debug/deps/libpyo3-b1c63cababa068d5.rlib --cap-lints allow`
error[E0432]: unresolved import `pyo3::PyObjectWithToken`
 --> /usr/local/cargo/git/checkouts/rust-numpy-1fb78b81c057f15a/baa3fe8/src/array.rs:6:29
  |
6 | use pyo3::{PyDowncastError, PyObjectWithToken, ToPyPointer};
  |                             ^^^^^^^^^^^^^^^^^ no `PyObjectWithToken` in the root. Did you mean to use `PyObjectWithGIL`?

error[E0412]: cannot find type `T` in this scope
   --> /usr/local/cargo/git/checkouts/rust-numpy-1fb78b81c057f15a/baa3fe8/src/array.rs:104:13
    |
104 |     PyArray<T, D>,
    |             ^ not found in this scope

error[E0412]: cannot find type `D` in this scope
   --> /usr/local/cargo/git/checkouts/rust-numpy-1fb78b81c057f15a/baa3fe8/src/array.rs:104:16
    |
104 |     PyArray<T, D>,
    |                ^ not found in this scope

error[E0107]: wrong number of type arguments: expected 0, found 1
  --> /usr/local/cargo/git/checkouts/rust-numpy-1fb78b81c057f15a/baa3fe8/src/slice_box.rs:69:23
   |
69 | impl<T> PyObjectAlloc<SliceBox<T>> for SliceBox<T> {
   |                       ^^^^^^^^^^^ unexpected type argument

error: aborting due to 4 previous errors

Some errors occurred: E0107, E0412, E0432.
For more information about an error, try `rustc --explain E0107`.
error: Could not compile `numpy`.

while with pyo3 = "= 0.5.0" the example works fine.

Safe wrapper for NPyIter

I'm evaluating using rust-numpy for a project that will require extensive use of NPyIter as I have to re-implement a number of ufuncs and need access to the broadcasting implementation.

I'm currently using np.broadcast in Python code to implement this.

Would there be any interest in making a safe wrapper for NPyIter if I built one?

Does anyone have any advice or similar wrappers I could reference? This is my first experience with writing unsafe rust code.

avoid import numpy::* in examples

This makes it very hard to read and understand the code. I can understand (but don't like) the use pyo3::prelude::*;, but having two wildcard imports makes it impossible to guess what any given symbol might mean.

Operand order of numpy array and user defined `PyClass`

When implementing arithmetic operations between an user defined class and numpy arrays there can be problems with regards to the return type of the operation depending on the order of the operands as described in this question.

There are several options to circumvent this problem:

  1. Subclassing numpy array
  2. __array_wrap__ and __array_priority__
  3. Setting the class variable __array_ufunc__ = None (see here)

I am running into the same problem as above when using pyO3 and rust-numpy. I built a PyClass (say MyClass) for which I implemented arithmetic operations where the rhs argument is a numpy array. When I use my class as left hand side argument in arithmetic operations where a numpy array is the right hand side argument, everything works fine - the result type is MyClass.
Since MyClass also implements arithmetic operations with the elements of the numpy array (float), when using MyClass as rhs argument returns a numpy array of objects since it uses elementwise operations.

My question is: Is there currently a way to change operand order in a way similar (preferably without subclassing) to the "pure" python way?

Thanks alot!
In pure python it is possible to change order

can't compile with rust-ndarray 0.10

error[E0599]: no method named into_pyarray found for type ndarray::ArrayBase<ndarray::OwnedRepr<f64>, ndarray::Dim<[usize; 2]>> in the current scope

59 | let temp = temp.into_pyarray(py, &np);

disable write access for python

Hi! maybe I'm missing something, but I found no function in this API to lock the write access to the numpy arrays passed to python. I see 2 useful cases:

  1. we want to expose some rust-owned data
    I know that in this case, the rust ownership principles may say that it should not happend as we are not sure that the python view will be destroyed when the rust data will be. But what about exposing data hold with a Arc<Mutex<>> (so a data shared bt python and rust, hold by the two of them) ?

  2. we want to move some data ownership to python, but we don't want python to modify it
    moving data to python is possible with PyArray::from_owned_array only, but even so we can't specify that we don't want python to write in it. For instance, let's imagine a rust function F1 producing an array that is intended to be used by an other rust function F2. But we want to read the intermediate data in python. Currently as python could have corrupted the data, F2 must check the integrity before processing. With write lock added, it wouldn't be necessary

I'm in the second case for a personal project, and the first case would be very usefull in the future for any kind of project.

So what do you think about adding a method PyArray::lock_write() ?

Slicing and `to_pyarray()` leads to incorrect results

pub fn slice() -> Py<PyArray2<f32>> {
        let matrix = Array2::from_shape_vec([4,2], vec![1.,2.,3.,4.,5.,6.,7.,8.]).unwrap();
        let matrix_view = matrix.view();
        matrix_view.slice(s![1..4; -1, ..]).to_pyarray(gil.python()).to_owned()
}
slice()
array([[7.0e+00, 8.0e+00],
       [4.6e-44, 0.0e+00],
       [0.0e+00, 0.0e+00]], dtype=float32)

matrix_view.to_owned() fixes this.

I assume that the conversion doesn't check whether the underlying array is contiguous and just copies from the pointer?

Extraction from PyAny fails with numpy import failure

I put together a fairly minimal example reproducing the problem over at jgosmann/py03-numpy-bug. Essentially I am trying to extract a PyArrayDyn from a &PyAny:

let _array: &PyArrayDyn<f64> = array.extract()?;

This fails with an import failure of NumPy:

thread 'tests::test' panicked at 'Failed to import numpy module', /Volumes/Home/blubb/.cargo/git/checkouts/rust-numpy-ea13d29ac0b89ab6/323a3e7/src/npyffi/mod.rs:16:9

I can, however, use Python::import(py, "numpy") to import NumPy and use when executing Python code via the Rust interface.

Backtrace

thread 'tests::test' panicked at 'Failed to import numpy module', /Volumes/Home/blubb/.cargo/git/checkouts/rust-numpy-ea13d29ac0b89ab6/323a3e7/src/npyffi/mod.rs:16:9
stack backtrace:
   0:        0x107bb3d7f - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::h83d53b696ac99295
   1:        0x107bd93be - core::fmt::write::hf81c429634e1f3ed
   2:        0x107b34879 - std::io::Write::write_fmt::h53fe50e3fff0275d
   3:        0x107bae13c - std::io::impls::<impl std::io::Write for alloc::boxed::Box<W>>::write_fmt::h352c9db3a02449c0
   4:        0x107bb5e7a - std::panicking::default_hook::{{closure}}::ha991e4eca34b4afa
   5:        0x107bb5b58 - std::panicking::default_hook::h722aa3f5c1c31788
   6:        0x107bb6448 - std::panicking::rust_panic_with_hook::h2cd47f71d6d55501
   7:        0x107be06a6 - std::panicking::begin_panic::h7efaef4e36ffc64a
   8:        0x107b72852 - numpy::npyffi::get_numpy_api::h98be504f57c3be61
   9:        0x107b6fff9 - numpy::npyffi::array::PyArrayAPI::get::h0c7ca843541ebbb5
  10:        0x107b7012c - numpy::npyffi::array::PyArrayAPI::get_type_object::hb9b385b5ce0a23b3
  11:        0x107b70089 - numpy::npyffi::array::PyArray_Check::h5ac623a5504efbb0
  12:        0x107b17ff2 - <&numpy::array::PyArray<T,D> as pyo3::conversion::FromPyObject>::extract::h97c652a49899d9c2
  13:        0x107b22ae8 - pyo3::types::any::PyAny::extract::hf4c553423b3ec21f
  14:        0x107b1d96f - pyo3_numpy_bug::MyPyClass::new::h75d9032154f441cf
  15:        0x107b2f358 - pyo3_numpy_bug::__init18004438298088000734::__init18004438298088000734::__wrap::{{closure}}::hf02ecd4750c385b3
  16:        0x107b32884 - std::panicking::try::do_call::h2b3ba1c7f7846e5f
  17:        0x107b3299d - __rust_try
  18:        0x107b32729 - std::panicking::try::h55857f565c753f20
  19:        0x107b137ff - std::panic::catch_unwind::h0ed4ca859a19684b
  20:        0x107b1db67 - pyo3_numpy_bug::__init18004438298088000734::__init18004438298088000734::__wrap::h6d26eebae25fd729
  21:        0x107d73ece - type_call
  22:        0x107d38c7a - _PyObject_FastCallKeywords
  23:        0x107dcdc8e - call_function
  24:        0x107dc7331 - _PyEval_EvalFrameDefault
  25:        0x107dce489 - _PyEval_EvalCodeWithName
  26:        0x107dc5b80 - PyEval_EvalCode
  27:        0x107df364c - run_mod
  28:        0x107df27a9 - PyRun_StringFlags
  29:        0x107b8e3f3 - pyo3::python::Python::run_code::hf397a0fe7392fa69
  30:        0x107b8e063 - pyo3::python::Python::run::h0e1429160cf4929e
  31:        0x107b23261 - pyo3_numpy_bug::tests::test::h966af346dfceadc0
  32:        0x107b31dd1 - pyo3_numpy_bug::tests::test::{{closure}}::h646a3e36b6bc3cb3
  33:        0x107b18b41 - core::ops::function::FnOnce::call_once::hc0532253f2c027d4
  34:        0x107b59b53 - test::run_test::run_test_inner::{{closure}}::hf35455f67ec1e4ed
  35:        0x107b33d7b - std::sys_common::backtrace::__rust_begin_short_backtrace::hffd4a983e423c33e
  36:        0x107b39255 - core::ops::function::FnOnce::call_once{{vtable.shim}}::hd45267100ae6c7ce
  37:        0x107bbc51d - std::sys::unix::thread::Thread::new::thread_start::h2b28b74d30bce841
  38:     0x7fff6caee109 - _pthread_start

Numpy import error using get_array_module(py)

Using the example from the Readme:

np.abs() or np.absolute modules cannot be found:

error! :PyErr { type: Py(0x7f6a4e15e1a0, PhantomData) }
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: module 'numpy.core.multiarray' has no attribute 'absolute'
Error: ()
extern crate numpy;
extern crate pyo3;
use numpy::{PyArray1, get_array_module};
use pyo3::prelude::{ObjectProtocol, PyResult, Python};
use pyo3::types::PyDict;

fn main() -> Result<(), ()> {
    let gil = Python::acquire_gil();
    main_(gil.python()).map_err(|e| {
        eprintln!("error! :{:?}", e);
        // we can't display python error type via ::std::fmt::Display
        // so print error here manually
        e.print_and_set_sys_last_vars(gil.python());
    })
}

fn main_<'py>(py: Python<'py>) -> PyResult<()> {
    let np = get_array_module(py)?;
    let dict = PyDict::new(py);
    dict.set_item("np", np)?;
    let pyarray: &PyArray1<i32> = py
        .eval("np.absolute(np.array([1, 2, 3], dtype='int32'))", Some(&dict), None)?
        .extract()?;
    let slice = pyarray.as_slice();
    assert_eq!(slice, &[1, 2, 3]);
    Ok(())
}

However,

let np = py.import("numpy")?;

works.

class SliceBox initialization panic with PyO3 0.7

After I upgraded my project in rth/vtext#56 to use rust-numpy 0.6.0 (and PyO3 0.7), I started to see panics in python tests with the following message,

thread '<unnamed>' panicked at 'An error occurred while initializing class SliceBox'

that is raised in pyo3 here

full backtrace below,

tests/test_vectorize.py::test_count_vectorizer thread '<unnamed>' panicked at 'An error occurred while initializing class SliceBox', C:\Users\Administrator\.cargo\registry\src\github.com-1ecc6299db9ec823\pyo3-0.7.0\src\type_object.rs:260:17
stack backtrace:
   0: std::sys::windows::backtrace::set_frames
             at /rustc/7e001e5c6c7c090b41416a57d4be412ed3ccd937\/src\libstd\sys\windows\backtrace\mod.rs:94
   1: std::sys::windows::backtrace::unwind_backtrace
             at /rustc/7e001e5c6c7c090b41416a57d4be412ed3ccd937\/src\libstd\sys\windows\backtrace\mod.rs:81
   2: std::sys_common::backtrace::_print
             at /rustc/7e001e5c6c7c090b41416a57d4be412ed3ccd937\/src\libstd\sys_common\backtrace.rs:70
   3: std::sys_common::backtrace::print
             at /rustc/7e001e5c6c7c090b41416a57d4be412ed3ccd937\/src\libstd\sys_common\backtrace.rs:58
   4: std::panicking::default_hook::{{closure}}
             at /rustc/7e001e5c6c7c090b41416a57d4be412ed3ccd937\/src\libstd\panicking.rs:200
   5: std::panicking::default_hook
             at /rustc/7e001e5c6c7c090b41416a57d4be412ed3ccd937\/src\libstd\panicking.rs:215
   6: std::panicking::rust_panic_with_hook
             at /rustc/7e001e5c6c7c090b41416a57d4be412ed3ccd937\/src\libstd\panicking.rs:478
   7: std::panicking::continue_panic_fmt
             at /rustc/7e001e5c6c7c090b41416a57d4be412ed3ccd937\/src\libstd\panicking.rs:385
   8: std::panicking::begin_panic_fmt
             at /rustc/7e001e5c6c7c090b41416a57d4be412ed3ccd937\/src\libstd\panicking.rs:340
   9: <T as pyo3::type_object::PyTypeObject>::init_type::{{closure}}
  10: <numpy::slice_box::SliceBox<T>>::new
  11: <numpy::array::PyArray<T, D>>::from_boxed_slice
  12: <ndarray::ArrayBase<ndarray::OwnedRepr<A>, D> as numpy::convert::IntoPyArray>::into_pyarray
  13: <T as pyo3::type_object::PyTypeObject>::init_type::{{closure}}
  14: <T as pyo3::type_object::PyTypeObject>::init_type::{{closure}}
  15: PyMethodDef_RawFastCallKeywords
  16: PyMethodDef_RawFastCallKeywords
  17: PyEval_EvalFrameDefault
  18: PyMethodDef_RawFastCallKeywords
  19: PyEval_EvalFrameDefault
  20: PyFunction_FastCallDict
  21: PySlice_New
  22: PyEval_EvalFrameDefault
  23: PyEval_EvalCodeWithName
  24: PyFunction_FastCallDict
  25: PySlice_New
  26: PyEval_EvalFrameDefault
  27: PyEval_EvalCodeWithName
  28: PyMethodDef_RawFastCallKeywords
  29: PyEval_EvalFrameDefault
  30: PyEval_EvalCodeWithName
  31: PyMethodDef_RawFastCallKeywords
  32: PyEval_EvalFrameDefault
  33: PyMethodDef_RawFastCallKeywords
  34: PyEval_EvalFrameDefault
  35: PyEval_EvalCodeWithName
  36: PyFunction_FastCallDict
  37: PyObject_Call_Prepend
  38: PyType_FromSpecWithBases
  39: PyObject_FastCallKeywords
  40: PyMethodDef_RawFastCallKeywords
  41: PyEval_EvalFrameDefault
  42: PyMethodDef_RawFastCallKeywords
  43: PyEval_EvalFrameDefault
  44: PyFunction_FastCallDict
  45: PySlice_New
  46: PyEval_EvalFrameDefault
  47: PyEval_EvalCodeWithName
  48: PyMethodDef_RawFastCallKeywords
  49: PyEval_EvalFrameDefault
  50: PyEval_EvalCodeWithName
  51: PyMethodDef_RawFastCallKeywords
  52: PyEval_EvalFrameDefault
  53: PyMethodDef_RawFastCallKeywords
  54: PyEval_EvalFrameDefault
  55: PyEval_EvalCodeWithName
  56: PyFunction_FastCallDict
  57: PyObject_Call_Prepend
  58: PyType_FromSpecWithBases
  59: PySlice_New
  60: PyEval_EvalFrameDefault
  61: PyEval_EvalCodeWithName
  62: PyMethodDef_RawFastCallKeywords
  63: PyEval_EvalFrameDefault
  64: PyEval_EvalCodeWithName
  65: PyMethodDef_RawFastCallKeywords
  66: PyEval_EvalFrameDefault
  67: PyEval_EvalCodeWithName
  68: PyFunction_FastCallDict
  69: PySlice_New
  70: PyEval_EvalFrameDefault
  71: PyEval_EvalCodeWithName
  72: PyMethodDef_RawFastCallKeywords
  73: PyEval_EvalFrameDefault
  74: PyEval_EvalCodeWithName
  75: PyMethodDef_RawFastCallKeywords
  76: PyEval_EvalFrameDefault
  77: PyFunction_FastCallDict
  78: PySlice_New
  79: PyEval_EvalFrameDefault
  80: PyEval_EvalCodeWithName
  81: PyMethodDef_RawFastCallKeywords
  82: PyEval_EvalFrameDefault
  83: PyEval_EvalCodeWithName
  84: PyMethodDef_RawFastCallKeywords
  85: PyEval_EvalFrameDefault
  86: PyMethodDef_RawFastCallKeywords
  87: PyEval_EvalFrameDefault
  88: PyEval_EvalCodeWithName
  89: PyFunction_FastCallDict
  90: PyObject_Call_Prepend
  91: PyType_FromSpecWithBases
  92: PyObject_FastCallKeywords
  93: PyMethodDef_RawFastCallKeywords
  94: PyEval_EvalFrameDefault
  95: PyFunction_FastCallDict
  96: PySlice_New
  97: PyEval_EvalFrameDefault
  98: PyEval_EvalCodeWithName
  99: PyMethodDef_RawFastCallKeywords
Windows fatal exception: code 0xc000001d

I think it must be happening in this function that mostly does something along the lines of,

ndarray::arr1(..).mapv(|elem| elem as i32).into_pyarray(py).to_owned()

(sorry for not providing a more self-contained example).

Interestingly this happens somewhat erratically. A week or so ago, this error was consistently happening on Windows builds while Linux and MacOS builds were fine. Now Windows builds pass, and Linux build fails.

I managed to reproduce this on a separate windows machine, but only when building wheels (i.e. this error did not happen when running python setup.py develop).

I'm not sure if this is an issue with rust-numpy, Pyo3 or if I'm doing something wrong.

Feature request: example converting to ndarray

We're experiencing a lot of friction just figuring out how to get our data all of the way out of a pyarray.

rust-ndarray/ndarray#493

I suspect the ergonomics could be improved (or better documented) in the ndarray crate, but since this might be a more common pain point for new users of rust-numpy (compared to people just learning ndarray in isolation), I thought I'd mention it here as well.

New users of the ndarray crate are probably going to start out with owned arrays of static number of dimensions, whereas a numpy users starting from the rust-numpy example get handed an array of dynamic dimensions and non-owned data with lifetime bound to the python interpreter. I imagine that would be nice for performance if we ever manage to get our code to compile, but for a numpy user trying to get up to speed in rust, it really throws you into the deep end of the pool.

Thanks!

PyArray::to_owned_array() panics for reversed array

use numpy;
use pyo3::types::IntoPyDict;

fn main() {
    let gil = pyo3::Python::acquire_gil();
    let py = gil.python();
    let locals = [("np", numpy::get_array_module(py).unwrap())].into_py_dict(py);
    let not_contiguous: &numpy::PyArray1<f32> = py
        .eval("np.zeros(2)[::-1]", Some(locals), None)
        .unwrap()
        .downcast()
        .unwrap();
    let _owned = not_contiguous.to_owned_array();
}
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ShapeError/Overflow: arithmetic overflow', /Users/hombit/.cargo/registry/src/github.com-1ecc6299db9ec823/ndarray-0.13.1/src/impl_raw_views.rs:75:13
stack backtrace:
   0: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
   1: core::fmt::write
   2: std::io::Write::write_fmt
   3: std::panicking::default_hook::{{closure}}
   4: std::panicking::default_hook
   5: std::panicking::rust_panic_with_hook
   6: rust_begin_unwind
   7: core::panicking::panic_fmt
   8: core::option::expect_none_failed
   9: core::result::Result<T,E>::unwrap
  10: ndarray::impl_raw_views::<impl ndarray::ArrayBase<ndarray::RawViewRepr<*const A>,D>>::from_shape_ptr
  11: ndarray::impl_views::constructors::<impl ndarray::ArrayBase<ndarray::ViewRepr<&A>,D>>::from_shape_ptr
  12: numpy::array::PyArray<T,D>::as_array
  13: numpy::array::PyArray<T,D>::to_owned_array
  14: pyo3_panics::main
  15: std::rt::lang_start::{{closure}}
  16: std::rt::lang_start_internal
  17: std::rt::lang_start
  18: main

Document copy behaviour

This is great! I think making it easier to write Rust extensions for Python scientific code might interest a lot of people.

After a superficial glance at the readme and issues, I wasn't able to figure out whether passing arrays between Rust and Python (or vice versa) currently involves copying. The only comment I saw was #5 (comment),

  • if this does zero copies this should be clearly documented
  • if copies are made it might be worth opening an issue about what would be needed in the long-term to get rid of those.

Maintenance

This project has been started as an experiment to show Rust can write NumPy extension. However, I have little motivation to manage this repository since I do not use NumPy and Python itself recently. If you are interesting to manage and develop this project, please let me know.

Any suggestions are welcome :)

'as_slice()' successfully reports incorrect data on noncontiguous arrays

https://github.com/rust-numpy/rust-numpy/blob/b54bb166c1c075bd730f20f60bb0f45c6cffd1f3/src/array.rs#L407-L410

This function is defined to always return a slice, but this is impossible, because arrays can be discontiguous.

>>> x = np.array([1, 2, 3, 4], dtype='int32')
>>> x.strides
(4,)
>>> x = x[::2]
>>> x.strides
(8,)

Rust example

extern crate numpy;
extern crate pyo3;
use numpy::{PyArray1};
use pyo3::prelude::{ObjectProtocol, PyResult, Python};
use pyo3::types::PyDict;

fn main() -> Result<(), ()> {
    let gil = Python::acquire_gil();
    _main(gil.python()).map_err(|e| {
        eprintln!("error! :{:?}", e);
        // we can't display python error type via ::std::fmt::Display
        // so print error here manually
        e.print_and_set_sys_last_vars(gil.python());
    })
}

fn _main(py: Python<'_>) -> PyResult<()> {
    let np = py.import("numpy")?;
    let dict = PyDict::new(py);
    dict.set_item("np", np)?;

    let pyarray: &PyArray1<i32> = py
        .eval("np.array([1, 2, 3, 4], dtype='int32')[::2]", Some(&dict), None)?
        .extract()?;

    println!("{:?}", pyarray.as_slice());

    Ok(())
}
[1, 2]

This method should either fail, return Option::None, panic, or not exist.

Update to ndarray-0.12

bluss released ndarray 0.12 just a few days ago, and when I upgraded the dependencies on my project, the Python library wouldn't compile anymore because the versions of ndarray were different. I'll make a fork to see if everything works ๐Ÿ‘Œ

Can't use with PyO3

Upon updating rust-numpy to 0.8.0 and pyo3 to 0.9.1 I get the following errors:

error[E0277]: the trait bound `&mut numpy::array::PyArray<u8, ndarray::dimension::dim::Dim<[usize; 2]>>: pyo3::PyClass` is not satisfied
  --> camd/camd_python/src/lib.rs:47:1
   |
47 | #[pymethods]
   | ^^^^^^^^^^^^ the trait `pyo3::PyClass` is not implemented for `&mut numpy::array::PyArray<u8, ndarray::dimension::dim::Dim<[usize; 2]>>`
   |
   = note: required because of the requirements on the impl of `pyo3::FromPyObject<'_>` for `&mut numpy::array::PyArray<u8, ndarray::dimension::dim::Dim<[usize; 2]>>`
   = note: required because of the requirements on the impl of `pyo3::derive_utils::ExtractExt<'_>` for `&mut numpy::array::PyArray<u8, ndarray::dimension::dim::Dim<[usize; 2]>>`
   = note: this error originates in an attribute macro (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0277]: the trait bound `&mut numpy::array::PyArray<u8, ndarray::dimension::dim::Dim<[usize; 2]>>: std::clone::Clone` is not satisfied
  --> camd/camd_python/src/lib.rs:47:1
   |
47 | #[pymethods]
   | ^^^^^^^^^^^^ the trait `std::clone::Clone` is not implemented for `&mut numpy::array::PyArray<u8, ndarray::dimension::dim::Dim<[usize; 2]>>`
   |
   = note: `std::clone::Clone` is implemented for `&numpy::array::PyArray<u8, ndarray::dimension::dim::Dim<[usize; 2]>>`, but not for `&mut numpy::array::PyArray<u8, ndarray::dimension::dim::Dim<[usize; 2]>>`
   = note: required because of the requirements on the impl of `pyo3::FromPyObject<'_>` for `&mut numpy::array::PyArray<u8, ndarray::dimension::dim::Dim<[usize; 2]>>`
   = note: required because of the requirements on the impl of `pyo3::derive_utils::ExtractExt<'_>` for `&mut numpy::array::PyArray<u8, ndarray::dimension::dim::Dim<[usize; 2]>>`
   = note: this error originates in an attribute macro (in Nightly builds, run with -Z macro-backtrace for more info)

I feel like those should be clonable?

Rust2018

This project should migrate to Rust2018

Undefined behavior in safe code: &self -> &mut T methods permit aliasing

With this crate it is possible to invoke undefined behavior in safe code:

extern crate numpy;
extern crate pyo3;
use numpy::{PyArray1};
use pyo3::prelude::{ObjectProtocol, PyResult, Python};
use pyo3::types::PyDict;

fn main() -> Result<(), ()> {
    let gil = Python::acquire_gil();
    show_ub(gil.python()).map_err(|e| {
        eprintln!("error! :{:?}", e);
        // we can't display python error type via ::std::fmt::Display
        // so print error here manually
        e.print_and_set_sys_last_vars(gil.python());
    })
}

// a function that behaves differently in dev and release builds
fn show_ub(py: Python<'_>) -> PyResult<()> {
    let np = py.import("numpy")?;
    let dict = PyDict::new(py);
    dict.set_item("np", np)?;

    let pyarray: &PyArray1<i32> = py
        .eval("np.absolute(np.array([-1, -2, -3], dtype='int32'))", Some(&dict), None)?
        .extract()?;

    let slice = pyarray.as_slice();
    let mut_slice = pyarray.as_slice_mut();
    lol(&slice[0], &mut mut_slice[0]);
    assert_eq!(slice, &[1, 2, 3]);

    Ok(())
}

#[inline(never)]
fn lol(x: &i32, y: &mut i32) {
    let original_x = *x;
    *y = *x - 1;
    assert_eq!(*x, original_x);
}

Debug result:

thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `0`,
 right: `1`', src/main.rs:39:5

Release result:

thread 'main' panicked at 'assertion failed: `(left == right)`
  left: `[0, 2, 3]`,
 right: `[1, 2, 3]`', src/main.rs:30:5

Test failure in into_obj_vec_to_pyarray with ndarray 0.13

I believe, because Cargo.toml specifies,

ndarray = ">=0.12"

with the release of 0.13, one test started to fail,

---- into_obj_vec_to_pyarray stdout ----
thread 'into_obj_vec_to_pyarray' panicked at 'assert arr[1] == 'Hello python :)': ()', src/libcore/result.rs:1165:5

also occasionally for me that test passes and error happens later,

test into_pyarray_vec ... ok
Traceback (most recent call last):
test into_pyarray_cant_resize ... ok
  File "<string>", line 1, in <module>
test into_pyarray_array ... ok
AssertionError
test from_small_array ... ok
Fatal Python error: GC object already tracked

Thread 0x00007fd2116ee700 (most recent call first):
  File "<frozen importlib._bootstrap>", line 98 in acquire
  File "<frozen importlib._bootstrap>", line 149 in __enter__
  File "<frozen importlib._bootstrap>", line 980 in _find_and_load

Current thread 0x00007fd210edd700 (most recent call first):

Thread 0x00007fd21028b700 (most recent call first):
  File "<frozen importlib._bootstrap>", line 98 in acquire
  File "<frozen importlib._bootstrap>", line 149 in __enter__
  File "<frozen importlib._bootstrap>", line 980 in _find_and_load

Thread 0x00007fd2134d1700 (most recent call first):
error: process didn't exit successfully: `/home/rth/src/rust-numpy/target/debug/deps/array-32ad574c39fda83e` (signal: 6, SIGABRT: process abort signal)

I quickly looked at the release notes for ndarray 0.13 but wasn't able to find the source of the problem. Tests run fine for me with ndarray 0.12.

PyArray::is_fotran_contiguous() is misspelled

This seems nitpicky but since it's a public api, changing it may be more of an issue.

pub fn is_fotran_contiguous(&self) -> bool {}
pub fn is_fortran_contiguous(&self) -> bool {}

Implement IntoPyErr for ndarray::ShapeError

I came across a use case for such a thing.

Say you've got a function

fn my_func(py: Python, arr: &PyArray<f64>) -> PyResult<f64>

and you need to convert the argument arr to a ndarray array. You get an ArrayViewD<f64> with dynamic dimension, but you've also got a function libfunc in some library that only takes ArrayView2, so you need to do

let arg: Result<ArrayView2<f64>, ShapeError> = arr.into_dimensionality<Ix2>();

and get a Result type.

You can either unwrap and let your code panic if the user passes a one-dimensional array (which is bad in Jupyter Notebooks because it forces a kernel reboot), or handle it in the import logic of your __init__.py, or do a big match arm.

The alternative is implementing the IntoPyErr trait for ShapeError, and you can just do

let arg: ArrayView2<f64> = arr.into_dimensionality<Ix2>()?;

I'll make a fork to implement that and do a pull request if it all works

Memoryleak

The following code grows in mem constantly:

extern crate pyo3;
extern crate numpy;

use pyo3::prelude::*;
use numpy::{PyArray2};

fn main() {
    let v = vec![0; 100_000];
    let gil = Python::acquire_gil();
    let py = gil.python();

    loop {
        let py_arr = PyArray2::from_vec2(py, &vec![v.clone()]).unwrap();
        ();
    }
}

PyArray::from_vec and PyArray::from_slice do it also.

pyo3 = "0.5.2"
numpy = "0.4.0"

os: windows-10+ I see the same behavior in WSL

--added--
I was trying to add

   let gc = py.import("gc").unwrap();
...
   loop {
...
        gc.call0("collect").unwrap();
   }

but it did not help.

pylist of pyarray2?

Not sure this is an issue! but didnt know where to ask the question.

I am very new to rust and I am trying to write a pyfunction which will take an input as a python list (PyList from pyo3) and each item is a PyArray2 (numpy array). I want ot send this item to another function as a reference. but PyList iter() returns PyAny. How do I convert that to PyArray?

if I directly call preprocess from python and provide a numpy array, it works.
but I cannot send a list of numpy arrays to pylist_implementation

I get error

mismatched types

expected struct `numpy::PyArray`, found reference

note: expected type `&mut numpy::PyArray<f64, ndarray::Dim<[usize; 2]>>`
         found type `&mut &pyo3::types::PyAny`rustc(E0308)
lib.rs(89, 25): expected struct `numpy::PyArray`, found reference

Which I understand, that there is not way for the compiler to know what kind of data is in the list. but how to "cast it"?

Any help is appreciated, if this is a wrong place to ask the question apologies in advance.

#[pyfunction]
fn pylist_implementation(data: & mut PyList) -> PyResult<(&PyList)> {
    // -> &PyArray2<f64>
    for d in data.iter() {
        preprocess_data(&mut d, 100);
    }
    Ok(data)
}

#[pyfunction]
fn preprocess_data(
    data: &mut PyArray2<f64>,
    index: i32,
) -> PyResult<(Vec<f64>, Vec<f64>, Vec<usize>)> {
    let d = data.shape();
    remove_baseline(data, index);

    let mut peaks: Vec<f64> = Vec::new();
    let mut peaks_index: Vec<usize> = Vec::new();
    //println!("data in preprocess: {:?}", d);
    for subdata in data.as_slice_mut().chunks_exact_mut(d[0]) {
        let max = subdata.iter().max_by(|a, b| a.partial_cmp(b).unwrap());
        peaks.push(*max.unwrap());
        let index = subdata.iter().position(|&r| r == *max.unwrap()).unwrap();
        //println!("INDEX {}", index);
        peaks_index.push(index);
    }

    let not_outlier: Vec<bool> = mad_based_outlier(peaks.to_vec());
    let mut i = 0;
    peaks.retain(|_| (not_outlier[i], i += 1).0);
    let mut j = 0;
    peaks_index.retain(|_| (not_outlier[j], j += 1).0);
    let mut mean_data: Vec<f64> = Vec::new();
    for subdata in data.as_array_mut().genrows_mut() {
        let mut j = 0;
        subdata.to_vec().retain(|_| (not_outlier[j], j += 1).0);
        let sum_col: f64 = subdata.iter().sum();
        let no_signals = not_outlier.iter().filter(|&n| *n == true).count() as f64;
        mean_data.push(sum_col / no_signals);
    }
    //let interpolated_signal: Vec<f64> = mean_data.interpolate_hermit;
    Ok((mean_data, peaks, peaks_index))
}

Test on Python image

Installation of Rust on Python image (e.g. python:3.6) is enough fast (~1m). Test should be done on such general condition.

Memory leak

I found a memory leak-like behavior while testing #18

import numpy as np
import rust_ext

N = 100000
a = np.random.random(N)
b = np.random.random(N)

for i in range(10000):
    rust_ext.axpy(3.0, a, b)

This code crashes with consuming large amount of memory.
It seems to be memory leak (ยดใƒปฯ‰ใƒป๏ฝ€)

Overhead of import

// wrapper of `axpy`
fn axpy_py(py: Python, a: f64, x: PyArray, y: PyArray) -> PyResult<PyArray> {
    let np = PyArrayModule::import(py)?;
    let x = x.as_array().into_pyresult(py, "x must be f64 array")?;
    let y = y.as_array().into_pyresult(py, "y must be f64 array")?;
    Ok(axpy(a, x, y).into_pyarray(py, &np))
}

This example imports np for every axpy_py call. It may be a non-negligible overhead.

TODO

  • Profile the cost of PyArrayModule::import
  • If it is expensive, consider how to remove it.

Is it possible for rust-numpy to work with python2.7?

I noticed that this crate requires python3-sys, so by default it is only for python 3, is that right?

Is it possible to simply change python3-sys to python-sys in the Cargo.toml to enable it work with python 2.7?

Thanks.

ImportError with bindings (no export function)

I created bindings using the examples and docs as reference for a function and got this when I try to import my new python library: ImportError: dynamic module does not define module export function (PyInit_emo_audio)

Code can be seen here, the bindings are in src/bindings.rs and everything looks the same to me so not sure what to do.

Below is also a minified version of what's in bindings.rs to remove all the logic and just return an empty array

#[pymodule]
fn emo_audio(_py: Python<'_>, m: &PyModule) -> PyResult<()> {
    type StftResult = Py<PyArray2<Complex32>>;

    #[pyfn(m, "stft")]
    fn stft(
        py: Python<'_>,
        x: &PyArray1<f32>,
        n_fft: Option<usize>,
        win_length: Option<usize>,
        hop_length: Option<usize>,
    ) -> PyResult<StftResult> {
        Ok(arr2(&[[]]).to_pyarray(py).to_owned())
    }

    Ok(())
}

pyclass errors when I use numpy=0.9.0?

If I go to examples/simple-extension and compile with python setup.py build, for example, everything is fine. But if I change Cargo.toml from

[dependencies]
numpy = { path = "../.." }
ndarray = ">= 0.12"

to

[dependencies]
numpy = "0.9.0"
ndarray = ">= 0.12"

then it all goes pear shaped, with this mysterious error:

6 | #[pymodule]
  | ^^^^^^^^^^^ the trait `pyo3::PyClass` is not implemented for `&numpy::PyArray<f64, ndarray::Dim<ndarray::IxDynImpl>>`

Does this mean I have to download and compile the source code for rust-numpy by myself everywhere I want to use it?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.