josiahparry / arrow-extendr Goto Github PK

Arrow uses a cargo deny approach to prevent compatability across arrow-rs versions. So if a library is using Arrow 47 and another 48, those are not compatible. The suggested approach is to create a release for each version of arrow we want to support. This means that I need to, at minimum, use 48.0.1 (DataFusion) and 49.0.0 current support.

• 48.0.1
• 49.0.0
• 49-dev for geoarrow

Thanks for taking on the task of creating this!
Perhaps this is because this is a very early experiment, but am I correct in understanding that ultimately this is aimed to be a Rust crate, not an R package? (as in arrow-rs' pyarrow feature)

What I had imagined was a crate something like "extendr-nanoarrow".
If such a crate does not exist, type conversion from arrow-rs to R would have to be created on individual R packages, correct?

I would probably encourage writers of Rust extensions to go through nanoarrow (e.g., via as_nanoarrow_array_stream() or as_nanoarrow_array()) rather than arrow directly.

@paleolimbot

Since Rust on CRAN is always outdated, it is very useful to have MSRV confirmed on CI.
Like this: apache/arrow-rs#4490

Probably should be created as an optional feature.

I want to try having two traits IntoArrowRobj and IntoNanoArrowRobj. I think there is at least still some utility in being able to return an {arrow} R package object still.

I think it may be slower to return a nano-arrow stream and then cast as a data.frame than it might be to return a recordbatch reader directly.