Comments (9)
You're ahead of me! Lol yeah, correct. Rust crate only. The R package is strictly for me the test what I'm doing as I go along. It's mostlyyyy working though! Except conversions from RecordBatch Robj -> arrow-rs record batch. It relies on the arrow R package but @paleolimbot pointed out that it's probably best to rely on the nanoarrow R package which I'll probably try swapping over to on Friday (tomorrow is a holiday).
See #1 for related dependency convo!
from arrow-extendr.
I suppose we can put the R package for integration tests at the tests folder.
from arrow-extendr.
Yes, DataType
, Field
, and Schema
all can be losslessly represented as a nanoarrow_schema
/ArrowSchema
.
from arrow-extendr.
I've moved the package and restructured. Need to work on creating integration tests now.
from arrow-extendr.
Oh sorry, I think most of the above is written in the README...
from arrow-extendr.
I think it would be fine if only objects of the nanoarro_array_stream class existed on the interface. (The development version DBI package has recently been switched from arrow to nanoarrow and can be exchanged using nanoarrow_array_stream)
from arrow-extendr.
Good to know! I didn't know that DBI was using nanoarrow! I'll give it a look. Dewey provided some pseudo code for me to reference:
array <- nanoarrow::nanoarow_allocate_array()
schema <- nanoarrow::nanoarrow_allocate_schema()
nanoarrow::nanoarrow_pointer_move(<address of the FFI struct as a string>, array)
nanoarrow::nanoarrow_pointer_move(<address of the FFI struct as a string>, schema)
nanoarrow::nanoarrow_array_set_schema(array, schema)
from arrow-extendr.
Yah, I was wondering what the best way to handle that would be.
Baby steps!
arrow-rs -> nanoarrow implemented
6dab964
from arrow-extendr.
@paleolimbot:
When you get a chance, can you confirm that DataType
and Field
are both represented in nanoarrow
as a schema?
library(nanoarrow)
library(arrowextendr)
test_f64() |>
as_nanoarrow_array()
#> <nanoarrow_array double[3]>
#> $ length : int 3
#> $ null_count: int 1
#> $ offset : int 0
#> $ buffers :List of 2
#> ..$ :<nanoarrow_buffer validity<bool>[8][1 b]> `TRUE FALSE TRUE FALSE FALS...`
#> ..$ :<nanoarrow_buffer data<double>[3][24 b]> `1 0 3`
#> $ dictionary: NULL
#> $ children : list()
test_schema() |>
as_nanoarrow_schema()
#> <nanoarrow_schema struct>
#> $ format : chr "+s"
#> $ name : chr ""
#> $ metadata : list()
#> $ flags : int 0
#> $ children :List of 3
#> ..$ a:<nanoarrow_schema date64>
#> .. ..$ format : chr "tdm"
#> .. ..$ name : chr "a"
#> .. ..$ metadata : list()
#> .. ..$ flags : int 0
#> .. ..$ children : list()
#> .. ..$ dictionary: NULL
#> ..$ a:<nanoarrow_schema int64>
#> .. ..$ format : chr "l"
#> .. ..$ name : chr "a"
#> .. ..$ metadata : list()
#> .. ..$ flags : int 2
#> .. ..$ children : list()
#> .. ..$ dictionary: NULL
#> ..$ b:<nanoarrow_schema bool>
#> .. ..$ format : chr "b"
#> .. ..$ name : chr "b"
#> .. ..$ metadata : list()
#> .. ..$ flags : int 0
#> .. ..$ children : list()
#> .. ..$ dictionary: NULL
#> $ dictionary: NULL
test_field() |>
as_nanoarrow_schema()
#> <nanoarrow_schema binary>
#> $ format : chr "z"
#> $ name : chr "field_name"
#> $ metadata : list()
#> $ flags : int 2
#> $ children : list()
#> $ dictionary: NULL
test_datatype() |>
as_nanoarrow_schema()
#> <nanoarrow_schema timestamp('s', '')>
#> $ format : chr "tss:"
#> $ name : chr ""
#> $ metadata : list()
#> $ flags : int 2
#> $ children : list()
#> $ dictionary: NULL
test_record_batch() |>
as_nanoarrow_array_stream()
#> <nanoarrow_array_stream struct<id: int32>>
#> $ get_schema:function ()
#> $ get_next :function (schema = x$get_schema(), validate = TRUE)
#> $ release :function ()
from arrow-extendr.
Related Issues (7)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from arrow-extendr.