Giter Club home page Giter Club logo

Comments (9)

JosiahParry avatar JosiahParry commented on June 14, 2024 1

You're ahead of me! Lol yeah, correct. Rust crate only. The R package is strictly for me the test what I'm doing as I go along. It's mostlyyyy working though! Except conversions from RecordBatch Robj -> arrow-rs record batch. It relies on the arrow R package but @paleolimbot pointed out that it's probably best to rely on the nanoarrow R package which I'll probably try swapping over to on Friday (tomorrow is a holiday).

See #1 for related dependency convo!

from arrow-extendr.

eitsupi avatar eitsupi commented on June 14, 2024 1

I suppose we can put the R package for integration tests at the tests folder.

from arrow-extendr.

paleolimbot avatar paleolimbot commented on June 14, 2024 1

Yes, DataType, Field, and Schema all can be losslessly represented as a nanoarrow_schema/ArrowSchema.

from arrow-extendr.

JosiahParry avatar JosiahParry commented on June 14, 2024 1

I've moved the package and restructured. Need to work on creating integration tests now.

from arrow-extendr.

eitsupi avatar eitsupi commented on June 14, 2024

Oh sorry, I think most of the above is written in the README...

from arrow-extendr.

eitsupi avatar eitsupi commented on June 14, 2024

I think it would be fine if only objects of the nanoarro_array_stream class existed on the interface. (The development version DBI package has recently been switched from arrow to nanoarrow and can be exchanged using nanoarrow_array_stream)

from arrow-extendr.

JosiahParry avatar JosiahParry commented on June 14, 2024

Good to know! I didn't know that DBI was using nanoarrow! I'll give it a look. Dewey provided some pseudo code for me to reference:

array <- nanoarrow::nanoarow_allocate_array()
schema <- nanoarrow::nanoarrow_allocate_schema()
nanoarrow::nanoarrow_pointer_move(<address of the FFI struct as a string>, array)
nanoarrow::nanoarrow_pointer_move(<address of the FFI struct as a string>, schema)
nanoarrow::nanoarrow_array_set_schema(array, schema)

from arrow-extendr.

JosiahParry avatar JosiahParry commented on June 14, 2024

Yah, I was wondering what the best way to handle that would be.

Baby steps!

arrow-rs -> nanoarrow implemented
6dab964

from arrow-extendr.

JosiahParry avatar JosiahParry commented on June 14, 2024

@paleolimbot:
When you get a chance, can you confirm that DataType and Field are both represented in nanoarrow as a schema?

library(nanoarrow)
library(arrowextendr)

test_f64() |> 
  as_nanoarrow_array()
#> <nanoarrow_array double[3]>
#>  $ length    : int 3
#>  $ null_count: int 1
#>  $ offset    : int 0
#>  $ buffers   :List of 2
#>   ..$ :<nanoarrow_buffer validity<bool>[8][1 b]> `TRUE FALSE TRUE FALSE FALS...`
#>   ..$ :<nanoarrow_buffer data<double>[3][24 b]> `1 0 3`
#>  $ dictionary: NULL
#>  $ children  : list()

test_schema() |> 
  as_nanoarrow_schema()
#> <nanoarrow_schema struct>
#>  $ format    : chr "+s"
#>  $ name      : chr ""
#>  $ metadata  : list()
#>  $ flags     : int 0
#>  $ children  :List of 3
#>   ..$ a:<nanoarrow_schema date64>
#>   .. ..$ format    : chr "tdm"
#>   .. ..$ name      : chr "a"
#>   .. ..$ metadata  : list()
#>   .. ..$ flags     : int 0
#>   .. ..$ children  : list()
#>   .. ..$ dictionary: NULL
#>   ..$ a:<nanoarrow_schema int64>
#>   .. ..$ format    : chr "l"
#>   .. ..$ name      : chr "a"
#>   .. ..$ metadata  : list()
#>   .. ..$ flags     : int 2
#>   .. ..$ children  : list()
#>   .. ..$ dictionary: NULL
#>   ..$ b:<nanoarrow_schema bool>
#>   .. ..$ format    : chr "b"
#>   .. ..$ name      : chr "b"
#>   .. ..$ metadata  : list()
#>   .. ..$ flags     : int 0
#>   .. ..$ children  : list()
#>   .. ..$ dictionary: NULL
#>  $ dictionary: NULL

test_field() |> 
  as_nanoarrow_schema()
#> <nanoarrow_schema binary>
#>  $ format    : chr "z"
#>  $ name      : chr "field_name"
#>  $ metadata  : list()
#>  $ flags     : int 2
#>  $ children  : list()
#>  $ dictionary: NULL

test_datatype() |> 
  as_nanoarrow_schema()
#> <nanoarrow_schema timestamp('s', '')>
#>  $ format    : chr "tss:"
#>  $ name      : chr ""
#>  $ metadata  : list()
#>  $ flags     : int 2
#>  $ children  : list()
#>  $ dictionary: NULL

test_record_batch() |> 
  as_nanoarrow_array_stream()
#> <nanoarrow_array_stream struct<id: int32>>
#>  $ get_schema:function ()  
#>  $ get_next  :function (schema = x$get_schema(), validate = TRUE)  
#>  $ release   :function ()

from arrow-extendr.

Related Issues (7)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.