elixir-explorer / fss Goto Github PK

Livebook v0.12 may now set this field. We need to:

1. Add this field to S3 here
2. Pass down this field in Explorer
3. Receive this field from Livebook in Kino

I'm interested in integrating Elixir with a Data Lakehouse architected data layer. Currently, I'm looking at Apache Iceberg. The specification is open, which makes it attractive. In the process of looking into how I might implement it on Elixir, I determined that I would be served well by having a library like fsspec, as that's what the PyIceberg project uses to abstract the required filesystem operations from the spec.

So my first question is, would this project be amenable to PRs to build out those features?

I'm also curious if anyone has insight into any development in general Elixir community. For example, this seems like it would work well with Broadway. I know that Broadway has a Kafka connector, but I'm unable to find anything related to to an iceberg connector. I'm fairly new to the "data engineering" side of things, so this may be an issue of naïveté. If there are other ways of dealing with this in Elixir, I'd be appreciative of those being pointed out.

I'd be happy to work on creating a road map of the necessary features, pending that work isn't ongoing via some other project.