Can Substrait help in any way? about puffin HOT 4 CLOSED

@ghalimi first of all this is a great initiative!!! Thank you for coordinating 😄

You may or may not be aware, but duckdb has a substrait extension: https://duckdb.org/docs/extensions/substrait

Demo repo (disclaimer: I haven't ever looked at this before): https://github.com/duckdblabs/duckdb-substrait-demo

install substrait;
load substrait;

call <some substrait>;

I'm not exactly sure what your idea is with the issue; so please feel free to call me out if I'm going in the wrong direction! I could see a future where puffin might not only receive sql but also a substrait blob / possibly other things that can turn into substrait. In such cases duckdb might do the heavy lifting to actually use this substrait.

from puffin.

@jpmmcneill Thanks for the kind words, much appreciated.

No, I did not realize that DuckDB had this extension, this is great! I think you're onto something here, but you'll have to bear with me. I'm really new at that game, and I have pretty much everything to learn when it comes to query planners. I think that I understand the main concepts, but I've never built one, so I'm on a really steep learning curve.

I am operating with some basic intuitions:

1. DuckDB's own non-distributed query planner can be used to help in the generation of distributed query plans.
2. A distributed query planner must deal with some metrics that are different from the ones used by a non-distributed one.
3. PuffinDB's distributed query planner should be outside of DuckDB, ideally in userland.

The last point is really important: we want to foster innovation, and if new query planning techniques must follow a traditional software development process to be implemented, tested, and deployed, things are going to move really slowly. Instead, I'd like anyone to be able to optimize the query planner, from the outside. I don't yet know how to go about this, but I have the intuition that it could lead to some really interesting results.

from puffin.

We should probably consider Apache Calcite as well.

from puffin.

Closing this issue and replacing it with a more precise one: #7

from puffin.