Feature: Consider supporting flink sql about coral HOT 22 OPEN

Thank you for your answer. I will try this part of the work myself first. If there is any progress, I will be happy to contribute PR.

from coral.

What I thought before was that maybe it would be easier to use a standard SQL method to convert to any other SQL dialect. But obviously your goals are much bigger

from coral.

I hope I can make a contribution to the project in the future, because this is what I wanted to do before

from coral.

@aastha25 Thanks for the tip, I'll try it

from coral.

Coral to Spark SQL is fully supported. If you notice small differences in the class organization compared to what @aastha25 shared above, expect Spark module to converge to that pattern (which @aastha25 shared) soon.

from coral.

Yes, it is in the roadmap, but as far as I know no one is actively working on it at the moment. Do you have a use case in mind?

from coral.

Since Flink supports streaming, you might check out Coral's incremental view maintenance features as well. It complements streaming on the batch side.

from coral.

FYI @aastha25 @ljfgem @KevinGe00 @yiqiangin

from coral.

Are you thinking of Flink SQL as a dialect on the input or output side? (or which one will you focus on)?

from coral.

Strictly speaking, I don’t think this is a conversion between dialects. I think it is a conversion between models. If it is just a conversion between dialects, why not use ANTLR to extract key information to spell SQL (if you do this, obviously this is a trap)

from coral.

你是否认为 Flink SQL 是输入端或输出端的一种方言？（或者你会关注哪一个）？

I have tried the conversion of spark sql before, and I am still learning, and I am more concerned about spark sql and flink sql

from coral.

Sorry should have clarified my question. When you said you would consider implementing Flink SQL integration with Coral, did you refer to integration on the input side (Flink SQL to Coral) or the output side (Coral to Flink SQL)?

from coral.

sink

from coral.

As in (Coral to Flink SQL)?

from coral.

如（珊瑚到Flink SQL）？

yes

from coral.

Thanks @caicancai. Happy to collaborate.

from coral.

Thank you for your patient reply

from coral.

@caicancai thank you for picking this line of work. Please refer to the provided sample pseudo-code below to structure your code:

Convert(RelNode coralRelNode) {
          // Step 1: Convert CoralRel IR to CoralSqlNode
          CoralRelToSqlNodeConverter rel2sql = new CoralRelToSqlNodeConverter();
          SqlNode coralSqlNode = rel2sql.convert(coralRelNode);

         // Step 2: Apply Coral to Flink SQL transformations using a custom SqlShuttle
          SqlNode flinkSqlNode = coralSqlNode.accept(new CoralToFlinkSqlNodeConverter())

         // Step 3: Convert the resulting Flink SQL to a string
          String flinkSQL = flinkSqlNode.toSqlString();
}

In this code:

1. CoralRelToSqlNodeConverter is an existing class in Coral that converts Coral IR represented as a RelNode to its SqlNode representation.
2. To implement the CoralToFlinkSqlNodeConverter, please use DataTypeDerivedSqlCallConverter and CoralToTrinoSqlCallConverter as a reference. Each transformer in the SqlShuttle for this converter should extend SqlCallTransformer to modify the SqlCalls in the input.

By following this structure, you can effectively model your code for converting Coral IR to Flink SQL.

from coral.

Re-opening since Flink is not yet supported.

from coral.

I'm going to start trying this part out

from coral.

I want to ask if sink is fully supported by spark sql.

from coral.

@wmoustafa Sorry, I was a little busy a while ago, I will start to try now.
I want to put this idea into practice

from coral.