@zhouyuan @xuechendi

A tiny issue here: Would you please check if there were reasons we use SparkConf rather than SQLConf in columnar plugin? If possible we may be able to migrate to SQLConf as it's more specific to SQL usages.

columnar SMJ should support expression as join keys

We split the original single repository OAP project to a more flexible multi-repositories. OAP project becomes the top level organization at github.

This is the issue to track the work for reorganizing the source code structure.

We should guard the empty results code path

java.io.IOException: nativeEvaluate: evaluate failed with error msg Invalid: RecordBatch must be non-empty.
	at com.intel.oap.vectorized.ExpressionEvaluatorJniWrapper.nativeEvaluate(Native Method)
	at com.intel.oap.vectorized.ExpressionEvaluator.evaluate(ExpressionEvaluator.java:137)
	at com.intel.oap.vectorized.ExpressionEvaluator.evaluate(ExpressionEvaluator.java:107)
	at com.intel.oap.execution.ColumnarShuffledHashJoinExec.$anonfun$doExecuteColumnar$1(ColumnarShuffledHashJoinExec.scala:229)
	at org.apache.spark.rdd.ZippedPartitionsRDD2.compute(ZippedPartitionsRDD.scala:89)

the link of pictures are not correctly displayed, some contents seems out of date also.

We tested current fallback logic on the generated decimal dataset, and found it is not sufficient enough. Therefore, we need to add more fallback logic for decimal type to make TPC-DS and TPC-H queries runnable.

• SHJ
• Union
• Sort
• update the ArrowWritableColumnarVector in arrow-datasource also.
• fix the error caused by row-based exchange and columnar-BHJ.
• SMJ
• TPC-H

the issue linking is broken on new repo

The issue seems due to multiple SMJ in on columnar wscg

Stack: [0x00007f2ead550000,0x00007f2ead651000],  sp=0x00007f2ead64c9e0,  free space=1010k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [spark-columnar-plugin-codegen-ad4b5a2b868bedea.so+0x9aff]  TypedWholeStageCodeGenImpl::WholeStageCodeGenResultIterator::SetDependencies(std::vector<std::shared_ptr<ResultIteratorBase>, std::allocator<std::shared_ptr<ResultIteratorBase> > > const&)+0x16bf

Columnar SMJ does not support all join features as we are using columnar SHJ to handle most queries.

• Update and put documents in docs/ dir
• create gh-pages, and use Github Pages and Mkdocs to build different versions on website

We noticecd duplicate IsNull check in codegen sort.

When package dependency libraries, there are several new features that can be supported,

1. Build from Source for Arrow(current address is under intel-bigdata repo)
2. Package dependency libraries into the jar file
3. Support 3 Options to set Static/Shared/Use Environment for arrow library setting.

number of output rows will be wrong if it's a reused exchange

To publish our jar file in official mvn repository can help to deliver binary to the users directly without compile by themselves.

• scan
• shuffle
• projector
• joins
• sort
• hasaggregate
• window

See a relevant issue from previous repository:

https://github.com/Intel-bigdata/OAP/issues/2034

• early release sort input
• early release sort output in columnar SMJ

Noticed some queries may fail when DPP enabled and manually disabled AQE, bug is related to columnarSort

on DPP enabled platform, the query is not correctly fallback to row based SMJ

Currently TPC-H CI is running based on BHJ (RAM size of GitHub Action VMs are quite limited so BHJ is automatically used). We should add some execution rounds for SMJ too.

Some ArrowVectorWriter implementations doesn't implement setNulls method

stream_rb is shadowed and not get released.

https://github.com/oap-project/native-sql-engine/blob/master/core/src/main/scala/com/intel/oap/expression/ColumnarSortMergeJoin.scala#L170-L186

java.lang.ClassCastException: org.apache.spark.sql.execution.ColumnarHashedRelation cannot be cast to org.apache.spark.sql.execution.joins.HashedRelation (state=,code=0)

native parquet reader/writer is not used as we changed to arrow data source

TPC-DS Q51 Q97

To save the time of codegen in multiple-key sort, a non-codegen version of multiple-key sort is needed.

We should have a CI case using at least 1 master + 1 slave node to run TPC-H queries to emulate cluster enviroment.

spark 3.0.1 has some minor changes over 3.0.0, we should test with 3.0.1

as two repo shares some piece of codes, it's much easier to merge these two repo together

• using non-codegen operator for columnar SMJ in some cases
• using data ref in comparing tuple on multiple key based cases

driver failed to do sort codegen as there's some issue on sortorder. Quick fix as below can bring it back

diff --git a/core/src/main/scala/com/intel/oap/execution/ColumnarSortExec.scala b/core/src/main/scala/com/intel/oap/execution/ColumnarSortExec.scala
index 5302bf3b..9d7b16c5 100644
--- a/core/src/main/scala/com/intel/oap/execution/ColumnarSortExec.scala
+++ b/core/src/main/scala/com/intel/oap/execution/ColumnarSortExec.scala
@@ -135,10 +135,7 @@ case class ColumnarSortExec(

   /***********************************************************/
   def getCodeGenSignature =
-    if (!sortOrder
-          .filter(
-            expr => bindReference(expr.child, child.output, true).isInstanceOf[BoundReference])
-          .isEmpty) {
+    if (true) {
       ColumnarSorter.prebuild(
         sortOrder,
         child.output,

travis is broken on new repo

TPC-DS q24a failed by the following exception:
Error: Error running query: java.io.IOException: nativeEvaluate: evaluate failed with error msg Capacity error: array cannot contain more than 2147483646 bytes, have 2147483672 (state=,code=0)

JNIEnv::ThrowNew doesn't actually throw exceptions. The following of the codes will still be executed. Thus we should protect our JNI code from execution against any possible exceptions. Otherwise termination of whole process due to something like SIGSEGV
will be easily faced.

We noticed that there is memleak in sort when SMJ is disabled, caused by batch close was not called by the consumer of sort operator (C2R).

We have a strong dependency on https://github.com/oap-project/arrow-data-source, it's better to make this as one submodule

Need to add more cmake parameters to support mvn install with more detail configuration.
For example, cmake can set up BUILD_PROTOBUF or BUILD_ARROW from source, but mvn can only run a default setting with compile.sh

We noticed that in Q72 native WSCG is still slower than vanilla spark, some ideas below may improve performance.

• postpone GetValue in WSCG when value is required.
• BHJ build : we may decide hashmap initial size according to build table size
• BHJ Get: optimize hashmap probe and add a fast cache
• SMJ optimization: TBD