teeyog / iql Goto Github PK

产品是否已完成的，可部署使用的了？

在执行包含 offset in top 等spark不支持 的SQL语句或者错误的SQL语句时，会导致 zookeeper中的/iql/valid_engine 的actor丢失 在并且在spark的8088端口访问出错

Readme里只有使用步骤，没有安装步骤啊！

Hi Team,

I am trying following SQL:-

var work__store_level_vend_pack_loc_final_data =

sparksession.read.format("csv")
.option("header", "true")
.option("delimiter", "|")
.option("inferSchema", "true")
.load("C:\Users\jabin\Desktop\project_files\work__store_level_vend_pack_loc_final_data.txt");

work__store_level_vend_pack_loc_final_data.registerTempTable("work__store_level_vend_pack_loc_final_data_table");

var r1 = sparksession.sqlContext.sql( "SELECT shc_item_id ,'K' as source_owner_cd,item_purchase_status_cd, vendor_package_id,vendor_package_purchase_status_cd,flow_type_cd as vendor_package_flow_type_cd,vendor_carton_qty,vendor_stock_nbr,ksn_package_id,ksn_purchase_status_cd,import_ind,sears_divission_nbr,sears_item_nbr,sears_sku_nbr,scan_based_trading_ind,cross_merchandising_cd,retail_carton_vendor_package_id,vendor_package_owner_cd,can_carry_model_id,'' AS days_to_check_begin_day_qty,'' AS days_to_check_end_day_qty ,dotcom_allocation_ind ,retail_carton_internal_package_qty,allocation_replenishment_cd,shc_item_type_cd,idrp_order_method_cd,source_package_qty as store_source_package_qty,order_duns_nbr FROM work__store_level_vend_pack_loc_final_data_table WHERE flow_type_cd = 'JIT' OR servicing_dc_nbr > '0' ")
// .collect.foreach(println)

now i want to do distinct all column of the r1 using sparksession.sqlContext.sql("")
how to do above thing?

您好，执行得时候报错

Exception in thread "main" com.typesafe.config.ConfigException$UnresolvedSubstitution: reference.conf @ jar:file:/home/zhangxin/iql-engine-1.0-SNAPSHOT-jar-with-dependencies.jar!/reference.conf: 846: Could not resolve substitution to a value: ${akka.stream.materializer}
at com.typesafe.config.impl.ConfigReference.resolveSubstitutions(ConfigReference.java:111)
at com.typesafe.config.impl.ResolveContext.realResolve(ResolveContext.java:179)
at com.typesafe.config.impl.ResolveContext.resolve(ResolveContext.java:142)
at com.typesafe.config.impl.SimpleConfigObject$ResolveModifier.modifyChildMayThrow

执行命令

spark-submit
--name IQL
--master local
--deploy-mode client
--conf spark.executor.extraJavaOptions="-Xms1024m -XX:MetaspaceSize=256m -XX:MaxMetaspaceSize=256m"
--num-executors 1
--driver-memory 3G
--executor-memory 1g
--class iql.engine.main.IqlMain
iql-engine-1.0-SNAPSHOT-jar-with-dependencies.jar

Hi, is there anything required to do to support Spark 3.x?

如何和cdh6兼容

启动模块的顺序，及初始化配置大概需要怎么配置？