housepower / clickhouse-native-jdbc Goto Github PK

View Code? Open in Web Editor NEW

518.0 21.0 147.0 2.11 MB

ClickHouse Native Protocol JDBC implementation

Home Page: https://housepower.github.io/ClickHouse-Native-JDBC/

License: Apache License 2.0

Java 97.42% Scala 2.32% Shell 0.27%

clickhouse analytics database jdbc clickhouse-client spark tcp-protocol

clickhouse-native-jdbc's Introduction

ClickHouse Native JDBC

English | 简体中文

Home Page | GitHub | Gitee

A Native JDBC library for accessing ClickHouse in Java, also provide a library for integrating with Apache Spark.

CONTRIBUTE

We welcome anyone that wants to help out in any way, whether that includes reporting problems, helping with documentations, or contributing code changes to fix bugs, add tests, or implement new features. Please follow Contributing Guide.

Supported by JetBrains Open Source License 2020-2021.

JDBC Driver

Requirements

Java 8/11.

Notes: We only do test with Java LTS versions.

Differences from yandex/clickhouse-jdbc

Data is organized and compressed by columns.
Implemented in the TCP Protocol, with higher performance than HTTP, here is the benchmark report.

Limitations

Not support insert complex values expression, like INSERT INTO test_table VALUES(toDate(123456)), but query is ok.
Not support insert non-values format, like TSV.
Not support more compression method, like ZSTD.

Import

Gradle

// (recommended) shaded version, available since 2.3-stable
compile "com.github.housepower:clickhouse-native-jdbc-shaded:${clickhouse_native_jdbc_version}"

// normal version
compile "com.github.housepower:clickhouse-native-jdbc:${clickhouse_native_jdbc_version}"

Maven

<!-- (recommended) shaded version, available since 2.3-stable -->
<dependency>
    <groupId>com.github.housepower</groupId>
    <artifactId>clickhouse-native-jdbc-shaded</artifactId>
    <version>${clickhouse-native-jdbc.version}</version>
</dependency>

<!-- normal version -->
<dependency>
    <groupId>com.github.housepower</groupId>
    <artifactId>clickhouse-native-jdbc</artifactId>
    <version>${clickhouse-native-jdbc.version}</version>
</dependency>

Integration with Spark

Requirements

Java 8, Scala 2.11/2.12, Spark 2.4
Or Java 8/11, Scala 2.12, Spark 3.0/3.1

For Spark 3.2, Spark ClickHouse Connector is recommended.

Notes: Spark 2.3.x(EOL) should also work fine. Actually we do test on both Java 8 and Java 11, but Spark official support on Java 11 since 3.0.0.

Import

Gradle

// available since 2.4.0
compile "com.github.housepower:clickhouse-integration-spark_2.11:${clickhouse_native_jdbc_version}"

Maven

<!-- available since 2.4.0 -->
<dependency>
    <groupId>com.github.housepower</groupId>
    <artifactId>clickhouse-integration-spark_2.11</artifactId>
    <version>${clickhouse-native-jdbc.version}</version>
</dependency>

License

This project is distributed under the terms of the Apache License (Version 2.0). See LICENSE for details.

clickhouse-native-jdbc's People

Contributors

Stargazers

Watchers

Forkers

shijiyu dulong keymanhuang bondy123 shangshujie365 tburch blinkov niharpradhan the-alchemist adubovikov wchch cosote clw87 lauri-nomme shunpeizhang yrwanziqi blankrain xxj123go daidd2019 kuyusea andygogo15 comphead stayhsf fanglietao liuaixi200 cfbin song1279 a615549958 alxkolm wangpengpeng qq459673705 wsy120 aliakseihaidukou waters-souls windaworld sunyuanpinggithub connieyang kifile dino1010 raghu777 drizzt321 forestlzj yc2213211 nauu ukchuvrus sannysanoff dnsang junneyang mmsmdali liekkassmile lizhanyang505 zwylovejava rowen110 sqleasy yujinyan heroonline uschindler frank-dkvan dcastanier kone-net wosow guangshu100 chengxianglibra andy1xx8 ddingcham kschubert pan3793 feiyuewuyue microserviceprovider dingxuefei rongyj frankswu clickhouse-java aries-ckt liuneng1994 kristguo jeromehj baibaichen ioyogi xing947933994 tangyibo lccbiluox2 elgca auguszero junyangz two8g misselvexu luj15 nerverbe sunny19930321 clickhouse-team zhbdesign xxxscarlxrd404 ganymedewuke zhao-jj help-lixin thexs nysyxxg wforget zcw5116

clickhouse-native-jdbc's Issues

Question: clickhouse-jdbc comparison?

Hi. How is this different from https://github.com/yandex/clickhouse-jdbc ?

Add DatabaseMetadata Support

SQLDatabaseMetadata

java.sql.SQLFeatureNotSupportedException
:at [com.github.housepower.jdbc.wrapper.SQLConnection getMetaData SQLConnection.java

server 1.1.54289 jdbc 1.3 insert data

Exception in thread "main" com.github.housepower.jdbc.ClickHouseSQLException: DB::ExceptionDB::Exception: Unknown compression method: 226. Stack trace:

clickhouse-server(StackTrace::StackTrace()+0x16) [0x2fe5936]
clickhouse-server(DB::Exception::Exception(std::string const&, int)+0x1f) [0x1293c2f]
clickhouse-server(DB::CompressedReadBufferBase::readCompressedData(unsigned long&, unsigned long&)+0x7ca) [0x3003cea]
clickhouse-server(DB::CompressedReadBuffer::nextImpl()+0x24) [0x3002f54]
clickhouse-server() [0x2963ff0]
clickhouse-server(DB::NativeBlockInputStream::readData(DB::IDataType const&, DB::IColumn&, DB::ReadBuffer&, unsigned long, double)+0x125) [0x29287d5]
clickhouse-server(DB::NativeBlockInputStream::readImpl()+0x7b6) [0x2929276]
clickhouse-server(DB::IProfilingBlockInputStream::read()+0x1e6) [0x2925116]
clickhouse-server(DB::TCPHandler::receiveData()+0x58) [0x129fc78]
clickhouse-server(DB::TCPHandler::receivePacket()+0x1b5) [0x12a0335]
clickhouse-server(DB::TCPHandler::readData(DB::Settings const&)+0x142) [0x12a06a2]
clickhouse-server(DB::TCPHandler::processInsertQuery(DB::Settings const&)+0x241) [0x12a1311]
clickhouse-server(DB::TCPHandler::runImpl()+0x647) [0x12a1ab7]
clickhouse-server(DB::TCPHandler::run()+0x1c) [0x12a263c]
clickhouse-server(Poco::Net::TCPServerConnection::start()+0xf) [0x384ddef]
clickhouse-server(Poco::Net::TCPServerDispatcher::run()+0x10b) [0x384f5fb]
clickhouse-server(Poco::PooledThread::run()+0x87) [0x39559b7]
clickhouse-server(Poco::ThreadImpl::runnableEntry(void*)+0x96) [0x3950546]
/lib64/libpthread.so.0(+0x7dc5) [0x7f2889116dc5]
/lib64/libc.so.6(clone+0x6d) [0x7f2886bf373d]

at com.github.housepower.jdbc.protocol.ExceptionResponse.readExceptionFrom(ExceptionResponse.java:32)
at com.github.housepower.jdbc.protocol.RequestOrResponse.readFrom(RequestOrResponse.java:38)
at com.github.housepower.jdbc.connect.PhysicalConnection.receiveResponse(PhysicalConnection.java:86)
at com.github.housepower.jdbc.connect.PhysicalConnection.receiveEndOfStream(PhysicalConnection.java:78)
at com.github.housepower.jdbc.ClickHouseConnection.sendInsertRequest(ClickHouseConnection.java:132)
at com.github.housepower.jdbc.statement.ClickHousePreparedInsertStatement.executeBatch(ClickHousePreparedInsertStatement.java:61)
at lyd.ai.ck.NativeC2CK.generateRowsList(NativeC2CK.java:59)
at lyd.ai.ck.NativeC2CK.main(NativeC2CK.java:68)

java.lang.ClassCastException: class [B cannot be cast to class [C ([B and [C are in module java.base of loader 'bootstrap')

Hello!

Not sure why, but I'm getting an error with insert batch query:

java.lang.ClassCastException: class [B cannot be cast to class [C ([B and [C are in module java.base of loader 'bootstrap')
	at com.github.housepower.jdbc.misc.StringViewCoding.getValues(StringViewCoding.java:25)
	at com.github.housepower.jdbc.misc.SQLLexer.<init>(SQLLexer.java:11)
	at com.github.housepower.jdbc.stream.ValuesWithParametersInputFormat.<init>(ValuesWithParametersInputFormat.java:27)
	at com.github.housepower.jdbc.statement.ClickHousePreparedInsertStatement.executeBatch(ClickHousePreparedInsertStatement.java:61)
	....

java.sql.SQLFeatureNotSupportedException is thrown for ResultSetMetaData.getColumnTypeName() method

val stmt = con.createStatement()
val rs = stmt.executeQuery("select 1")
val meta = rs.getMetaData
meta.getColumnTypeName(1))

Performance and memory usage issues

I tried to do some simple performance tests to compare clickhouse-native-jdbc and official clickhouse-jdbc drivers in terms of speed and memory usage. Below are two examples for making a simple select query from CH numbers function.

clickhouse-jdbc:

Register http driver..
Executing sql: SELECT number as n1 from numbers(100000000)
  0% Total memory (MB):   960, used memory (MB):    20, time elapsed:    354(+   354 ms)
 10% Total memory (MB):  1211, used memory (MB):   386, time elapsed:   1425(+  1071 ms)
 20% Total memory (MB):  1676, used memory (MB):   777, time elapsed:   2433(+  1008 ms)
 30% Total memory (MB):  1595, used memory (MB):   674, time elapsed:   3313(+   880 ms)
 40% Total memory (MB):  1525, used memory (MB):   740, time elapsed:   4197(+   884 ms)
 50% Total memory (MB):  1430, used memory (MB):   140, time elapsed:   5038(+   841 ms)
 60% Total memory (MB):  1375, used memory (MB):   524, time elapsed:   5868(+   830 ms)
 70% Total memory (MB):  1301, used memory (MB):   358, time elapsed:   6698(+   830 ms)
 80% Total memory (MB):  1236, used memory (MB):   401, time elapsed:   7529(+   831 ms)
 90% Total memory (MB):  1164, used memory (MB):   121, time elapsed:   8358(+   829 ms)
100% Total memory (MB):  1103, used memory (MB):   116, time elapsed:   9193(+   835 ms)
4999999950000000

real    0m9.652s
user    0m9.728s
sys     0m0.388s

clickhouse-native-jdbc:

Register native driver..
Executing sql: SELECT number as n1 from numbers(100000000)
  0% Total memory (MB):  9092, used memory (MB):  6802, time elapsed:  21052(+ 21052 ms)
 10% Total memory (MB): 12213, used memory (MB):  6598, time elapsed:  30561(+  9509 ms)
 20% Total memory (MB): 12282, used memory (MB):  6706, time elapsed:  30930(+   369 ms)
 30% Total memory (MB): 12390, used memory (MB):  6786, time elapsed:  31252(+   322 ms)
 40% Total memory (MB): 12412, used memory (MB):  6757, time elapsed:  31576(+   324 ms)
 50% Total memory (MB): 12510, used memory (MB):  6713, time elapsed:  31897(+   321 ms)
 60% Total memory (MB): 12523, used memory (MB):  6585, time elapsed:  32222(+   325 ms)
 70% Total memory (MB): 12523, used memory (MB):  8020, time elapsed:  32521(+   299 ms)
 80% Total memory (MB): 12608, used memory (MB):  7876, time elapsed:  32836(+   315 ms)
 90% Total memory (MB): 12619, used memory (MB):  7639, time elapsed:  33166(+   330 ms)
100% Total memory (MB): 12530, used memory (MB):  7413, time elapsed:  33472(+   306 ms)
4999999950000000

real    0m33.958s
user    3m8.092s
sys     0m3.816s

The first lines in the above examples (marked as 0%) mark the moment when the driver has returned a ResultSet object for the sql (ResultSet ClickHouseStatement.executeQuery(String query)) and it is ready to be iterated.

In these particular examples it can be seen that the official http jdbc driver returns the result set almost instantly (~300ms). Next lines with increasing percentages show the processing of the result set by iterating it via boolean ClickHouseResultSet.next() and long ClickHouseResultSet.getLong(int columnIndex).

For clickhouse-native-jdbc driver there is a considerable delay before the ResultSet object is ready (also, what is not seen here, since these tests are only for one particular result size, but this delay grows as the amount of rows returned by the query increases). By the time the ResultSet is ready a significant amount of memory is used (particularly in comparison to the http driver).

Overall, on this particular example the clickhouse-native-jdbc driver is 3 times slower in total and uses an order of magnitude more memory.

I tried to dig into sources and found that while preparing the result set in ClickHouseStatement.executeUpdate(String query) the QueryResponse object is being built by receiving all responses from PhysicalConnection.receiveResponse(..) and storing the results in ArrayList<RequestOrResponse>. So it looks like all the data is being read from the DB in advance prior to returning the ResultSet object which takes time and memory. As a result I wonder why it is implemented like this. Maybe I'm missing something or I do not understand some details or maybe it was just not a priority to make the driver memory-efficient for large data sets and it is planned to be improved upon in the future.

Would be grateful for any comments and clarifications.

Date type is wrong when insert into clickhouse

How to access the MySQL directly?

ClickHouse is pretty cool.

I want to use the SQL with ClickHouse-Native-JDBC to connect to the MySQL by using ClickHouse.

How to do? Any idea about ClickHouse and MySQL.

Maybe is it slow for ClickHouse with ODBC?

I want to use JDBC directly with ClickHouse and MySQL.

Freezing on Read

Hi there,
I'm unable to get this lib to work. It is always freezing for me. Do you have any idea why?

Date type is wrong when insert into clickhouse

when I insert into clickhouse table , I met a problem：
code：

`
Statement stmt = connection.createStatement();
stmt.executeQuery("drop table if exists test_jdbc_example");
stmt.executeQuery("create table test_jdbc_example(day Date, name String, age UInt8) Engine=Log");

    PreparedStatement pstmt = connection.prepareStatement("INSERT INTO test_jdbc_example VALUES(?, ?, ?)");

    for (int i = 0; i < 10; i++) {
        long time = 1530374400;
        System.out.println(new Date((time) * 1000));
        pstmt.setDate(1, new Date(time*1000));
        pstmt.setString(2, "Zhang San" + i);
        pstmt.setByte(3, (byte)i);
        pstmt.addBatch();
    }
    pstmt.executeBatch();

1530374400 convert to Date is ‘2018-07-01’， but in clickhouse Table，it show ‘2018-06-30’，I use it the wrong way？thank you.

SELECT *
FROM test_jdbc_example

┌────────day─┬─name───────┬─age─┐
│ 2018-06-30 │ Zhang San0 │ 0 │
│ 2018-06-30 │ Zhang San1 │ 1 │
│ 2018-06-30 │ Zhang San2 │ 2 │

Support isValid for Connection

Support Simple JDBC Interface

JDBC Driver
JDBC Connection
JDBC Statement
JDBC ResultSet

Use travis for CI

Support SQLStatement Batch

Batch insert into CK but get more records.

JDBC version in pom.xml

com.github.housepower clickhouse-native-jdbc 1.6-stable net.jpountz.lz4 lz4

ClickHouse version

ru.yandex.clickhouse clickhouse-jdbc 0.1.52 net.jpountz.lz4 lz4 guava com.google.guava

Error logs

no error logs

Steps to reproduce

I use Java language.
First, I use spark to query Hive table to produce a DataSet.Which count is 2w record.
Then, I use spark.forearchPartition ,and use iterator to insert into ck which is preparestatement ,and I count the number I have inserted . It's also 20000 record.
Finally, when the job have completed, there is no error logs. It seems every thing is ok (Because I have catch all exception and print messages but get nothing error). But when I query clickhouse use client , it show it has 360736 record in db.

Other descriptions

At first, I use distribute table , and I thought it is the reason which cause this problem . Then I alter the target table as a normal table,but it has no use. And I have try to insert about 17,000,000 record to a distribute table ,but I query about 200,000,000+ records in clickhouse client.

Support ResultSetMetaData

How to set the database in the JDBC connection url.

Class.forName("com.github.housepower.jdbc.ClickHouseDriver");
Connection connection = 
       DriverManager.getConnection("jdbc:clickhouse://127.0.0.1:9001/test");

Error, Can not the database named test.

Error in sending request.

Class.forName("com.github.housepower.jdbc.ClickHouseDriver");
    Connection connection = DriverManager.getConnection("jdbc:clickhouse://115.29.37.214:9000");
    Statement stmt = connection.createStatement();
    ResultSet rs = stmt.executeQuery("SELECT (number % 3 + 1) as n, sum(number) FROM numbers(10000000) GROUP BY n");

    while (rs.next()) {
      System.out.println(rs.getInt(1) + "\t" + rs.getLong(2));
    }

Exception in thread "main" java.sql.SQLException: Broken pipe (Write failed)
	at com.github.housepower.jdbc.connect.PhysicalConnection.disPhysicalConnection(PhysicalConnection.java:103)
	at com.github.housepower.jdbc.ClickHouseConnection.getHealthyPhysicalConnection(ClickHouseConnection.java:146)

java.sql.SQLException occurs when using nullIf() with DateTime type

When using the driver in Scala with the following code, a java.sql.SQLException is thrown:

val stmt = connection.createStatement()
stmt.executeQuery("SELECT nullIf(toDateTime('2018-06-01 00:00:00'), toDateTime(0))")

Support timezone for DateTime datatype

Support Enum Data Type

Support Failover Connection

Support complex clickhouse data types

How to read arrays, tuples, etc.

Maybe this is what you mean by Non-values format, but how do I access array, tuple, etc. values? This makes me think that it just hasn't been implemented yet?

https://github.com/housepower/ClickHouse-Native-JDBC/blob/master/src/main/java/com/github/housepower/jdbc/wrapper/SQLArray.java

Support INSERT INTO with TCP Protocol

UUID support

Hi,

when using clickhouses UUID conversion with a fixed String I'm ending up with this error (for write and read):

[04:55:41] [pool-3-thread-1/INFO] [STDERR]: [com.helion3.prism.storage.clickhouse.ClickhouseRecords:write:68]:  at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[04:55:41] [pool-3-thread-1/INFO] [STDERR]: [com.helion3.prism.storage.clickhouse.ClickhouseRecords:write:68]:  at java.lang.Thread.run(Unknown Source)
[04:55:42] [pool-3-thread-1/INFO] [STDERR]: [com.helion3.prism.storage.clickhouse.ClickhouseRecords:write:68]: java.sql.SQLException
[04:55:42] [pool-3-thread-1/INFO] [STDERR]: [com.helion3.prism.storage.clickhouse.ClickhouseRecords:write:68]:  at com.github.housepower.jdbc.misc.Validate.isTrue(Validate.java:14)
[04:55:42] [pool-3-thread-1/INFO] [STDERR]: [com.helion3.prism.storage.clickhouse.ClickhouseRecords:write:68]:  at com.github.housepower.jdbc.misc.Validate.isTrue(Validate.java:8)
[04:55:42] [pool-3-thread-1/INFO] [STDERR]: [com.helion3.prism.storage.clickhouse.ClickhouseRecords:write:68]:  at com.github.housepower.jdbc.misc.SQLLexer.stringLiteral(SQLLexer.java:72)
[04:55:42] [pool-3-thread-1/INFO] [STDERR]: [com.helion3.prism.storage.clickhouse.ClickhouseRecords:write:68]:  at com.github.housepower.jdbc.data.type.complex.DataTypeFixedString.deserializeTextQuoted(DataTypeFixedString.java:60)
[04:55:42] [pool-3-thread-1/INFO] [STDERR]: [com.helion3.prism.storage.clickhouse.ClickhouseRecords:write:68]:  at com.github.housepower.jdbc.stream.ValuesWithParametersInputFormat.next(ValuesWithParametersInputFormat.java:53)
[04:55:42] [pool-3-thread-1/INFO] [STDERR]: [com.helion3.prism.storage.clickhouse.ClickhouseRecords:write:68]:  at com.github.housepower.jdbc.ClickHouseConnection.sendInsertRequest(ClickHouseConnection.java:134)
[04:55:42] [pool-3-thread-1/INFO] [STDERR]: [com.helion3.prism.storage.clickhouse.ClickhouseRecords:write:68]:  at com.github.housepower.jdbc.statement.ClickHousePreparedInsertStatement.executeBatch(ClickHousePreparedInsertStatement.java:61)
[04:55:42] [pool-3-thread-1/INFO] [STDERR]: [com.helion3.prism.storage.clickhouse.ClickhouseRecords:write:68]:  at com.helion3.prism.storage.clickhouse.ClickhouseRecords.write(ClickhouseRecords.java:65)
[04:55:42] [pool-3-thread-1/INFO] [STDERR]: [com.helion3.prism.storage.clickhouse.ClickhouseRecords:write:68]:  at com.helion3.prism.queues.RecordingQueueManager.run(RecordingQueueManager.java:49)
[04:55:42] [pool-3-thread-1/INFO] [STDERR]: [com.helion3.prism.storage.clickhouse.ClickhouseRecords:write:68]:  at org.spongepowered.api.scheduler.Task$Builder.lambda$execute$0(Task.java:139)
[04:55:42] [pool-3-thread-1/INFO] [STDERR]: [com.helion3.prism.storage.clickhouse.ClickhouseRecords:write:68]:  at org.spongepowered.common.scheduler.SchedulerBase.lambda$startTask$0(SchedulerBase.java:192)
[04:55:42] [pool-3-thread-1/INFO] [STDERR]: [com.helion3.prism.storage.clickhouse.ClickhouseRecords:write:68]:  at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
[04:55:42] [pool-3-thread-1/INFO] [STDERR]: [com.helion3.prism.storage.clickhouse.ClickhouseRecords:write:68]:  at java.util.concurrent.FutureTask.run(Unknown Source)
[04:55:42] [pool-3-thread-1/INFO] [STDERR]: [com.helion3.prism.storage.clickhouse.ClickhouseRecords:write:68]:  at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
[04:55:42] [pool-3-thread-1/INFO] [STDERR]: [com.helion3.prism.storage.clickhouse.ClickhouseRecords:write:68]:  at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[04:55:42] [pool-3-thread-1/INFO] [STDERR]: [com.helion3.prism.storage.clickhouse.ClickhouseRecords:write:68]:  at java.lang.Thread.run(Unknown Source)

When doing the conversion my self, I'm ending up with setBytes not being supported:

[05:03:05] [pool-3-thread-1/INFO] [STDERR]: [com.helion3.prism.storage.clickhouse.ClickhouseRecords:write:68]: java.sql.SQLFeatureNotSupportedException
[05:03:05] [pool-3-thread-1/INFO] [STDERR]: [com.helion3.prism.storage.clickhouse.ClickhouseRecords:write:68]:  at com.github.housepower.jdbc.wrapper.SQLStatement.setBytes(SQLStatement.java:295)
[05:03:05] [pool-3-thread-1/INFO] [STDERR]: [com.helion3.prism.storage.clickhouse.ClickhouseRecords:write:68]:  at com.helion3.prism.storage.clickhouse.ClickhouseRecords.write(ClickhouseRecords.java:54)
[05:03:05] [pool-3-thread-1/INFO] [STDERR]: [com.helion3.prism.storage.clickhouse.ClickhouseRecords:write:68]:  at com.helion3.prism.queues.RecordingQueueManager.run(RecordingQueueManager.java:49)
[05:03:05] [pool-3-thread-1/INFO] [STDERR]: [com.helion3.prism.storage.clickhouse.ClickhouseRecords:write:68]:  at org.spongepowered.api.scheduler.Task$Builder.lambda$execute$0(Task.java:139)
[05:03:05] [pool-3-thread-1/INFO] [STDERR]: [com.helion3.prism.storage.clickhouse.ClickhouseRecords:write:68]:  at org.spongepowered.common.scheduler.SchedulerBase.lambda$startTask$0(SchedulerBase.java:192)
[05:03:05] [pool-3-thread-1/INFO] [STDERR]: [com.helion3.prism.storage.clickhouse.ClickhouseRecords:write:68]:  at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
[05:03:05] [pool-3-thread-1/INFO] [STDERR]: [com.helion3.prism.storage.clickhouse.ClickhouseRecords:write:68]:  at java.util.concurrent.FutureTask.run(Unknown Source)
[05:03:05] [pool-3-thread-1/INFO] [STDERR]: [com.helion3.prism.storage.clickhouse.ClickhouseRecords:write:68]:  at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
[05:03:05] [pool-3-thread-1/INFO] [STDERR]: [com.helion3.prism.storage.clickhouse.ClickhouseRecords:write:68]:  at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
[05:03:05] [pool-3-thread-1/INFO] [STDERR]: [com.helion3.prism.storage.clickhouse.ClickhouseRecords:write:68]:  at java.lang.Thread.run(Unknown Source)

Can this be supported or is there a workaround?

Add README doc

Error "Unknown compression method: 108" when doing arrayMap() with tuple() over two distinctly-typed arrays.

Reproduced on the version 1.5-stable with:

Class.forName("com.github.housepower.jdbc.ClickHouseDriver");
Connection connection = DriverManager.getConnection("jdbc:clickhouse://localhost:9000","default","");
Statement stmt = connection.createStatement();
val rs = stmt.executeQuery("select distinct arrayMap((x,y) -> tuple(x,y), nums, names) from (select arrayJoin([1,2]) as id, id = 1 ? [1,2,3] : [10,20,30] as nums, id = 1 ? ['one', 'two', 'three'] : ['a','b','c'] as names)")

[bug] Host not found

Server resolve client host error

Update readme

Add travis logo
Add batch examples

Does it support read data in terms for stream?

like mysql setting prepareStatement.setFetchSize(Integer.MIN_VALUE);

java.io.EOFException: Attempt to read after eof.

Hello!

I'm getting an error with insert batch query once in a while:

Exception in thread "main" java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:65)
	at org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala)
Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 118 in stage 5221.0 failed 4 times, most recent failure: Lost task 118.3 in stage 5221.0 (TID 626642, 10.111.65.213, executor 35): java.sql.SQLException: Attempt to read after eof.
	at com.github.housepower.jdbc.connect.PhysicalConnection.receiveResponse(PhysicalConnection.java:88)
	at com.github.housepower.jdbc.connect.PhysicalConnection.receiveEndOfStream(PhysicalConnection.java:78)
	at com.github.housepower.jdbc.ClickHouseConnection.sendInsertRequest(ClickHouseConnection.java:138)
	at com.github.housepower.jdbc.statement.ClickHousePreparedInsertStatement.executeBatch(ClickHousePreparedInsertStatement.java:61)
	at com.hellobike.SoaToClickHouse$$anonfun$main$1$$anonfun$apply$1.apply(SoaToClickHouse.scala:107)
	at com.hellobike.SoaToClickHouse$$anonfun$main$1$$anonfun$apply$1.apply(SoaToClickHouse.scala:75)
	at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:929)
	at org.apache.spark.rdd.RDD$$anonfun$foreachPartition$1$$anonfun$apply$29.apply(RDD.scala:929)
	at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
	at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
	at org.apache.spark.scheduler.Task.run(Task.scala:109)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.EOFException: Attempt to read after eof.
	at com.github.housepower.jdbc.buffer.SocketBuffedReader.refill(SocketBuffedReader.java:67)
	at com.github.housepower.jdbc.buffer.SocketBuffedReader.readBinary(SocketBuffedReader.java:34)
	at com.github.housepower.jdbc.serializer.BinaryDeserializer.readVarInt(BinaryDeserializer.java:23)
	at com.github.housepower.jdbc.protocol.RequestOrResponse.readFrom(RequestOrResponse.java:32)
	at com.github.housepower.jdbc.connect.PhysicalConnection.receiveResponse(PhysicalConnection.java:86)
	... 15 more

FixedString with Nullable broken

When retrieving a string from the ResultSet which is a Nullable(FixedString(n)) Column it returns ???????????????????????????????????? (probably something different and just encoded to ?) instead of null.

Release the first preview

Wildcard query throwing exception

A simple query like this is throwing the following exception.

SELECT * FROM prism_records ORDER BY created DESC

[05:45:49] [pool-3-thread-1/INFO] [STDERR]: [com.helion3.prism.util.AsyncUtil:lambda$async$1:110]: java.sql.SQLException
[05:45:49] [pool-3-thread-1/INFO] [STDERR]: [com.helion3.prism.util.AsyncUtil:lambda$async$1:110]:  at com.github.housepower.jdbc.misc.Validate.isTrue(Validate.java:14)
[05:45:49] [pool-3-thread-1/INFO] [STDERR]: [com.helion3.prism.util.AsyncUtil:lambda$async$1:110]:  at com.github.housepower.jdbc.misc.Validate.isTrue(Validate.java:8)
[05:45:49] [pool-3-thread-1/INFO] [STDERR]: [com.helion3.prism.util.AsyncUtil:lambda$async$1:110]:  at com.github.housepower.jdbc.data.Block.getPositionByName(Block.java:68)
[05:45:49] [pool-3-thread-1/INFO] [STDERR]: [com.helion3.prism.util.AsyncUtil:lambda$async$1:110]:  at com.github.housepower.jdbc.ClickHouseResultSet.findColumn(ClickHouseResultSet.java:235)
[05:45:49] [pool-3-thread-1/INFO] [STDERR]: [com.helion3.prism.util.AsyncUtil:lambda$async$1:110]:  at com.github.housepower.jdbc.ClickHouseResultSet.getString(ClickHouseResultSet.java:88)
[05:45:49] [pool-3-thread-1/INFO] [STDERR]: [com.helion3.prism.util.AsyncUtil:lambda$async$1:110]:  at com.helion3.prism.storage.clickhouse.ClickhouseRecords.query(ClickhouseRecords.java:106)

Pentaho PDI use Native JDBC TO ClickHouse net/jpountz/lz4/LZ4Factory

Error connecting to database [ClickHouse] :org.pentaho.di.core.exception.KettleDatabaseException:
Error occurred while trying to connect to the database

Error connecting to database: (using class com.github.housepower.jdbc.ClickHouseDriver)
net/jpountz/lz4/LZ4Factory

org.pentaho.di.core.exception.KettleDatabaseException:
Error occurred while trying to connect to the database

Error connecting to database: (using class com.github.housepower.jdbc.ClickHouseDriver)
net/jpountz/lz4/LZ4Factory

at org.pentaho.di.core.database.Database.normalConnect(Database.java:472)
at org.pentaho.di.core.database.Database.connect(Database.java:370)
at org.pentaho.di.core.database.Database.connect(Database.java:341)
at org.pentaho.di.core.database.Database.connect(Database.java:331)
at org.pentaho.di.core.database.DatabaseFactory.getConnectionTestReport(DatabaseFactory.java:80)
at org.pentaho.di.core.database.DatabaseMeta.testConnection(DatabaseMeta.java:2786)
at org.pentaho.ui.database.event.DataHandler.testDatabaseConnection(DataHandler.java:619)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.pentaho.ui.xul.impl.AbstractXulDomContainer.invoke(AbstractXulDomContainer.java:313)
at org.pentaho.ui.xul.impl.AbstractXulComponent.invoke(AbstractXulComponent.java:157)
at org.pentaho.ui.xul.impl.AbstractXulComponent.invoke(AbstractXulComponent.java:141)
at org.pentaho.ui.xul.swt.tags.SwtButton.access$500(SwtButton.java:43)
at org.pentaho.ui.xul.swt.tags.SwtButton$4.widgetSelected(SwtButton.java:137)
at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source)
at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Display.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)
at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)
at org.eclipse.jface.window.Window.runEventLoop(Window.java:820)
at org.eclipse.jface.window.Window.open(Window.java:796)
at org.pentaho.di.ui.xul.KettleDialog.show(KettleDialog.java:80)
at org.pentaho.di.ui.xul.KettleDialog.show(KettleDialog.java:47)
at org.pentaho.di.ui.core.database.dialog.XulDatabaseDialog.open(XulDatabaseDialog.java:118)
at org.pentaho.di.ui.core.database.dialog.DatabaseDialog.open(DatabaseDialog.java:60)
at org.pentaho.di.ui.spoon.delegates.SpoonDBDelegate.editConnection(SpoonDBDelegate.java:95)
at org.pentaho.di.ui.spoon.Spoon.doubleClickedInTree(Spoon.java:3202)
at org.pentaho.di.ui.spoon.Spoon.doubleClickedInTree(Spoon.java:3137)
at org.pentaho.di.ui.spoon.Spoon.access$2200(Spoon.java:370)
at org.pentaho.di.ui.spoon.Spoon$29.widgetDefaultSelected(Spoon.java:6286)
at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source)
at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Display.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)
at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)
at org.pentaho.di.ui.spoon.Spoon.readAndDispatch(Spoon.java:1376)
at org.pentaho.di.ui.spoon.Spoon.waitForDispose(Spoon.java:8161)
at org.pentaho.di.ui.spoon.Spoon.start(Spoon.java:9523)
at org.pentaho.di.ui.spoon.Spoon.main(Spoon.java:702)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.pentaho.commons.launcher.Launcher.main(Launcher.java:92)

Caused by: org.pentaho.di.core.exception.KettleDatabaseException:
Error connecting to database: (using class com.github.housepower.jdbc.ClickHouseDriver)
net/jpountz/lz4/LZ4Factory

at org.pentaho.di.core.database.Database.connectUsingClass(Database.java:585)
at org.pentaho.di.core.database.Database.normalConnect(Database.java:456)
... 47 more

Caused by: java.lang.NoClassDefFoundError: net/jpountz/lz4/LZ4Factory
at com.github.housepower.jdbc.buffer.CompressedBuffedWriter.(CompressedBuffedWriter.java:16)
at com.github.housepower.jdbc.serializer.BinarySerializer.(BinarySerializer.java:21)
at com.github.housepower.jdbc.connect.PhysicalConnection.openPhysicalConnection(PhysicalConnection.java:131)
at com.github.housepower.jdbc.ClickHouseConnection.createPhysicalInfo(ClickHouseConnection.java:173)
at com.github.housepower.jdbc.ClickHouseConnection.createClickHouseConnection(ClickHouseConnection.java:169)
at com.github.housepower.jdbc.NonRegisterDriver.connect(NonRegisterDriver.java:23)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at org.pentaho.di.core.database.Database.connectUsingClass(Database.java:567)
... 48 more
Caused by: java.lang.ClassNotFoundException: net.jpountz.lz4.LZ4Factory
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 57 more

Custom URL :jdbc:clickhouse://IP:9000/default
Custom Driver Class:com.github.housepower.jdbc.ClickHouseDriver

Port 9000 is for clickhouse-client program

I user ClickHouse-Native-JDBC to send data into clickhouse,but get expection like this:
java.lang.RuntimeException: ru.yandex.clickhouse.except.ClickHouseUnknownException: ClickHouse exception, code: 1002, host: 172.18.164.98, port: 9000; Port 9000 is for clickhouse-client program.
You must use port 8123 for HTTP.

I don't know that i should add Clickhouse-Jdbc dependency into pom.xml or not .But when i don't add Clickhouse-Jdbc dependency,there is a exception that is No suitable driver found for jdbc url.

The version of clickhouse is 18.1.0

Cannot use syntax for insert with multiple values via jdbc driver, an error occurs

Version 1.5-stable, a way to reproduce:

Class.forName("com.github.housepower.jdbc.ClickHouseDriver");
Connection connection = DriverManager.getConnection("jdbc:clickhouse://localhost:9000","default","");
Statement stmt = connection.createStatement();
stmt.execute("CREATE TEMPORARY TABLE tmp_debug_jdbc(id Int32) ENGINE = Memory")
stmt.execute("INSERT INTO tmp_debug_jdbc values (1), (2)")

Support SQLStatement FetchSize Injection

Can we support SQLStatement FetchSize Injection?

ClickHouse-Native-JDBC/src/main/java/com/github/housepower/jdbc/wrapper/SQLStatement.java

Lines 113 to 115 in 7ece148

 public void setFetchSize(int rows) throws SQLException { 

 throw new SQLFeatureNotSupportedException(); 

 }

  @Override
    public void setFetchSize(int rows) throws SQLException {
        throw new SQLFeatureNotSupportedException();
    }

Better identification of `INSERT`

What about: select * from test where name = 'VALUES (a,b,c,d)' ?
It's not an insert query, but matches the Pattern, should use QuotedLexer to check it's an insert query.

reference: cf51d65

Support PreparedStatement

Pentaho PDI use JDBC to ClickHouse，Could not initialize class ru.yandex.clickhouse.response.ClickHouseLZ4Stream

Error connecting to database [ClickHouse] :org.pentaho.di.core.exception.KettleDatabaseException:
Error occurred while trying to connect to the database

Error connecting to database: (using class ru.yandex.clickhouse.ClickHouseDriver)
Could not initialize class ru.yandex.clickhouse.response.ClickHouseLZ4Stream

org.pentaho.di.core.exception.KettleDatabaseException:
Error occurred while trying to connect to the database

Error connecting to database: (using class ru.yandex.clickhouse.ClickHouseDriver)
Could not initialize class ru.yandex.clickhouse.response.ClickHouseLZ4Stream

at org.pentaho.di.core.database.Database.normalConnect(Database.java:472)
at org.pentaho.di.core.database.Database.connect(Database.java:370)
at org.pentaho.di.core.database.Database.connect(Database.java:341)
at org.pentaho.di.core.database.Database.connect(Database.java:331)
at org.pentaho.di.core.database.DatabaseFactory.getConnectionTestReport(DatabaseFactory.java:80)
at org.pentaho.di.core.database.DatabaseMeta.testConnection(DatabaseMeta.java:2786)
at org.pentaho.ui.database.event.DataHandler.testDatabaseConnection(DataHandler.java:619)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.pentaho.ui.xul.impl.AbstractXulDomContainer.invoke(AbstractXulDomContainer.java:313)
at org.pentaho.ui.xul.impl.AbstractXulComponent.invoke(AbstractXulComponent.java:157)
at org.pentaho.ui.xul.impl.AbstractXulComponent.invoke(AbstractXulComponent.java:141)
at org.pentaho.ui.xul.swt.tags.SwtButton.access$500(SwtButton.java:43)
at org.pentaho.ui.xul.swt.tags.SwtButton$4.widgetSelected(SwtButton.java:137)
at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source)
at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Display.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)
at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)
at org.eclipse.jface.window.Window.runEventLoop(Window.java:820)
at org.eclipse.jface.window.Window.open(Window.java:796)
at org.pentaho.di.ui.xul.KettleDialog.show(KettleDialog.java:80)
at org.pentaho.di.ui.xul.KettleDialog.show(KettleDialog.java:47)
at org.pentaho.di.ui.core.database.dialog.XulDatabaseDialog.open(XulDatabaseDialog.java:118)
at org.pentaho.di.ui.core.database.dialog.DatabaseDialog.open(DatabaseDialog.java:60)
at org.pentaho.di.ui.spoon.delegates.SpoonDBDelegate.editConnection(SpoonDBDelegate.java:95)
at org.pentaho.di.ui.spoon.Spoon.editConnection(Spoon.java:2787)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.pentaho.ui.xul.impl.AbstractXulDomContainer.invoke(AbstractXulDomContainer.java:313)
at org.pentaho.ui.xul.impl.AbstractXulComponent.invoke(AbstractXulComponent.java:157)
at org.pentaho.ui.xul.impl.AbstractXulComponent.invoke(AbstractXulComponent.java:141)
at org.pentaho.ui.xul.jface.tags.JfaceMenuitem.access$100(JfaceMenuitem.java:43)
at org.pentaho.ui.xul.jface.tags.JfaceMenuitem$1.run(JfaceMenuitem.java:106)
at org.eclipse.jface.action.Action.runWithEvent(Action.java:498)
at org.eclipse.jface.action.ActionContributionItem.handleWidgetSelection(ActionContributionItem.java:545)
at org.eclipse.jface.action.ActionContributionItem.access$2(ActionContributionItem.java:490)
at org.eclipse.jface.action.ActionContributionItem$5.handleEvent(ActionContributionItem.java:402)
at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Display.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)
at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)
at org.pentaho.di.ui.spoon.Spoon.readAndDispatch(Spoon.java:1376)
at org.pentaho.di.ui.spoon.Spoon.waitForDispose(Spoon.java:8161)
at org.pentaho.di.ui.spoon.Spoon.start(Spoon.java:9523)
at org.pentaho.di.ui.spoon.Spoon.main(Spoon.java:702)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.pentaho.commons.launcher.Launcher.main(Launcher.java:92)

Caused by: org.pentaho.di.core.exception.KettleDatabaseException:
Error connecting to database: (using class ru.yandex.clickhouse.ClickHouseDriver)
Could not initialize class ru.yandex.clickhouse.response.ClickHouseLZ4Stream

at org.pentaho.di.core.database.Database.connectUsingClass(Database.java:585)
at org.pentaho.di.core.database.Database.normalConnect(Database.java:456)
... 56 more

Caused by: java.lang.NoClassDefFoundError: Could not initialize class ru.yandex.clickhouse.response.ClickHouseLZ4Stream
at ru.yandex.clickhouse.ClickHouseStatementImpl.checkForErrorAndThrow(ClickHouseStatementImpl.java:728)
at ru.yandex.clickhouse.ClickHouseStatementImpl.getInputStream(ClickHouseStatementImpl.java:551)
at ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:114)
at ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:97)
at ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:92)
at ru.yandex.clickhouse.ClickHouseStatementImpl.executeQuery(ClickHouseStatementImpl.java:88)
at ru.yandex.clickhouse.ClickHouseConnectionImpl.initTimeZone(ClickHouseConnectionImpl.java:86)
at ru.yandex.clickhouse.ClickHouseConnectionImpl.(ClickHouseConnectionImpl.java:75)
at ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:58)
at ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:50)
at ru.yandex.clickhouse.ClickHouseDriver.connect(ClickHouseDriver.java:32)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at org.pentaho.di.core.database.Database.connectUsingClass(Database.java:567)
... 57 more

Custom URL :jdbc:clickhouse://IP:9000/default
Custom Driver Class:ru.yandex.clickhouse.ClickHouseDriver

when batch insert FixedString type ,it can't work!

connect_timeout exception

Connection connection = DriverManager.getConnection("jdbc:clickhouse://127.0.0.1:9000?connect_timeout=10");

use connect_timeout throw exception:
java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.Integer

No support for unsigned int values.

Hi, unfortunately, the driver doesn't support UINT.

i.e. UINT16 - value 36199 will be return as -29337.

Looks like the getObject->Number makes wrong byte shift i.e. ("36199 - 65535 (max uin16))) and as result we got wrong value. IMHO the best way implement getUINT8/16/32 in the ResultSet, because the beauty of CH is unsigned numbers ;-)

java.lang.ArrayIndexOutOfBoundsException

批量插入数据有异常：

java.lang.ArrayIndexOutOfBoundsException: 0
	at com.github.housepower.jdbc.stream.ValuesWithParametersInputFormat.quotedParameter(ValuesWithParametersInputFormat.java:62)
	at com.github.housepower.jdbc.stream.ValuesWithParametersInputFormat.next(ValuesWithParametersInputFormat.java:52)
	at com.github.housepower.jdbc.ClickHouseConnection.sendInsertRequest(ClickHouseConnection.java:144)
	at com.github.housepower.jdbc.statement.ClickHousePreparedInsertStatement.executeBatch(ClickHousePreparedInsertStatement.java:61)
	at com.dp.kuafu.sdk.SdkDau$$anonfun$1.apply(SdkDau.scala:140)
	at com.dp.kuafu.sdk.SdkDau$$anonfun$1.apply(SdkDau.scala:79)
	at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:797)
	at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:797)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
	at org.apache.spark.scheduler.Task.run(Task.scala:99)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

代码如下：

val sql = "INSERT INTO mk_dw.thirdappclientfunc(dhid, uhid, appId, productId, verCode, chanId, origChanId, " +
                              "lang, ip, createDt, imei, mac, capSsid, capBssid, capApRefId, longi, lati, mapSP, netModel, ts, " +
                              "funid, ext, cts, manuf, model, seq, vcode, vname, chid, dur, sid, sdk, province, pt, hour, minute) " +
                              "VALUES(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"

                            println("sql:" + sql)

                            val pstmt = connection.prepareStatement(sql);

                            iter.foreach(sdkDauReq => {
                                pstmt.setString(1, sdkDauReq.dhid)
                                pstmt.setString(2, sdkDauReq.uhid)
                                pstmt.setString(3, sdkDauReq.appId)
                                pstmt.setInt(4, sdkDauReq.productId)
                                pstmt.setString(5, sdkDauReq.verCode)
                                pstmt.setString(6, sdkDauReq.chanId)
                                pstmt.setString(7, sdkDauReq.origChanId)
                                pstmt.setString(8, sdkDauReq.lang)
                                pstmt.setString(9, sdkDauReq.ip)
                                pstmt.setString(10, sdkDauReq.createDt)
                                pstmt.setString(11, sdkDauReq.imei)
                                pstmt.setString(12, sdkDauReq.mac)
                                pstmt.setString(13, sdkDauReq.capSsid)
                                pstmt.setString(14, sdkDauReq.capBssid)
                                pstmt.setString(15, sdkDauReq.capApRefId)
                                pstmt.setString(16, sdkDauReq.longi)
                                pstmt.setString(17, sdkDauReq.lati)
                                pstmt.setString(18, sdkDauReq.mapSP)
                                pstmt.setString(19, sdkDauReq.netModel)
                                pstmt.setString(20, sdkDauReq.ts)
                                pstmt.setString(21, sdkDauReq.funid)
                                pstmt.setString(22, sdkDauReq.ext)
                                pstmt.setString(23, sdkDauReq.cts)
                                pstmt.setString(24, sdkDauReq.manuf)
                                pstmt.setString(25, sdkDauReq.model)
                                pstmt.setString(26, sdkDauReq.seq)
                                pstmt.setString(27, sdkDauReq.vcode)
                                pstmt.setString(28, sdkDauReq.vname)
                                pstmt.setString(29, sdkDauReq.chid)
                                pstmt.setString(30, sdkDauReq.dur)
                                pstmt.setString(31, sdkDauReq.sid)
                                pstmt.setString(32, sdkDauReq.sdk)
                                pstmt.setString(33, sdkDauReq.province)
                                pstmt.setDate(34, sdkDauReq.pt)
                                pstmt.setString(35, sdkDauReq.hour)
                                pstmt.setString(36, sdkDauReq.minute)

                                pstmt.addBatch()
                            })
                            pstmt.executeBatch();

Add JDK6 7 integration testing in traivis

Support SQLConnection.getCatalog API

we try to integrate this clickhouse JDBC connection with c3p0 data source pool. But we encounter the below issue

java.sql.SQLFeatureNotSupportedException
	at com.github.housepower.jdbc.wrapper.SQLConnection.getCatalog(SQLConnection.java:88)
	at com.mchange.v2.c3p0.impl.NewPooledConnection.<init>(NewPooledConnection.java:108)
	at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:159)
	at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:132)
	at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
	at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
	at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
	at com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
	at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)

related source code

 private final ComboPooledDataSource cpds = new ComboPooledDataSource();

  public Connection getConnection() throws SQLException {
    return cpds.getConnection();
  }

  C3poDataSource() {
    try {
      cpds.setDriverClass("com.github.housepower.jdbc.ClickHouseDriver");
      cpds.setJdbcUrl("jdbc:clickhouse://10.8.121.33:9000");
      cpds.setMinPoolSize(10);
      cpds.setMaxPoolSize(10);

    } catch (PropertyVetoException e) {
      // handle the exception
    }
  }

ClickHouseResultSet.getObject() works incorrectly for Nullable data types.

ResultSet incorrectly returns 0.0 instead of null for my Nullable(Float32) column. Looking into the sources I have found that getObject() returns default column value, which is taken from nested data type in case of Nullable.

ClickHouse-Native-JDBC/src/main/java/com/github/housepower/jdbc/ClickHouseResultSet.java

Line 185 in fa6b1d3

return rowData == null ? column.type().defaultValue() : rowData;

	public void setFetchSize(int rows) throws SQLException {
	throw new SQLFeatureNotSupportedException();
	}

housepower / clickhouse-native-jdbc Goto Github PK

clickhouse-native-jdbc's Introduction

ClickHouse Native JDBC

Home Page | GitHub | Gitee

CONTRIBUTE

JDBC Driver

Requirements

Differences from yandex/clickhouse-jdbc

Limitations

Import

Integration with Spark

Requirements

Import

License

clickhouse-native-jdbc's People

Contributors

Stargazers

Watchers

Forkers

clickhouse-native-jdbc's Issues

JDBC version in pom.xml

ClickHouse version

Error logs

Steps to reproduce

Other descriptions

Recommend Projects

Recommend Topics

Recommend Org