Comments (7)
How about adding "logical type" attribute in connect.schema?
- Hive schema depends on "type" and "logical type";
- Column type stored in target file system only depends on "type".
The code translating between connect.schema and avro.schema is hard to be understood.
Adding "DATE"/"TIMESTAMP" type in connect.schema can be it more complex.
from kafka-connect-hdfs.
@xianzhen The logical type information is stored in the name of the schema. The HDFS connector can potentially discover the logical type from that can covert it to a proper Hive data type.
from kafka-connect-hdfs.
Hello,
From my point of view, this issue is: when I have timestamp type in ORACLE, after data is extracted into kafka, it is still timestamp type in schema registry, but when I use HDFS sink to put data into the HIVE, the type goes wrong, it became BIGINT.
If so, I am facing this problem too.
from kafka-connect-hdfs.
Hi,
Any updates on this issue, from which version it will be available?
from kafka-connect-hdfs.
Hi @ewencp
I have updated the code
static {
LOGICAL_TO_TYPEINFO = new HashMap<>();
LOGICAL_TO_TYPEINFO.put(Date.LOGICAL_NAME, TypeInfoFactory.dateTypeInfo);
LOGICAL_TO_TYPEINFO.put(Timestamp.LOGICAL_NAME, TypeInfoFactory.timestampTypeInfo);
LOGICAL_TO_TYPEINFO.put(Decimal.LOGICAL_NAME, TypeInfoFactory.decimalTypeInfo);
}
if (schema.name() != null) {
switch (schema.name()) {
case Date.LOGICAL_NAME:
return LOGICAL_TO_TYPEINFO.get(schema.name());
case Decimal.LOGICAL_NAME:
return LOGICAL_TO_TYPEINFO.get(schema.name());
case Timestamp.LOGICAL_NAME:
return LOGICAL_TO_TYPEINFO.get(schema.name());
}
}
Updated HiveSchemaConverter to add above code when creating Avro Hive table. So, this handle schema conversion. Data conversion where to be handled and how it needs to be handled.
But When table is getting created still i am seeing table has int column. So, further looked in to Avro Hive Conversion. Also, Avro 1.7.7 does not have logical date data type from 1.8 logical date type is available.
https://cwiki.apache.org/confluence/display/Hive/AvroSerDe
But there is no date data type in that. So, could you please explain more and help me on this to fix this issue. Thanks in advance.
from kafka-connect-hdfs.
Any updates on this issue?
We're also facing this problem with kafka-connect-hdfs 4.0.0.
from kafka-connect-hdfs.
+1
Any update on this ? Right now we are facing the issue for DATE, DATETIME and DECIMAL.
from kafka-connect-hdfs.
Related Issues (20)
- using wrong user/keytab while there are multiple hdfs-sink connections HOT 1
- template file isn't committed and uploaded to storage when using AvroFormat
- java.util.ConcurrentModificationException during task rebalancing HOT 1
- log4j update schedule HOT 1
- Hive table does not match column names present in the parquet data
- Exception when reading Decimal types written by connector
- Hive Merge Feature
- Incremental Co-operative Rebalancing Support for HDFS Connector
- Error after install and unistall connect-transforms
- Adding Hive partition threw unexpected error
- HDFS2 connect compatibility with HDFS3 server
- CVE-2021-34538 HIGH vulnerability HOT 2
- Task is being killed and will not recover until manually restarted
- Allow to limit retry write errors by timeout
- Kafka Issue while running on docker and adding new connector HOT 1
- can't build because repo conjars is down
- multiple keytab kerberos issue HOT 1
- OzoneFileSystem
- Non-resolvable parent POM io.confluent:common:[7.7.0, 7.7.1) for io.confluent:kafka-connect-storage-common-parent:11.2.9
- [2024-05-30 10:25:31,403] ERROR [hdfs3_sink-test_v4|task-0] WorkerSinkTask{id=hdfs3_sink-test_v4-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:237) java.lang.NullPointerException: Cannot invoke "io.confluent.connect.hdfs3.DataWriter.open(java.util.Collection)" because "this.hdfsWriter" is null
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kafka-connect-hdfs.