Comments (2)
Caused by: org.apache.kafka.connect.errors.DataException: Invalid Java object for schema type STRUCT: class java.lang.String for field: "null"
@rhatlapa Can you share the complete schema you're using to reproduce? The type STRUCT:
and field: "null"
makes it look like this default value is applied on the struct level, not the field level. This would mean that the default that is causing problems is not the one you've shown, but one somewhere else in the containing schema.
Also providing non-trivial default values to structs is currently not supported, see https://issues.apache.org/jira/browse/KAFKA-12694 for more details about that.
The issue seems to be caused by using too old versions of org.apache.kafka:connect-api:1.1.0 & io.confluent:kafka-avro-serializer:4.1.4 => I suggest to update the dependencies to resolve the issue.
Do you know what patch is missing from those dependencies?
We can upgrade the connect-api, but that is almost certainly not causing the issue, as the connect worker substitutes it's own connect-api at runtime. If there was a bug in that artifact, then running the connector on a modern Kafka Connect version would fix it.
Unfortunately we cannot upgrade the kafka-avro-serializer to a more modern version due to licensing changes. On a quick inspection I do not know what change has been made to the serializer which would address this bug, but please let me know if you've already looked into it.
from s3-connector-for-apache-kafka.
Per remote debugger the osType is the proplematic field and also I can confirm that removing the "default": "UNKNOWN"
helped. Other tools we are using (e.g. Kafka UI ) had no issues to handle the schema as shown bellow including the "default": "UNKNOWN"
.
From what I can say, when I download the s3 connector plugin from https://github.com/aiven/s3-connector-for-apache-kafka/releases/download/v2.12.0/aiven-kafka-connect-s3-2.12.0.zip, it already contains the connect-api-1.1.0.jar (my assumption of this version being used). I need to setup local run to be able to verify the actual behavior (unfortunately I didn't have time to do so yet).
Here is the schema (slightly made smaller to show all types of fields defined there, no other field with "UNKNOWN" string is there).
"type": "record",
"namespace": "com.company.kafka.avro",
"name": "AppInstallationEventAvro",
"fields": [
{
"name": "packageName",
"type": [
"null",
"string"
],
"default": null
},
{
"name": "osType",
"type": [
"string"
],
"default": "UNKNOWN"
}
]
```
from s3-connector-for-apache-kafka.
Related Issues (20)
- file in s3 without return line for each event HOT 1
- file.name.prefix doesn't seem to work
- Random UnknownHostException resolving S3 hostname HOT 1
- Just hold latest 10 record of each key or just hold data of 10 minutes ago HOT 4
- The s3 file only has startoffset. How to obtain the endoffset HOT 2
- AWS ECS Task or (likely) EKS pod with pre-mounted IAM credentials unsupported
- Support dynamic fields partitioned without requiring Avro scheme
- java.lang.NoClassDefFoundError HOT 1
- S3 Source Connector HOT 7
- Consumer offsets not committed when using KeyRecordGrouper
- Out-of-Memory errors when sinking large topics HOT 4
- Use environment variables for AWS Access Key ID/Secret Access Key Config
- master incompatible with latest cp-kafka-connect Docker image HOT 1
- AivenKafkaConnectS3SinkConnector should inherit from SinkConnector
- Published artifacts available for download? HOT 3
- does csv output require the input to be ByteArrayConverter ? HOT 1
- csv output with headers as 1st row HOT 2
- Delivery semantic HOT 1
- Issue on "Date Format" using the S3 Sink Connector from Avro to Parquet HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from s3-connector-for-apache-kafka.