Comments (17)
It progressed a bit further but at some stage run into this error:
Error Could not write JSON: Not a map: {"type":"record","name":..... rest of the schema follows.
When I set the formats to String, String. The messages are displayed. Its just the portions of the data are not human readable. This would mean that the format is correct.
from kafka-webview.
This looks good. I can see the data now. The only problem (And I am not sure if you can fix it. It could be that the bug or limitation is with the Avro deserialiser) is that the data that has been encoded in BYTES type is not converted to the logical type (decimal) and is not readable.
from kafka-webview.
Here is condensed Avro schema. I have removed some info and changed field names.
`{
"type": "record",
"name": "Future",
"namespace": "au.com.xxx.models",
"fields": [
{
"name": "index",
"type": "int"
},
{
"name": "description",
"type": "string"
},
{
"name": "inv_return",
"type": [
{
"type": "record",
"name": "Dollars",
"fields": [
{
"name": "dollars",
"type": {
"type": "bytes",
"logicalType": "decimal",
"precision": 20,
"scale": 2
}
}
]
},
{
"type": "record",
"name": "LinePay",
"fields": [
{
"name": "money",
"type": {
"type": "bytes",
"logicalType": "decimal",
"precision": 20,
"scale": 2
}
}
]
}
]
},
{
"name": "place_return",
"type": [
"Dollars",
"LinePay"
]
},
{
"name": "take",
"type": {
"type": "bytes",
"logicalType": "decimal",
"precision": 20,
"scale": 2
}
},
{
"name": "liability",
"type": {
"type": "bytes",
"logicalType": "decimal",
"precision": 20,
"scale": 2
}
},
{
"name": "laid_off_liability",
"type": {
"type": "bytes",
"logicalType": "decimal",
"precision": 20,
"scale": 2
}
},
{
"name": "limit",
"type": {
"type": "bytes",
"logicalType": "decimal",
"precision": 20,
"scale": 2
}
},
{
"name": "status",
"type": "string"
},
{
"name": "official",
"type": "boolean"
},
{
"name": "last_tkt_time",
"type": [
"null",
{
"type": "long",
"logicalType": "timestamp-millis"
}
]
},
{
"name": "take",
"type": {
"type": "bytes",
"logicalType": "decimal",
"precision": 20,
"scale": 2
}
},
{
"name": "liability",
"type": {
"type": "bytes",
"logicalType": "decimal",
"precision": 20,
"scale": 2
}
},
{
"name": "last_take",
"type": {
"type": "bytes",
"logicalType": "decimal",
"precision": 20,
"scale": 2
}
},
{
"name": "sort_order",
"type": "int"
},
{
"name": "deleted",
"type": "boolean"
},
{
"name": "override_reduction_percent",
"type": "int"
},
{
"name": "diff_price",
"type": "boolean"
},
{
"name": "session_take",
"type": {
"type": "bytes",
"logicalType": "decimal",
"precision": 20,
"scale": 2
}
},
{
"name": "session_liability",
"type": {
"type": "bytes",
"logicalType": "decimal",
"precision": 20,
"scale": 2
}
},
{
"name": "feed_id",
"type": "string"
},
{
"name": "value_type_str",
"type": "string"
},
{
"name": "group_id",
"type": "string"
},
{
"name": "last_official_time",
"type": [
"null",
{
"type": "long",
"logicalType": "timestamp-millis"
}
]
},
{
"name": "future_id",
"type": "int"
},
{
"name": "feed_return",
"type": [
"Dollars",
"LinePay",
"Odds"
]
}
]
}
from kafka-webview.
Hey! Are you able to send me the JAR so I can test locally, or even better fork the examples project and commit your changes there for review?
Thanks!
from kafka-webview.
@cezarywisniewski When I build your JAR and upload it to my local web kafkaview instance using the same configuration as you've defined above (only entering a schema.registry.url property), I end up getting this underlying error.
Invalid value io.confluent.kafka.serializers.subject.TopicNameStrategy for configuration key.subject.name.strategy: Class io.confluent.kafka.serializers.subject.TopicNameStrategy could not be found.
Since I'm not super familiar with the Schema Registry product, I'll need to do some additional research to understand what exactly is going on.
I also need to better expose the real underlying exception cause in these scenarios.
from kafka-webview.
So I've narrowed this down to a classloader issue. The classloader Kafka Webview is using to load the deserializer from the uploaded JAR, is not the same class loader that the deserializer attempts to load its own classes from.
The AvroDeserializer uses Thread.getCurrentThread().getContextClassloader()
which references a different classloader.
I'm still working through the best possible solution to this issue.
from kafka-webview.
from kafka-webview.
@cezarywisniewski So after poking around a bit more at confluent's source code it looks like the fact that they were using the ContextClassloader instead of the current classloader was a bug that was resolved in version ~5.x.
Are you able to try using version 5.1.0 of their deserializer package? When I built and upload a JAR using version 5.1.0 I seem to get further along in the consuming process. Unfortunately at the moment I don't have a copy of schema registry setup anywhere to completely validate that version works as expected end to end.
Let me know if you're able to test using version 5.1.0 and if things seem to work, or you run into other errors.
Thanks
from kafka-webview.
Thanks for the update. I'll stand up a schema registry service today and see if I can replicate it as well.
from kafka-webview.
I've made some progress locally, but still not in an ideal situation. It looks like its just serializing it a flat string vs json. A little more playing and hopefully get it sorted out :)
from kafka-webview.
I think I've put together a solution.
from kafka-webview.
I'm debating if I should just include/package confluent's schema registry deserializer with Kafka WebView. I need to check to see how it's licensed.
from kafka-webview.
I've published a pre-release package which (hopefully) will resolve the issue for you here: https://github.com/SourceLabOrg/kafka-webview/releases/tag/v2.1.2-RC1
You should be able to download and extract the release, and overwrite your existing install's .jar file. Doing so, all your existing settings should carry over.
Let me know if you're able to validate this works end to end for you or not, and I'll cut a proper release if so.
Thanks for your help!
Stephen
from kafka-webview.
Are you able to share a condensed snippet of the avro schema that I could import locally?
from kafka-webview.
Do you use a particular library for populating your logical data types when generating the AVRO messages? It looks like the generated class files only deal w/ ByteBuffers for logical types using the standard AVRO 1.8.2 java library. That or I'm missing something. The AVRO docs seem to indicate logical types are for extending custom or non-native data types, so unfortunately there may not be much I can add to generically support those types :/
Schema file:
{"namespace": "io.confluent.examples.clients.basicavro",
"type": "record",
"name": "Payment",
"fields": [
{"name": "id", "type": "string"},
{"name": "amount", "type": "double"},
{
"name": "take",
"type": {
"type": "bytes",
"logicalType": "decimal",
"precision": 20,
"scale": 2
}
}
]
}
from kafka-webview.
You are right. I do not think you can improve it any more due to limitations of avro library.
Anyway, thanks for your help.
from kafka-webview.
Yea you may be able to do better by adding a wrapper around the avro deserializer and handling deserialization of those types yourself?
Anyhow thanks for your continued help! If you have other issues or ideas feel free to post up an issue!
from kafka-webview.
Related Issues (20)
- Lets add warning log msgs when ignoring or overwriting user defined configuration values for deserializers HOT 1
- Kafka-Webview should provide option to "Stringify" ByteArray messages HOT 2
- Failing to connect to GSSAPI/SSL cluster HOT 2
- Custom protobuf deserializer failed because of pb version conflict HOT 2
- On View page, the "previous button" jump is twice as big as the "next button" jump. HOT 1
- use custom config.yml with docker
- Repository with ID="orgsourcelab-1031" not found HOT 1
- Expose actuator/health path without login HOT 1
- Selecting a Partition filter from a 'Stream' persists the partition as an enforced filter.
- Allow Filtering on Views Page HOT 3
- Consumer poll timeout is hardcoded
- How to Build this as a Single Jar/War file to deploy in server which doesn't have MVN/JDK HOT 2
- Disable server host name verification HOT 3
- "Unknown magic byte!" when deserializing avro message with TopicRecordNameStrategy HOT 2
- Mistake in environment variable name HOT 2
- Feature Request: Make `requestTimeoutMs` configurable per cluster HOT 3
- Security Update HOT 3
- [Docker] Execute web application fail while upgrade from v2.4.0 to 2.8.1 HOT 5
- Is webview vulnerable to Spring4Shell vulnerablity (CVE-2022-22965)? HOT 3
- unable to connect to AWS MSK clusters HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from kafka-webview.