Comments (7)
@hhromic In general that is correct.
from avro.
Its not that you cannot generate the schema from objects, but that the schema is the source of truth for encoding and decoding data in Avro. There are of course use cases where the objects make up a central definition of data exchange (a good example of this is Kubernetes) but this is not the general use case. And even in these cases, the object is only used when working in the same language as API Server. For example, were Kubernetes to support Avro in its API, the object is only useful in Go, in any other language I must depend instead of the fixed Schema generated from those objects.
from avro.
👍 thanks @nrwiersma @hhromic
from avro.
What is the question/proposal here?
from avro.
I thinks this is a duplicate of #263.
@longbai , here is my advise after our own experience.
We tried to think in this way in the past and realized this is not the correct way to think about it.
With Avro, you should always start with the schema first. Then, for convenience, you can generate Go structs from that schema so you can create typed objects to serialize/deserialize for that schema in particular. But this is technically optional as you can also serialize/deserialize to/from generic maps as well without Go structs.
If using structs, your program should always have a copy of the schema in the binary to use for serialization/deserialization (that is, you need a parsed schema object). If using generic maps, you can have the schemas external to the binary, e.g. from a schema registry or on disk and read them in runtime.
In our experience, an easy way to manage this in Go is to use the go embed tooling. In this way you can have the schema on disk and make sure the binary always contains the correct schema for which it was designed to work.
The biggest reason to embed the schema into the binary, is that it does not make much sense to change the schema and not the binary. If you change the schema, is highly likely that you have to change something in your program as well. In other words, Avro schemas are highly coupled to the program using them.
@nrwiersma please correct me if the above is still not the right way to think about it. This is the way we are using this great library with lot of success in our software components.
from avro.
In my current project, I chose to use Avro mainly because of its flexibility regarding schema generation.
I tend to disagree and consider Avro a friendly format when following a code-first approach compared to other formats such as Protocol buffer (because of the absence of tag numbers)
The Encoding and Evolution chapter in the Designing Data-intensive applications book (by Martin Kleppmann) highlights this point.
from avro.
I have a code to dynamic generate a go Struct from a avro.Schema. I use this to a code to read and write avro with schema input only.
avroSchema2Struct.go
Let me know if it can help you.
from avro.
Related Issues (20)
- [BUG] array schema cannot be correctly parsed HOT 4
- Local timestamp logical types HOT 1
- Add support for Zstandard compression
- Enum schema evolution for missing value in reader schema but with default HOT 3
- Question about max byte slice HOT 6
- Performance degradation in v2.19.0 HOT 4
- Wrong decoding of nested map HOT 1
- Infinite loop parsing recursive array type HOT 1
- "unknown union type long" error HOT 2
- Support nested array of record HOT 3
- Bug with encoding union + fixed + decimal HOT 6
- Problem with empty slices vs nil. HOT 8
- Decode Array - Panic: Allocation size out of range Error HOT 1
- I struggle to have more than one "registered" type in the same field of a schema HOT 9
- Reasoning behind swallowing EOF errors? HOT 2
- Decoding of map[string]any behavior changed HOT 3
- [avrogen] tags - having omitEmpty in the json generated object HOT 8
- [avrogen] Avro int32s are mapped to Go ints HOT 2
- Serialize schemas feature HOT 3
- SchemaCompatibility.Compatible has one differ from apache/avro java implementation HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from avro.