bobbyrauchenberg / avronaut Goto Github PK
View Code? Open in Web Editor NEWScala library for Avro schema generation, encoding and decoding
License: Apache License 2.0
Scala library for Avro schema generation, encoding and decoding
License: Apache License 2.0
Schema is currently working, but i think the code could be cleaned up in the instances by moving to an AST
Magnolia would generate an AST which was then parsed to a schema, keeping the instances simple and clean and also making things like shuffling schema elements, handling annotations etc... a lot cleaner
Currently no date formats are supported
Need to decide which we'll support as default and add instances for
schema
encoder
decoder
Schema tests are currently comparing Strings, this is fragile
On the whole we should be able to convert to json and compare json objects (this may not apply to, for example Union schemas, where we want to enforce field order)
When you give a default for a field in the schema creation, the value needs to be converted to a Java structure
Currently there is a case match on possible types to use for defaults in SchemaHelper.transformDefault
- it would be nice to make it possible for users to add their own default conversions - possibly in the shape of a map from DefaultType -> JavaType
(which in reality would probably be sth like Map[String, Any] => Any
But this could then be passed in, or added to some config object when schema generation happened
Currently all apis return Either[Error, T]
. This can translate nicely to cats Validation
instances, and given Magnolia iterates over all params we can build a very descriptive error report if a typeclass generation fails
A nice to have would be to be able to also have this in json format
We need schema / encoder / decoder instances for UUID
Currently Encoder and Decoder will run thru a load of extra steps each time they're called
Encoder generates an AvroSchema, and also translates to Avro
in the magnolia step. We should
ToRecord
, then pass this to the parser when we have the recordwill need to change this
def encode[A](a: A)(implicit encoder: Encoder[A], schema: AvroSchema[A]): Either[Error, GenericData.Record] =
for {
s <- schema.schema
encoded <- encoder.encode(a)
genRec <- encoded match {
case AvroRecord(schema, values) =>
Parser(new GenericData.Record(s)).parse(AvroRoot(schema, values))
case _ => Error(s"Can only encode records, got $encoded").asLeft
}
} yield genRec
similar with Decoder which currently does everything on each invocation, need to split this so that the interface is sth like
trait Decoder[A] {
def apply(operation: DecodeOperation): Result[A]
}
GenericRecord returns Any
when you lookup values. I didn't want to pollute the API passing Any
about. However given the way the typeclass derivation works, i wasn't able to write a Decoder
instance for, for example List
, without passing the Any
value from the GenericRecord
As a solution I have written a parser which traverses the schema passed by the reader, and pulls the relevant values from the generic record, this keeps Any
out of any public facing APIs
However because this step happens before Magnolia derives typeclass instances, I don't have access to default parameter values (as Magnolia provides these) at parse time. So the parser currently can fail to parse a field because there is no value in the provided GenericRecord
, but the overall process needs to still succeed, setting the field that parsed tp type ParseFail
, this means that when the Magnolia based Decoder
runs, ParseFails
are propogated and it is possible to provide a default value for those fields (if a default exists).
This feels wrong because:
ParseFail
has been added as an AvroType
to allow me to return Result[AvroType]
from Parser
It would be nice to find a cleaner way to do this.
Need schema / encoder / decoder instances for
NonEmptyList
NonEmptyMap
(maybe some others)
Currently blows up with stackoverflow
Worth looking at magnolia in more detail, as this should be handled by magnolia
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.