salsify / avro-patches Goto Github PK

Thanks for creating this gem! This repo has been very useful rather than waiting for upstream patches to make it into avro.

I understand that this patch might not be needed when upstream avro is > 1.8.2. However, is it possible to include AVRO-2001 as part of this gem? I put a lot of effort to document fields in my schemas and it's a shame that they get stripped off when they make it into my schema registry

As per Avro specification:
https://avro.apache.org/docs/1.8.2/spec.html#json_encoding

unions in json-format records should be encoded as a single object containing a single property where its name should be the tag-type refering to the union definition and its value being of that type.

Currently validate_union just iterates through all member types and uses the first that fits.

given the following schema

{
    "type":"record",
    "name":"ShouldSucceed",
    "namespace":"com.scigility.schemadrivenload",
    "fields":[
        {"name":"a","type":"int"},
        {"name":"b","type":"string"},
        {"name":"c","type":"boolean"},
        {"name":"d","type":["null","string"]}
    ]
}

this should be valid:

{"a":1,"b":"","c":true,"d":{"string":""}}

but leads to an error

at .d expected union of ['null', 'string'], got record with value {"string"=>""}

whereas

{"a":1,"b":"","c":true,"d":"asdf"}

should be rejected but isn't.

Hello,

I've encountered an issue with encoding and the timestamp-millis logical type - when we use that logical type, throughout our codebase we've already converted the underlying Time or DateTime value to a Numeric, so it fails when trying to call to_time on it when encoding: https://github.com/salsify/avro-patches/blob/master/lib/avro-patches/logical_types/logical_types.rb#L19

Would it be possible to add a guard in this encode method which just returns the value itself if it is already 'encoded'? Something like:

module Avro
  module LogicalTypes
    module TimestampMillis
      def self.encode(value)
        return value if value.kind_of?(Numeric)

        time = value.to_time
        time.to_i * 1000 + time.usec / 1000
      end
    end
  end
end

Cheers,
Stefan