Avro with Kafka Connect sink connector about pubsub HOT 8 OPEN

Can you explain a little more what you are trying to do? What is the format you want the data written in when sent to Pub/Sub? In what format is the data you've written to Kafka stored?

from pubsub.

Thanks @kamalaboulhosn for your help!

Yes, so I'm trying to sink data from Kafka topics that has been written using kafka-avro-serializer, so basically there are byte arrays within the topics, containing the schema ID registered within the Confluent Schema Registry then the payload of the event itself.

I would like to copy the data from those topics to Pub/Sub, in JSON target format for instance.

I would like the connector to read the data, deserialize it then convert it to json and send to PubSub topic.

That's what does for instance the GCS sink connector using the following confs

      "value.converter": "io.confluent.connect.avro.AvroConverter",
      "value.converter.schema.registry.url": "http://localhost:8081",

from pubsub.

I believe the notion of the schema registry is a concept specific to Confluent Kafka and not part of the generic, open-source Kafka connect infrastructure. The GCE connector you link to is provided by Confluent, whereas this one is not. At this time, we do not support any lookup of schema in a schema registry via this connector.

from pubsub.

Ok so for now the only way to read avro from kafka topics would be to provide avro file with the schema embedded within every kafka message?

from pubsub.

You can dump the Avro messages into the Cloud Pub/Sub as-is since they are just bytes. You'd then have to rely on your subscribers to decode the messages. If all messages on a topic use the same schema, then you could potentially take advantage of Pub/Sub's schema support.

from pubsub.

Actually we use Kafka topics with several schemas. For instance UserDeleted & UserCreated messages have different schemas but need to be stored in the same topic partition to ensure ordering.

from pubsub.

Yeah, so in that case, there is no way to convert the Avro into another format within the connector. You could store your schema in Pub/Sub and then manually attach the path to it as an attribute in your messages so that you can pull the schema and decode messages, though this would require your Kafka publisher to publish with the metadata in the headers.

from pubsub.