lensesio / kafka-connect-common Goto Github PK

NPE happens at the moment. See line 61

Kafka Version: kafka_2.11-2.1.1
export KAFKA_HEAP_OPTS="-Xms256M -Xmx2G"
Ubuntu Server Config = 16GB RAM, 30 GB HD

In connect-avro-distributed.properties console i was getting a following bug

MqttException (0) - java.lang.OutOfMemoryError: Java heap space
at org.eclipse.paho.client.mqttv3.internal.CommsCallback.run(CommsCallback.java:220)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.apache.avro.util.Utf8.setByteLength(Utf8.java:78)
at org.apache.avro.io.BinaryDecoder.readString(BinaryDecoder.java:261)
at org.apache.avro.io.ResolvingDecoder.readString(ResolvingDecoder.java:201)
at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:422)
at org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:414)
at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:181)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
at org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:232)
at org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:222)
at org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
at org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:145)
at com.datamountaineer.streamreactor.connect.converters.source.AvroConverter.convert(AvroConverter.scala:52)
at com.datamountaineer.streamreactor.connect.mqtt.source.MqttManager.messageArrived(MqttManager.scala:73)
at org.eclipse.paho.client.mqttv3.internal.CommsCallback.deliverMessage(CommsCallback.java:499)
at org.eclipse.paho.client.mqttv3.internal.CommsCallback.handleMessage(CommsCallback.java:402)
at org.eclipse.paho.client.mqttv3.internal.CommsCallback.run(CommsCallback.java:206)

should be here, clean up dependencies.

Found this issue using JsonSimpleConverter

java.lang.NoClassDefFoundError: com.datamountaineer.streamreactor.connect.converters.MsgKey$ (initialization failure)\n\tat java.base/java.lang.J9VMInternals.initializationAlreadyFailed(J9VMInternals.java:143)\n\tat com.datamountaineer.streamreactor.connect.converters.source.JsonSimpleConverter.convert(JsonSimpleConverter.scala:56)\n\tat com.datamountaineer.streamreactor.connect.jms.source.readers.JMSReader.convert(JMSReader.scala:55)\n\tat com.datamountaineer.streamreactor.connect.jms.source.readers.JMSReader$$anonfun$3$$anonfun$apply$2.apply(JMSReader.scala:47)\n\tat com.datamountaineer.streamreactor.connect.jms.source.readers.JMSReader$$anonfun$3$$anonfun$apply$2.apply(JMSReader.scala:47)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)\n\tat scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)\n\t

i guess the import statement should be
import com.datamountaineer.streamreactor.connect.converters.source.MsgKey
rather than
import com.datamountaineer.streamreactor.connect.converters.MsgKey

Allow JSON contain optional nulls.
JsonSimpleConverter set null values as Schema.STRING_SCHEMA

Hi,
we are using JsonConverterWithSchemaEvolution.
It is a great converter because it automatically solves issues related to malformed data sources (null values or float with no decimal digits).
The issue is that i does not support subtopics in the mqtt source address:

• "machineNamePLCNameSensorName" works
• "machineName/PLCName/SensorName" does not work

JsonSimpleConverter supports subtopics.
Thanks for support.

The kafka-connect-common version 0.8.2.7 in maven is a shadowJar and includes dependencies. It contains the com.google.common.* classes. These conflict with the Guava version used by Confluent 3.3 and cause the following error:

confluent-connect: Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.collect.Sets$SetView.iterator()Lcom/google/common/collect/UnmodifiableIterator;
confluent-connect: at org.reflections.Reflections.expandSuperTypes(Reflections.java:380)
confluent-connect: at org.reflections.Reflections.<init>(Reflections.java:126)
confluent-connect: at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanPluginPath(DelegatingClassLoader.java:221)
confluent-connect: at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.scanUrlsAndAddPlugins(DelegatingClassLoader.java:198)
confluent-connect: at org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader.initLoaders(DelegatingClassLoader.java:159)
confluent-connect: at org.apache.kafka.connect.runtime.isolation.Plugins.<init>(Plugins.java:47)
confluent-connect: at org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:63)
confluent-connect.service: main process exited, code=exited, status=1/FAILURE
Unit confluent-connect.service entered failed state.
confluent-connect.service failed.