apache / pekko-connectors-kafka Goto Github PK

Hi folks,

Thanks for the work on maintaining the project. Recently I've had to migrate a project away from Akka due to the license change and the Pekko project was a life saver!

The latest package has Kafka 3.0.1 as dependency https://mvnrepository.com/artifact/org.apache.pekko/pekko-connectors-kafka_2.13/1.0.0
Are there plans to release a new package with support for Kafka >= 3.5?

• see use of sbt-source-dist in main pekko repo

The following innocent log.debug statement dumps the full contents of the consumer settings. If SASL is used it will also log the user and password in clear text. Secrets in the consumer or producer settings should be obfuscated. The same issue exists in Akka.

https://github.com/apache/incubator-pekko-connectors-kafka/blob/b83b10af1fa8d2afd23d9cd2143db8f2ed0807df/core/src/main/scala/org/apache/pekko/kafka/internal/KafkaConsumerActor.scala#L431

https://akka.io/security/alpakka-kafka-cve-2023-29471.html

• pekko-connectors-kafka is not released yet - so we will not require a CVE as long as we fix this before a release
• do not print the values associated with properties because they might have sensitive values (like passwords, usernames, etc)

I am using DrainingControl to toggle my kafka subscribe state.

public pause() {
	this.drainingControl
                .drainAndShutdown(system.executionContext())
                 .....
}

public start() {
        this.drainingControl = this.kafkaGraph.run(this.system);
}

when I call the pause() method, the actor system complains StopFromStage wasn't delivered to kafka-consumer-actor...

https://akka.io/alpakka-samples/ -- we don't have a pekko equivalent yet

• https://github.com/apache/incubator-pekko-connectors-samples needs work
• We might want to just change the link text to acknowledge that the alpakka-samples are useful for now (as they show the concepts and migrating to pekko shouldn't be hard)
• In a few weeks, we might have the pekko-samples anyway

This module has copied from from Apache Kafka.

• we need to update the LICENSE and NOTICE for this project to mention
• we may also need to update the source file headers where we have copies of Kafka code

Search for based on Apache Kafka's.

https://github.com/apache/incubator-pekko-connectors-kafka/blob/44d9bb307fbfbecaf87e4be5b9de28058bdae183/cluster-sharding/src/main/scala/org/apache/pekko/kafka/cluster/sharding/KafkaClusterSharding.scala#L55

• Keep the page but remove the detail pages about each release - they are Akka history
• Just add a link from main release notes page to the Akka release notes (making it clear it is Akka history)

For the time being CsvFormatting duplicated in PR#17

Duplicated class needs to be removed after migrating CsvFormatting to pekko

See comments in #98

• needed so people can review our releases (and generally useful too)
• one key issue is that users will need to have Kafka installed and running when running tests

• we've already implemented something in incubator-pekko repo

Some .md have links to https://github.com/apache/incubator-pekko-connectors-kafka/tree/main/ files

These files are copies of previous akka files but we have added license headers etc. So the line numbers in the links are likely to be wrong. Look for links with segments like #L169-L170

In #8 we removed the return function type because scalafmt has a bug with tuples as return function types in order to unblock the repo.

When a new version of scalafmt is out we should revert the change, just to be on the safe side to ensure that no important code has been changed from Akka latest Apache release.

See https://github.com/sbt/sbt/releases/tag/v1.9.0-RC3, specifically

Deprecation of IntegrationTest configuration
sbt 1.9.0 deprecates IntegrationTest configuration. (RFC-3 proposes to deprecate general use of configuration axis beyond Compile and Test, and this is the first installment of the change.)

The recommended migration path is to create a subproject named "integration", or "foo-integration" etc.

lazy val integration = (project in file("integration"))
  .dependsOn(core) // your current subproject
  .settings(
    publish / skip := true,
    // test dependencies
    libraryDependencies += something % Test,
  )

• remove release-train-issue-template.md - Lightbend specific - our release docs will probably end up on cwiki.apache.org
• remove create-release-issue.sh

Relate to: apache/pekko#1246

Need to fix these line: https://github.com/search?q=repo%3Aapache%2Fpekko-connectors-kafka%20ExternalShardAllocationStrategy&type=code

• https://github.com/apache/pekko-connectors-kafka/actions/runs/9352674736
• https://github.com/apache/pekko-connectors-kafka/actions/runs/9360144516

Integration tests were missing revision, and other tests unable to download dependency.

Some have been changed but there is still a lot left to do

Applying scalafmt is currently blocked by scalameta/scalafmt#3369

• affects our LICENSE/NOTICE files

This is constrained by a Confluent lib that we use.
https://github.com/apache/pekko-connectors-kafka/blob/main/project/Versions.scala#L40

That lib has dependencies on a 'provided' version of the kafka client that is a Confluent release, not the Apache one.

@mdedetrich I believe that you are a Kafka contributor. Do you have any thoughts on how we should approach this?

see apache/pekko#160

• follow up to #58
• that PR makes the code compile
• unfortunately something about the scala 3 generated classes upsets the Java jupiter test framework
• one option would be be to take the Java tests out of the 'tests' module and create a Java only 'java-tests' module
• we might still be able to run that java-tests module using sbt but we may be forced into using a Java build tool like maven or gradle

• reference.conf and application.conf files in this repo
• related scala/java code that uses these confs
• docs