karafka / wiki Goto Github PK

I think there will be some false positives as there is no code, but I think it would be still good to have it under control.

• Getting started
• New section on consumer perstistence (or explanation somewhere)
• Articles and references
• Auto reload of code changes in development
• CLI
• Concurrency and multithreading
• Configuration
• Consumer extra event based methods
• Consuming messages / consumers
• Deployment
• Error handling and backoff policy
• Events monitoring and logging
• Instrumentation
• Exit codes
• FAQ
• Responders references removed
• Rails integration docs
• Testing
• Consumer mappers (topics mappers removed)
• Serialization to Deserialization and serializer removal (as responders are removed)
• License and comm license
• ActiveJob + ActiveJob pro
• Virtual Partitions
• Long Running Jobs
• Remove backend references
• Routing
• Producing messages
• Setting up Kafka locally with Docker
• Testing
• Provide footer to the wiki via _Footer
• Run al the docs through Grammarly
• Check all the links in the Home.md after merging
• Once stable, generate TOC for sections that are complex

ref karafka/karafka#798

Coming to Karafka with only a very rudimentary understanding of Kafka, I was very confused at first on how to get the same topic to be processed by different consumers in the same application. (Our use case is wanting to reformat and deliver events to 3 different external services, 2 of which are being handled in the same Karafka app.)

I finally got the hint I needed in this FAQ: https://karafka.io/docs/FAQ/#what-are-consumer-groups-used-for
Unfortunately, it doesn't have a code example currently!

Do you think it would be beneficial to write an entire advanced doc on consumer groups, both in general for Kafka, as well as use in Karafka code and best practices recommendations? (for example: where do I put topic config when it appears multiple times in the routes?)

We have something like

class ApplicationConsumer < Karafka::BaseConsumer
  def self.on_unknown(message)
    …
  end

  def consume
    messages.each { |message| consume_single(message) }
  end

  def consume_single(message)
    return self.class.on_unknown(message) unless validate(message)

    MyApp::Metrics.with_context(**metrics_context(message)) do
      process(message)
    end
  end

with a bunch of code handling unknown messages, errors and adding some context for metrics and error reporting in AppicationConsumer and all other consumers inherited from this one.
In specs it's tested (or was tested in karafka-1) like

RSpec.describe ApplicationConsumer do
   let(:consumer) do
      Class.new(described_class) do
        def initialize; end # rubocop:disable Style/RedundantInitialize required to avoid exception with different arity
      end.new
    end
end

and with karafka-2 it fails because karafka-testing implies that consumer has topic set, for example here and here.
It could be workarounded by

before
  consumer_group = Karafka::Routing::ConsumerGroup.new('foo')  
  consumer.topic = Karafka::Routing::Topic.new('foo', consumer_group)            
end

but would be good to retain the ability to test topic-less consumers or at least simplify/document how consumer group should be created.

ref karafka/karafka#774

First of all: great gen. Love working with karafka.

I think I found an error on a rakafka-testing` example.

Apparently Karafka::Routing::Topic has a method #find.

Change suggestion:

-  let(:topic) { group.topics.find { |ts| ts.name == 'xml_data' } } 
+  let(:topic) { group.topics.find('xml_data' ) }