Giter Club home page Giter Club logo

Comments (6)

horkhe avatar horkhe commented on August 28, 2024

What version are you using?

from kafka-pixy.

horkhe avatar horkhe commented on August 28, 2024

Have you acknowledged all consumed messages before stopping? Alternatively you could have used auto_ack=true when consuming. Before stopping kafka-pixy waits for acknowledgments of consumed messages for proxies.<proxy-name>.consumer.ack_timeout which is 5 minutes by default.

from kafka-pixy.

timbunce avatar timbunce commented on August 28, 2024

What version are you using?

v0.17.0

Before stopping kafka-pixy waits for acknowledgments of consumed messages for proxies..consumer.ack_timeout which is 5 minutes by default.

That seems like a long time for a process to wait after a SIGTERM (especially without logging an explanatory message).

I'd expect kafka-pixy to stop promptly, dropping the connection, and for the client's Ack to fail with a gRPC error so the client would know the Ack didn't succeed.

What does "before retrying" mean in this context:

      # Period of time that Kafka-Pixy should wait for an acknowledgement
      # before retrying.
      ack_timeout: 5m

from kafka-pixy.

horkhe avatar horkhe commented on August 28, 2024

That seems like a long time for a process to wait after a SIGTERM (especially without logging an explanatory message).

If kafka-pixy stops immediately that would result in unacknowledged messages to be offered to another client later, possibly by another kafka-pixy instance, hence it is advisable to set this parameter to be greater than the longest it takes to handle your messages.

What does "before retrying" mean in this context:

It means that if a message is not acknowledged in ack_timeout time, it will be offered by kafka-pixy again.

from kafka-pixy.

timbunce avatar timbunce commented on August 28, 2024

Thanks for the explanation. I agree it's working as designed.

My suggested changes are:

  • clarify the wording of the ack_timeout comments by explaining the effect, both on messages and on shutdown.
  • if a shutdown is blocked waiting on ack_timeout log a message to that effect.
  • add an config option to shutdown more aggressively and document the consequences.

from kafka-pixy.

horkhe avatar horkhe commented on August 28, 2024

I agree with the first two suggestions, however we will not implement the third, because a user have an option to auto ack consumed messages.

from kafka-pixy.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.