Giter Club home page Giter Club logo

Comments (10)

ozangunalp avatar ozangunalp commented on June 2, 2024

Looking at the stacktrace it happens accessing the MappedByteBuffer

from kafka-native.

k-wall avatar k-wall commented on June 2, 2024

I was running a kafka-producer-perf-test.sh based test at the time with 10,000,000 x 1024 byte records. I ran this test successfully against the 3.6.1 image. I'm just seeing if the problem is reproducible.

edit: yes - it is reproducible!

from kafka-native.

ozangunalp avatar ozangunalp commented on June 2, 2024

With which --throughput ?

from kafka-native.

ozangunalp avatar ozangunalp commented on June 2, 2024

We may want to include such a test to a CI job

from kafka-native.

ozangunalp avatar ozangunalp commented on June 2, 2024

@k-wall we can also test the #150 against your scenario if it fixes the issue

from kafka-native.

k-wall avatar k-wall commented on June 2, 2024

like this:

 bin/kafka-producer-perf-test.sh --topic ${TOPIC} --throughput -1 --num-records ${NUM_RECORDS} --record-size ${RECORD_SIZE} \
      --producer-props acks=all bootstrap.servers=${ENDPOINT}

from kafka-native.

k-wall avatar k-wall commented on June 2, 2024

Looking at the stacktrace it happens accessing the MappedByteBuffer

I don't see changes in the way that MappedByteBuffer is handled in 3.7.0

@k-wall we can also test the #150 against your scenario if it fixes the issue

Yes, I'll do that.

from kafka-native.

ozangunalp avatar ozangunalp commented on June 2, 2024

I couldn't reproduce it in the first run. Running a single node on podman.

from kafka-native.

k-wall avatar k-wall commented on June 2, 2024

The container is running out of diskspace. The same behaviour happens on the old image 3.6.1 if I increase the number of messages. It seems the disk utilisation is slightly different.

This will reproduce it:

version: '3.5'

name: kafka-native
services:
  broker1:
    storage_opt:
      size: '1G'
    image: quay.io/ogunalp/kafka-native:latest-kafka-3.6.1
    ports:
      - "9095:9092"
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9095
      SERVER_HOST: broker1
      KAFKA_BROKER_ID: 1
      SERVER_CLUSTER_ID: BmMQZPq6Qoua1-yvgGJFbA
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@broker1:9094
    networks:
      - kafka-network
networks:
  kafka-network:
    name: kafka-native
docker run --network kafka-native --rm quay.io/strimzi/kafka:0.39.0-kafka-3.6.1 bin/kafka-producer-perf-test.sh  --topic foo --throughput -1 --num-records 20000000 --record-size 1024   --producer-props acks=all bootstrap.servers=broker1:9093

I would expect an exception to bring the broker down when the disk space fills up, so it is not clear why it isn't happening in this case. I would expect an IOException to be logged too.

I'll have a dig soon.

from kafka-native.

k-wall avatar k-wall commented on June 2, 2024

Odd, I'm now failing to reproduce the issue at all. The VM of my Docker had almost run up against its virtual disk limit, so I'm wonder if that had an effect on things? I've since cleanup up my Docker environment.

Trying now with a constrained kafka.log.dir shows no sign of a general problem:

version: '3.5'

name: kafka-native
services:
  broker1:
    storage_opt:
      size: '1G'
    image: quay.io/ogunalp/kafka-native:latest-kafka-3.7.0
    ports:
      - "9095:9092"
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9095
      KAFKA_LOG_DIR: /tiny
      SERVER_HOST: broker1
      KAFKA_BROKER_ID: 1
      SERVER_CLUSTER_ID: BmMQZPq6Qoua1-yvgGJFbA
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@broker1:9094
    networks:
      - kafka-network
    volumes:
    - type: tmpfs
      target: /tiny
      tmpfs:
        size: "3221225472"
networks:
  kafka-network:
    name: kafka-native

It fails in the expected way when the disk fills up:

kafka-native-broker1-1  | 2024-03-08 16:42:33,730 ERROR [org.apa.kaf.sto.int.log.LogDirFailureChannel] (data-plane-kafka-request-handler-6) Error while appending records to foo-0 in dir /tiny: java.io.IOException: No space left on device
kafka-native-broker1-1  | 	at [email protected]/sun.nio.ch.FileDispatcherImpl.write0(FileDispatcherImpl.java)
kafka-native-broker1-1  | 	at [email protected]/sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:62)
kafka-native-broker1-1  | 	at [email protected]/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:132)
kafka-native-broker1-1  | 	at [email protected]/sun.nio.ch.IOUtil.write(IOUtil.java:97)
kafka-native-broker1-1  | 	at [email protected]/sun.nio.ch.IOUtil.write(IOUtil.java:67)
kafka-native-broker1-1  | 	at [email protected]/sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:288)
kafka-native-broker1-1  | 	at org.apache.kafka.common.record.MemoryRecords.writeFullyTo(MemoryRecords.java:90)
kafka-native-broker1-1  | 	at org.apache.kafka.common.record.FileRecords.append(FileRecords.java:188)
kafka-native-broker1-1  | 	at org.apache.kafka.storage.internals.log.LogSegment.append(LogSegment.java:259)
kafka-native-broker1-1  | 	at kafka.log.LocalLog.append(LocalLog.scala:410)
kafka-native-broker1-1  | 	at kafka.log.UnifiedLog.append(UnifiedLog.scala:906)
kafka-native-broker1-1  | 	at kafka.log.UnifiedLog.appendAsLeader(UnifiedLog.scala:722)
kafka-native-broker1-1  | 	at kafka.cluster.Partition.$anonfun$appendRecordsToLeader$1(Partition.scala:1364)
kafka-native-broker1-1  | 	at kafka.cluster.Partition.appendRecordsToLeader(Partition.scala:1352)
kafka-native-broker1-1  | 	at kafka.server.ReplicaManager.$anonfun$appendToLocalLog$6(ReplicaManager.scala:1529)
kafka-native-broker1-1  | 	at scala.collection.StrictOptimizedMapOps.map(StrictOptimizedMapOps.scala:28)
kafka-native-broker1-1  | 	at scala.collection.StrictOptimizedMapOps.map$(StrictOptimizedMapOps.scala:27)
kafka-native-broker1-1  | 	at scala.collection.mutable.HashMap.map(HashMap.scala:35)
kafka-native-broker1-1  | 	at kafka.server.ReplicaManager.appendToLocalLog(ReplicaManager.scala:1515)
kafka-native-broker1-1  | 	at kafka.server.ReplicaManager.appendRecords(ReplicaManager.scala:860)
kafka-native-broker1-1  | 	at kafka.server.KafkaApis.handleProduceRequest(KafkaApis.scala:720)
kafka-native-broker1-1  | 	at kafka.server.KafkaApis.handle(KafkaApis.scala:184)
kafka-native-broker1-1  | 	at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:160)
kafka-native-broker1-1  | 	at [email protected]/java.lang.Thread.run(Thread.java:833)
kafka-native-broker1-1  | 	at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:775)
kafka-native-broker1-1  | 	at org.graalvm.nativeimage.builder/com.oracle.svm.core.posix.thread.PosixPlatformThreads.pthreadStartRoutine(PosixPlatformThreads.java:203)

I'll close this issue.

from kafka-native.

Related Issues (13)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.