ozangunalp / kafka-native Goto Github PK
View Code? Open in Web Editor NEWKafka broker compiled to native using Quarkus and GraalVM.
License: Apache License 2.0
Kafka broker compiled to native using Quarkus and GraalVM.
License: Apache License 2.0
Hi Ozan!
I am working with @k-wall and we would like to discuss with you a topic we have in mind that is very useful for us.
We are very interested in having container images in quay for different versions of kafka-native, because it is important for us to demonstrate that our project works against many versions of the kafka broker.
Our test framework currently relies on your repo to provide an image containing the kafka native binary.
We need a mechanism to produce many different kafka-native images (one per kafka version). Perhaps the naming scheme could be:
quay.io/projectName/kafka-native-3.3.1:latest
quay.io/projectName/kafka-native-3.2.0:latest
For that we will appreciate a lot your experience and your work in this repo to help us with our topic. What do you suggest that could be the best approach for that?
Would it be possible to adapt the build to produce kafka and zookeeper native images for other architectures (s390x, ppc64le etc) in addition to the amd64.
quarkusio/quarkus#23281 (comment) gives me hope that support is in place.
I was also wondering if cross-compilation is possible is Quarkus/GraalVM? oracle/graal#407 makes me think 'no'.
Question came up from this https://github.com/orgs/strimzi/discussions/8801
The opinionated nature of BrokerConfig.defaultCoreConfig
sometimes gets in the way of some use-cases.
For instance, when bringing up a multi node kraft based cluster a configuration with 1 broker being in the controller,broker
and the second being only a broker
gives a substantial start-up saving over two brokers in the controller,broker
role (controller leader election avoided). It is currently impossible to configure a broker in the broker only role - defaultCoreConfig
turns it into a controller.
A simple way to avoid the problem is to allow the defaultCoreConfig
to be skipped by config property. I'd default this to false maintaining current behaviour. Alternatively, we could have a much richer set of flags, but to my mind, this would tend towards a re-expression of server.properties
file itself - which would be pointless.
What do you think @ozangunalp ?
Hi Ozan,
this is interesting, did you have time to perform some benchmarks that you could share?
Thanks,
Raffaele
With the last merge (pr #120) the integration tests have started to fail on CI.
https://github.com/ozangunalp/kafka-native/actions/runs/7196947318/job/19603932764#step:6:3963
Error: Tests run: 10, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 112.0 s <<< FAILURE! -- in com.ozangunalp.kafka.test.container.KafkaNativeContainerIT
Error: com.ozangunalp.kafka.test.container.KafkaNativeContainerIT.testKerberosContainer -- Time elapsed: 63.85 s <<< ERROR!
org.testcontainers.containers.ContainerLaunchException: Container startup failed for image quay.io/ogunalp/kafka-native:main-4a8689368ac5241e0ad9ac5f49cbe4ccd1f5c185
I'm trying to reproduce the issue in my local environment.
I think the container is failing with an issue like this:
(edit: removed misleading stack trace)
I haven't yet worked out why this system property is failing to be mapped to the bean.
[root@vm-3-20 ~]# ./kafka-server-amd64 -Dkafka.advertised.listeners=inside://:9093,outside://172.24.3.20:9092 -Dkafka.listeners=BROKER://:9093,PLAINTEXT://:9092,CONTROLLER://:9094 -Dkafka.inter.broker.listener.name=inside -Dkafka.controller.listener.names=CONTROLLER -Dkafka.listener.security.protocol.map=CONTROLLER:PLAINTEXT,inside:PLAINTEXT,outside:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL -Dkafka.inter.broker.listener.name=inside -Dkafka.process.roles=broker,controller
The error is as follows:
2023-06-20 17:04:26,601 WARN [com.oza.kaf.ser.BrokerConfig] (main) Broker configs controller.listener.names, listeners, inter.broker.listener.name, listener.security.protocol.map will not be configured automatically, make sure to provide necessary configuration manually.
2023-06-20 17:04:26,605 ERROR [io.qua.run.Application] (main) Failed to start application (with profile [prod]): java.lang.IllegalArgumentException: No security protocol defined for listener BROKER
I'd like to build native zookeeper server only, and I build it at CentOS 7.2009 successfully.
Although I had through process entirely experience, I follow that process at Windows 10 ran an error.
It's confused that my win10 actually is x64 architecture, while the error message show me currently only supports target architecture: AMD64.
systeminfo
Host Name: DESKTOP-MMR0SH0
OS Name: Microsoft Windows 10 专业版
OS Version: 10.0.19044 N/A Build 19044
OS Manufacturer: Microsoft Corporation
OS Configuration: Standalone Workstation
OS Build Type: Multiprocessor Free
Registered Owner: appadmin
Registered Organization:
Product ID: 00330-80000-00000-AA055
Original Install Date: 2023/7/5, 14:42:18
System Boot Time: 2023/7/12, 15:10:45
System Manufacturer: VMware, Inc.
System Model: VMware Virtual Platform
System Type: x64-based PC
Processor(s): 2 Processor(s) Installed.
[01]: Intel64 Family 6 Model 106 Stepping 6 GenuineIntel ~2594 Mhz
[02]: Intel64 Family 6 Model 106 Stepping 6 GenuineIntel ~2594 Mhz
BIOS Version: Phoenix Technologies LTD 6.00, 2020/11/12
Windows Directory: C:\Windows
System Directory: C:\Windows\system32
Boot Device: \Device\HarddiskVolume1
System Locale: en-us;English (United States)
Input Locale: zh-cn;Chinese (China)
Time Zone: (UTC+08:00) Beijing, Chongqing, Hong Kong, Urumqi
Total Physical Memory: 16,383 MB
Available Physical Memory: 12,328 MB
Virtual Memory: Max Size: 18,815 MB
Virtual Memory: Available: 14,810 MB
Virtual Memory: In Use: 4,005 MB
Page File Location(s): C:\pagefile.sys
Domain: WORKGROUP
Logon Server: \\DESKTOP-MMR0SH0
Hotfix(s): 6 Hotfix(s) Installed.
[01]: KB5027122
[02]: KB5003791
[03]: KB5027215
[04]: KB5014032
[05]: KB5014035
[06]: KB5026879
Network Card(s): 1 NIC(s) Installed.
[01]: Intel(R) 82574L Gigabit Network Connection
Connection Name: Ethernet0
DHCP Enabled: No
IP address(es)
[01]: 192.168.81.88
[02]: fe80::7be2:7223:9efa:b5ad
Hyper-V Requirements: A hypervisor has been detected. Features required for Hyper-V will not be displayed.
vcvars64
**********************************************************************
** Visual Studio 2022 Developer Command Prompt v17.6.4
** Copyright (c) 2022 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'
Graalvm native-image build error msg
========================================================================================================================
GraalVM Native Image: Generating 'zookeeper-server-999-SNAPSHOT-runner' (executable)...
========================================================================================================================
[1/7] Initializing... (0.0s @ 0.27GB)
Error: Native-image building on Windows currently only supports target architecture: AMD64 (? unsupported)
Error: To prevent native-toolchain checking provide command-line option -H:-CheckToolchain
com.oracle.svm.core.util.UserError$UserException: Native-image building on Windows currently only supports target architecture: AMD64 (? unsupported)
To prevent native-toolchain checking provide command-line option -H:-CheckToolchain
at org.graalvm.nativeimage.builder/com.oracle.svm.core.util.UserError.abort(UserError.java:139)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.c.codegen.CCompilerInvoker.addSkipCheckingInfo(CCompilerInvoker.java:104)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.c.codegen.CCompilerInvoker.verifyCompiler(CCompilerInvoker.java:95)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.setupNativeImage(NativeImageGenerator.java:911)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.doRun(NativeImageGenerator.java:575)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGenerator.run(NativeImageGenerator.java:535)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.buildImage(NativeImageGeneratorRunner.java:403)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.build(NativeImageGeneratorRunner.java:580)
at org.graalvm.nativeimage.builder/com.oracle.svm.hosted.NativeImageGeneratorRunner.main(NativeImageGeneratorRunner.java:128)
------------------------------------------------------------------------------------------------------------------------
0.9s (5.8% of total time) in 11 GCs | Peak RSS: 0.88GB | CPU load: 1.07
========================================================================================================================
Failed generating 'zookeeper-server-999-SNAPSHOT-runner' after 12.6s.
Error: Image build request failed with exit status 1
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:27 min
[INFO] Finished at: 2023-07-12T15:13:46+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal io.quarkus.platform:quarkus-maven-plugin:2.16.5.Final:build (default) on project zookeeper-server: Failed to build quarkus application: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
[ERROR] [error]: Build step io.quarkus.deployment.pkg.steps.NativeImageBuildStep#build threw an exception: io.quarkus.deployment.pkg.steps.NativeImageBuildStep$ImageGenerationFailureException: Image generation failed. Exit code: 1
[ERROR] at io.quarkus.deployment.pkg.steps.NativeImageBuildStep.imageGenerationFailed(NativeImageBuildStep.java:422)
[ERROR] at io.quarkus.deployment.pkg.steps.NativeImageBuildStep.build(NativeImageBuildStep.java:263)
[ERROR] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[ERROR] at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
[ERROR] at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[ERROR] at java.base/java.lang.reflect.Method.invoke(Method.java:568)
[ERROR] at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:909)
[ERROR] at io.quarkus.builder.BuildContext.run(BuildContext.java:281)
[ERROR] at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
[ERROR] at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2449)
[ERROR] at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1478)
[ERROR] at java.base/java.lang.Thread.run(Thread.java:833)
[ERROR] at org.jboss.threads.JBossThread.run(JBossThread.java:501)
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
Hi,
We are interested in using the native image to test a cluster with some controller-only nodes, so process.roles=controller
. We have:
broker,controller
controller
controller
Here's our server.properties
for one of the controller-only containers:
node.id=1
offsets.topic.num.partitions=1
metrics.jmx.exclude=.*
listeners=CONTROLLER\://0.0.0.0\:9091
transaction.state.log.replication.factor=1
group.initial.rebalance.delay.ms=0
offsets.topic.replication.factor=1
transaction.state.log.min.isr=1
controller.quorum.voters=0@//broker-0\:9091,1@//broker-1\:9091,2@//broker-2\:9091
controller.listener.names=CONTROLLER
process.roles=controller
broker.id=1
early.start.listeners=CONTROLLER
listener.security.protocol.map=CONTROLLER\:PLAINTEXT
When we start it up we see a failure like:
docker run -it --rm -p 19092:9091 \
-v $(pwd):/conf \
-e SERVER_PROPERTIES_FILE=/conf/server.properties \
quay.io/ogunalp/kafka-native:latest
2023-08-24 02:21:49,407 ERROR [io.qua.run.Application] (main) Failed to start application (with profile [prod]): java.lang.IllegalArgumentException: No security protocol defined for listener PLAINTEXT
at kafka.cluster.EndPoint$.$anonfun$createEndPoint$2(EndPoint.scala:49)
at scala.collection.immutable.Map$Map1.getOrElse(Map.scala:248)
at kafka.cluster.EndPoint$.securityProtocol$1(EndPoint.scala:49)
at kafka.cluster.EndPoint$.createEndPoint(EndPoint.scala:54)
at kafka.utils.CoreUtils$.$anonfun$listenerListToEndPoints$6(CoreUtils.scala:270)
at scala.collection.StrictOptimizedIterableOps.map(StrictOptimizedIterableOps.scala:100)
at scala.collection.StrictOptimizedIterableOps.map$(StrictOptimizedIterableOps.scala:87)
at scala.collection.mutable.ArraySeq.map(ArraySeq.scala:37)
at kafka.utils.CoreUtils$.listenerListToEndPoints(CoreUtils.scala:270)
at kafka.server.KafkaConfig.effectiveAdvertisedListeners(KafkaConfig.scala:2034)
at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:2148)
at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:2107)
at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1534)
at kafka.server.KafkaConfig.fromProps(KafkaConfig.scala:1457)
at com.ozangunalp.kafka.server.EmbeddedKafkaBroker.start(EmbeddedKafkaBroker.java:182)
at com.ozangunalp.kafka.server.Startup.startup(Startup.java:36)
at com.ozangunalp.kafka.server.Startup_Observer_startup_ba142133319fbb85a5c3c9a901d8261b72ab2d8b.notify(Unknown Source)
It looks like advertised.listeners is being defaulted here if we don't specify it, using PLAINTEXT as listener name by default. So it fails when it tries to lookup the security protocol.
I tried setting advertised listeners to match listeners
via the env var, and then get a failure:
docker run -it --rm -p 19092:9091 \
-v $(pwd):/conf \
-e KAFKA_ADVERTISED_LISTENERS="CONTROLLER://0.0.0.0:9091" \
-e SERVER_PROPERTIES_FILE=/conf/server.properties \
quay.io/ogunalp/kafka-native:latest
2023-08-24 02:23:05,459 ERROR [io.qua.run.Application] (main) Failed to start application (with profile [prod]): java.lang.IllegalArgumentException: requirement failed: The advertised.listeners config must be empty when process.roles=controller
at scala.Predef$.require(Predef.scala:337)
at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:2222)
at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:2107)
at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:1534)
at kafka.server.KafkaConfig.fromProps(KafkaConfig.scala:1457)
at com.ozangunalp.kafka.server.EmbeddedKafkaBroker.start(EmbeddedKafkaBroker.java:182)
at com.ozangunalp.kafka.server.Startup.startup(Startup.java:36)
at com.ozangunalp.kafka.server.Startup_Observer_startup_ba142133319fbb85a5c3c9a901d8261b72ab2d8b.notify(Unknown Source)
And I couldn't see a way to unset it, smallrye complains if you set the env var to an empty string.
docker run -it --rm -p 19092:9091 \
-v $(pwd):/conf \
-e KAFKA_ADVERTISED_LISTENERS="" \
-e SERVER_PROPERTIES_FILE=/conf/server.properties \
quay.io/ogunalp/kafka-native:latest
2023-08-24 02:10:12,680 ERROR [io.qua.run.Application] (main) Failed to start application (with profile [prod]): java.util.NoSuchElementException: SRCFG00040: The config property kafka.advertised.listeners is defined as the empty String ("") which the following Converter considered to be null: io.smallrye.config.Converters$BuiltInConverter
at io.smallrye.config.SmallRyeConfig.convertValue(SmallRyeConfig.java:300)
at io.smallrye.config.SmallRyeConfig.getValue(SmallRyeConfig.java:242)
at io.smallrye.config.SmallRyeConfig.getValue(SmallRyeConfig.java:168)
at com.ozangunalp.kafka.server.BrokerConfig.providedConfig(BrokerConfig.java:190)
So it appears advertised.listeners is incompatible with controller only mode, would it make sense to add a check here to make it not set up the default advertised.listeners
if the only role is controller
?
CI failed with this error. The error suggests that the brokers weren't able to find one and other.
I haven't been able to reproduce it so far.
but was: 1 within 30 seconds.
at org.awaitility.core.ConditionAwaiter.await(ConditionAwaiter.java:167)
at org.awaitility.core.AssertionCondition.await(AssertionCondition.java:119)
at org.awaitility.core.AssertionCondition.await(AssertionCondition.java:31)
at org.awaitility.core.ConditionFactory.until(ConditionFactory.java:985)
at org.awaitility.core.ConditionFactory.untilAsserted(ConditionFactory.java:769)
at com.ozangunalp.kafka.test.container.KafkaNativeContainerIT.verifyClusterMembers(KafkaNativeContainerIT.java:65)
at com.ozangunalp.kafka.test.container.KafkaNativeContainerIT.testKraftClusterBothControllers(KafkaNativeContainerIT.java:183)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
I just hit a segfault when using the new Kafka 3.7 image.
LogSegment was one of the areas changed by #148.
2024-02-28 10:34:18 Stacktrace for the failing thread 0x00007f7f00000b80:
2024-02-28 10:34:18 SP 0x00007f7f24ff8630 IP 0x0000000000dfe2f1 com.oracle.svm.core.code.CodeInfo@0x7f8019a6c8d8 name = image code
2024-02-28 10:34:18 SP 0x00007f7f24ff8650 IP 0x00000000023fae7e [image code] org.apache.kafka.storage.internals.log.OffsetIndex.append(OffsetIndex.java:151)
2024-02-28 10:34:18 SP 0x00007f7f24ff86b0 IP 0x00000000023e6597 [image code] org.apache.kafka.storage.internals.log.LogSegment.append(LogSegment.java:267)
2024-02-28 10:34:18 SP 0x00007f7f24ff8740 IP 0x000000000151e106 [image code] kafka.log.LocalLog.append(LocalLog.scala:410)
2024-02-28 10:34:18 SP 0x00007f7f24ff8780 IP 0x0000000001595c4d [image code] kafka.log.UnifiedLog.append(UnifiedLog.scala:906)
2024-02-28 10:34:18 SP 0x00007f7f24ff8890 IP 0x00000000015990ec [image code] kafka.log.UnifiedLog.appendAsLeader(UnifiedLog.scala:722)
2024-02-28 10:34:18 SP 0x00007f7f24ff88f0 IP 0x000000000134655b [image code] kafka.cluster.Partition.$anonfun$appendRecordsToLeader$1(Partition.scala:1364)
2024-02-28 10:34:18 SP 0x00007f7f24ff8970 IP 0x000000000135a676 [image code] kafka.cluster.Partition.appendRecordsToLeader(Partition.scala:1352)
2024-02-28 10:34:18 SP 0x00007f7f24ff89d0 IP 0x0000000001838ed0 [image code] kafka.server.ReplicaManager.$anonfun$appendToLocalLog$6(ReplicaManager.scala:1529)
2024-02-28 10:34:18 SP 0x00007f7f24ff8a80 IP 0x00000000018327c9 [image code] kafka.server.ReplicaManager$$Lambda$f4a93bcaae8878dda4af003cd9ae7d643a6b6b6f.apply(Unknown Source)
2024-02-28 10:34:18 SP 0x00007f7f24ff8aa0 IP 0x0000000002d32b73 [image code] scala.collection.StrictOptimizedMapOps.map(StrictOptimizedMapOps.scala:28)
2024-02-28 10:34:18 SP 0x00007f7f24ff8b00 IP 0x00000000026b0753 [image code] scala.collection.StrictOptimizedMapOps.map$(StrictOptimizedMapOps.scala:27)
2024-02-28 10:34:18 SP 0x00007f7f24ff8b00 IP 0x00000000026b0753 [image code] scala.collection.mutable.HashMap.map(HashMap.scala:35)
2024-02-28 10:34:18 SP 0x00007f7f24ff8b10 IP 0x00000000018619f1 [image code] kafka.server.ReplicaManager.appendToLocalLog(ReplicaManager.scala:1515)
2024-02-28 10:34:18 SP 0x00007f7f24ff8b60 IP 0x0000000001860c0b [image code] kafka.server.ReplicaManager.appendRecords(ReplicaManager.scala:860)
2024-02-28 10:34:18 SP 0x00007f7f24ff8c40 IP 0x00000000017a32d3 [image code] kafka.server.KafkaApis.handleProduceRequest(KafkaApis.scala:720)
2024-02-28 10:34:18 SP 0x00007f7f24ff8d20 IP 0x0000000001784dfd [image code] kafka.server.KafkaApis.handle(KafkaApis.scala:184)
2024-02-28 10:34:18 SP 0x00007f7f24ff8d80 IP 0x00000000017dcac1 [image code] kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:160)
2024-02-28 10:34:18 SP 0x00007f7f24ff8e40 IP 0x0000000000c848b4 [image code] java.lang.Thread.run(Thread.java:833)
2024-02-28 10:34:18 SP 0x00007f7f24ff8e50 IP 0x000000000083982d [image code] com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:775)
2024-02-28 10:34:18 SP 0x00007f7f24ff8e80 IP 0x0000000000803827 [image code] com.oracle.svm.core.posix.thread.PosixPlatformThreads.pthreadStartRoutine(PosixPlatformThreads.java:203)
2024-02-28 10:34:18 SP 0x00007f7f24ff8ea0 IP 0x00000000007592bf [image code] com.oracle.svm.core.code.IsolateEnterStub.PosixPlatformThreads_pthreadStartRoutine_cf47bd191b082a3631657d369b9255c828b9a95c(IsolateEnterStub.java:0)
2024-02-28 10:34:18
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.