confluentinc / ksql-images Goto Github PK

View Code? Open in Web Editor NEW

34.0 149.0 20.0 4.48 MB

KSQL platform docker images

License: Apache License 2.0

Shell 56.31% Python 36.99% Groovy 6.70%

ksql-images's Introduction

Docker images for KSQL

This repo provides build files for KSQL Docker images.

Properties

Properties are inherited from a top-level POM. Properties may be overridden on the command line (-Ddocker.registry=testing.example.com:8080/), or in a subproject's POM.

docker.skip-build: (Optional) Set to false to include Docker images as part of build. Default is 'false'.
docker.skip-test: (Optional) Set to false to include Docker image integration tests as part of the build. Requires Python 2.7, tox. Default is 'true'.
docker.registry: (Optional) Specify a registry other than placeholder/. Used as DOCKER_REGISTRY during docker build and testing. Trailing / is required. Defaults to placeholder/.
docker.tag: (Optional) Tag for built images. Used as DOCKER_TAG during docker build and testing. Defaults to the value of project.version.
docker.upstream-registry: (Optional) Registry to pull base images from. Trailing / is required. Used as DOCKER_UPSTREAM_REGISTRY during docker build. Defaults to the value of docker.registry.
docker.upstream-tag: (Optional) Use the given tag when pulling base images. Used as DOCKER_UPSTREAM_TAG during docker build. Defaults to the value of docker.tag.
docker.test-registry: (Optional) Registry to pull test dependency images from. Trailing / is required. Used as DOCKER_TEST_REGISTRY during testing. Defaults to the value of docker.upstream-registry.
docker.test-tag: (Optional) Use the given tag when pulling test dependency images. Used as DOCKER_TEST_TAG during testing. Defaults to the value of docker.upstream-tag.
docker.os_type: (Optional) Specify which operating system to use as the base image by using the Dockerfile with this extension. Valid values are ubi8. Default value is ubi8.

Building

This project uses maven-assembly-plugin and dockerfile-maven-plugin to build Docker images via Maven.

To build SNAPSHOT images, configure .m2/settings.xml for SNAPSHOT dependencies. These must be available at build time.

mvn clean package -Pdocker -DskipTests # Build local images

License

Usage of these images is subject to the license terms of the software contained within. Please refer to Confluent's Docker images documentation reference for further information. The software to extend and build the custom Docker images is available under the Apache 2.0 License.

ksql-images's People

Contributors

Stargazers

Watchers

Forkers

mikebin harinirajendran isabella232 global-localhost global19 global19-atlassian-net xli1996 pauloscarin-dev hercules261188 fritzss im-pratham saonam rharriso guilamb kaitsu4 lucasbru vihcle elicharlese alexrogalskiy

ksql-images's Issues

why this ksql image does not support TIMESTAMP,DATE and TIME data type?

When i am using confluentinc/ksqldb-server:0.23+ and it support timestamp data type.

But cp-ksqldb-server:6.1.0 is not supporting and i got

error resolve timestamp error

5.4.4 "Some problems were encountered while building the effective model for"

18:19:52 + cd ksql-images
18:19:52 + git checkout 5.4.4-post
18:19:52 Already on '5.4.4-post'
18:19:52 + mvn install:install-file -Dpackaging=pom -Dfile=pom.xml -DpomFile=pom.xml -Dmaven.test.skip=true
18:19:52 [INFO] Scanning for projects...
18:19:54 [WARNING]
18:19:54 [WARNING] Some problems were encountered while building the effective model for io.confluent.ksql:cp-ksql-cli:jar:5.4.4
18:19:54 [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: ${project.groupId}:ksql-functional-tests:jar -> duplicate declaration of version ${project.version} @ line 101, column 14
18:19:54 [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: ${project.groupId}:ksql-functional-tests:jar -> duplicate declaration of version ${project.version} @ line 107, column 21
18:19:54 [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.apache.kafka:kafka-clients:jar:test -> duplicate declaration of version ${kafka.version} @ line 113, column 21
18:19:54 [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.apache.kafka:kafka-streams-test-utils:jar -> duplicate declaration of version ${kafka.version} @ line 121, column 21
18:19:54 [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: io.confluent.ksql:ksql-metastore:test-jar -> duplicate declaration of version ${project.version} @ line 127, column 21
18:19:54 [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: io.confluent.ksql:ksql-engine:test-jar -> duplicate declaration of version ${project.version} @ line 134, column 21
18:19:54 [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: io.confluent.ksql:ksql-common:test-jar -> duplicate declaration of version ${project.version} @ line 141, column 21
18:19:54 [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.hamcrest:hamcrest-all:jar -> duplicate declaration of version 1.3 @ line 148, column 21
18:19:54 [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: junit:junit:jar -> duplicate declaration of version 4.12 @ line 155, column 21
18:19:54 [WARNING]
18:19:54 [WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
18:19:54 [WARNING]
18:19:54 [WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
18:19:54 [WARNING]

Vulnerability issue with 5.4.11-1 kafka images CVE-2023-4911 & CVE-2023-44487

The following image(5.4.11-1) is vulnerable to

Please provide a resolution. Can you please check if there are any insecure uses of glibc or nghttp2 packages?

Remove inconsistent, expensive, unprofessional-looking self-health-check function

PR #22, shortly prior to the CP5.4 release, added a new self-health-check function to the ksql-server image. This new function is invoked every 5 seconds by default (see https://github.com/confluentinc/ksql-images/blob/master/cp-ksql-server/Dockerfile.deb9#L41-L42) and works by posting a show topics; query to the running ksql server.
There are several problems with this:
1 - it is inconsistent with all the other CP docker images, with the possible exception of Connect which looks to have something similar added. This means that when you bring up a CP stack with docker-compose up and then check it's status with docker-compose up, only ksql-server (and ppossibly Connect) report their status as Up (healthy). This gives the misleading impression that the other platform components are NOT healthy - a check like this needs to be all or none of the images.
2 - running show topics; on a "real" cluster is potentially an expensive and time-consuming operation - certainly more than should be running every 5 seconds from every server.
3 - the "woohoo ksql is up!" message is less than professional looking
4 - the check runs too frequently (every 5 seconds by default) and spams the docker logs with verbose output at default logging levels
5 - this check is undocumented - you have to spelunk this private repo code to figure out how to adjust or disable it.

Example log output:

ksql-server        | [2020-01-17 01:46:29,759] INFO 127.0.0.1 - - [17/Jan/2020:01:46:29 +0000] "POST /ksql HTTP/1.1" 200 4000  81 (io.confluent.rest-utils.requests:62)
ksql-server        | [2020-01-17 01:46:34,982] INFO Received: KsqlRequest{ksql='SHOW TOPICS;', streamsProperties={}, commandSequenceNumber=Optional.empty} (io.confluent.ksql.rest.server.resources.KsqlResource:200)
ksql-server        | [2020-01-17 01:46:34,984] INFO AvroDataConfig values:
ksql-server        |    connect.meta.data = true
ksql-server        |    enhanced.avro.schema.support = false
ksql-server        |    schemas.cache.config = 1000
ksql-server        |  (io.confluent.connect.avro.AvroDataConfig:347)

ClassNotFoundException KafkaLog4jAppender

When using the cp-ksqldb-server image I am getting this message written to stdout:

log4j:ERROR Could not instantiate class [org.apache.kafka.log4jappender.KafkaLog4jAppender].
java.lang.ClassNotFoundException: org.apache.kafka.log4jappender.KafkaLog4jAppender
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:315)
at org.apache.log4j.helpers.Loader.loadClass(Loader.java:190)
at org.apache.log4j.helpers.OptionConverter.instantiateByClassName(OptionConverter.java:304)
at org.apache.log4j.helpers.OptionConverter.instantiateByKey(OptionConverter.java:123)
at org.apache.log4j.PropertyConfigurator.parseAppender(PropertyConfigurator.java:755)
at org.apache.log4j.PropertyConfigurator.parseCategory(PropertyConfigurator.java:738)
at org.apache.log4j.PropertyConfigurator.parseCatsAndRenderers(PropertyConfigurator.java:652)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:518)
at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:577)
at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:504)
at org.apache.log4j.LogManager.(LogManager.java:119)
at org.slf4j.impl.Reload4jLoggerFactory.(Reload4jLoggerFactory.java:67)
at org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:72)
at org.slf4j.impl.StaticLoggerBinder.(StaticLoggerBinder.java:45)
at org.slf4j.LoggerFactory.bind(LoggerFactory.java:150)
at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:124)
at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:417)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:362)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:388)
at io.confluent.admin.utils.cli.KafkaReadyCommand.(KafkaReadyCommand.java:53)
log4j:ERROR Could not instantiate appender named "kafka_appender".
SLF4J: A number (165) of logging calls during the initialization phase have been intercepted and are
SLF4J: now being replayed. These are subject to the filtering rules of the underlying logging system.
SLF4J: See also http://www.slf4j.org/codes.html#replay

I do see kafka-log4j-appender-7.4.1-ccs.jar in the /usr/share/java/ksqldb-server directory.

`ksql-tools.jar` not present in the `cp-ksqldb-server` image

Hi, I wanted to try the KSQL migration tool but looks like the confluentinc/cp-ksqldb-server image does not embed the ksqldb-tools jar...

By comparing the following pom's

https://github.com/confluentinc/ksql-images/blob/master/cp-ksqldb-server/pom.xml
https://github.com/confluentinc/ksql/blob/master/ksqldb-docker/pom.xml
I can see the ksqlsb-tools dependency is not present in the first but it is in the latter.

Version 6.1.0 has not been deployed to docker hub

I can only find version 6.0.0 on docker hub even though I think version 6.1.0 should be in docker hub.
https://hub.docker.com/r/confluentinc/cp-ksqldb-server/tags?page=1&ordering=last_updated&name=6

Am I doing something wrong?

ANTLR Version Mismatch

Cross posting this bug report here, at this might be the better place for it

confluentinc/ksql#8980

Describe the bug
After upgrading from confluentinc/cp-ksqldb-server:6.1.0 to confluentinc/cp-ksqldb-server:7.1.0 I've noticed warnings about ANTLR in the logs and after running SHOW CONNECTORS

ANTLR Tool version 4.7.1 used for code generation does not match the current runtime version 4.9.2
ANTLR Runtime version 4.7.1 used for parser compilation does not match the current runtime version 4.9.2

To Reproduce
Steps to reproduce the behavior, include:
We use Kubernetes for deployment with ArgoCD and a fork of the helm chart here.

Update chart to use image tag 7.1.0
Deploy

ksql> show connectors;
ANTLR Tool version 4.7.1 used for code generation does not match the current runtime version 4.9.2
ANTLR Runtime version 4.7.1 used for parser compilation does not match the current runtime version 4.9.2
ANTLR Tool version 4.7.1 used for code generation does not match the current runtime version 4.9.2
ANTLR Runtime version 4.7.1 used for parser compilation does not match the current runtime version 4.9.2

 Connector Name | Type | Class | Status 
----------------------------------------
----------------------------------------
ksql>

Expected behavior
No warnings. ANTLR versions should match

Actual behaviour
A clear and concise description of what actually happens, including:
See above

Additional context
Log example

[2022-04-05 21:23:46,591] INFO Adding function ComplexFunction for method public java.lang.Object io.confluent.ksql.function.UdfLoaderTest$ComplexUdf.foo(java.lang.String) (io.confluent.ksql.function.UdfLoader)
ANTLR Tool version 4.7.1 used for code generation does not match the current runtime version 4.9.2
ANTLR Runtime version 4.7.1 used for parser compilation does not match the current runtime version 4.9.2

This is a new image, so shouldn't be polluted with any changes we've made. This is a list off all the ANTLR JARs I found on the image. Only the bottom one is in the class path. I tried deleting it with rm, but that had no effect.

['./usr/share/doc/cp-ksqldb-server/licenses/LICENSE-antlr4-runtime-4.7.1.txt',
 './usr/share/java/confluent-security/connect/antlr-runtime-3.5.2.jar',
 './usr/share/java/confluent-security/connect/antlr4-4.9.2.jar',
 './usr/share/java/confluent-security/connect/antlr4-runtime-4.9.2.jar',
 './usr/share/java/confluent-security/kafka-rest/antlr-runtime-3.5.2.jar',
 './usr/share/java/confluent-security/kafka-rest/antlr4-4.9.2.jar',
 './usr/share/java/confluent-security/kafka-rest/antlr4-runtime-4.9.2.jar',
 './usr/share/java/confluent-security/ksql/antlr-runtime-3.5.2.jar',
 './usr/share/java/confluent-security/ksql/antlr4-4.9.2.jar',
 './usr/share/java/confluent-security/ksql/antlr4-runtime-4.9.2.jar',
 './usr/share/java/confluent-security/schema-registry/antlr-runtime-3.5.2.jar',
 './usr/share/java/confluent-security/schema-registry/antlr4-4.9.2.jar',
 './usr/share/java/confluent-security/schema-registry/antlr4-runtime-4.9.2.jar',
 './usr/share/java/ksqldb-server/antlr4-runtime-4.7.1.jar']

Unable to run on k8s with readOnlyRootFilesystem: true

This line and this line apparently make it impossible to run on k8s with readOnlyRootFilesystem: true

It would be better if the templates were in a different dir (perhaps a parent) so we can mount a writable dir at the target site; as-is, this forces a less-secure deployment.

I am thinking that instead of this

dub template "/etc/ksqldb/ksqldb-server.properties.template" "/etc/ksqldb/ksqldb-server.properties"

maybe something like this?

dub template "/etc/ksqldb/ksqldb-server.properties.template" "/etc/ksqldb/config/ksqldb-server.properties"

Provide linux/arm64 based container image for ksqlDB Server

First off, thank you for these wonderful container images.

When running the ksqlDB quickstart example locally on my MacBook Pro M1 (which has an arm64 processor), I experienced problems with performance, resulting in connection problems when trying to connect from the ksql-cli to the ksql-server. Also, queries/commands on the cli (when it does connect to the ksqlDB server) sometimes just freeze and if they execute, they take longer than necessary.

For clarification, in Docker Desktop it shows the orange exclamation mark with the ksqldb-cli image based containers:

I have identified that this is caused by the fact that for the ksqldb-server image no linux/arm64 version is available (in contrast to other images like images for kafka and zookeeper). Docker Desktop tries to 'help' by using QEMU under the hood to allow this image to 'run', which it does, but with a terrible performance.

Could you please add support for the ARM64 architecture for this ksqlDB server image? That would allow me and other Apple M1 users to use the quickstart in a local fashion as intended.