confluentinc / cli Goto Github PK
View Code? Open in Web Editor NEWCLI for Confluent Cloud and Confluent Platform
Home Page: https://docs.confluent.io/confluent-cli/current/overview.html
License: Other
CLI for Confluent Cloud and Confluent Platform
Home Page: https://docs.confluent.io/confluent-cli/current/overview.html
License: Other
We use servers with Debian Linux with ARM64 bits on AWS and it would be great to have support for using Confluent with Linux ARM64
I am trying to use confluent cli against a onprem cluster, and the bootstrap option is not recognised.
Version: v3.1.1
Git Ref: 336dc3b
Build Date: 2023-02-10T00:08:09Z
Go Version: go1.20 X:boringcrypto (darwin/amd64)
Development: false
I'm seeing the following error when trying to update confluent locally on Apple M1 using macOS 13.3.1 (a) (Ventura):
➜ confluent update
Checking for updates...
New version of confluent is available
Current Version: v3.11.0
Latest Version: v3.12.0
[5/4/2023] Confluent CLI v3.12.0 Release Notes
==============================================
New Features
------------
- Add `--topics` flag to `confluent asyncapi export` to only export specified topics
Do you want to download and install this update? (y/n): y
Downloading confluent version 3.12.0...
Done. Downloaded 43.31 MB in 1 seconds. (32.41 MB/s)
Error: update client failure: error updating CLI binary: unable to copy /var/folders/0_/nkh8mcg92pq1fft8mrzcg98w0000gn/T/confluent3475877262/confluent-v3.12.0-darwin-arm64 to /usr/local/bin/confluent: remove /usr/local/bin/confluent: permission denied
Suggestions:
Please submit a support ticket.
In the meantime, see link for other ways to download the latest CLI version:
https://docs.confluent.io/current/cli/installing.html
However, I'm not seeing the issue Intel macOS 11.7.6 (Big Sur):
➜ confluent update
Checking for updates...
New version of confluent is available
Current Version: v3.11.0
Latest Version: v3.12.0
[5/4/2023] Confluent CLI v3.12.0 Release Notes
==============================================
New Features
------------
- Add `--topics` flag to `confluent asyncapi export` to only export specified topics
Do you want to download and install this update? (y/n): y
Downloading confluent version 3.12.0...
Done. Downloaded 45.67 MB in 3 seconds. (14.25 MB/s)
In short, one needs do sudo confluent update
instead confluent update
. Thus, I believe that the confluent
binary executable should invoke sudo
in such situations.
I try to run confluent login
within wsl
and it opens the browser within wsl.
The problem with this is that I cannot login with SSO, because I get an Azure error my device is unmanaged.
For other commands I solved this by setting BROWSER=wslview
, but it looks like confluent cli
does not take this environment variable into account.
The workaround is to use the --no-browser
flag, but it would be still nice if it can be fixed.
My second workaround is removing Firefox within wsl.
I think there may be a bug in Confluent CLI for Windows: I installed the last available version (file confluent_3.25.1_windows_amd64.zip), and It works ok for some orders (getting the list of topics from the cluster, creating a topic), so connectivity and credentials are ok, but I get an error trying to read from an existing topic with confluent kafka topic consume -b my-topic
:
%3|1691140847.189|FAIL|Confluent-CLI_v3.25.1#consumer-1| [thrd:sasl_ssl://pkc-1wvvj.westeurope.azure.confluent.cloud:9092/boot]: sasl_ssl://pkc-1wvvj.westeurope.azure.confluent.cloud:9092/bootstrap: SASL authentication error: Authentication failed (after 5283ms in state AUTH_REQ)
From Linux, I don't get this error.
Cannot produce integer keys with confluent cli version v3.30.1
Steps to reproduce:
Init the producer with integer key and string value
confluent kafka topic produce test.integers --key-format integer --value-format string --parse-key
Send a record:
1:Test
Init the consumer and receive a wrong key:
confluent kafka topic consume test.integers --key-format integer --value-format string --print-key --from-beginning
With the new confluent plugin
support it would be very useful to be able to extend some of the features found in the official cli by reusing some of your code. Unfortunately the cli hasn't been released as a Go module the last 3 years.
go list -versions -m github.com/confluentinc/cli
github.com/confluentinc/cli v0.1.0 v0.2.0 v0.3.0 v0.4.0 v0.5.0 v0.6.0 v0.7.0 v0.8.0 v0.9.0 v0.10.0 v0.11.0 v0.12.0 v0.13.0 v0.14.0 v0.15.0 v0.16.0 v0.17.0 v0.18.0 v0.19.0 v0.20.0 v0.21.0 v0.22.0 v0.23.0 v0.24.0 v0.25.0 v0.25.1 v0.25.2 v0.25.3 v0.26.0 v0.26.1 v0.27.0 v0.28.0 v0.29.0 v0.30.0 v0.31.0 v0.32.0 v0.33.0 v0.34.0 v0.35.0 v0.36.0 v0.37.0 v0.38.0 v0.39.0 v0.40.0 v0.41.0 v0.42.0 v0.43.0 v0.44.0 v0.45.0 v0.46.0 v0.47.0 v0.48.0 v0.49.0 v0.50.0 v0.51.0 v0.52.0 v0.53.0 v0.54.0 v0.55.0 v0.56.0 v0.57.0 v0.58.0 v0.59.0 v0.60.0 v0.61.0 v0.62.0 v0.63.0 v0.64.0 v0.65.0 v0.66.0 v0.67.0 v0.68.0 v0.69.0 v0.70.0 v0.71.0 v0.72.0 v0.73.0 v0.74.0 v0.75.0 v0.76.0 v0.77.0 v0.78.0 v0.79.0 v0.80.0 v0.81.0 v0.82.0 v0.83.0 v0.84.0 v0.85.0 v0.86.0 v0.87.0 v0.88.0 v0.89.0 v0.90.0 v0.91.0 v0.92.0 v0.93.0 v0.94.0 v0.95.0 v0.95.1 v0.96.0 v0.97.0 v0.98.0 v0.99.0 v0.100.0 v0.101.0 v0.102.0 v0.103.0 v0.104.0 v0.105.0 v0.106.0 v0.107.0 v0.108.0 v0.109.0 v0.110.0 v0.111.0 v0.112.0 v0.113.0 v0.114.0 v0.115.0 v0.116.0 v0.117.0 v0.118.0 v0.119.0 v0.120.0 v0.121.0 v0.122.0 v0.123.0 v0.124.0 v0.125.0 v0.126.0 v0.127.0 v0.128.0 v0.129.0 v0.130.0 v0.131.0 v0.132.0 v0.133.0 v0.134.0 v0.135.0 v0.136.0 v0.137.0 v0.138.0 v0.139.0 v0.140.0 v0.141.0 v0.142.0 v0.143.0 v0.144.0 v0.145.0 v0.146.0 v0.147.0 v0.148.0 v0.149.0 v0.150.0 v0.151.0 v0.152.0 v0.153.0 v0.154.0 v0.155.0 v0.156.0 v0.157.0 v0.158.0 v0.159.0 v0.160.0 v0.161.0 v0.162.0 v0.163.0 v0.164.0 v0.165.0 v0.166.0 v0.167.0 v0.168.0 v0.169.0 v0.170.0 v0.171.0 v0.172.0 v0.173.0 v0.174.0 v0.175.0 v0.176.0 v0.177.0 v0.178.0 v0.179.0 v0.180.0 v0.181.0 v0.182.0 v0.183.0 v0.184.0 v0.185.0 v0.186.0 v0.187.0 v0.188.0 v0.188.1 v0.189.0 v0.190.0 v0.191.0 v0.192.0 v0.193.0 v0.194.0 v0.195.0 v0.196.0 v0.197.0 v0.198.0 v0.199.0 v0.200.0 v0.201.0 v0.202.0 v0.203.0 v0.204.0 v0.205.0 v0.206.0 v0.207.0 v0.208.0 v0.209.0 v0.210.0 v0.211.0 v0.212.0 v0.213.0 v0.214.0 v0.215.0 v0.216.0 v0.217.0 v0.218.0 v0.219.0 v0.220.0 v0.221.0 v0.222.0 v0.223.0 v0.224.0 v0.225.0 v0.226.0 v0.227.0 v0.228.0 v0.229.0 v0.230.0 v0.231.0 v0.232.0 v0.233.0 v0.234.0 v0.235.0 v0.236.0 v0.237.0 v0.238.0 v0.239.0 v0.240.0 v0.241.0 v0.242.0 v0.243.0 v0.244.0 v0.245.0 v0.246.0 v0.247.0 v0.248.0 v0.249.0 v0.250.0 v0.251.0 v0.252.0 v0.253.0 v0.254.0 v0.255.0 v0.256.0 v0.257.0 v0.258.0 v0.259.0 v0.260.0 v0.261.0 v0.262.0 v0.263.0 v0.264.0 v0.265.0 v1.0.0 v1.1.0 v1.2.0 v1.3.0 v1.4.0 v1.5.0 v1.6.0 v1.7.0 v1.8.0 v1.9.0 v1.10.0 v1.11.0 v1.12.0 v1.12.1 v1.13.0 v1.13.1 v1.14.0 v1.14.1 v1.15.0 v1.16.0 v1.16.1 v1.16.2 v1.16.3 v1.16.4 v1.17.0 v1.18.0 v1.19.0 v1.19.1 v1.20.0 v1.20.1 v1.21.0 v1.21.1 v1.22.0 v1.22.1 v1.23.0 v1.24.0 v1.25.0 v1.26.0 v1.27.0 v1.28.0 v1.28.1 v1.29.0 v1.30.0 v1.31.0 v1.32.0 v1.33.0 v1.34.0 v1.35.0 v1.36.0 v1.37.0 v1.37.1 v1.38.0 v1.39.0 v1.39.1 v1.40.0 v1.40.1 v1.40.2 v1.41.0 v1.42.0 v1.43.0 v1.43.1 v1.43.2 v1.100.0
https://pkg.go.dev/github.com/confluentinc/cli
Published: Sep 15, 2020
Also for anyone to be able to use this code it would be good if the code was published with a well known OSS licence. Otherwise everyone needs to involve the legal department to scrutinize your special license.
There's an error when consuming a topic using api keys in the command
confluent kafka topic consume <topicName> --from-beginning --schema-registry-api-key <srApiKey> --schema-registry-api-secret <srApiSecret> --api-key <kafkaApiKey> --api-secret <kafkaApiSecret> --environment <env> --cluster <cluster>
Error: failed to create consumer: decryptbytes: procdecryptdata: The parameter is incorrect.
confluent - Confluent CLI
Version: v3.40.0
Git Ref: 299e722ef
Build Date: 2023-11-02T23:32:56Z
Go Version: go1.21.0 X:boringcrypto (windows/amd64)
Development: false
I'm trying to deploy the confluent-cli pod into an EKS cluster with OPA Gatekeeper. It has an OPA constraint that enforces readOnlyRootFilesystem
. Therefore, we must set the readOnlyRootFilesystem: true
to comply with the OPA constraint.
However, it prevents writing to the pod's /etc
folder. When we run the confluent login --no-browser
, it doesn't create the /etc/machine-id
.
Hence, we get the following error after a successful login.
cipher: message authentication failed
.
Is storing the machine-id
in the confluent user home (/home/confluent
) possible?
Currently I have a service account with a specific role and a cloud api key.
I would like to run a command (e.g. creating a role binding) by using this cloud api key.
confluent login
is only meant to login a user (with email+password or SSO) and confluent api-key store/use
is only meant for kafka clusters, if I'm not mistaken.
My current workaround would be to use the Confluent Cloud API I think, but it would be nice if we could use the confluent cli
as well. Did I miss something or is this not possible?
I have this shell script which uses the Confluent CLI to delete all subjects in a concurrent fashion:
#!/usr/bin/env bash
delete_subject() {
echo "Soft deleting subject $1"
confluent schema-registry schema delete --force --version all --subject "$1"
echo "Hard deleting subject $1"
confluent schema-registry schema delete --force --permanent --version all --subject "$1"
}
subjects=$(confluent schema-registry schema list -o json | jq -r ".[].subject")
# Perform deletes in parallel
for subject in ${subjects}; do
delete_subject "$subject" &
sleep 0.2
done
wait
echo "All subjects deleted."
It works most of the time, but I sometimes get these error messages:
Successfully soft deleted all versions for subject "insurance_customer_activity-value".
Version
-----------
1
Successfully soft deleted all versions for subject "payment_transaction-value".
Version
-----------
1
Hard deleting subject shoestore_clickstream-value
Successfully hard deleted all versions for subject "gaming_player-value".
Version
-----------
2
Hard deleting subject shoestore_shoe-value
Hard deleting subject fleetmgmt_location-value
Error: unable to read configuration file "/Users/gphilippart/.confluent/config.json": invalid character ']' after top-level value
Hard deleting subject pizzastore_order_cancelled-value
Hard deleting subject insurance_customer_activity-value
Error: unable to read configuration file "/Users/gphilippart/.confluent/config.json": invalid character ']' after top-level value
Error: unable to read configuration file "/Users/gphilippart/.confluent/config.json": invalid character ']' after top-level value
Error: unable to read configuration file "/Users/gphilippart/.confluent/config.json": invalid character ']' after top-level value
Hard deleting subject payment_transaction-value
What is probably happening is that another process reads the config.json file in the middle of the config write operation.
It would be best to write the config.json
changes in a temporary file and then renaming it, which is an atomic operation at the OS level.
Os ubuntu 22 64 bit
I am getting the below msg when I tried installing the latest
❯ curl -sL --http1.1 https://cnfl.io/cli | sh -s -- latest
confluentinc/cli info checking S3 for tag 'latest'- latest
confluentinc/cli crit unable to find 'latest' - use 'latest' or see https://docs.confluent.io/confluent-cli/current/release-notes.html for available versions.
Even installing 3.6.0 gives me the below error
❯ curl -sL https://raw.githubusercontent.com/confluentinc/cli/main/install.sh | sh -s -- -b /usr/local/bin 3.6.0
confluentinc/cli info checking S3 for tag '3.6.0'luentinc/cli/main/install.sh | sh -s -- -b /usr/local/bin 3.6.0
confluentinc/cli crit unable to find '3.6.0' - use 'latest' or see https://docs.confluent.io/confluent-cli/current/release-notes.html for available versions.
There's an error when consuming from a topic that seems related to a lack of snappy support:
confluent kafka topic consume --from-beginning --value-format avro
% Error: Local: Not implemented: Decompression (codec 0x2) of message at 152711835 of 165 bytes failed: Local: Not implemented
% Error: Local: Not implemented: Decompression (codec 0x2) of message at 152711835 of 165 bytes failed: Local: Not implemented
% Error: Local: Not implemented: Decompression (codec 0x2) of message at 152711835 of 165 bytes failed: Local: Not implemented
% Error: Local: Not implemented: Decompression (codec 0x2) of message at 152711835 of 165 bytes failed: Local: Not implemented
% Error: Local: Not implemented: Decompression (codec 0x2) of message at 152711835 of 165 bytes failed: Local: Not implemented
Version: v3.40.0
Git Ref: 299e722ef
Build Date: 2023-11-02T23:32:56Z
Go Version: go1.21.0 X:boringcrypto (windows/amd64)
Development: false
We find very useful this new feature, but I doesn't 100% in our use case. To be honest is a great idea, but maybe having and stand alone module to obtain the schema from asyncapi definition could be enough for us.
This is our use case :
1) A custom library is called that merge all files with the definitions, including both the common metadata for all company events and the specific payload for each project, and generates the schema to be published in the schema registry.
2) The schema is validated and compatibility (FORWARD_TRANSITIVE) is checked. If compatibility condition is not met the job fails
3) Once everything has been validated the schema is published in the schema registry
Being able to allow users to define their events only in AsyncAPI form instead of avro will make their job easier so we would like to be able to use the AsyncAPI CLI tool, having just the tool that converts from AsyncAPI definition into an schema as a stand alone module would also be quite helpful as we would be able to integrate that functionality into our process.
If it's not possible, maybe could be a good point to:
Please don't hesitate to ask anything,
Thanks
Hi again.
I'm seeing an odd error when attempting to have both a consumer and producer using the CLI running at the same time.
I start a consumer using the confluent kafka topic consume
command. I'm using an avro schema that is configured already on the topic via the Schema Registry in Confluent Cloud.
I think go to a new terminal to create a producer using the confluent kafka topic produce
command. I'm sending the raw Avro schema (.avsc file). I successfully produce the message (and I see it produced properly in Confluent Cloud).
However, the consumer
now breaks. I get the following error (which is from the golang stdlib)
Error: open /<temp-directory>/ccloud-schema/<topic-name>-<registered-schema-id>.txt: no such file or directory
Interestingly, if I restart the consumer and use the --from-begginning
option, I successfully get the message
I'm wondering if there is some attempt at an optimization to cache the registered schema on produce that doesn't jive properly when attempting to consume.
Below are the exact commands I'm using (env vars used for secrets)
# Consume
confluent kafka topic consume <topic> \
--bootstrap $(KAFKA_BOOTSTRAP_SERVER) \
--print-offset \
--delimiter "|" \
--value-format avro \
--api-key $(KAFKA_USERNAME) \
--api-secret $(KAFKA_PASSWORD) \
--schema-registry-endpoint $(SCHEMA_REGISTRY_URL) \
--schema-registry-api-key $(SCHEMA_REGISTRY_USER) \
--schema-registry-api-secret $(SCHEMA_REGISTRY_PASSWORD)
# Produce
confluent kafka topic produce <topic> \
--bootstrap $(KAFKA_BOOTSTRAP_SERVER) \
--value-format avro \
--schema "schemas/avro/avsc/value.avsc" \
--key-format avro \
--parse-key \
--delimiter "|" \
--key-schema "schemas/avro/avsc/key.avsc" \
--api-key $(KAFKA_USERNAME) \
--api-secret $(KAFKA_PASSWORD) \
--schema-registry-endpoint $(SCHEMA_REGISTRY_URL) \
--schema-registry-api-key $(SCHEMA_REGISTRY_USER) \
--schema-registry-api-secret $(SCHEMA_REGISTRY_PASSWORD)
I've started digging into the code myself and think there is something going on in the StoreSchemasReferences
(or related funcs) that store a reference to the schema, but not the schema itself.
I'm following the Kafka-Connect-101 (https://developer.confluent.io/courses/kafka-connect/connect-api-hands-on/) and I can't figure out what's wrong with the following:
Command 14 at tutorial:
confluent kafka client-config create java --sr-apikey <sr API key> --sr-apisecret <sr API secret> | tee $HOME/.confluent/java.config
The script is outdated since the documentation here has changed --sr-apikey
to full form and they are not consistent (https://docs.confluent.io/confluent-cli/current/command-reference/kafka/client-config/create/confluent_kafka_client-config_create_java.html)
I'm basically getting command not found for client-config, I couldn't find any information on whether the API has been moved since the documentation still says it's the same.
VERSION: confluent-7.1.1
3456253:~# confluent kafka client-config
Manage Apache Kafka.
Usage:
confluent kafka [command]
Available Commands:
acl Manage Kafka ACLs.
cluster Manage Kafka clusters.
link Manages inter-cluster links.
mirror Manages cluster linking mirror topics.
region Manage Confluent Cloud regions.
topic Manage Kafka topics.
Global Flags:
-h, --help Show help for this command.
-v, --verbose count Increase verbosity (-v for warn, -vv for info, -vvv for debug, -vvvv for trace).
Use "confluent kafka [command] --help" for more information about a command.
I have simple local installation docs.confluent.io/platform/current/installation/installing_cp/… and it working perfectly, but important parameters can not changed.
I use standard home path for confluent center
export CONFLUENT_HOME=/home/kafka/confluent/confluent-7.4.0
Unfortunately, I can not change path to store data. I found Kafka config in path and set up needed storage
cat /home/kafka/confluent/confluent-7.4.0/etc/kafka/server.properties | grep log.dir
log.dirs=/storage/kafka-logs
Than I restart Confluent Platform, however Path is wrong
./confluent local services start
The local commands are intended for a single-node development environment only, NOT for production usage. See more: https://docs.confluent.io/current/cli/index.html
As of Confluent Platform 8.0, Java 8 is no longer supported.
Using CONFLUENT_CURRENT: /tmp/confluent.430818
Starting ZooKeeper
ZooKeeper is [UP]
Kafka is [UP]
Starting Schema Registry
Schema Registry is [UP]
Starting Kafka REST
Kafka REST is [UP]
Starting Connect
Connect is [UP]
Starting ksqlDB Server
ksqlDB Server is [UP]
Starting Control Center
Control Center is [UP]
I try to use Clonfluent CLI to change path to store data, but not found this parameter. I try to use
./ confluent context list
in order to update and receive answer - Context None found.
Second possible to manage default setting
./confluent kafka acl list
answer is Error: Kafka REST URL not found, but in reality REST API found
If I try to use
./confluent kafka topic list
give me "No session token found, please enter user credentials. To avoid being prompted, run "confluent login". - login to what? Login to confluent cloud need credit card firstly, even to free plan. What this is trash? To change configuration on my local installation I must set up my credit card to Confluent company? Or what login means this command line?
Hi,
I tried building confluentinc/cli on FreeBSD 13.1-RELEASE-p3 and found this:
The Makefile
is GNU Make specific (not very unusual). The README.md
says to use make deps
and make build
and the Makefile
also calls make
. Replacing make
with gmake
in both the README.md
and Makefile
fixes that. The gmake
command is also available on Linux so this should make this more portable without requiring FreeBSD specific things.
For the cgo dependency for rdkafa
I needed to do a pkg install librdkafka
to get librdkafka 1.9.2.
Then to build I used gmake CGO_CFLAGS=`pkg-config --cflags rdkafka` CGO_LDFLAGS=`pkg-config --libs rdkafka` TAGS='dynamic' build
. Afterwards I noticed that building with gmake TAGS='dynamic' build
also works, but I'm not sure if that also works for fresh builds.
% go version -m ./dist/confluent_freebsd_amd64_v1/confluent | grep -vw dep
./dist/confluent_freebsd_amd64_v1/confluent: go1.19.5
path command-line-arguments
build -asmflags=all=-trimpath=/home/dveeden/git
build -compiler=gc
build -gcflags=all=-trimpath=/home/dveeden/git
build -ldflags="-s -w -X main.version=v3.0.0-dirty-dveeden -X main.commit=83072d47 -X main.date=2023-01-26T08:38:18Z"
build -tags=dynamic
build CGO_ENABLED=1
build CGO_CFLAGS=
build CGO_CPPFLAGS=
build CGO_CXXFLAGS=
build CGO_LDFLAGS=
build GOARCH=amd64
build GOOS=freebsd
build GOAMD64=v1
% file ./dist/confluent_freebsd_amd64_v1/confluent
./dist/confluent_freebsd_amd64_v1/confluent: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 13.1, FreeBSD-style, Go BuildID=nbtGuL4rOvzdWcchfu4k/FWCY7oI7U3W9T22Jk4pU/SjSsSKJdis-nENQXiNjp/mXRzoVc2_k0MVOozlSpq, stripped
% ./dist/confluent_freebsd_amd64_v1/confluent version
confluent - Confluent CLI
Version: v3.0.0-dirty-dveeden
Git Ref: 83072d47
Build Date: 2023-01-26T08:38:18Z
Go Version: go1.19.5 (freebsd/amd64)
Development: true
I recently just upgraded from 3.38 to 3.45.
Using the confluent kafka topic consume
command, when specifying details for Schema Registry API Key/secret and endpoint, I get a 401 with an API key I have been using before without error.
Concretely I get the following error output
Starting Kafka Consumer. Use Ctrl-C to exit.
Error: failed to validate Schema Registry client: 401 Unauthorized
I imagine the same bug exists for produce
as well.
When launching 'confluent local kafka start', I get this error:
confluent local -vvvv kafka start
The local commands are intended for a single-node development environment only, NOT for production usage. See more: https://docs.confluent.io/current/cli/index.html
Pulling from confluentinc/confluent-local
Digest: sha256:8e391de42cfcd3498e7317dcf159790f1f1cc3f3ffce900b30d7da23888687fd
Status: Image is up to date for confluentinc/confluent-local:latest
2023-09-19T10:04:18.214+0200 [TRACE] Successfully pulled Confluent Local image
Error: runtime error: index out of range [0] with length 0
Find below my CLI information
confluent - Confluent CLI
Version: v3.33.0
Git Ref: 53da89b2
Build Date: 2023-09-14T22:43:04Z
Go Version: go1.21.0 X:boringcrypto (darwin/amd64)
Development: false
Find below my docker environment information
Client:
Version: 24.0.6
Context: desktop-linux
Debug Mode: false
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 59
Server Version: 24.0.6
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
runc version: v1.1.8-0-g82f18fe
init version: de40ad0
Security Options:
seccomp
Profile: unconfined
cgroupns
Kernel Version: 6.3.13-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.769GiB
Name: docker-desktop
ID: 746e542e-f404-4449-b366-ba2a54e391a8
Docker Root Dir: /var/lib/docker
Debug Mode: false
HTTP Proxy: http.docker.internal:3128
HTTPS Proxy: http.docker.internal:3128
No Proxy: hubproxy.docker.internal
Experimental: false
Insecure Registries:
hubproxy.docker.internal:5555
127.0.0.0/8
Live Restore Enabled: false
WARNING: daemon is not using the default seccomp profile
this error is popping up on running this command
curl -sL --http1.1 https://cnfl.io/cli | sh -s -- latest
confluentinc/cli crit uname_arch_check 'i686' got converted to 'i686' which is not a GOARCH value.
There is an error when using contexts when consuming from a topic
confluent kafka topic <topicName> --context dev
Error: runtime error: invalid memory address or nil pointer dereference
CLI details:
confluent - Confluent CLI
Version: v3.40.0
Git Ref: 299e722ef
Build Date: 2023-11-02T23:32:56Z
Go Version: go1.21.0 X:boringcrypto (windows/amd64)
Development: false
As a security-aware developer,
I would like to get rid of API Keys and use the OAuth2 flow
to consume and produce messages from Confluent cloud.
This flow is already possible when using Confluent cloud and Java, see link above, bit not possible for SSO users when using the Confluent CLI. It would be a major security improvement.
This does not have to be a breaking change, it is just another possibility to authenticate.
Looks like it's possible to delete a service account in Confluent Cloud with ACLs still assigned to it. CLI doesn't throw an error in that case but also doesn't delete ACLs automatically (however, cluster API keys related to the service account ARE deleted automatically)
This leads to orphan ALCs left behind but still counted into cluster-type based ALCs limit.
It would be nice if those ACLs are also deleted automatically, just like API keys.
Hello,
I have a few general remarks and questions in regard to the docker version for what used to be confluent local services start
.
From my understanding of previous use, the bootstrap servers have been default available at localhost:9092. From the logged KafkaRestConfig I can see that: bootstrap.servers = broker:44535
. Is this documented somewhere or is it expected from the users that we manually check the KafkaRestConfig?
We have our local development environment running in Docker Containers, and needed to manually get the docker run
command and modify it with the --net
flag to facilitate communication. Is there any plan to support additional configuration to avoid this issue?
This enabled us to get a connection with the bootstrap servers at broker:44535, but we fail to get a response from the Schema Registry which we would expect to be available at http://localhost:8081
. Any idea why?
Additionally, we do not get any feedback in the terminal when running confluent local kafka start
. You are left with a blank terminal until the image is finished downloading, and I miss the printed output previously available as:
Starting Zookeeper Zookeeper is [UP] Starting Kafka Kafka is [UP] Starting Schema Registry Schema Registry is [UP] Starting Kafka REST Kafka REST is [UP] Starting Connect Connect is [UP] Starting KSQL Server KSQL Server is [UP] Starting Control Center Control Center is [UP]
And it sometimes fail to create a docker container when running "confluent local kafka start`, and multiple retries are needed without any console output.
I do also believe that this documentation: https://docs.confluent.io/platform/6.2/quickstart/ce-quickstart.html is outdated after the 3.16 release.
Thanks in advance!
BR
According to the confluent kafka topic produce
command --ca-location
is required for On Premise interactions.
Other confluent kafka topic <subcommand>
do not require this property.
It does not make sense, to me, to require this property for only one of these commands, especially since you can choose to establish connection via the --protocol
flag which allows for PLAINTEXT
(which is fine for local Docker containers).
Hi,
first of all I would like to thank you for implementing confluent asyncapi import
command. It's extremely useful!
Unfortunately, it looks like it doesn't handle $ref
too well. Check out the following example.
confluent asyncapi import --file asyncapi.yaml --overwrite --unsafe-trace
asyncapi: 2.6.0
info:
title: Example
version: 1.0.0
channels:
entitlements-v1:
x-messageCompatibility: NONE
subscribe:
operationId: EntitlementsV1Subscribe
message:
$ref: "#/components/messages/EntitlementsV1Message"
components:
messages:
EntitlementsV1Message:
name: EntitlementsV1Message
contentType: application/json
payload:
title: EntitlementsV1Message
type: object
required:
- uuid
- level
properties:
uuid:
type: string
format: uuid
level:
$ref: "#/components/schemas/EntityLevel"
schemas:
EntityLevel:
type: string
enum:
- team
- org
The result is as follows:
POST /subjects/entitlements-v1-value/versions HTTP/1.1
User-Agent: Confluent-CLI/v3.10.0 (https://confluent.io; [email protected])
Content-Length: 251
Accept: application/vnd.schemaregistry.v1+json,application/vnd.schemaregistry+json; qs=0.9,application/json; qs=0.5
Content-Type: application/json
Accept-Encoding: gzip
{"schemaType":"JSON","schema":"{\"properties\":{\"level\":{\"$ref\":\"#/components/schemas/EntityLevel\"},\"uuid\":{\"format\":\"uuid\",\"type\":\"string\"}},\"required\":[\"uuid\",\"level\"],\"title\":\"EntitlementsV1Message\",\"type\":\"object\"}"}
HTTP/2.0 422 Unprocessable Entity
Content-Length: 390
Content-Type: application/vnd.schemaregistry.v1+json
Date: Tue, 18 Apr 2023 11:34:24 GMT
{"error_code":42201,"message":"Invalid schema {subject=entitlements-v1-value,version=0,id=-1,schemaType=JSON,references=[],schema={\"properties\":{\"level\":{\"$ref\":\"#/components/schemas/EntityLevel\"},\"uuid\":{\"format\":\"uuid\",\"type\":\"string\"}},\"required\":[\"uuid\",\"level\"],\"title\":\"EntitlementsV1Message\",\"type\":\"object\"}}, details: #: key [components] not found"}
[WARN] unable to register schema: 422 Unprocessable Entity
As far as I can see, the problem is caused by the fact that the body of POST /subjects/entitlements-v1-value/versions
request contains message schema only - without components/schema
part.
Hello
We are looking for a deploy automation tool to import the topic and schema configuration via asycapi Spec to our on-prem Confluent Platform. The asyncapi import / export function is powerful which could help us to address our need. Howeve after studied a bit on the official Confluent Cli document, seems the async cli does not support on-prem yet.
May i know if there has any roadmap on this?
The command confluent local services start
is not working anymore. Is there anything else we are supposed to use?
Since V3 of the CLI has been released everything I try and execute hangs, even running -h does this. I have showed the full output below. I have also update the config with:
"version": "1.0.0",
"disable_update_check": true,
"disable_updates": true,
"disable_plugins": true,
output
[root@apptuv72fd ~]# /opt/app/software/confluent-7.3.1/bin/confluent login --url https://xxxxx.xxxx.test.group:8090 -vvvvv
2023-02-08T13:28:55.939Z [DEBUG] Did not find full credential set from environment variables
2023-02-08T13:28:55.940Z [DEBUG] Searching for netrc machine with filter: {IgnoreCert:true IsCloud:false IsSSO:false Name: URL:https://xxxx.xxxxx.cloud.test.group:8090}
2023-02-08T13:28:55.940Z [DEBUG] Get netrc machine error: open /root/.netrc: no such file or directory
I have a topic with a simple avro schemas for key and value
Key schema:
{
"fields": [
{
"name": "CustomerId",
"type": "int"
}
],
"name": "CustomerKey",
"type": "record"
}
Value schema:
{
"fields": [
{
"name": "FirstName",
"type": "string"
},
{
"name": "LastName",
"type": "string"
}
],
"name": "CustomerValue",
"type": "record"
}
When trying to produce to the topic, I get an error:
confluent kafka topic produce <topic> --key-format avro --value-format avro --schema <value-schema-id> --key-schema <key-schema-id> --parse-key
{"CustomerId": 1}:{ "FirstName": "Maria", "LastName": "Garcia"}
The error reads:
Error: cannot decode textual record ... : short buffer
Hello,
When I execute confluent local services start I am facing the error:
The local commands are intended for a single-node development environment only, NOT for production usage. See more: https://docs.confluent.io/current/cli/index.html As of Confluent Platform 8.0, Java 8 will no longer be supported.
Using CONFLUENT_CURRENT: /var/folders/65/yd73hltj4_lc72d20ppndnsm0000gn/T/confluent.957436
ZooKeeper is [UP]
Kafka is [UP]
Error: the Confluent CLI requires Java version 1.8 or 1.11.
See https://docs.confluent.io/current/installation/versions-interoperability.html .
If you have multiple versions of Java installed, you may need to set JAVA_HOME to the version you want Confluent to use.
I am using the following Java aversion:
openjdk version "17.0.9" 2023-10-17 OpenJDK Runtime Environment Homebrew (build 17.0.9+0) OpenJDK 64-Bit Server VM Homebrew (build 17.0.9+0, mixed mode, sharing)
My operational system is a MacBook Pro MacOS 13.6.4.
And at last I am downloaded conluent-7.6.0.
Please can you help me?
Kind regards
I have installed Confluent Platform 7.5.2 per the [Install Confluent Platform using Systemd on RHEL and CentOS](https://install/ Confluent Platform using Systemd on RHEL and CentOS) page. I’m installing on Amazon Linux 2023 AWS instance. I’ve completed all steps up to
“Test that you set the CONFLUENT_HOME variable correctly by running the confluent command: confluent --help”.
When I run confluent --help, I get the following error:
confluentinc/cli crit platform linux/arm64 is not supported. Make sure this script is up-to-date and file request at Sign in to GitHub · GitHub
As I just extracted the Confluent from the confluent-7.5.2.tar.gz tarball, I believe the script should be up-to-date. Can anyone provide some direction on how I might correct this situation?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.