Comments (1)
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
name: kafka-connect
annotations:
strimzi.io/use-connector-resources: "true"
spec:
version: 3.0.0
replicas: 1
bootstrapServers: kafka1-cluster-kafka-bootstrap:9093
tls:
trustedCertificates:
- secretName: kafka1-cluster-cluster-ca-cert
certificate: ca.crt
authentication:
type: tls
certificateAndKey:
secretName: kafka-int-tls
certificate: user.crt
key: user.key
config:
group.id: connect-cluster
offset.storage.topic: connect-cluster-offsets
config.storage.topic: connect-cluster-configs
status.storage.topic: connect-cluster-status
key.converter: org.apache.kafka.connect.json.JsonConverter
value.converter: org.apache.kafka.connect.json.JsonConverter
# key.converter.schemas.enable: true
# value.converter.schemas.enable: true
config.providers: directory
config.providers.directory.class: org.apache.kafka.common.config.provider.DirectoryConfigProvider
build:
output:
type: docker
image: asia.gcr.io/my-gcp-project-name/kafka-connect:latest # will auto create and push the image to gcr with this name
pushSecret: docker-credentials
plugins:
- name: gcp-bigquery
artifacts:
- type: jar
url: https://github.com/GoogleCloudPlatform/pubsub/releases/download/v0.11-alpha/pubsub-kafka-connector.jar
sha512sum: ee07f6334f5ea77d39689b0e764aaf32f9324ad90811bfe41c5039cd212bbd715e913bc7c9a8c3157b0f91726e1b2aba6b7e53bfacb5e558eeab870c6dc6f60e
---
# gcloud auth print-access-token | docker login -u oauth2accesstoken --password-stdin https://asia.gcr.io
# OR
# cat keyfile.json | docker login -u _json_key --password-stdin https://asia.gcr.io
# kubectl create secret generic docker-credentials --from-file=.dockerconfigjson=~/.docker/config.json --type=kubernetes.io/dockerconfigjson
apiVersion: v1
kind: Secret
metadata:
name: docker-credentials
labels:
app: connect
namespace: kafka
type: kubernetes.io/dockerconfigjson
data:
.dockerconfigjson: ewogICJ0eXBlIjogInNlcnZpY2Vf # your docker config file in base64 encoded, I have given two commads above to generate this secret. Srimzi Doc link: https://strimzi.io/docs/operators/latest/deploying.html#creating-new-image-using-kafka-connect-build-str
Strimzi kafka connector resource
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
name: source-pubsub
labels:
strimzi.io/cluster: kafka-connect
spec:
class: com.google.pubsub.kafka.source.CloudPubSubSourceConnector
tasksMax: 2
config:
cps.project: my-gcp-project-name
cps.subscription: test-topic-sub # pubsub subscription name
kafka.topic: topic2 # kafka topic name where messages will be consumed
Now when i consume messages from kafka cluster they look like this.
$ ./bin/kafka-console-consumer.sh --bootstrap-server kafka1-cluster-kafka-bootstrap:9097 --topic topic2 --from-beginning --consumer.config /tmp/config.properties
{"schema":{"type":"struct","fields":[{"type":"bytes","optional":false,"field":"message"},{"type":"string","optional":false,"field":"_stream"},{"type":"string","optional":false,"field":"_namespace"}],"optional":false},"payload":{"message":"eyJfYWlyYnl0ZV9hYl9pZCI6IjA1YjA3MzgxLWE2NmEtNDMwOC05Mjg0LWU1ZTBiZjk4NmYxNyIsIl9haXJieXRlX2RhdGEiOnsic2Vzc2lvbl9kYXRlIjoiMjAyMS0xMi0xM1QwMDowMDowMFoiLCJjb3VudHJ5IjoiSW5kaWEiLCJjaXR5IjoiSmFpcHVyIiwic2Vzc2lvbl9jb3VudCI6MzI2NTM3fSwiX2FpcmJ5dGVfZW1pdHRlZF9hdCI6MTY0MzIxNDY5NzU1NX0","_stream":"pb_test_table","_namespace":"behaviour_us"}}
Next challenge is to decode the base64 encoded payload received from pubsub OR is there any way to directly receive base64 decoded json payload.
On the plus note - Able to receive messages in kafka from pubsub.
from pubsub.
Related Issues (20)
- Issue running copy_tool.py for kafka connector in a single-machine HOT 2
- Connectors failing with NoClassDefFoundError on CDP clusters
- GCEComputeResourceController creates VM incompatible default Python version
- CMake build got broken in one of the latest versions (vcpkg) HOT 1
- netty dependency is too old to compile on aarch64 HOT 1
- Avro with Kafka Connect sink connector HOT 8
- Pub sub connector not working with 2.8.1
- PubSub Sink Task Flush issues with clearing partition tracking.
- Receiving base64 encoded messages in kafka from pubsub HOT 1
- A potential Denial of Service issue in protobuf-java HOT 3
- Kafka Connector [Quickstart - copy_tool.py] doesn't verify the integrity of the downloaded files
- Intermittent error related to AuthMetadataPluginCallback : ValueError: None could not be converted to unicode
- Question: How Can I enable encrypted transmission between the connector and On Premise Kafka
- loadtest: unable to connect to client
- Can't reach GCP PubSub Emulator
- Ordering Key prober does not build
- TopicAdminClient UnauthenticatedException When Settings Endpoints HOT 1
- Load test framework doesn't use pagination when calling instanceGroupManagers.listManagedInstances
- TypeError: this.auth.getUniverseDomain is not a function HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pubsub.