Goodday
I was very glad with the solution you are creating. But unfortionately it doesnt work entire;ly correct.
First up i was struggling with the auth keys. The enviroment variables i cant get working and the credenitials i had to change the "config.providers.aws.param.aws.client.key=your-client-key" into "config.providers.aws.param.aws.access.key" to get the connection working with aws.
Then when i changed the connection.password into "${aws:ems-dev-eventmanagement-db-secret:password}" it unfortunately failed.
In trace logging i found this part which i think might be the problem but i'm not sure:
[2020-06-15 05:47:08,261] DEBUG http-outgoing-7 << "{"ARN":"arn:aws:secretsmanager:eu-west-1:577868951105:secret:ems-dev-eventmanagement-db-secret-B0TWsl","CreatedDate":1.591854819443E9,"Name":"ems-dev-eventmanagement-db-secret","SecretString":"{"username":"eventmanagement_user","password":"xxxxxx","engine":"mysql","host":"dev-ems.coollnhgrzes.eu-west-1.rds.amazonaws.com","port":"3306","dbname":"eventmanagement"}","VersionId":"2f3cc10f-b194-4a1f-bef5-523a03f68a70","VersionStages":["AWSCURRENT"]}" (org.apache.http.wire)
[2020-06-15 05:47:08,261] DEBUG http-outgoing-7 << HTTP/1.1 200 OK (org.apache.http.headers)
[2020-06-15 05:47:08,261] DEBUG http-outgoing-7 << Date: Mon, 15 Jun 2020 05:47:08 GMT (org.apache.http.headers)
[2020-06-15 05:47:08,261] DEBUG http-outgoing-7 << Content-Type: application/x-amz-json-1.1 (org.apache.http.headers)
[2020-06-15 05:47:08,261] DEBUG http-outgoing-7 << Content-Length: 491 (org.apache.http.headers)
[2020-06-15 05:47:08,261] DEBUG http-outgoing-7 << Connection: keep-alive (org.apache.http.headers)
[2020-06-15 05:47:08,261] DEBUG http-outgoing-7 << x-amzn-RequestId: fef90d70-733b-470f-8987-17ad2c6f38a3 (org.apache.http.headers)
[2020-06-15 05:47:08,261] DEBUG Connection can be kept alive for 60000 MILLISECONDS (org.apache.http.impl.execchain.MainClientExec)
[2020-06-15 05:47:08,261] TRACE Parsing service response JSON (com.amazonaws.request)
[2020-06-15 05:47:08,261] DEBUG Connection [id: 7][route: {s}->https://secretsmanager.eu-west-1.amazonaws.com:443] can be kept alive for 60.0 seconds (org.apache.http.impl.conn.PoolingHttpClientConnectionManager)
[2020-06-15 05:47:08,261] DEBUG http-outgoing-7: set socket timeout to 0 (org.apache.http.impl.conn.DefaultManagedHttpClientConnection)
[2020-06-15 05:47:08,261] DEBUG Connection released: [id: 7][route: {s}->https://secretsmanager.eu-west-1.amazonaws.com:443][total kept alive: 1; route allocated: 1 of 50; total allocated: 1 of 50] (org.apache.http.impl.conn.PoolingHttpClientConnectionManager)
[2020-06-15 05:47:08,262] TRACE Done parsing service response (com.amazonaws.request)
[2020-06-15 05:47:08,262] DEBUG Received successful response: 200, AWS Request ID: fef90d70-733b-470f-8987-17ad2c6f38a3 (com.amazonaws.request)
[2020-06-15 05:47:08,262] DEBUG x-amzn-RequestId: fef90d70-733b-470f-8987-17ad2c6f38a3 (com.amazonaws.requestId)
I run kafka connect in kubernetes with the following plugins - versions:
FROM confluentinc/cp-kafka-connect:5.5.0
COPY extras/mariadb-java-client-2.1.0.jar /usr/share/java/kafka/
COPY extras/mysql-connector-java-8.0.15.jar /usr/share/java/kafka-connect-jdbc
COPY extras/ojdbc8.jar /usr/share/java/kafka-connect-jdbc
COPY extras/secret-provider-0.0.2-all.jar /usr/share/java/
and the connector i'm trying to install:
curl -X PUT
http://ems-kafka-dev-cp-kafka-connect.ems-dev:8083/connectors/JdbcSourceConnectorEventManagement/config
-H 'Content-Type: application/json'
-H 'Accept: application/json'
-d '{
"connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
"timestamp.column.name": "modified_at",
"config.providers.aws.param.aws.access.key": "xxxxxxxxxx",
"config.providers.aws.param.aws.secret.key": "xxxxxxxxx",
"connection.password": "${aws:ems-dev-eventmanagement-db-secret:password}",
"tasks.max": "1",
"config.providers": "aws",
"table.types": "VIEW",
"table.whitelist": "trigger_recipe_event_view,recipe_business_rules_with_parameters_view,event_linked_active_recipes_view",
"mode": "timestamp",
"topic.prefix": "connected-keyless-",
"poll.interval.ms": "60000",
"config.providers.aws.param.aws.auth.method": "default",
"config.providers.aws.param.aws.region": "eu-west-1",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"key.converter": "org.apache.kafka.connect.json.JsonConverter",
"config.providers.aws.class": "io.lenses.connect.secrets.providers.AWSSecretProvider",
"validate.non.null": "false",
"connection.attempts": "3",
"batch.max.rows": "100",
"connection.backoff.ms": "10000",
"timestamp.delay.interval.ms": "0",
"table.poll.interval.ms": "60000",
"connection.user": "${aws:ems-dev-eventmanagement-db-secret:username}",
"config.providers.aws.param.file.dir": "/connector-files/aws",
"connection.url": "jdbc:mysql://dev-ems.coollnhgrzes.eu-west-1.rds.amazonaws.com:3306/eventmanagement",
"numeric.precision.mapping": "false"
}'