Kong hub includes all plugins, integrations and tooling that exists around Kong. Adding decK to Kong Hub would let users find decK easily and use it.

If the same certificate is duplicated associated with SNIs, deduplicate the data and apply.

See Kong/kubernetes-ingress-controller#327 for more details.

decK currently throws back an error if the same cert is duplicated.
Logging a warning for this and then de-duplicating the cert and SNIs and applying them should solve the problem.

Current version does not seem to allow TCP stream service configuration.

Kong 1.0 added TCP Stream support https://konghq.com/blog/tcp-stream-support-kong/. However, despite Deck can dump TCP stream services and routes config from Kong, it is not able to generate TCP services from yaml file.

It looks this is due to the default addition of "/" when path is not specified in a service declaration.

For example, when a TCP service is previously configured via Admin API

$ curl localhost:8001/services -d name=tcp-service-name -d url=tcp://192.168.1.127:21 | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   308  100   260  100    48  28888   5333 --:--:-- --:--:-- --:--:-- 34222
{
  "host": "192.168.1.127",
  "created_at": 1565077649,
  "connect_timeout": 60000,
  "id": "4a244da8-a941-4cc5-b5fa-3221f428cf00",
  "protocol": "tcp",
  "name": "tcp-service-name",
  "read_timeout": 60000,
  "port": 21,
  "path": null,
  "updated_at": 1565077649,
  "retries": 5,
  "write_timeout": 60000
}

and given the following yaml file which applies the same service config:

services:
- connect_timeout: 60000
  host: 192.168.1.127
  name: tcp-service-name
  port: 21
  protocol: tcp
  read_timeout: 60000
  retries: 5
  write_timeout: 60000
  routes:
  - name: tcp-route-name
    protocols:
    - tcp
    regex_priority: 0

Then deck diff execution results in

updating service tcp-service-name {
   "connect_timeout": 60000,
   "host": "192.168.1.127",
   "id": "2933a858-9394-402b-b338-a3593974a1a4",
   "name": " tcp-service-name",
   "port": 21,
   "protocol": "tcp",
   "read_timeout": 60000,
   "retries": 5,
   "write_timeout": 60000
+  "path": "/"
 }
creating route tcp-route-name

And deck sync --verbose=2 execution results in

GET /services?size=1000 HTTP/1.1
Host: localhost:8001
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip


HTTP/1.1 200 OK
Content-Length: 23
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Tue, 06 Aug 2019 07:38:40 GMT
Server: kong/1.0.3

{"next":null,"data":[]}
GET /routes?size=1000 HTTP/1.1
Host: localhost:8001
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip


HTTP/1.1 200 OK
Content-Length: 23
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Tue, 06 Aug 2019 07:38:40 GMT
Server: kong/1.0.3

{"next":null,"data":[]}
GET /plugins?size=1000 HTTP/1.1
Host: localhost:8001
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip


HTTP/1.1 200 OK
Content-Length: 23
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Tue, 06 Aug 2019 07:38:40 GMT
Server: kong/1.0.3

{"next":null,"data":[]}
GET /certificates?size=1000 HTTP/1.1
Host: localhost:8001
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip


HTTP/1.1 200 OK
Content-Length: 23
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Tue, 06 Aug 2019 07:38:40 GMT
Server: kong/1.0.3

{"next":null,"data":[]}
GET /snis?size=1000 HTTP/1.1
Host: localhost:8001
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip


HTTP/1.1 200 OK
Content-Length: 23
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Tue, 06 Aug 2019 07:38:40 GMT
Server: kong/1.0.3

{"next":null,"data":[]}
GET /consumers?size=1000 HTTP/1.1
Host: localhost:8001
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip


HTTP/1.1 200 OK
Content-Length: 23
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Tue, 06 Aug 2019 07:38:40 GMT
Server: kong/1.0.3

{"next":null,"data":[]}
GET /upstreams?size=1000 HTTP/1.1
Host: localhost:8001
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip


HTTP/1.1 200 OK
Content-Length: 23
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Tue, 06 Aug 2019 07:38:40 GMT
Server: kong/1.0.3

{"next":null,"data":[]}
Error: 2 errors occured:
        while processing event: {Create} failed: validation on entity in the request failed
        could not find service 'tcp-route-name' for route : entity not found

The same result is obtained when a new TCP service is configured.

Used Kong version is v1.0.3

oauth2 credentials were missed while adding support for credentials.
I'm always confused with oauth2 specs and all the flows.

Even if strip_path is set to false in config, the value in kong will still be true

Add --workspace flag to dump command to dump the configuration of a single workspace for Kong Enterprise.

• Add headers property support in the Route object
• Add support for the new ca_certificates core entity

decK was designed keeping in mind compatibility with Kong's native declarative config file.

Currently, all that is needed to make the file compatible with Kong's declarative config file is to add the following line:

_format_version: "1.1"

It would be useful if we could pass multiple state files in a specific order, and override individual values in the same way that docker compose supports.

for reference: https://docs.docker.com/compose/extends/#understanding-multiple-compose-files

If I have a production state yaml file, I may want to override an individual value for development, experimenting with a new feature.

Lets say I have the following state file:

kong.yaml

services:
- name: svc1
  host: my-api:8080
  routes:
  - name: r1
    paths:
    - /r1
      plugins:
      - name: jwt
        config:
          secret_is_base64: false
          key_claim_name: iss
          claims_to_verify:
          - exp
          cookie_names:
          - auth_jwt
          anonymous: null
          run_on_preflight: true

Example scenario 1:
I want to set an anonymous user for use in local development/testing

kong.local.yaml

services:
- name: svc1
  routes:
  - name: r1
      plugins:
      - name: jwt
        config:
          anonymous: bc92eeda-c35b-11e9-9cb5-2a2ae2dbcce4

deck sync --kong-addr="http://kong:8001" -s=kong.yaml -s=kong.local.yaml

In this case I would expect the value of anonymous in route r1 to be 'bc92eeda-c35b-11e9-9cb5-2a2ae2dbcce4'.

Example scenario 2:
I want to include an additional route with debug logs on my development environment

kong.dev.yaml

services:
- name: svc1
  routes:
  - name: r1-debug
    paths:
    - /debug/r1
      plugins:
      - name: file-log
        config:
          path: /tmp/file.log
      - name: jwt
        config:
          secret_is_base64: false
          key_claim_name: iss
          claims_to_verify:
          - exp
          cookie_names:
          - auth_jwt
          anonymous: null
          run_on_preflight: true

deck sync --kong-addr="http://kong:8001" -s=kong.yaml -s=kong.dev.yaml

In this case I would expect a new route to be added to svc1.

Kong allows configuring plugins for any combination of Consumer, Service and Route.
These are currently not configurable via deck.

Currently, default values for plugins need to be hard-coded into the YAML file.
It would be nice if users can set only the values they need in the file and then the rest of the defaults is something deck figures out on it's own.

Kong Ingress Controller already does something similar to this.

If _workspace is specified in a config file, during a sync, attempt to create the workspace if the workspace doesn't exist.

This eases the sync process as the user need not ensure that all the workspaces exist.

Kong 1.3 introduced support for upstream client certificates.
decK doesn't read a certificate specified in the client_certificate entity.
Also, decK cannot reference a certificate already defined in the certificates section.

#29 is a blocker for this feature.

decK v0.5.0 had a very simple bug that rendered the release unusable.
This could have been easily could with an automated end-to-end testing setup, which we don't have at the moment.
As more changes are introduced, manual testing is becoming more expesive and proving not to be sufficient.
Figure out a path forward for how do we want to test decK in an automated fashion.

If I want to configure the anonymous field on a jwt plugin, is there a way to handle this in deck?

Its tricky because this field requires the consumer id, which we cannot know until the consumer has been created already.

It would be useful if we could set this field using 'username' in deck, and to have the ability to list some users in the config so that deck will ensure they exist, even when the --skip-consumers flag is used (or maybe have another flag to enable this).

Update:

I see that using the kong admin api I can create a consumer and specify the id so i guess that part should be possible to get working in deck

As for the issue of synching only certain consumers, I have 2 suggestions.

1. include a parameter to only create consumers and ignore updates or deletions --create-consumers
2. alternatively, a flag to sync consumers with a particular tag(s). For example, --sync-consumer-tags=anonymous will perform synchronize actions on users with the anonymous tag and ignore others.

• Metrics around decK's adoption in the community
• What is compatibility promise of decK v1.0? #27
• What are the features necessary to get decK to v1.0?

Hi @hbagdi,

I have a request in regards to the sync command. I've noticed in the code that the first thing done as part of the "sync" logic is reading all the entities of Kong, including the consumers. I've noticed that this is done even though consumers are currently not being used as part of syncing Kong's configuration.

We will shortly be deploying Kong 1.0.3 into our production environment which has ~20,000 consumers and we're a little concerned that that many consumers may cause difficulties when using "deck" to configure Kong.

What are your thoughts of refactoring the sync logic to avoid reading consumers when dumping or syncing Kong's configuration; either by doing something similar to kongfig by providing a flag, or by ignoring them altogether until there is a need to add support for configuring plugins on consumers?

If you like, I can contribute a PR which does one of these things, but I would be interested in knowing which direction you would like to take before I start changing things.

Thanks,

Leon

Currently, GetStateFromContent returns a duplicate target definitions found for: ... error when different upstreams have same target.

It's not rare that different services sharing the same group of pods in Kubernetes. Such restrictions causes the kubernetes-ingress-controller stop working. Maybe we can use ${upstream.id}/${target.Target} here as the id in TargetsCollection?

Currently, IDs cannot be specified in decK for any entitiy, and if they are specified, they are just ignored.
This feature will also benefit Kong's Ingress Controller as the IDs can be deterministic.

How can I use workspace with diff and sync. I exported file from one workspace and I need to make changes to URLs for services and then need to import into another workspace. Can we do that

After exporting my config from 1.1.0 with the deck 1.5.1 I get the following error when importing it on kong 1.3.1:

in 'services':
  - in entry 1 of 'services':
    in 'plugins':
      - in entry 1 of 'plugins':
        in 'config':
          in 'http_method': expected a string

Snippet of my config:

_format_version: "1.1"
services:
- connect_timeout: 60000
  host: example.com
  name: eod_historical_data
  path: /api/
  port: 443
  protocol: https
  read_timeout: 60000
  retries: 5
  write_timeout: 60000
  routes:
  - hosts:
    - api.example.com
    name: real-time
    paths:
    - /real-time
    preserve_host: false
    protocols:
    - https
    regex_priority: 0
    strip_path: false
  plugins:
  - name: request-transformer
    config:
      add:
        body: []
        headers: []
        querystring:
        - api_token:abc
        - fmt:json
      append:
        body: []
        headers: []
        querystring: []
      http_method: null

decK follows semantic versioning and hence it is important to document what is considered as an API surface and what is a breaking change.

• The config file and CLI are part of the public API
• Go code and modules are not part of the public API and will change

When --all-workspaces is supplied, all workspaces in Kong Enterprise should be exported out.
A file should be created for each workspace.

@hbagdi here is the issue we discussed on slack
Not sure if it could also affect other cases beside cookie_names, maybe there is some general solution. Thanks a lot!!

When select_tags feature is used to manage only a subset of Kong configuration, then any credential created for a consumer created outside of the managed config is loaded, decK errors out.

The problem here being that credentials are not tagged in Kong (yet) and hence Kong tries to load all the credentials.

First of all, loving this! See a bright future ahead. But first...ideally, decK could handle all Kong configurations. Barring that, perhaps a pointer to the first plugin definition that decK is panicking about?

$ deck dump
panic: plugins on a combination of route/service/consumer are not yet supported by decK

goroutine 1 [running]:
github.com/hbagdi/deck/dump.GetState(0xc000154000, 0x0, 0xdd5438, 0x0, 0x0, 0x0, 0x0, 0x0)
/Users/harry/deck/dump/dump.go:114 +0x1a44
github.com/hbagdi/deck/cmd.glob..func3(0xdaf9e0, 0xdd5438, 0x0, 0x0, 0x0, 0x0)
/Users/harry/deck/cmd/dump.go:29 +0xa0
github.com/spf13/cobra.(*Command).execute(0xdaf9e0, 0xdd5438, 0x0, 0x0, 0xdaf9e0, 0xdd5438)
/Users/harry/go/pkg/mod/github.com/spf13/[email protected]/command.go:762 +0x465
github.com/spf13/cobra.(*Command).ExecuteC(0xdaf060, 0xa285a0, 0xa3c040, 0xa285a8)
/Users/harry/go/pkg/mod/github.com/spf13/[email protected]/command.go:852 +0x2ec
github.com/spf13/cobra.(*Command).Execute(...)
/Users/harry/go/pkg/mod/github.com/spf13/[email protected]/command.go:800
github.com/hbagdi/deck/cmd.Execute()
/Users/harry/deck/cmd/root.go:50 +0x2e
main.main()
/Users/harry/deck/main.go:31 +0x25

Following credential types can be created in Kong, that are associated with a consumer in Kong:

• key-auth
• hmac-auth
• basic-auth
• jwt
• acl

Currently, decK cannot manage these plugin specific entities.

While key-auth, jwt and hmac-auth store credential in plain-text and return them, basic-auth plugin hashes the password. This results in a hash being returned on the Admin API. In such a situation, the credential cannot be compared for a diff and updated if it is out of sync. It is not clear the path that Kong will take in future and if other credentials and secrets will also be hashed or encrypted or both.

There are a couple of solutions possible:

• Add support for the 3 credentials that are in plain-text today and then later if Kong hashes them, drop support for them.
• Add support for all credentials, but then warn users that update operation is not supported, meaning deck will take care of creating a credential if one doesn't exist, but will not diff them to apply them again. There can be an annotation on a credential which can force a resync, meaning, the credential will be PUT. This will result in PUT request to credentials in every sync.

In either of the solution, hashed information can not be exported from Kong.

Developer documentation on the steps to perform a release.
The release procedure is semi-automated using Goreleaser.
There are some manual steps and guidelines which should be noted.

_workspace meta tag should represent the workspace for the file.

After doing a deck dump and then running a kong config parse kong.yaml it has showns a few discrepencies:

1. No '_format_version': - needed at the top of the file
2. A null value was added for an empty field:

  plugins:
  - name: request-transformer
    config:
      add:
        body: []
        headers: []
        querystring:
        - fmt:json
        - period:d
      append:
        body: []
        headers: []
        querystring: []
      http_method: null

Kong version before 1.3 do not have ca_certificate entity and an attempt to sync this entity results in an error.

bash-4.4$ ./deck dump
Error: route 'route id' does not have a name. decK needs routes to be named.

decK's select_tags is a feature that is very useful for managing Kong's configuration in a distributed team but is undocumented.

Hi,

It would be nice if the sync and diff commands would read try to read from stdin if a '-' (dash) is indicated on the command line.

Hi,

Is there an option to ignore SSL certs errors?

I am running kong on openshift with an external route, and I have this error:

$ ./deck dump --kong-addr https://esteemed-hedgehog-kong-admin.kong.svc.cluster.local:8444
Error: Get https://esteemed-hedgehog-kong-admin.kong.svc.cluster.local:8444/services?size=1000: x509: certificate is valid for localhost, not esteemed-hedgehog-kong-admin.kong.svc.cluster.local

I made a similar script yesterday to dump the config:

https://github.com/zoobab/kongdump

with curl -k to ignore SSL problems.

Would be great to publish decK to docker hub eg. for job definitions in k8s or smth

Is there currently any flag or strategy available for not deleting all the certificates inside Kong on each deck sync run?

If not, what would be a possible strategy for managing the certificates separately?

I applied deck to kong k8s.
here is my configmap:

here is my Dockerfile

here is my kong-deployment
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: kong-rc
spec:
replicas: 1
template:
metadata:
labels:
name: kong-rc
app: kong
spec:
containers:
- name: kong
image: kong-test:latest
imagePullPolicy: IfNotPresent
env:
- name: KONG_ADMIN_LISTEN
value: "0.0.0.0:8001, 0.0.0.0:8444 ssl"
- name: KONG_PG_PASSWORD
value: kong
- name: KONG_PG_HOST
value: postgres.default.svc.cluster.local
- name: KONG_PROXY_ACCESS_LOG
value: "/dev/stdout"
- name: KONG_ADMIN_ACCESS_LOG
value: "/dev/stdout"
- name: KONG_PROXY_ERROR_LOG
value: "/dev/stderr"
- name: KONG_ADMIN_ERROR_LOG
value: "/dev/stderr"
ports:
- name: admin
containerPort: 8001
protocol: TCP
- name: proxy
containerPort: 8000
protocol: TCP
- name: proxy-ssl
containerPort: 8443
protocol: TCP
- name: admin-ssl
containerPort: 8444
protocol: TCP
command:
- "/bin/sh"
- "-ecx"
- |
cd /etc/kong &&
# sleep 111111 &&
kong start --v &&
./deck sync &&
kong stop -v &&
cd / &&
./docker-entrypoint.sh kong docker-start
volumeMounts:
- name: kongconfig
mountPath: /etc/kong/kong.yaml
subPath: kong.yaml
volumes:
- name: kongconfig
configMap:
name: kongconfig
and the result:

seams that services r1,r2,r3 all fail.
could you please help me to figure out what's wrong?
@davidcv5

Hi,

It would be nice to be able to deck dump to stdout by using a command line flag.

Great work with this tool, It actually helped us a lot keeping track of the various changes we were doing in Kong and have configuration as source.

In some situations, DB-less seems to be a better option, as Cassandra is very heavy and PosgreSQL needs more config to be replicated.

So, having kong.yaml as a ConfigMap in a Kubernetes deployment, automatically replicated by ETCD and mounted in the Kong pods - is an attractvie solution.

Doing a dump using deck, is not immediately usable in Kong. To make it work, I had to:

• Add the header _format_version: "1.1".
• Remove all the keys having as value null string.
• SNI must be in the format name: sni-value instead of just the sni value as it is currently, because Kong "expecst a record".

So, could you consider implementing for dump:

1. Adding the header by default when writing, ignore during reading
2. Empty/null values should actually be omitted from the dump
3. SNIs preceeded by name: keyword:

   snis:
   - name: some-host.com
   - name: other-host.com

Add a warning log when decK is being used to sync basic-auth credential, notifying the user that decK will not detect change in the password

With version 0.5.0 I cannot execute any commands other than ping, has there been some change to how I need to call it?

Tested with kong versions 1.1.2 and 1.2