sagikazarmark / curiefense-emissary-poc Goto Github PK

An attempt to integrate Curiefense into Emissary Ingress

Dockerfile 27.59% Shell 72.41%

curiefense-emissary-poc's Introduction

Curiefense WAF + Emissary Ingress POC

Clone the repository with the following command:

git clone --recurse-submodules https://github.com/sagikazarmark/curiefense-emissary-poc.git

Preparations

Build container image:

docker build .

If you use Kind (proceed to the instructions below if you are here for the first time), you can build a local image and load into Kind:

docker build -t curiefense-emissary .
kind load docker-image curiefense-emissary:latest

Setup

Gain access to a Kubernetes cluster. Check out the Using Kind section for a local setup.

Using Kind

Create a new Kind cluster:

kind create cluster --config kind/kind.yaml

Run the setup script to install required components:

./kind/setup.sh

Deploy Curiefense

Deploy Curiefense:

kubectl create namespace curiefense

kubectl apply -f curiefense/secret.yaml

cd curiefense/curiefense-helm/curiefense-helm
DOCKER_TAG=v1.5.0 ./deploy.sh -f curiefense/use-minio.yaml --set "global.proxy.frontend=envoy" --set "global.settings.curiefense_minio_insecure=true"
cd -

TODO: quality of life improvement: push (prod) chart to a chart repo? Use Kustomize to install components (uiserver, confserver) separately?

Deploy Emissary Ingress

Deploy Emissary:

# If you run into any error, run it again
kustomize build emissary | k apply -f -

kubectl -n emissary wait --for condition=available --timeout=90s deploy emissary-ingress

Deploy the echo app

kubectl apply -f app/app.yaml

Usage

First, you might want to create some configuration that proves the system works.

For example, you could create a Global Filter that matches requests with a specific header (eg. breakme: true).

Check out the documentation to learn about the vast number of features Curiefense has.

First, port-forward into the Curiefense UI server:

kubectl -n curiefense port-forward deploy/uiserver 8080:80

Then follow these steps to setup a simple deny rule:

Go to Policies & Rules
Choose Global Filters
Click the + (plus) sign in the right upper corner
Give the new filter a name
Add a new match for a Header (eg. breakme: true)
Choose 503 Service Unavailable as action
Hit save (floppy icon)
Go to Publish Changes
Hit Publish configuration

Next, port-forward into Emissary Ingress:

kubectl -n emissary port-forward deploy/emissary-ingress 8888:8080

Finally, send a request to the ingress:

curl -H "Host: host2.example.com" -H "breakme: true" localhost:8888

You should get an 503 from the server.

Cleanup

Ideally, delete the cluster.

In case of Kind:

kind delete cluster

Best effort attempt to delete resources:

kubectl delete namespace emissary
kubectl delete namespace curiefense

curiefense-emissary-poc's People

Contributors

Stargazers

Watchers

curiefense-emissary-poc's Issues

Curiefense GitOps flow

How can we use a custom (private) Git repository?

Emissary + Curieproxy production deployment

Emissary comes with two installation methods:

Helm
Kubernetes YAML / Kustomize

In order to make Curieproxy work, the following changes have to be made:

the Module CR has to be extended with the necessary lua_scripts (note: this may change from version to version (Curiefense))
the image should be changed to an image containing Curieproxy
a sidecar container has to be added (Curiesync)
an emptyDir volume has to be mounted to both containers (Emissary, Curiesync)

In addition to the above, the Curiesync container needs a ConfigMap or a Secret containing credentials to the config store bucket.

Also, a LogService CR can optionally be created to send logs to Curielogger (although this may change in 1.6).

The Kubernetes YAML method either requires patched YAML files, or use Kustomize (see this repo).

The Helm chart provides all the necessary extension points, so it should be a matter of documentation.

Keep in mind that Curieproxy also requires Redis for features, like rate limiting. The credentials and information for that has to be injeceted.....probably to the Curieproxy (Emissary) container?

Curieconf server + UI + Curielogger deployment

The current deployment mechanism is using a Helm chart in https://github.com/curiefense/curiefense-helm

This is in no way suitable for a production environment:

Chart is heavy with all kinds of dependencies usually not installed in a production environment (Prometheus, Grafana, Elasticsearch, Kibana, etc)
Chart supports various environments which aren't necessary (eg. Istio)
a Helm chart may not even be the right way to install these components (Kustomize might be better)

Create container image integrating Curiefense into Emissary

Access log forwarding

In the Kubernetes world a common pattern for collecting and forwarding logs is using FluentBit and Fluentd:

Drop filebeat
Mark access logs with a marker, so Fluentd can route them accordingly

This may not be an issue once Curiefense 1.6 is out.

Automate container image build

Make collab easier

Emissary integration questions

Lua scripts filter ordering: ideally Curiefense should always come first. Is that currently possible?
LogService: are requests logged after going through all filters (lua included)?

Production Dockerfile

Create a Dockerfile that contains the minimal files and build steps required for a production image.

Geo subregion logging

We need to be able to prove that access to certain subregions is blocked, so we are going to need curiefense/curiefense#694

Does curielogger implement Envoy's ALS?

We could use the LogService CRD of Emissary to configure envoy to send logs to curielogger: https://www.getambassador.io/docs/emissary/latest/topics/running/services/log-service/

I have no idea though if this comes after the requests are passed through Curiefense or not, so it may require some help from Emissary.

Production image build process for Emissary + Curieproxy

We need to figure out a build process for production images. A couple things to consider:

Emissary support policy (how many versions do they support)?
Curiefense support policy
Emissary release cycle
Curiefense release cycle

An optimal scenario: support the latest patch versions, two minor versions and all supported major versions of each. That would result in a matrix build, combining all support versions (minus all incompatible versions).

For example:

Emissary supported versions: 2.3.x, 3.2.x
Curiefense supported versions: 1.5.x, 1.6.x

Emissary 2.3.x and Curiefense 1.6.x are known NOT to work (eg. too old Envoy version, etc).

The build matrix would look like this:

Emissary 2.3.x, Curiefense 1.5.x
Emissary 2.4.x, Curiefense 1.5.x
Emissary 2.4.x, Curiefense 1.6.x

Builds should be automated as much as possible (for example: use Dependabot for automatically updating and releasing patch versions)

Requires #8

References

MaxMind database update

How is the MaxMind database handled at the moment? Is it updated? If so, how?

Do we need a custom update process?

Can't see any logs in Kibana

I suspect Curiefense (proxy) expects the curielogger component to be in the same namespace (at least I vaguely remember that I had to install nginx ingress in the same namespace as curiefense back when I tried it last time).