saarasio / enroute Goto Github PK

EnRoute Ingress API Gateway - Kubernetes-native, Ingress API Gateway Powered using Envoy

License: Apache License 2.0

Dockerfile 0.02% Makefile 0.28% Go 95.15% Shell 1.62% PLpgSQL 1.50% CMake 0.55% C++ 0.32% Smarty 0.52% Lua 0.04%

envoyproxy envoy-xds api-gateway apigateway ssl kubernetes ingress-controller openapi-spec jwt kubernetes-ingress-gateway

enroute's People

Contributors

Stargazers

Watchers

Forkers

dantodor easyfmxu maniacs-oss siemasusel neomantra jdixoncs aeimer little-chili happysky2046 johhnydinh eternalerrors bluebamboostudios huangyingting muharihar anthonyhewins kg-ops

enroute's Issues

Doc: Cookbook:Show a way to stream routes to envoy

Packaging: Build an docker image that packages envoy and controller

SaarasCloudCache: Add support for upstream fields - strategy, ca, subjectname

Currently, the following upstream fields read from the cloud are not programmed for Envoy. Need to add support for these fields.


  "upstream_strategy": null,
  "upstream_validation_cacertificate": null,
  "upstream_validation_subjectname": null,

Webapp: Run webapp automatically when control plane comes up

Right now we

cd enroute-cp && go run webapp.go

This is related to #2 . When an image is created, this should also be run

DB: Make DB/GQL container port only accessible on localhost

Doc: Write up documentation on integration with ECS on AWS

Webapp: Fix error messages during upstream creation

Note that when upstream_ip is not provided, the error message is - "No JSON object could be decoded". This is not very helpful.

[ec2-user@ip-172-31-18-10 enroute-cp]$ curl -s -X POST localhost:1323/upstream -H "Authorization: Bearer treehugger" -H "Content-Type: application/json" -d '{"Upstream_name":"db_serve_process", "upstream_hc_host":"127.0.0.1", "upstream_hc_path":"/v1/version", "upstream_port":"8081"}' | python -m json.tool
No JSON object could be decoded
[ec2-user@ip-172-31-18-10 enroute-cp]$ curl -s -X POST localhost:1323/upstream -H "Authorization: Bearer treehugger" -H "Content-Type: application/json" -d '{"Upstream_name":"db_serve_process", "upstream_hc_host":"127.0.0.1", "upstream_hc_path":"/v1/version", "upstream_port":"8081"}' | python -m json.tool
No JSON object could be decoded
[ec2-user@ip-172-31-18-10 enroute-cp]$ curl -s -X POST localhost:1323/upstream -H "Authorization: Bearer treehugger" -H "Content-Type: application/json" -d '{"Upstream_name":"db_serve_process", "upstream_ip":"127.0.0.1", "upstream_hc_path":"/v1/version", "upstream_port":"8081"}' | python -m json.tool
{
    "Error": "Please provide a valid weight value."
}
[ec2-user@ip-172-31-18-10 enroute-cp]$ curl -s -X POST localhost:1323/upstream -H "Authorization: Bearer treehugger" -H "Content-Type: application/json" -d '{"Upstream_name":"db_serve_process", "upstream_ip":"127.0.0.1", "upstream_hc_path":"/v1/version", "upstream_port":"8081", "upstream_weight":"98"}' | python -m json.tool
{
    "data": {
        "insert_saaras_db_upstream": {
            "affected_rows": 1
        }
    }
}

Enroute-dp: convert fmt.Printf to log statements in builder.go

Webapp: Error message for service creation not accurate

The error message below should say "Service_name" and not "Name"

[ec2-user@ip-172-31-18-10 enroute-cp]$ curl -s -X POST localhost:1323/service -H "Authorization: Bearer treehugger" -H "Content-Type: application/json" -d '{"Name":"adminservice"}' | python -m'' json.tool
{
    "Error": "Please provide service name using Name field"
}
[ec2-user@ip-172-31-18-10 enroute-cp]$ curl -s -X POST localhost:1323/service -H "Authorization: Bearer treehugger" -H "Content-Type: application/json" -d '{"Service_Name":"adminservice"}' | python -m'' json.tool
{
    "Error": "Please provide fqdn using Fqdn field"
}

Demo: Setup control plane on AWS for demo

Webapp: Missing Content-Type fails update

For PATCH calls, Content-Type should be set to application/json when making such API calls. However, when it is missing, updates fail and there is no error message. There should be a way to check for Content-Type and throw an error. Maybe this should be settable at a group of API level.

[ec2-user@ip-172-31-18-10 ~]$ curl -s -X PATCH https://ingresspipe.io:8443/upstream/default -H "Authorization: Bearer treehugger" -d '{"Upstream_ip":"172.17.0.1"}' | python -m json.tool
{
    "data": {
        "update_saaras_db_upstream": {
            "affected_rows": 1
        }
    }
}
[ec2-user@ip-172-31-18-10 ~]$ curl -s -X PATCH https://ingresspipe.io:8443/upstream/default -H "Authorization: Bearer treehugger" -d '{"Upstream_ip":"172.17.0.1"}' | python -m json.tool
{
    "data": {
        "update_saaras_db_upstream": {
            "affected_rows": 1
        }
    }
}
[ec2-user@ip-172-31-18-10 ~]$ curl -s -X PATCH https://ingresspipe.io:8443/upstream/default -H "Authorization: Bearer treehugger" -H "Content-Type: application/json" -d '{"Upstream_ip":"172.17.0.1"}' | python -m json.tool
{
    "data": {
        "update_saaras_db_upstream": {
            "affected_rows": 1
        }
    }
}

Webapp: Support PATCH for service

Webapp: Error message not friendly when creating service

v5s-iMac:~ v5$ curl -s -X POST https://ingresspipe.io:8443/service -H "Content-Type: application/json" -d '{"Service_Name":"proxy_admin"}' | python -m json.tool
No JSON object could be decoded
v5s-iMac:~ v5$ curl -s -X POST https://ingresspipe.io:8443/service -H "Content-Type: application/json" -d '{"Service_Name":"proxy_admin", "Fqdn":"blah"}' | python -m json.tool
{
    "data": {
        "insert_saaras_db_service": {
            "affected_rows": 1
        }
    }
}

Doc:Cookbook:Program two proxies - one with https another with http

Webapp: Add auth key based middleware for demo

Enroute-dp: Listener ports should be configurable

Right now upstream ports are set to values provided by contour. They are not configurable. This value needs to be configurable from the webapp API

Webapp: Support PATCH for upstream

Packaging: Rearrange directories to follow project

Build directory structure to follow project -

enroute
├── certs
├── docs
├── enroute-cp
├── enroute-dp
├── packaging
└── scripts

Webapp: API to copy an upstream

Related to #8

Webapp: Create an API to deepdelete a service

Right now service deletion requires -

(1) Remove route-upstream relationships
(2) Remove route
(3) Delete service

A deep delete operation is needed which avoids this three level operations

Webapp: GET of upstream for a route prints all upstreams

Webapp: Canned templates for quick setup - HTTP listener -> "/" -> Cluster -> HTTP member

Enroute-dp: Make port configurable

Webapp: dump service

Webapp: Validation: Middleware: Evaluate middleware to validate config

Webapp: API to perform deep copy of an application

We need a way to copy a (service -> route -> upstream). The name of the new entities will have a prefix that will be provided in the REST call.

Webapp: API to copy a route corresponding to a service

Related to #8

Webapp: Canned templates for quick setup - HTTPs listener -> "/" -> HTTP Cluster

Enroute-dp: Add support for additional route matching

We need to fix up the webapp to make route a separate object. Besides, we also need to add support for additional route matching.

Packaging: Build control plane docker image

The image should -

Have hasura components
Have postgres
Support migrations
Install golang to run webapp
Run webapp

We should be able to download it from docker hub.

When run, the port and listen IP should be configurable

Webapp: Support PATCH for route

Enroute-dp: Enable HTTPS communication between CP <-> DP

We should be able to define the protocol used to communicate between CP and DP. Externally connecting DP should talk to CP over HTTPS.

Add additional parameter to enroute to take the protocol used to talk to CP.

Webapp: Query for upstream does not show upstream_weight

ubuntu@ip-172-31-17-83:~/enroute$ curl localhost:1323/upstream | python -m json.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1042  100  1042    0     0   149k      0 --:--:-- --:--:-- --:--:--  169k
{
    "data": {
        "saaras_db_upstream": [
            {
                "create_ts": "2019-09-09T17:38:44.516337+00:00",
                "update_ts": "2019-09-09T18:32:55.590527+00:00",
                "upstream_hc_healthythresholdcount": null,
                "upstream_hc_host": "127.0.0.1",
                "upstream_hc_intervalseconds": null,
                "upstream_hc_path": "/",
                "upstream_hc_timeoutseconds": null,
                "upstream_hc_unhealthythresholdcount": null,
                "upstream_id": 2,
                "upstream_ip": "127.0.0.1",
                "upstream_name": "webapp_serve_process",
                "upstream_port": 1323,
                "upstream_strategy": null,
                "upstream_validation_cacertificate": null,
                "upstream_validation_subjectname": null
            },
            {
                "create_ts": "2019-09-09T18:28:18.684243+00:00",
                "update_ts": "2019-09-10T00:06:01.290403+00:00",
                "upstream_hc_healthythresholdcount": 0,
                "upstream_hc_host": "127.0.0.1",
                "upstream_hc_intervalseconds": 0,
                "upstream_hc_path": "/v1/version",
                "upstream_hc_timeoutseconds": 0,
                "upstream_hc_unhealthythresholdcount": 0,
                "upstream_id": 3,
                "upstream_ip": "127.0.0.1",
                "upstream_name": "db_serve_process",
                "upstream_port": 8081,
                "upstream_strategy": "",
                "upstream_validation_cacertificate": "",
                "upstream_validation_subjectname": ""
            }
        ]
    }
}

Webapp: API to copy a service

Related to #8

Demo: Reorganize site with admin site running on ec2 and demo on ingresspipe

Right now demo and admin are both running on ingresspipe.io. Break up and reorganize so that only demo pieces are running on ingresspipe.io

Feature: Support tags for objects

Doc: Document APIs

Logging: Enable logging for enroute-dp

Should be able to control logging using the webapp API. Should be configurable per application

Webapp: Design and provide an API to automatically generate and attach a letsencrypt cert

We have the domain name and we can try and automatically generate a let's encrypt certificate

Webapp: Canned templates for quick setup - HTTP listener -> "/" -> Cluster - gRPC member

Webapp: Implement API to dump all config starting from a proxy

Enroute-cp: Make webhttp listen on localhost only (not 0.0.0.0)

Enroute-dp: Add support for gRPC

gRPC is something that is causing a lot of people to use Envoy. The gRPC configuration also makes a lot of people struggle with config.

Webapp: Associating route with a service doesn't check for existence of service

If service is absent, a default value is created with uninitialized values. When we attach a route to a service, we need to add validation to check if the service is indeed present.

Logging: Enable logging for enroute-cp

Should be able to control log level using the webapp API. Also logging should be per application

Adding upstream without health checks results in erroneous cluster/endpoint on Envoy

[2019-09-05 04:11:28.823][15][debug][upstream] [source/common/upstream/cds_api_impl.cc:78] cds: add/update cluster 'test/test/9001/42099b4af0'                                                                                                                                                                                 [1924/3125]
[2019-09-05 04:11:28.823][15][debug][misc] [source/common/protobuf/utility.cc:88] Proto validation error; throwing Proto constraint validation failed (ClusterValidationError.HealthChecks[i]: ["embedded message failed validation"] | caused by HealthCheckValidationError.HttpHealthCheck: ["embedded message failed validation"] | cau
sed by HttpHealthCheckValidationError.Path: ["value length must be at least " '\x01' " bytes"]): name: "test/test2/9001/da39a3ee5e"
type: EDS
eds_cluster_config {
  eds_config {
    api_config_source {
      api_type: GRPC
      grpc_services {
        envoy_grpc {
          cluster_name: "contour"
        }
      }
    }
  }
  service_name: "test/test2"
}
connect_timeout {
  nanos: 250000000
}
health_checks {
  timeout {
    seconds: 2
  }
  interval {
    seconds: 10
  }
  unhealthy_threshold {
    value: 3
  }
  healthy_threshold {
    value: 2
  }
  http_health_check {
    host: "contour-envoy-healthcheck"
  }
}
common_lb_config {
  healthy_panic_threshold {
  }
}
alt_stat_name: "test_test2_9001"
drain_connections_on_host_removal: true

[2019-09-05 04:11:28.823][15][warning][config] [source/common/config/grpc_mux_subscription_impl.cc:72] gRPC config for type.googleapis.com/envoy.api.v2.Cluster rejected: Error adding/updating cluster(s) test/test2/9001/da39a3ee5e: Proto constraint validation failed (ClusterValidationError.HealthChecks[i]: ["embedded message fail
ed validation"] | caused by HealthCheckValidationError.HttpHealthCheck: ["embedded message failed validation"] | caused by HttpHealthCheckValidationError.Path: ["value length must be at least " '\x01' " bytes"]): name: "test/test2/9001/da39a3ee5e"
type: EDS
eds_cluster_config {
  eds_config {
    api_config_source {
      api_type: GRPC
      grpc_services {
        envoy_grpc {
          cluster_name: "contour"
        }
      }
    }
  }
  service_name: "test/test2"
}
connect_timeout {
  nanos: 250000000
}
health_checks {
  timeout {
    seconds: 2
  }
  interval {
    seconds: 10
  }
  unhealthy_threshold {
    value: 3
  }
  healthy_threshold {
    value: 2
  }
  http_health_check {
    host: "contour-envoy-healthcheck"
  }
}
common_lb_config {
  healthy_panic_threshold {
  }
}
alt_stat_name: "test_test2_9001"
drain_connections_on_host_removal: true

[2019-09-05 04:11:28.823][15][trace][config] [source/common/config/grpc_mux_impl.cc:60] Sending DiscoveryRequest for type.googleapis.com/envoy.api.v2.Cluster: version_info: "7"
node {
  id: "service-node"
  cluster: "test"
  build_version: "e349fb6139e4b7a59a9a359be0ea45dd61e589c5/1.11.1/Clean/RELEASE/BoringSSL"
}
type_url: "type.googleapis.com/envoy.api.v2.Cluster"
response_nonce: "8"
error_detail {
  code: 13
  message: "Error adding/updating cluster(s) test/test2/9001/da39a3ee5e: Proto constraint validation failed (ClusterValidationError.HealthChecks[i]: [\"embedded message failed validation\"] | caused by HealthCheckValidationError.HttpHealthCheck: [\"embedded message failed validation\"] | caused by HttpHealthCheckValidationError$
Path: [\"value length must be at least \" \'\\x01\' \" bytes\"]): name: \"test/test2/9001/da39a3ee5e\"\ntype: EDS\neds_cluster_config {\n  eds_config {\n    api_config_source {\n      api_type: GRPC\n      grpc_services {\n        envoy_grpc {\n          cluster_name: \"contour\"\n        }\n      }\n    }\n  }\n  service_name:
\"test/test2\"\n}\nconnect_timeout {\n  nanos: 250000000\n}\nhealth_checks {\n  timeout {\n    seconds: 2\n  }\n  interval {\n    seconds: 10\n  }\n  unhealthy_threshold {\n    value: 3\n  }\n  healthy_threshold {\n    value: 2\n  }\n  http_health_check {\n    host: \"contour-envoy-healthcheck\"\n  }\n}\ncommon_lb_config {\n  h$
althy_panic_threshold {\n  }\n}\nalt_stat_name: \"test_test2_9001\"\ndrain_connections_on_host_removal: true\n"
}

Webapp: Support to provide information about one proxy

This call fails -

v5s-iMac:~ v5$ curl -s -X GET https://ingresspipe.io:8443/proxy/adminproxy | python -m json.tool
{
    "message": "Not Found"
}

Webapp: Route deletions cannot be done without removing entries from route_upstream

Route id is a foreign key in route_upstream relationship table. Entry in this table should be deleted before the route is deleted.

When a route delete call is made, we could first delete the entries from route_upstream table.

9b1dd9:~ v3$ curl -s -X DELETE https://ingresspipe.io:8443/service/adminservice2/route/db_requests -H "Authorization: Bearer treehugger" | python -m json.tool
{
    "errors": [
        {
            "extensions": {
                "code": "constraint-violation",
                "path": "$"
            },
            "message": "Foreign key violation. update or delete on table \"route\" violates foreign key constraint \"route_upstream_route_id_fkey\" on table \"route_upstream\""
        }
    ]
}

Webapp: Take port and IP needed to talk to db configurable

Webhttp is a client to DB. It talks to DB over IP:Port. Make this IP:Port configurable

Enroute-cp: No listener programmed when upstream values are not valid

        "saaras_db_proxy": [                                                                                                                                                                        [21/903]
            {
                "create_ts": "2019-09-09T17:23:26.540791+00:00",
                "proxy_id": 1,
                "proxy_name": "adminproxy",
                "proxy_services": [
                    {
                        "service": {
                            "create_ts": "2019-09-09T17:31:56.327328+00:00",
                            "fqdn": "ingresspipe.io",
                            "routes": [
                                {
                                    "create_ts": "2019-09-09T17:34:50.108189+00:00",
                                    "route_id": 1,
                                    "route_name": "dp_requests",
                                    "route_prefix": "/v1/graphql",
                                    "route_upstreams": [],
                                    "update_ts": "2019-09-09T17:34:50.108189+00:00"
                                },
                                {
                                    "create_ts": "2019-09-09T17:35:05.120026+00:00",
                                    "route_id": 2,
                                    "route_name": "webapp_requests",
                                    "route_prefix": "/",
                                    "route_upstreams": [],
                                    "update_ts": "2019-09-09T18:32:55.590527+00:00"
                                },
                                {
                                    "create_ts": "2019-09-09T18:28:18.684243+00:00",
                                    "route_id": 3,
                                    "route_name": "db_requests",
                                    "route_prefix": null,
                                    "route_upstreams": [
                                        {
                                            "upstream": {
                                                "upstream_hc_healthythresholdcount": null,
                                                "upstream_hc_host": null,
                                                "upstream_hc_intervalseconds": null,
                                                "upstream_hc_path": null,
                                                "upstream_hc_timeoutseconds": null,
                                                "upstream_hc_unhealthythresholdcount": null,
                                                "upstream_id": 3,
                                                "upstream_ip": null,
                                                "upstream_name": "db_serve_process",
                                                "upstream_port": null,
                                                "upstream_strategy": null,
                                                "upstream_validation_cacertificate": null,
                                                "upstream_validation_subjectname": null,
                                                "upstream_weight": 100
                                            }
                                        }
                                    ],
                                    "update_ts": "2019-09-09T18:33:56.980135+00:00"
                                }
                            ],
                            "service_id": 1,
                            "service_name": "adminservice",

For the above config, a listener is not configured. It is probably because health check is not programmed for the upstream. Check if this is expected behavior? The upstream here is hasura and doesn't necessarily have a health check. Should we program a listener in absence of a health check?

A work around is to setup a health check for path /v1/version right now, but this may not be the case in production.

saarasio / enroute Goto Github PK

enroute's People

Contributors

Stargazers

Watchers

Forkers

enroute's Issues

Recommend Projects

Recommend Topics

Recommend Org