Giter Club home page Giter Club logo

googlecloudplatform / esp-v2 Goto Github PK

View Code? Open in Web Editor NEW
261.0 37.0 167.0 9.09 MB

A service proxy that provides API management capabilities using Google Service Infrastructure.

Home Page: https://cloud.google.com/endpoints/

License: Apache License 2.0

Python 6.54% Makefile 0.51% Shell 4.84% Go 62.29% C++ 23.44% Lua 0.09% Dockerfile 0.09% JavaScript 0.39% Starlark 1.82%
envoyproxy cpp go proxy api google cloud-endpoints api-gateway google-cloud

esp-v2's Introduction

Google Cloud Platform ESPv2

Google Cloud Platform ESPv2 is a general-purpose L7 service proxy that enables API management capabilities for JSON/REST or gRPC API services. ESPv2 integrates with Google Service Infrastructure to provide policy checks and telemetry reports.

ESPv2 is the next iteration of ESP. The current implementation of ESPv2 uses Envoy as a service proxy.

Table of Contents

Introduction

Google Cloud Endpoints and ESPv2 provide:

  • Easy Adoption: The API service can be implemented in any coding language using any IDLs.

  • Multiple Deployment Modes: Deploy ESPv2 as a sidecar for local backends or as a reverse proxy for remote backends.

  • Platform Flexibility: Support the deployment on any cloud or hybrid environment.

  • Superb Performance and Scalability: ESPv2 has low latency and high throughput.

ESPv2 includes two components:

  • Config Manager: Control plane to configure the Envoy proxy
  • Envoy: Data plane to process API requests/responses

Config Manager configures the data plane's Envoy filters dynamically via the Google Service Management API and flags specified by the API producer.

Envoy (with our custom filters) handles API calls using Service Infrastructure, Google's foundational platform for creating, managing, and consuming APIs and services.

Architecture

Features

ESPv2 provides powerful enterprise-ready features, such as:

  • Multiple Authentication Methods: Authenticate applications using API Keys. Authenticate users using any authentication platform that conforms to JSON Web Token RFC 7519. This includes Firebase Authentication, Auth0, Okta, Google ID tokens, and Google Service Accounts.

  • Quotas and Rate Limiting: Control the rate at which applications can call your APIs.

  • gRPC Transcoding: Allow clients to use HTTP/JSON to call backends that only support gRPC. ESPv2 will map HTTP/JSON requests (and their parameters) to gRPC methods (and their parameters and return types).

  • API Telemetry: Monitor API metrics such as error rates, response latencies, request size, etc. on Google Cloud Platform. ESPv2 also writes access logs for each request, providing insight into HTTP headers and response codes.

  • Application Observability: Understand and debug problems with your API or ESPv2 by viewing traces and application logs.

  • Multi Transport Protocols: ESPv2 supports HTTP/1.1, HTTP/2, web sockets, gRPC and gRPC-web.

  • Security: ESPv2 supports TLS for downstream and upstream, also mTLS for upstream connections, for both HTTP and gRPC.

  • Dynamic Routing: When deployed on non-serverless platforms (GCE, GKE, K8S) as a sidecar, ESPv2 routes all traffic to a single local backend. When deployed on serverless platforms (Cloud Run) as a reverse proxy, ESPv2 can route traffic across multiple remote upstream services.

  • Health Checks: ESPv2 can actively monitor your services.

Getting Started

The official Google Cloud Endpoints documentation contains tutorials and detailed documentation on deploying ESPv2 and managing API traffic.

Getting started with Extensible Service Proxy V2(ESPv2) tutorials:

Configuring ESPv2:

Understanding ESPv2:

ESPv2 Releases

ESPv2 is released as two docker images:

ESPv2 vs ESP

ESPv2 is the next iteration of ESP. Instead of an nginx-based data plane, ESPv2 uses Envoy. See the architecture overview for more info on ESPv2.

ESPv2 is designed to be mostly backwards-compatible with ESP (with the exception of a few startup flags). API producers do not need to modify the Endpoints Service Configuration to use ESPv2. Please refer to the migration guide for more details on the incompatible changes.

For serverless reverse proxy deployments, it is recommended to use ESPv2 instead of ESP. For sidecar deployments, either ESPv2 or ESP can be used, but ESPv2 will receive more features.

Repository Structure

  • api: Envoy Filter Configurations developed in ESPv2

  • doc: Extended documentation (tutorials, architecture, use cases, etc.)

  • docker: Scripts for packaging ESPv2 in a Docker image for releases

  • examples: Examples to configure ESPv2

  • prow: Prow based test automation scripts

  • scripts: Scripts used for build and release ESPv2

  • src: ESPv2 source code, including Envoy Filters and Config Manager

  • tests: Integration and end-to-end tests for ESPv2

  • tools: Assorted tooling

Contributing and Support

Please join the google-cloud-endpoints Google group for announcements on ESPv2.

If you need support from the ESPv2 engineering team:

  • For issues using ESPv2 with Google Cloud Endpoints, post in the Google Group.
  • For feature requests and bug reports specific to the ESPv2 codebase, file a Github Issue.

Your contributions are welcome:

License

Apache v2 - Google LLC

esp-v2's People

Contributors

bochunz avatar brucearctor avatar dafudeng avatar dchakarwarti avatar elliot-xq avatar florayan avatar gopkarthik avatar ioggstream avatar jcking avatar jiachenwang avatar jlxia avatar kylewiese avatar maor-rz avatar nareddyt avatar numanelahi avatar paulhong01 avatar qiwzhang avatar rjerrems avatar same-id avatar sgammon avatar shuoyang2016 avatar siqidong avatar syangabq avatar taoxuy avatar terryc78 avatar toddbeckman-g avatar tomboc avatar wapa5pow avatar yaoluz avatar ygong avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

taoxuy bbamsch darrenchiu nareddyt rodened jlxia igornes toddbeckman-g blubnana qiwzhang vlampreia d4t4mon florayan danielwarnersmith mishamilovidov drowzy brucearctor masafumi-i hgarciao saraeriksen mshale josephglaspie tuskan julienrubyx mdlglobal-atlassian-net s3725150 zerofruit davidroossien-okta fiifibaidoo lurkercha bozz-srinikorn damadei-google georgetda ulfsv jacksalvador chanteckwei dev-asta chenkuansun sudeshwasnik istiaq107 longlehoang nathandiegues taksh018 gojohnkevin sabitha-kumar codecakes bhardwajparvesh marcodv jzdataminer valery-barysok enixdark yamamasa isabella232 wolfthom dkcpa68 rajeevdaidec abenamer kevjeon akira-isegawa kichlids bozzlab pojac3 takeshiemoto thiagobg ramaiahkk sgammon thinhvoxuan caibadal5 tuemu surabhisharma12 isaackolamide matthiasgasser darkpiv tomboc abhishek-anand-78 dio ioggstream vogelz salimzhulkhrni liammaru iva-marusic iamthatiam777 deus-service global-localhost global19 global19-atlassian-net klimaschkas alichnewsky shankarbcone mariemzyr drthornt connorbrett arpt fareeda0 varungbt frank00125 alx-andru armughan-derq lamhai1401 bmehta1307

esp-v2's Issues

Local testing, multiple backends?

Hello!

I use Google Endpoints in front of multiple cloud functions.
I have multiple routes specified that resolves to different x-google-backend addresses (for different cloud functions).

  • /a => cloud function "a"
  • /b => cloud function "b"

I assume this is a pretty common use case, and everything works fine running in GCP.

When developing locally, I run each cloud function on a separate port:

I have managed to get ESPv2 running locally using "--backend" argument.
But when I try to use "backend.rules" in my service.json (ENDPOINTS_SERVICE_PATH) config file - I get some 503 error from token_subscriber.cc:

[20][init][src/envoy/token/token_subscriber.cc:125] TokenSubscriber(http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/identity?format=standard&audience=https://localhost:3001/): failed: 503

How do I configure ESPv2 locally to support multiple local backend services?

Cloudrun script won't run when region limited to EU by policy

Hi, I'm trying to run endpoints for cloudrun with policy constraint for EU only. I'm getting this error when trying to tun script gcloud_build_image

ERROR: (gcloud.builds.submit) HTTPError 412: 'us' violates constraint 'constraints/gcp.resourceLocations'

Duplicate Alt-Svc HTTP response header

With ESPv2 in front of a Cloud Run backend service, a request returned from the ESPv2 service yields two identical Alt-Svc headers:

Dupe Alt-Svc

This is messy and adds to response sizes.
I don't think it's ESPv2 adding it, it looks to be the "Google Frontend" web server.
You may want to suggest to the Google Frontend team that they consider merging these headers into one if they are both identical.😉

Errors when building on macOS

Hello esteemed ESPv2 authors,

I am having trouble building ESPv2 via Bazel on macOS. Here is the error I get:

2020/08/03 01:36:35 Downloading https://releases.bazel.build/3.0.0/release/bazel-3.0.0-darwin-x86_64...
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
DEBUG: Rule 'io_bazel_rules_python' indicated that a canonical reproducible form can be obtained by modifying arguments shallow_since = "1523916428 -0700"
DEBUG: Call stack for the definition of repository 'io_bazel_rules_python' which is a git_repository (rule definition at /private/var/tmp/_bazel_sam.g/c5262b9dd666f2a9cad5b9bc4d2c9fe6/external/bazel_tools/tools/build_defs/repo/git.bzl:195:18):
 - <builtin>
 - /Volumes/KICKSTART/esp-v2/bazel/bazel_rules_python.bzl:22:9
 - /Volumes/KICKSTART/esp-v2/bazel/repositories.bzl:22:5
 - /Volumes/KICKSTART/esp-v2/WORKSPACE:91:1
ERROR: /Volumes/KICKSTART/esp-v2/tests/endpoints/grpc_interop/proto/BUILD:1:1: at least one argument must be given to the 'package' function
ERROR: /Volumes/KICKSTART/esp-v2/tests/env/testdata/BUILD:1:1: at least one argument must be given to the 'package' function
ERROR: /Volumes/KICKSTART/esp-v2/examples/BUILD:1:1: at least one argument must be given to the 'package' function
ERROR: package contains errors: tests/env/testdata
ERROR: package contains errors: tests/endpoints/grpc_interop/proto
ERROR: package contains errors: examples
ERROR: error loading package 'examples': Package 'examples' contains errors
Loading: 28 packages loaded

This, after running bazelisk query //.... Any ideas?

Don't use IMDS for backend auth when running with `--non_gcp`

This is a small usability enhancement based on customer confusion in issue #352.

When running ESPv2 with the --non_gcp flag, it's expected that we don't contact IMDS, as IMDS is a GPC compute-specific server. We honor this for most ESPv2 runtime components except backend authentication.

When backend.rules are configured for remote backends, ESPv2 by default contacts IMDS to retrieve the JWT for remote backend authentication. It will always contact IMDS for backend auth, unless the BackendRule has disable_auth = true.

So if there is a backend rule that requires a JWT but we are running ESPv2 with the --non_gcp flag, we should either:

  1. Reject the config, display an error to disable auth for the backend rule.
  2. Allow the config, but implicitly overwrite the backend rule to disable auth.
  3. Allow the config unmodified and display a warning. Preserves the current behavior, which results in Envoy never starting because the backend auth filter can't contact IMDS. I'm definitely against this behavior, we shouldn't be making requests that we know will fail.

1 and 2 are technically breaking changes, but it's probably OK to do so. The current behavior is not intended. I'm learning towards option 1, it's the clearest option. @qiwzhang @TAOXUY any thoughts?

Some implementation considerations:

  • Update this integration test. Currently it only tests ESPv2 with a local backend (sidecar mode), so it did not catch the backend auth filter making requests to IMDS.
  • Backend auth also supports contacting IAM instead of IMDS. This is only used by Cloud API Gateway, and they'll always run on GCP. But for consistency, we should still follow the same implementation decision for IAM as IMDS.

POST requests do not go through ESPv2

My iOS app makes POST requests to Google Cloud Run container which hosts my Django Rest Framework Project.

I've set up public ESP(deployed in Cloud Run with proper service account) to serve as a gateway for private Cloud Run Container. In ESP I'm using Firebase Security Definition (which I've deactivated for the testing, because it works and it's not the point of the test)

All GET requests to ESP work fine: they successfully get routed to Cloud Run Container (for example when I request non existent /hello at www.example.com/hello, I get Django Debug 404)

However, for some reason, POST requests are not routed by ESP (for example I'm trying to access existing /upload at www.example.com/upload I get standard 404). When I try POST request url in the browser I get this "Path does not match any requirement URI template." error message. Googling that message has led me to this GitHub. May be somebody can help me figure it out.

All POST requests work fine if sent directly to Cloud Run Container

I've tried disabling Django csrf, setting APPEND_SLASH to false, nothing seems to work. All POST requests result in 404.

All testing done with Postman.

Proto constraint validation failed (Duplicated pattern or invalid pattern)

The following error occurs even though the URL pattern is different.

esp    | I0731 10:22:58.984492       6 config_manager.go:297] respond type.googleapis.com/envoy.config.listener.v3.Listener[] version "" with version "2020-07-XXr1"
esp    | W0731 10:22:59.014 19 envoy] [19][config][external/envoy/source/common/config/grpc_subscription_impl.cc:100] gRPC config for type.googleapis.com/envoy.config.listener.v3.Listener rejected: Error adding/updating listener(s) ingress_listener: Proto constraint validation failed (Duplicated pattern or invalid pattern): uri_template: \"/v1/companies/{id}/users/{user_id}\"\nhttp_method: \"OPTIONS\"

My gRPC service is below.

service UserService {
  rpc Get(GetRequest) returns (User) {
    option (google.api.http) = {
      get: "/v1/companies/{id}/users/{user_id}"
    };
  }
  rpc Update(UpdateRequest) returns (User) {
    option (google.api.http) = {
      put: "/v1/companies/{id}/users/{user.id}"
      body: "user"
    };
  }
}

My protobuf messages below.

message GetRequest {
  string id = 1;
  string user_id = 2;
}

message User {
  string id = 1;
  string name = 2;
}

message UpdateRequest {
  string id = 1;
  User user = 2;
}

Is there a good way to avoid this error?

Support for a single app engine route (one route of one App Engine service in one GCP Project)?

Is it possible to get esp-v2 working with a single route in an app engine application? I thought perhaps this could be done by using a Serveless NEG and the IAP attached to that, but I get inexplicable errors with every variation of that strategy I've tried.

With or without IAP enabled, when I connect to the cloud endpoint, I get the following error:

{
    "message": "upstream connect error or disconnect/reset before headers. reset reason: connection failure",
    "code": 503
}

In the logs I see this, in case that helps:

D0917 04:32:00.835 28 envoy] [28][connection][external/envoy/source/common/network/connection_impl.cc:753] 
[C23] connecting to 127.0.0.1:8082
...
 upstream reset: reset reason connection failure

Any advice or pointers to docs would be welcome, thanks!

Incorrect padding for X-Endpoint-API-UserInfo header

Per docs, the X-Endpoint-API-UserInfo header is supposed to be base64 encoded list of the original claims. However, an example value I received just now is of length 354 - which is missing == at the end.

Confirmed by using Python base64.decodebytes(user_info_encoded.encode()).decode() to decode which fails unless I add ==.

Add start-up flag to set per_connection_buffer_limit_bytes setting in envoy

Description of problem

It seems that the response size of json-transcoded responses processed by espv2 is limited by the per_connection_buffer_limit_bytes setting parameter in envoy. This defaults to 1mb, and all json transcoded responses that are larger than this will fail. GRPC responses are

See envoy related issue in gitlab for more info: Size limit on gRPC-JSON transcoder with google.api.HttpBody as output?

Posted a more detailed description of the problem in the endpoints discussion group here:
Increasing max size of response for requests that are passed through json transcoding

Proposed resolution

Add a startup flag for espv2 that allows to set the per_connection_buffer_limit_bytes setting parameter in envoy to a custom buffer limit. This will allow for the espv2 gateway container to process and return larger responses.

API Key with IP address restriction blocks all IP including whitelisted

I've created ESPv2 Cloud Endpoint for my API, everything works fine except when I enabling IP address restriction in API Key, I get a message "PERMISSION_DENIED: IP address blocked." for all requests including whitelisted IP addresses.

What you expected to happen:
Block all requests except whitelisted.

Steps to reproduce:
Create IP restriction for API Key for Endpoint base on ESPv2.

I've checked the endpoint logs and see the internal google IP address instead of my IP address from which I send a request, a corresponding screenshot is attached.
Screenshot 2020-06-13 at 14 41 40

GoogleRE2.max_program_size field is deprecated in Envoy

I get a bunch of these warnings from Envoy in the ESPV2 logs:

2020-09-04 10:16:14.235 PDTW0904 17:16:14.035 16 envoy] [16][misc][external/envoy/source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.type.matcher.v3.RegexMatcher.GoogleRE2 Using deprecated option \'envoy.type.matcher.v3.RegexMatcher.GoogleRE2.max_program_size\' from file regex.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.

From Envoy v1.15.0 release notes:

The GoogleRE2.max_program_size field is now deprecated. Management servers are expected to validate regexp program sizes instead of expecting the client to do it. Alternatively, the max program size can be enforced by Envoy via runtime.

Feature request: allow passing the full OpenAPI config to the serverless runtime image as an environment variable

I'd like to request allowing passing the full value of the OpenAPI Service Management config as an environment variable, rather than just setting the path to the config as an environment variable.

Example:

gcloud run deploy ... \
  --image="gcr.io/endpoints-release/endpoints-runtime-serverless:2" \
  --set-env-vars=ENDPOINTS_SERVICE_CONFIG="{ ... }" \

This has several advantages, most notably that:

  • The step of building your own image is eliminated.
  • Updating the configuration becomes much faster because you only need to change an environment variable.
  • Overall simplification.

I also think you don't have to get rid of the old configuration methods, you can just add this as another option, similar to how both ENDPOINTS_SERVICE_NAME and ENDPOINTS_SERVICE_PATH are still available.

Add option to print áll fields to match grpc-gateway

Hi,

the grpc-gateway has an option called EmitDefaults that allows us to include JSON fields that have a default value. This includes non-primitive fields, namely arrays and objects.

Your option always_print_primitive_fields on the other hand is limited to primitive fields. Empty child objects are never included in the response.

This means that when running our services locally, the API is a little different than when we run on GCP. It would be ideal if we could expand all fields in both environments.

Vital D'haveloose
UZ Leuven

Bug in gcloud_build_image

If a user doesn't pass the -i parameter to gcloud_build_image then ESP_FULL_VERSION doesn't get defined.

esp-v2/docker/serverless/gcloud_build_image

Line 37 in db1ec6b

ESP_FULL_VERSION="custom"

In turn, this causes errors when tagging the resulting image.

esp-v2/docker/serverless/gcloud_build_image

Line 96 in db1ec6b

NEW_IMAGE="gcr.io/${PROJECT}/endpoints-runtime-serverless:${ESP_FULL_VERSION}-${SERVICE}-${CONFIG_ID}"

I removed ${ESP_FULL_VERSION}- from the image tag since that seems to be in line with the name convention referred to in the documentation here.

ESPv2 fails to start when service config has duplicated camel_case name to json name mapping.

Moving discussion from Google Group:

I'm trying to run ESPv2 locally to replace ESPv1 during development. However, although ESPv1 works just fine, ESPv2 keeps bouncing the container. I've set my service account to project owner to eliminate permission issues.

Here is my Docker compose:

esp:
    image: gcr.io/endpoints-release/endpoints-runtime:2
    restart: unless-stopped
    ports:
      - "127.0.0.1:8089:8089"
    volumes:
      - ${USERPROFILE}/.google-application-credentials.json:/.google-application-credentials.json:ro
    environment:
      - GOOGLE_APPLICATION_CREDENTIALS=/.google-application-credentials.json
    command:
      - "--service=<SERVICE_NAME>"
      - "--rollout_strategy=managed"
      - "--listener_port=8089"
      - "--backend=http://dev:8081"
      - "--service_account_key=/.google-application-credentials.json"
      - "--cors_preset=basic"
      - "--cors_allow_origin=http://localhost:8080"
      - "--non_gcp"
      - "--enable_debug"
    links:
      - dev

Error log output (this constantly repeats):

I0604 18:19:01.038510      10 main.go:29] Output path: /tmp/bootstrap.json
I0604 18:19:01.038623      10 flags.go:105] Common options: {AdminAddress:0.0.0.0 AdminPort:0 DiscoveryPort:8790 
Node:ESPv2 DisableTracing:true TracingProjectId: TracingStackdriverAddress: TracingSamplingRate:0.001 
TracingIncomingContext: TracingOutgoingContext: TracingMaxNumAttributes:32 TracingMaxNumAnnotations:32 
TracingMaxNumMessageEvents:128 TracingMaxNumLinks:128 NonGCP:false HttpRequestTimeout:5s 
MetadataURL:http://169.254.169.254/computeMetadata IamURL:https://iamcredentials.googleapis.com 
ServiceControlCredentials:<nil> BackendAuthCredentials:<nil>}
I0604 18:19:01.039175      10 flags.go:39] ADS Bootstrapper options: {CommonOptions:{AdminAddress:0.0.0.0 
AdminPort:0 DiscoveryPort:8790 Node:ESPv2 DisableTracing:true TracingProjectId: TracingStackdriverAddress: 
TracingSamplingRate:0.001 TracingIncomingContext: TracingOutgoingContext: TracingMaxNumAttributes:32 
TracingMaxNumAnnotations:32 TracingMaxNumMessageEvents:128 TracingMaxNumLinks:128 NonGCP:false 
HttpRequestTimeout:5s MetadataURL:http://169.254.169.254/computeMetadata 
IamURL:https://iamcredentials.googleapis.com ServiceControlCredentials:<nil> BackendAuthCredentials:<nil>} 
AdsConnectTimeout:10s DiscoveryAddress:127.0.0.1:8790}

CRITICAL: Config Manager is down, killing all processes.

And the standard log output (this also repeats):

Starting Config Manager with args: ['bin/configmanager', '--logtostderr', '--backend_address', 'http://dev:8081', '--rollout_strategy', 'managed', '--v', '1', '--listener_port', '8089', '--service', '<SERVICE NAME>', '--disable_tracing', '--cors_preset', 'basic', '--cors_allow_origin', 'http://localhost:8080', '--cors_allow_origin_regex', '', '--cors_allow_methods', 'GET, POST, PUT, PATCH, DELETE, OPTIONS', '--cors_allow_headers', 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization', '--cors_expose_headers', 'Content-Length,Content-Range', '--service_account_key', '/.google-application-credentials.json', '--non_gcp', '--suppress_envoy_headers=false']
['bin/bootstrap', '--logtostderr', '--admin_port', '0', '--disable_tracing', '/tmp/bootstrap.json']
I0604 18:19:01.034435       6 flags.go:105] Common options: {AdminAddress:0.0.0.0 AdminPort:8001 DiscoveryPort:8790 Node:ESPv2 DisableTracing:true TracingProjectId: TracingStackdriverAddress: TracingSamplingRate:0.001 TracingIncomingContext: TracingOutgoingContext: TracingMaxNumAttributes:32 TracingMaxNumAnnotations:32 TracingMaxNumMessageEvents:128 TracingMaxNumLinks:128 NonGCP:true HttpRequestTimeout:5s MetadataURL:http://169.254.169.254/computeMetadata IamURL:https://iamcredentials.googleapis.com ServiceControlCredentials:<nil> BackendAuthCredentials:<nil>}
I0604 18:19:01.034903       6 flags.go:173] Config Generator options: {CommonOptions:{AdminAddress:0.0.0.0 AdminPort:8001 DiscoveryPort:8790 Node:ESPv2 DisableTracing:true TracingProjectId: TracingStackdriverAddress: TracingSamplingRate:0.001 TracingIncomingContext: TracingOutgoingContext: TracingMaxNumAttributes:32 TracingMaxNumAnnotations:32 TracingMaxNumMessageEvents:128 TracingMaxNumLinks:128 NonGCP:true HttpRequestTimeout:5s MetadataURL:http://169.254.169.254/computeMetadata IamURL:https://iamcredentials.googleapis.com ServiceControlCredentials:<nil> BackendAuthCredentials:<nil>} CorsAllowCredentials:false CorsAllowHeaders:DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization CorsAllowMethods:GET, POST, PUT, PATCH, DELETE, OPTIONS CorsAllowOrigin:http://localhost:8080 CorsAllowOriginRegex: CorsExposeHeaders:Content-Length,Content-Range CorsPreset:basic BackendDnsLookupFamily:auto ClusterConnectTimeout:20s BackendAddress:http://dev:8081 ListenerAddress:0.0.0.0 Healthz: ServiceManagementURL:https://servicemanagement.googleapis.com ServiceControlURL:https://servicecontrol.googleapis.com ListenerPort:8089 SslServerCertPath: SslClientCertPath: SslMinimumProtocol: SslMaximumProtocol: EnableHSTS:false RootCertsPath:/etc/ssl/certs/ca-certificates.crt DnsResolverAddresses: ServiceAccountKey:/.google-application-credentials.json SkipJwtAuthnFilter:false SkipServiceControlFilter:false AccessLog: AccessLogFormat: EnvoyUseRemoteAddress:false EnvoyXffNumTrustedHops:2 LogJwtPayloads: LogRequestHeaders: LogResponseHeaders: MinStreamReportIntervalMs:0 SuppressEnvoyHeaders:false UnderscoresInHeaders:false ServiceControlNetworkFailOpen:true JwksCacheDurationInS:300 ScCheckTimeoutMs:0 ScQuotaTimeoutMs:0 ScReportTimeoutMs:0 ScCheckRetries:-1 ScQuotaRetries:-1 ScReportRetries:-1 ComputePlatformOverride: TranscodingAlwaysPrintPrimitiveFields:false TranscodingAlwaysPrintEnumsAsInts:false TranscodingPreserveProtoFieldNames:false TranscodingIgnoreQueryParameters: TranscodingIgnoreUnknownQueryParameters:false}
Starting Envoy with args: ['bin/envoy', '-c', '/tmp/bootstrap.json', '--disable-hot-restart', '--log-format %L%m%d %T.%e %t envoy] [%t][%n]%v', '--log-format-escaped', '-l debug']
I0604 18:19:01.089 22 envoy] [22][main][external/envoy/source/server/server.cc:293] initializing epoch 0 (hot restart version=disabled)
I0604 18:19:01.089 22 envoy] [22][main][external/envoy/source/server/server.cc:295] statically linked extensions:
I0604 18:19:01.089 22 envoy] [22][main][external/envoy/source/server/server.cc:297]   envoy.stats_sinks: envoy.metrics_service, envoy.stat_sinks.metrics_service
I0604 18:19:01.089 22 envoy] [22][main][external/envoy/source/server/server.cc:297]   envoy.retry_host_predicates: envoy.retry_host_predicates.previous_hosts
I0604 18:19:01.089 22 envoy] [22][main][external/envoy/source/server/server.cc:297]   envoy.retry_priorities: envoy.retry_priorities.previous_priorities
I0604 18:19:01.089 22 envoy] [22][main][external/envoy/source/server/server.cc:297]   envoy.udp_listeners: raw_udp_listener
I0604 18:19:01.089 22 envoy] [22][main][external/envoy/source/server/server.cc:297]   envoy.access_loggers: envoy.access_loggers.file, envoy.access_loggers.http_grpc, envoy.file_access_log, envoy.http_grpc_access_log
I0604 18:19:01.089 22 envoy] [22][main][external/envoy/source/server/server.cc:297]   envoy.clusters: envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns
I0604 18:19:01.089 22 envoy] [22][main][external/envoy/source/server/server.cc:297]   envoy.resolvers: envoy.ip
I0604 18:19:01.089 22 envoy] [22][main][external/envoy/source/server/server.cc:297]   envoy.grpc_credentials: envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata
I0604 18:19:01.089 22 envoy] [22][main][external/envoy/source/server/server.cc:297]   envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.tap, envoy.transport_sockets.tls, raw_buffer, tls
I0604 18:19:01.089 22 envoy] [22][main][external/envoy/source/server/server.cc:297]   envoy.tracers: envoy.tracers.opencensus
I0604 18:19:01.089 22 envoy] [22][main][external/envoy/source/server/server.cc:297]   envoy.filters.listener: envoy.filters.listener.original_dst, envoy.filters.listener.proxy_protocol, envoy.filters.listener.tls_inspector, envoy.listener.original_dst, envoy.listener.proxy_protocol, envoy.listener.tls_inspector
I0604 18:19:01.089 22 envoy] [22][main][external/envoy/source/server/server.cc:297]   envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.tap, envoy.transport_sockets.tls, raw_buffer, tls
I0604 18:19:01.089 22 envoy] [22][main][external/envoy/source/server/server.cc:297]   envoy.resource_monitors: envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource
I0604 18:19:01.089 22 envoy] [22][main][external/envoy/source/server/server.cc:297]   envoy.filters.http: envoy.buffer, envoy.cors, envoy.ext_authz, envoy.fault, envoy.filters.http.backend_auth, envoy.filters.http.backend_routing, envoy.filters.http.buffer, envoy.filters.http.cors, envoy.filters.http.ext_authz, envoy.filters.http.fault, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.gzip, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.jwt_authn, envoy.filters.http.path_matcher, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.service_control, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.gzip, envoy.health_check, envoy.ip_tagging, envoy.rate_limit, envoy.router
I0604 18:19:01.089 22 envoy] [22][main][external/envoy/source/server/server.cc:297]   envoy.filters.network: envoy.client_ssl_auth, envoy.ext_authz, envoy.filters.network.client_ssl_auth, envoy.filters.network.ext_authz, envoy.filters.network.http_connection_manager, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.sni_cluster, envoy.filters.network.tcp_proxy, envoy.http_connection_manager, envoy.ratelimit, envoy.tcp_proxy
W0604 18:19:01.098 22 envoy] [22][main][external/envoy/source/server/server.cc:384] No admin address given, so no admin HTTP server started.
D0604 18:19:01.098 22 envoy] [22][main][external/envoy/source/server/overload_manager_impl.cc:184] No overload action is configured for envoy.overload_actions.shrink_heap.
D0604 18:19:01.098 22 envoy] [22][main][external/envoy/source/server/overload_manager_impl.cc:184] No overload action is configured for envoy.overload_actions.stop_accepting_connections.
D0604 18:19:01.098 22 envoy] [22][main][external/envoy/source/server/overload_manager_impl.cc:184] No overload action is configured for envoy.overload_actions.stop_accepting_connections.
D0604 18:19:01.098 22 envoy] [22][main][external/envoy/source/server/overload_manager_impl.cc:184] No overload action is configured for envoy.overload_actions.stop_accepting_connections.
D0604 18:19:01.098 22 envoy] [22][main][external/envoy/source/server/overload_manager_impl.cc:184] No overload action is configured for envoy.overload_actions.stop_accepting_connections.
D0604 18:19:01.098 22 envoy] [22][main][external/envoy/source/server/overload_manager_impl.cc:184] No overload action is configured for envoy.overload_actions.stop_accepting_connections.
D0604 18:19:01.098 22 envoy] [22][main][external/envoy/source/server/overload_manager_impl.cc:184] No overload action is configured for envoy.overload_actions.stop_accepting_connections.
D0604 18:19:01.098 22 envoy] [22][main][external/envoy/source/server/overload_manager_impl.cc:184] No overload action is configured for envoy.overload_actions.stop_accepting_connections.
D0604 18:19:01.098 22 envoy] [22][main][external/envoy/source/server/overload_manager_impl.cc:184] No overload action is configured for envoy.overload_actions.stop_accepting_connections.
I0604 18:19:01.098 22 envoy] [22][main][external/envoy/source/server/server.cc:525] runtime: layers:\n  - name: base\n    static_layer:\n      {}\n  - name: admin\n    admin_layer:\n      {}
I0604 18:19:01.099 22 envoy] [22][config][external/envoy/source/server/configuration_impl.cc:103] loading tracing configuration
I0604 18:19:01.099 22 envoy] [22][config][external/envoy/source/server/configuration_impl.cc:69] loading 0 static secret(s)
I0604 18:19:01.099 22 envoy] [22][config][external/envoy/source/server/configuration_impl.cc:75] loading 1 cluster(s)
D0604 18:19:01.100 22 envoy] [22][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:978] adding TLS initial cluster ads_cluster
D0604 18:19:01.100 22 envoy] [22][upstream][external/envoy/source/common/upstream/upstream_impl.cc:278] transport socket match, socket default selected for host with address 127.0.0.1:8790
D0604 18:19:01.100 22 envoy] [22][upstream][external/envoy/source/common/upstream/strict_dns_cluster.cc:146] DNS hosts have changed for 127.0.0.1
D0604 18:19:01.100 22 envoy] [22][upstream][external/envoy/source/common/upstream/strict_dns_cluster.cc:167] DNS refresh rate reset for 127.0.0.1, refresh rate 5000 ms
D0604 18:19:01.100 22 envoy] [22][upstream][external/envoy/source/common/upstream/upstream_impl.cc:971] initializing Primary cluster ads_cluster completed
D0604 18:19:01.100 22 envoy] [22][init][external/envoy/source/common/init/manager_impl.cc:45] init manager Cluster ads_cluster contains no targets
D0604 18:19:01.100 22 envoy] [22][init][external/envoy/source/common/init/watcher_impl.cc:14] init manager Cluster ads_cluster initialized, notifying ClusterImplBase
D0604 18:19:01.100 22 envoy] [22][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:1137] membership update for TLS cluster ads_cluster added 1 removed 0
D0604 18:19:01.100 22 envoy] [22][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:105] cm init: init complete: cluster=ads_cluster primary=0 secondary=0
D0604 18:19:01.100 22 envoy] [22][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:125] maybe finish initialize state: 0
D0604 18:19:01.100 22 envoy] [22][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:77] cm init: adding: cluster=ads_cluster primary=0 secondary=0
D0604 18:19:01.100 22 envoy] [22][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:125] maybe finish initialize state: 1
D0604 18:19:01.100 22 envoy] [22][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:134] maybe finish initialize primary init clusters empty: true
D0604 18:19:01.101 22 envoy] [22][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:47] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.service.discovery.v3.DiscoveryRequest) returns (stream .envoy.service.discovery.v3.DiscoveryResponse);

D0604 18:19:01.101 22 envoy] [22][router][external/envoy/source/common/router/router.cc:435] [C0][S1482019712714766912] cluster \'ads_cluster\' match for URL \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'
D0604 18:19:01.101 22 envoy] [22][router][external/envoy/source/common/router/router.cc:590] [C0][S1482019712714766912] router decoding headers:\n\':method\', \'POST\'\n\':path\', \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'\n\':authority\', \'ads_cluster\'\n\':scheme\', \'http\'\n\'te\', \'trailers\'\n\'content-type\', \'application/grpc\'\n\'x-envoy-internal\', \'true\'\n\'x-forwarded-for\', \'172.18.0.8\'

D0604 18:19:01.101 22 envoy] [22][pool][external/envoy/source/common/http/conn_pool_base.cc:346] queueing request due to no available connections
D0604 18:19:01.101 22 envoy] [22][pool][external/envoy/source/common/http/conn_pool_base.cc:47] creating a new connection
D0604 18:19:01.101 22 envoy] [22][client][external/envoy/source/common/http/codec_client.cc:35] [C0] connecting
D0604 18:19:01.101 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:753] [C0] connecting to 127.0.0.1:8790
D0604 18:19:01.101 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:762] [C0] connection in progress
D0604 18:19:01.101 22 envoy] [22][http2][external/envoy/source/common/http/http2/codec_impl.cc:1011] [C0] updating connection-level initial window size to 268435456
I0604 18:19:01.101 22 envoy] [22][config][external/envoy/source/server/configuration_impl.cc:79] loading 0 listener(s)
I0604 18:19:01.101 22 envoy] [22][config][external/envoy/source/server/configuration_impl.cc:129] loading stats sink configuration
D0604 18:19:01.101 22 envoy] [22][init][external/envoy/source/common/init/manager_impl.cc:20] added target LDS to init manager Server
D0604 18:19:01.101 22 envoy] [22][init][external/envoy/source/common/init/manager_impl.cc:45] init manager RTDS contains no targets
D0604 18:19:01.101 22 envoy] [22][init][external/envoy/source/common/init/watcher_impl.cc:14] init manager RTDS initialized, notifying RDTS
I0604 18:19:01.101 22 envoy] [22][runtime][external/envoy/source/common/runtime/runtime_impl.cc:526] RTDS has finished initialization
D0604 18:19:01.101 22 envoy] [22][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:196] continue initializing secondary clusters
D0604 18:19:01.101 22 envoy] [22][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:125] maybe finish initialize state: 2
D0604 18:19:01.101 22 envoy] [22][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:134] maybe finish initialize primary init clusters empty: true
D0604 18:19:01.101 22 envoy] [22][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:149] maybe finish initialize secondary init clusters empty: true
D0604 18:19:01.101 22 envoy] [22][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:172] maybe finish initialize cds api ready: true
I0604 18:19:01.101 22 envoy] [22][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:174] cm init: initializing cds
D0604 18:19:01.101 22 envoy] [22][config][external/envoy/source/common/config/grpc_mux_impl.cc:75] gRPC mux addWatch for type.googleapis.com/envoy.config.cluster.v3.Cluster
D0604 18:19:01.101 27 envoy] [27][grpc][external/envoy/source/common/grpc/google_async_client_impl.cc:48] completionThread running
I0604 18:19:01.102 22 envoy] [22][main][external/envoy/source/server/server.cc:622] starting main dispatch loop
D0604 18:19:01.102 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:625] [C0] delayed connection error: 111
D0604 18:19:01.102 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:208] [C0] closing socket: 0
D0604 18:19:01.102 22 envoy] [22][client][external/envoy/source/common/http/codec_client.cc:92] [C0] disconnect. resetting 0 pending requests
D0604 18:19:01.102 22 envoy] [22][pool][external/envoy/source/common/http/conn_pool_base.cc:265] [C0] client disconnected, failure reason: 
D0604 18:19:01.102 22 envoy] [22][router][external/envoy/source/common/router/router.cc:1055] [C0][S1482019712714766912] upstream reset: reset reason connection failure
D0604 18:19:01.103 22 envoy] [22][http][external/envoy/source/common/http/async_client_impl.cc:99] async http request response headers (end_stream=true):\n\':status\', \'200\'\n\'content-type\', \'application/grpc\'\n\'grpc-status\', \'14\'\n\'grpc-message\', \'upstream connect error or disconnect/reset before headers. reset reason: connection failure\'

W0604 18:19:01.103 22 envoy] [22][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
D0604 18:19:01.103 22 envoy] [22][config][external/envoy/source/common/config/grpc_subscription_impl.cc:90] gRPC update for type.googleapis.com/envoy.config.cluster.v3.Cluster failed
D0604 18:19:01.132 22 envoy] [22][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:47] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.service.discovery.v3.DiscoveryRequest) returns (stream .envoy.service.discovery.v3.DiscoveryResponse);

D0604 18:19:01.132 22 envoy] [22][router][external/envoy/source/common/router/router.cc:435] [C0][S17192125370729511656] cluster \'ads_cluster\' match for URL \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'
D0604 18:19:01.132 22 envoy] [22][router][external/envoy/source/common/router/router.cc:590] [C0][S17192125370729511656] router decoding headers:\n\':method\', \'POST\'\n\':path\', \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'\n\':authority\', \'ads_cluster\'\n\':scheme\', \'http\'\n\'te\', \'trailers\'\n\'content-type\', \'application/grpc\'\n\'x-envoy-internal\', \'true\'\n\'x-forwarded-for\', \'172.18.0.8\'

D0604 18:19:01.132 22 envoy] [22][pool][external/envoy/source/common/http/conn_pool_base.cc:346] queueing request due to no available connections
D0604 18:19:01.132 22 envoy] [22][pool][external/envoy/source/common/http/conn_pool_base.cc:47] creating a new connection
D0604 18:19:01.132 22 envoy] [22][client][external/envoy/source/common/http/codec_client.cc:35] [C1] connecting
D0604 18:19:01.132 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:753] [C1] connecting to 127.0.0.1:8790
D0604 18:19:01.132 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:762] [C1] connection in progress
D0604 18:19:01.132 22 envoy] [22][http2][external/envoy/source/common/http/http2/codec_impl.cc:1011] [C1] updating connection-level initial window size to 268435456
D0604 18:19:01.133 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:625] [C1] delayed connection error: 111
D0604 18:19:01.133 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:208] [C1] closing socket: 0
D0604 18:19:01.133 22 envoy] [22][client][external/envoy/source/common/http/codec_client.cc:92] [C1] disconnect. resetting 0 pending requests
D0604 18:19:01.133 22 envoy] [22][pool][external/envoy/source/common/http/conn_pool_base.cc:265] [C1] client disconnected, failure reason: 
D0604 18:19:01.133 22 envoy] [22][router][external/envoy/source/common/router/router.cc:1055] [C0][S17192125370729511656] upstream reset: reset reason connection failure
D0604 18:19:01.133 22 envoy] [22][http][external/envoy/source/common/http/async_client_impl.cc:99] async http request response headers (end_stream=true):\n\':status\', \'200\'\n\'content-type\', \'application/grpc\'\n\'grpc-status\', \'14\'\n\'grpc-message\', \'upstream connect error or disconnect/reset before headers. reset reason: connection failure\'

W0604 18:19:01.133 22 envoy] [22][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
D0604 18:19:01.133 22 envoy] [22][config][external/envoy/source/common/config/grpc_subscription_impl.cc:90] gRPC update for type.googleapis.com/envoy.config.cluster.v3.Cluster failed
D0604 18:19:01.346 22 envoy] [22][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:47] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.service.discovery.v3.DiscoveryRequest) returns (stream .envoy.service.discovery.v3.DiscoveryResponse);

D0604 18:19:01.346 22 envoy] [22][router][external/envoy/source/common/router/router.cc:435] [C0][S13862197817187926874] cluster \'ads_cluster\' match for URL \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'
D0604 18:19:01.346 22 envoy] [22][router][external/envoy/source/common/router/router.cc:590] [C0][S13862197817187926874] router decoding headers:\n\':method\', \'POST\'\n\':path\', \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'\n\':authority\', \'ads_cluster\'\n\':scheme\', \'http\'\n\'te\', \'trailers\'\n\'content-type\', \'application/grpc\'\n\'x-envoy-internal\', \'true\'\n\'x-forwarded-for\', \'172.18.0.8\'

D0604 18:19:01.346 22 envoy] [22][pool][external/envoy/source/common/http/conn_pool_base.cc:346] queueing request due to no available connections
D0604 18:19:01.346 22 envoy] [22][pool][external/envoy/source/common/http/conn_pool_base.cc:47] creating a new connection
D0604 18:19:01.346 22 envoy] [22][client][external/envoy/source/common/http/codec_client.cc:35] [C2] connecting
D0604 18:19:01.346 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:753] [C2] connecting to 127.0.0.1:8790
D0604 18:19:01.346 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:762] [C2] connection in progress
D0604 18:19:01.346 22 envoy] [22][http2][external/envoy/source/common/http/http2/codec_impl.cc:1011] [C2] updating connection-level initial window size to 268435456
D0604 18:19:01.347 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:625] [C2] delayed connection error: 111
D0604 18:19:01.347 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:208] [C2] closing socket: 0
D0604 18:19:01.347 22 envoy] [22][client][external/envoy/source/common/http/codec_client.cc:92] [C2] disconnect. resetting 0 pending requests
D0604 18:19:01.347 22 envoy] [22][pool][external/envoy/source/common/http/conn_pool_base.cc:265] [C2] client disconnected, failure reason: 
D0604 18:19:01.347 22 envoy] [22][router][external/envoy/source/common/router/router.cc:1055] [C0][S13862197817187926874] upstream reset: reset reason connection failure
D0604 18:19:01.347 22 envoy] [22][http][external/envoy/source/common/http/async_client_impl.cc:99] async http request response headers (end_stream=true):\n\':status\', \'200\'\n\'content-type\', \'application/grpc\'\n\'grpc-status\', \'14\'\n\'grpc-message\', \'upstream connect error or disconnect/reset before headers. reset reason: connection failure\'

W0604 18:19:01.347 22 envoy] [22][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
D0604 18:19:01.347 22 envoy] [22][config][external/envoy/source/common/config/grpc_subscription_impl.cc:90] gRPC update for type.googleapis.com/envoy.config.cluster.v3.Cluster failed
D0604 18:19:01.553 22 envoy] [22][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:47] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.service.discovery.v3.DiscoveryRequest) returns (stream .envoy.service.discovery.v3.DiscoveryResponse);

D0604 18:19:01.553 22 envoy] [22][router][external/envoy/source/common/router/router.cc:435] [C0][S2820004545228429044] cluster \'ads_cluster\' match for URL \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'
D0604 18:19:01.553 22 envoy] [22][router][external/envoy/source/common/router/router.cc:590] [C0][S2820004545228429044] router decoding headers:\n\':method\', \'POST\'\n\':path\', \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'\n\':authority\', \'ads_cluster\'\n\':scheme\', \'http\'\n\'te\', \'trailers\'\n\'content-type\', \'application/grpc\'\n\'x-envoy-internal\', \'true\'\n\'x-forwarded-for\', \'172.18.0.8\'

D0604 18:19:01.553 22 envoy] [22][pool][external/envoy/source/common/http/conn_pool_base.cc:346] queueing request due to no available connections
D0604 18:19:01.553 22 envoy] [22][pool][external/envoy/source/common/http/conn_pool_base.cc:47] creating a new connection
D0604 18:19:01.553 22 envoy] [22][client][external/envoy/source/common/http/codec_client.cc:35] [C3] connecting
D0604 18:19:01.553 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:753] [C3] connecting to 127.0.0.1:8790
D0604 18:19:01.553 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:762] [C3] connection in progress
D0604 18:19:01.553 22 envoy] [22][http2][external/envoy/source/common/http/http2/codec_impl.cc:1011] [C3] updating connection-level initial window size to 268435456
D0604 18:19:01.553 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:625] [C3] delayed connection error: 111
D0604 18:19:01.553 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:208] [C3] closing socket: 0
D0604 18:19:01.553 22 envoy] [22][client][external/envoy/source/common/http/codec_client.cc:92] [C3] disconnect. resetting 0 pending requests
D0604 18:19:01.553 22 envoy] [22][pool][external/envoy/source/common/http/conn_pool_base.cc:265] [C3] client disconnected, failure reason: 
D0604 18:19:01.553 22 envoy] [22][router][external/envoy/source/common/router/router.cc:1055] [C0][S2820004545228429044] upstream reset: reset reason connection failure
D0604 18:19:01.553 22 envoy] [22][http][external/envoy/source/common/http/async_client_impl.cc:99] async http request response headers (end_stream=true):\n\':status\', \'200\'\n\'content-type\', \'application/grpc\'\n\'grpc-status\', \'14\'\n\'grpc-message\', \'upstream connect error or disconnect/reset before headers. reset reason: connection failure\'

W0604 18:19:01.554 22 envoy] [22][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
D0604 18:19:01.554 22 envoy] [22][config][external/envoy/source/common/config/grpc_subscription_impl.cc:90] gRPC update for type.googleapis.com/envoy.config.cluster.v3.Cluster failed
F0604 18:19:01.815178       6 server.go:52] fail to initialize config manager: fail to fetch and apply the startup service config, fail to initialize ServiceInfo, detected two types with same snake_name (number_value) but mistmatching json_name (numberValue, number_value)
D0604 18:19:03.198 22 envoy] [22][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:47] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.service.discovery.v3.DiscoveryRequest) returns (stream .envoy.service.discovery.v3.DiscoveryResponse);

D0604 18:19:03.198 22 envoy] [22][router][external/envoy/source/common/router/router.cc:435] [C0][S4438559899739963454] cluster \'ads_cluster\' match for URL \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'
D0604 18:19:03.198 22 envoy] [22][router][external/envoy/source/common/router/router.cc:590] [C0][S4438559899739963454] router decoding headers:\n\':method\', \'POST\'\n\':path\', \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'\n\':authority\', \'ads_cluster\'\n\':scheme\', \'http\'\n\'te\', \'trailers\'\n\'content-type\', \'application/grpc\'\n\'x-envoy-internal\', \'true\'\n\'x-forwarded-for\', \'172.18.0.8\'

D0604 18:19:03.198 22 envoy] [22][pool][external/envoy/source/common/http/conn_pool_base.cc:346] queueing request due to no available connections
D0604 18:19:03.198 22 envoy] [22][pool][external/envoy/source/common/http/conn_pool_base.cc:47] creating a new connection
D0604 18:19:03.198 22 envoy] [22][client][external/envoy/source/common/http/codec_client.cc:35] [C4] connecting
D0604 18:19:03.198 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:753] [C4] connecting to 127.0.0.1:8790
D0604 18:19:03.198 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:762] [C4] connection in progress
D0604 18:19:03.198 22 envoy] [22][http2][external/envoy/source/common/http/http2/codec_impl.cc:1011] [C4] updating connection-level initial window size to 268435456
D0604 18:19:03.198 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:625] [C4] delayed connection error: 111
D0604 18:19:03.198 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:208] [C4] closing socket: 0
D0604 18:19:03.198 22 envoy] [22][client][external/envoy/source/common/http/codec_client.cc:92] [C4] disconnect. resetting 0 pending requests
D0604 18:19:03.198 22 envoy] [22][pool][external/envoy/source/common/http/conn_pool_base.cc:265] [C4] client disconnected, failure reason: 
D0604 18:19:03.198 22 envoy] [22][router][external/envoy/source/common/router/router.cc:1055] [C0][S4438559899739963454] upstream reset: reset reason connection failure
D0604 18:19:03.198 22 envoy] [22][http][external/envoy/source/common/http/async_client_impl.cc:99] async http request response headers (end_stream=true):\n\':status\', \'200\'\n\'content-type\', \'application/grpc\'\n\'grpc-status\', \'14\'\n\'grpc-message\', \'upstream connect error or disconnect/reset before headers. reset reason: connection failure\'

W0604 18:19:03.198 22 envoy] [22][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
D0604 18:19:03.198 22 envoy] [22][config][external/envoy/source/common/config/grpc_subscription_impl.cc:90] gRPC update for type.googleapis.com/envoy.config.cluster.v3.Cluster failed
D0604 18:19:06.101 22 envoy] [22][upstream][external/envoy/source/common/upstream/upstream_impl.cc:278] transport socket match, socket default selected for host with address 127.0.0.1:8790
D0604 18:19:06.102 22 envoy] [22][upstream][external/envoy/source/common/upstream/strict_dns_cluster.cc:167] DNS refresh rate reset for 127.0.0.1, refresh rate 5000 ms
D0604 18:19:06.103 22 envoy] [22][main][external/envoy/source/server/server.cc:187] flushing stats
D0604 18:19:06.103 22 envoy] [22][main][external/envoy/source/server/server.cc:197] Envoy is not fully initialized, skipping histogram merge and flushing stats
D0604 18:19:06.412 22 envoy] [22][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:47] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.service.discovery.v3.DiscoveryRequest) returns (stream .envoy.service.discovery.v3.DiscoveryResponse);

D0604 18:19:06.412 22 envoy] [22][router][external/envoy/source/common/router/router.cc:435] [C0][S11388764262586255397] cluster \'ads_cluster\' match for URL \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'
D0604 18:19:06.412 22 envoy] [22][router][external/envoy/source/common/router/router.cc:590] [C0][S11388764262586255397] router decoding headers:\n\':method\', \'POST\'\n\':path\', \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'\n\':authority\', \'ads_cluster\'\n\':scheme\', \'http\'\n\'te\', \'trailers\'\n\'content-type\', \'application/grpc\'\n\'x-envoy-internal\', \'true\'\n\'x-forwarded-for\', \'172.18.0.8\'

D0604 18:19:06.412 22 envoy] [22][pool][external/envoy/source/common/http/conn_pool_base.cc:346] queueing request due to no available connections
D0604 18:19:06.412 22 envoy] [22][pool][external/envoy/source/common/http/conn_pool_base.cc:47] creating a new connection
D0604 18:19:06.412 22 envoy] [22][client][external/envoy/source/common/http/codec_client.cc:35] [C5] connecting
D0604 18:19:06.412 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:753] [C5] connecting to 127.0.0.1:8790
D0604 18:19:06.412 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:762] [C5] connection in progress
D0604 18:19:06.412 22 envoy] [22][http2][external/envoy/source/common/http/http2/codec_impl.cc:1011] [C5] updating connection-level initial window size to 268435456
D0604 18:19:06.412 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:625] [C5] delayed connection error: 111
D0604 18:19:06.412 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:208] [C5] closing socket: 0
D0604 18:19:06.412 22 envoy] [22][client][external/envoy/source/common/http/codec_client.cc:92] [C5] disconnect. resetting 0 pending requests
D0604 18:19:06.413 22 envoy] [22][pool][external/envoy/source/common/http/conn_pool_base.cc:265] [C5] client disconnected, failure reason: 
D0604 18:19:06.413 22 envoy] [22][router][external/envoy/source/common/router/router.cc:1055] [C0][S11388764262586255397] upstream reset: reset reason connection failure
D0604 18:19:06.413 22 envoy] [22][http][external/envoy/source/common/http/async_client_impl.cc:99] async http request response headers (end_stream=true):\n\':status\', \'200\'\n\'content-type\', \'application/grpc\'\n\'grpc-status\', \'14\'\n\'grpc-message\', \'upstream connect error or disconnect/reset before headers. reset reason: connection failure\'

W0604 18:19:06.413 22 envoy] [22][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
D0604 18:19:06.413 22 envoy] [22][config][external/envoy/source/common/config/grpc_subscription_impl.cc:90] gRPC update for type.googleapis.com/envoy.config.cluster.v3.Cluster failed
D0604 18:19:08.445 22 envoy] [22][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:47] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.service.discovery.v3.DiscoveryRequest) returns (stream .envoy.service.discovery.v3.DiscoveryResponse);

D0604 18:19:08.445 22 envoy] [22][router][external/envoy/source/common/router/router.cc:435] [C0][S7430516812441038328] cluster \'ads_cluster\' match for URL \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'
D0604 18:19:08.445 22 envoy] [22][router][external/envoy/source/common/router/router.cc:590] [C0][S7430516812441038328] router decoding headers:\n\':method\', \'POST\'\n\':path\', \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'\n\':authority\', \'ads_cluster\'\n\':scheme\', \'http\'\n\'te\', \'trailers\'\n\'content-type\', \'application/grpc\'\n\'x-envoy-internal\', \'true\'\n\'x-forwarded-for\', \'172.18.0.8\'

D0604 18:19:08.445 22 envoy] [22][pool][external/envoy/source/common/http/conn_pool_base.cc:346] queueing request due to no available connections
D0604 18:19:08.445 22 envoy] [22][pool][external/envoy/source/common/http/conn_pool_base.cc:47] creating a new connection
D0604 18:19:08.445 22 envoy] [22][client][external/envoy/source/common/http/codec_client.cc:35] [C6] connecting
D0604 18:19:08.445 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:753] [C6] connecting to 127.0.0.1:8790
D0604 18:19:08.446 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:762] [C6] connection in progress
D0604 18:19:08.446 22 envoy] [22][http2][external/envoy/source/common/http/http2/codec_impl.cc:1011] [C6] updating connection-level initial window size to 268435456
D0604 18:19:08.446 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:625] [C6] delayed connection error: 111
D0604 18:19:08.446 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:208] [C6] closing socket: 0
D0604 18:19:08.446 22 envoy] [22][client][external/envoy/source/common/http/codec_client.cc:92] [C6] disconnect. resetting 0 pending requests
D0604 18:19:08.446 22 envoy] [22][pool][external/envoy/source/common/http/conn_pool_base.cc:265] [C6] client disconnected, failure reason: 
D0604 18:19:08.446 22 envoy] [22][router][external/envoy/source/common/router/router.cc:1055] [C0][S7430516812441038328] upstream reset: reset reason connection failure
D0604 18:19:08.446 22 envoy] [22][http][external/envoy/source/common/http/async_client_impl.cc:99] async http request response headers (end_stream=true):\n\':status\', \'200\'\n\'content-type\', \'application/grpc\'\n\'grpc-status\', \'14\'\n\'grpc-message\', \'upstream connect error or disconnect/reset before headers. reset reason: connection failure\'

W0604 18:19:08.446 22 envoy] [22][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
D0604 18:19:08.446 22 envoy] [22][config][external/envoy/source/common/config/grpc_subscription_impl.cc:90] gRPC update for type.googleapis.com/envoy.config.cluster.v3.Cluster failed
D0604 18:19:11.101 22 envoy] [22][upstream][external/envoy/source/common/upstream/upstream_impl.cc:278] transport socket match, socket default selected for host with address 127.0.0.1:8790
D0604 18:19:11.101 22 envoy] [22][upstream][external/envoy/source/common/upstream/strict_dns_cluster.cc:167] DNS refresh rate reset for 127.0.0.1, refresh rate 5000 ms
D0604 18:19:11.103 22 envoy] [22][main][external/envoy/source/server/server.cc:187] flushing stats
D0604 18:19:11.103 22 envoy] [22][main][external/envoy/source/server/server.cc:197] Envoy is not fully initialized, skipping histogram merge and flushing stats
D0604 18:19:16.101 22 envoy] [22][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:125] maybe finish initialize state: 4
D0604 18:19:16.101 22 envoy] [22][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:134] maybe finish initialize primary init clusters empty: true
D0604 18:19:16.101 22 envoy] [22][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:149] maybe finish initialize secondary init clusters empty: true
D0604 18:19:16.101 22 envoy] [22][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:172] maybe finish initialize cds api ready: true
I0604 18:19:16.101 22 envoy] [22][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:178] cm init: all clusters initialized
D0604 18:19:16.102 22 envoy] [22][config][external/envoy/source/common/config/grpc_mux_impl.cc:98] Pausing discovery requests for type.googleapis.com/envoy.api.v2.RouteConfiguration
I0604 18:19:16.102 22 envoy] [22][main][external/envoy/source/server/server.cc:601] all clusters initialized. initializing init manager
D0604 18:19:16.102 22 envoy] [22][init][external/envoy/source/common/init/manager_impl.cc:49] init manager Server initializing
D0604 18:19:16.102 22 envoy] [22][init][external/envoy/source/common/init/target_impl.cc:15] init manager Server initializing target LDS
D0604 18:19:16.102 22 envoy] [22][config][external/envoy/source/common/config/grpc_mux_impl.cc:75] gRPC mux addWatch for type.googleapis.com/envoy.config.listener.v3.Listener
D0604 18:19:16.102 22 envoy] [22][config][external/envoy/source/common/config/grpc_mux_impl.cc:32] No stream available to sendDiscoveryRequest for type.googleapis.com/envoy.config.listener.v3.Listener
D0604 18:19:16.102 22 envoy] [22][config][external/envoy/source/common/config/grpc_mux_impl.cc:106] Resuming discovery requests for type.googleapis.com/envoy.api.v2.RouteConfiguration
D0604 18:19:16.102 22 envoy] [22][upstream][external/envoy/source/common/upstream/upstream_impl.cc:278] transport socket match, socket default selected for host with address 127.0.0.1:8790
D0604 18:19:16.102 22 envoy] [22][upstream][external/envoy/source/common/upstream/strict_dns_cluster.cc:167] DNS refresh rate reset for 127.0.0.1, refresh rate 5000 ms
D0604 18:19:16.104 22 envoy] [22][main][external/envoy/source/server/server.cc:187] flushing stats
D0604 18:19:16.104 22 envoy] [22][main][external/envoy/source/server/server.cc:197] Envoy is not fully initialized, skipping histogram merge and flushing stats
D0604 18:19:21.102 22 envoy] [22][upstream][external/envoy/source/common/upstream/upstream_impl.cc:278] transport socket match, socket default selected for host with address 127.0.0.1:8790
D0604 18:19:21.102 22 envoy] [22][upstream][external/envoy/source/common/upstream/strict_dns_cluster.cc:167] DNS refresh rate reset for 127.0.0.1, refresh rate 5000 ms
D0604 18:19:21.105 22 envoy] [22][main][external/envoy/source/server/server.cc:187] flushing stats
D0604 18:19:21.105 22 envoy] [22][main][external/envoy/source/server/server.cc:197] Envoy is not fully initialized, skipping histogram merge and flushing stats
D0604 18:19:26.102 22 envoy] [22][upstream][external/envoy/source/common/upstream/upstream_impl.cc:278] transport socket match, socket default selected for host with address 127.0.0.1:8790
D0604 18:19:26.102 22 envoy] [22][upstream][external/envoy/source/common/upstream/strict_dns_cluster.cc:167] DNS refresh rate reset for 127.0.0.1, refresh rate 5000 ms
D0604 18:19:26.105 22 envoy] [22][main][external/envoy/source/server/server.cc:187] flushing stats
D0604 18:19:26.105 22 envoy] [22][main][external/envoy/source/server/server.cc:197] Envoy is not fully initialized, skipping histogram merge and flushing stats
D0604 18:19:27.089 22 envoy] [22][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:47] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.service.discovery.v3.DiscoveryRequest) returns (stream .envoy.service.discovery.v3.DiscoveryResponse);

D0604 18:19:27.089 22 envoy] [22][router][external/envoy/source/common/router/router.cc:435] [C0][S6807818758608856795] cluster \'ads_cluster\' match for URL \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'
D0604 18:19:27.089 22 envoy] [22][router][external/envoy/source/common/router/router.cc:590] [C0][S6807818758608856795] router decoding headers:\n\':method\', \'POST\'\n\':path\', \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'\n\':authority\', \'ads_cluster\'\n\':scheme\', \'http\'\n\'te\', \'trailers\'\n\'content-type\', \'application/grpc\'\n\'x-envoy-internal\', \'true\'\n\'x-forwarded-for\', \'172.18.0.8\'

D0604 18:19:27.089 22 envoy] [22][pool][external/envoy/source/common/http/conn_pool_base.cc:346] queueing request due to no available connections
D0604 18:19:27.089 22 envoy] [22][pool][external/envoy/source/common/http/conn_pool_base.cc:47] creating a new connection
D0604 18:19:27.089 22 envoy] [22][client][external/envoy/source/common/http/codec_client.cc:35] [C7] connecting
D0604 18:19:27.089 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:753] [C7] connecting to 127.0.0.1:8790
D0604 18:19:27.089 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:762] [C7] connection in progress
D0604 18:19:27.089 22 envoy] [22][http2][external/envoy/source/common/http/http2/codec_impl.cc:1011] [C7] updating connection-level initial window size to 268435456
D0604 18:19:27.089 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:625] [C7] delayed connection error: 111
D0604 18:19:27.089 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:208] [C7] closing socket: 0
D0604 18:19:27.089 22 envoy] [22][client][external/envoy/source/common/http/codec_client.cc:92] [C7] disconnect. resetting 0 pending requests
D0604 18:19:27.089 22 envoy] [22][pool][external/envoy/source/common/http/conn_pool_base.cc:265] [C7] client disconnected, failure reason: 
D0604 18:19:27.089 22 envoy] [22][router][external/envoy/source/common/router/router.cc:1055] [C0][S6807818758608856795] upstream reset: reset reason connection failure
D0604 18:19:27.090 22 envoy] [22][http][external/envoy/source/common/http/async_client_impl.cc:99] async http request response headers (end_stream=true):\n\':status\', \'200\'\n\'content-type\', \'application/grpc\'\n\'grpc-status\', \'14\'\n\'grpc-message\', \'upstream connect error or disconnect/reset before headers. reset reason: connection failure\'

W0604 18:19:27.090 22 envoy] [22][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
D0604 18:19:27.090 22 envoy] [22][config][external/envoy/source/common/config/grpc_subscription_impl.cc:90] gRPC update for type.googleapis.com/envoy.config.listener.v3.Listener failed
D0604 18:19:27.090 22 envoy] [22][config][external/envoy/source/common/config/grpc_subscription_impl.cc:90] gRPC update for type.googleapis.com/envoy.config.cluster.v3.Cluster failed
D0604 18:19:31.102 22 envoy] [22][init][external/envoy/source/common/init/watcher_impl.cc:14] target LDS initialized, notifying init manager Server
D0604 18:19:31.102 22 envoy] [22][init][external/envoy/source/common/init/watcher_impl.cc:14] init manager Server initialized, notifying RunHelper
I0604 18:19:31.102 22 envoy] [22][config][external/envoy/source/server/listener_manager_impl.cc:836] all dependencies initialized. starting workers
D0604 18:19:31.102 22 envoy] [22][config][external/envoy/source/server/listener_manager_impl.cc:847] starting worker 0
D0604 18:19:31.102 22 envoy] [22][config][external/envoy/source/server/listener_manager_impl.cc:847] starting worker 1
D0604 18:19:31.102 34 envoy] [34][main][external/envoy/source/server/worker_impl.cc:111] worker entering dispatch loop
D0604 18:19:31.102 22 envoy] [22][config][external/envoy/source/server/listener_manager_impl.cc:847] starting worker 2
D0604 18:19:31.102 34 envoy] [34][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:978] adding TLS initial cluster ads_cluster
D0604 18:19:31.102 35 envoy] [35][main][external/envoy/source/server/worker_impl.cc:111] worker entering dispatch loop
D0604 18:19:31.102 36 envoy] [36][grpc][external/envoy/source/common/grpc/google_async_client_impl.cc:48] completionThread running
D0604 18:19:31.102 37 envoy] [37][main][external/envoy/source/server/worker_impl.cc:111] worker entering dispatch loop
D0604 18:19:31.102 34 envoy] [34][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:1137] membership update for TLS cluster ads_cluster added 1 removed 0
D0604 18:19:31.102 22 envoy] [22][config][external/envoy/source/server/listener_manager_impl.cc:847] starting worker 3
D0604 18:19:31.102 35 envoy] [35][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:978] adding TLS initial cluster ads_cluster
D0604 18:19:31.103 37 envoy] [37][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:978] adding TLS initial cluster ads_cluster
D0604 18:19:31.103 38 envoy] [38][grpc][external/envoy/source/common/grpc/google_async_client_impl.cc:48] completionThread running
D0604 18:19:31.103 22 envoy] [22][config][external/envoy/source/server/listener_manager_impl.cc:847] starting worker 4
D0604 18:19:31.103 37 envoy] [37][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:1137] membership update for TLS cluster ads_cluster added 1 removed 0
D0604 18:19:31.103 35 envoy] [35][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:1137] membership update for TLS cluster ads_cluster added 1 removed 0
D0604 18:19:31.103 22 envoy] [22][config][external/envoy/source/server/listener_manager_impl.cc:847] starting worker 5
D0604 18:19:31.103 40 envoy] [40][main][external/envoy/source/server/worker_impl.cc:111] worker entering dispatch loop
D0604 18:19:31.103 22 envoy] [22][config][external/envoy/source/server/listener_manager_impl.cc:847] starting worker 6
D0604 18:19:31.103 39 envoy] [39][grpc][external/envoy/source/common/grpc/google_async_client_impl.cc:48] completionThread running
D0604 18:19:31.103 42 envoy] [42][main][external/envoy/source/server/worker_impl.cc:111] worker entering dispatch loop
D0604 18:19:31.103 41 envoy] [41][main][external/envoy/source/server/worker_impl.cc:111] worker entering dispatch loop
D0604 18:19:31.103 22 envoy] [22][config][external/envoy/source/server/listener_manager_impl.cc:847] starting worker 7
D0604 18:19:31.103 43 envoy] [43][main][external/envoy/source/server/worker_impl.cc:111] worker entering dispatch loop
D0604 18:19:31.103 40 envoy] [40][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:978] adding TLS initial cluster ads_cluster
D0604 18:19:31.104 40 envoy] [40][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:1137] membership update for TLS cluster ads_cluster added 1 removed 0
D0604 18:19:31.103 44 envoy] [44][grpc][external/envoy/source/common/grpc/google_async_client_impl.cc:48] completionThread running
D0604 18:19:31.104 42 envoy] [42][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:978] adding TLS initial cluster ads_cluster
D0604 18:19:31.104 41 envoy] [41][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:978] adding TLS initial cluster ads_cluster
D0604 18:19:31.104 43 envoy] [43][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:978] adding TLS initial cluster ads_cluster
D0604 18:19:31.104 42 envoy] [42][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:1137] membership update for TLS cluster ads_cluster added 1 removed 0
D0604 18:19:31.104 46 envoy] [46][grpc][external/envoy/source/common/grpc/google_async_client_impl.cc:48] completionThread running
D0604 18:19:31.104 22 envoy] [22][upstream][external/envoy/source/common/upstream/upstream_impl.cc:278] transport socket match, socket default selected for host with address 127.0.0.1:8790
D0604 18:19:31.104 43 envoy] [43][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:1137] membership update for TLS cluster ads_cluster added 1 removed 0
D0604 18:19:31.104 47 envoy] [47][main][external/envoy/source/server/worker_impl.cc:111] worker entering dispatch loop
D0604 18:19:31.104 41 envoy] [41][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:1137] membership update for TLS cluster ads_cluster added 1 removed 0
D0604 18:19:31.104 22 envoy] [22][upstream][external/envoy/source/common/upstream/strict_dns_cluster.cc:167] DNS refresh rate reset for 127.0.0.1, refresh rate 5000 ms
D0604 18:19:31.104 45 envoy] [45][grpc][external/envoy/source/common/grpc/google_async_client_impl.cc:48] completionThread running
D0604 18:19:31.104 47 envoy] [47][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:978] adding TLS initial cluster ads_cluster
D0604 18:19:31.104 49 envoy] [49][grpc][external/envoy/source/common/grpc/google_async_client_impl.cc:48] completionThread running
D0604 18:19:31.104 47 envoy] [47][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:1137] membership update for TLS cluster ads_cluster added 1 removed 0
D0604 18:19:31.104 48 envoy] [48][grpc][external/envoy/source/common/grpc/google_async_client_impl.cc:48] completionThread running
D0604 18:19:31.107 22 envoy] [22][main][external/envoy/source/server/server.cc:187] flushing stats
D0604 18:19:36.101 22 envoy] [22][upstream][external/envoy/source/common/upstream/upstream_impl.cc:278] transport socket match, socket default selected for host with address 127.0.0.1:8790
D0604 18:19:36.102 22 envoy] [22][upstream][external/envoy/source/common/upstream/strict_dns_cluster.cc:167] DNS refresh rate reset for 127.0.0.1, refresh rate 5000 ms
D0604 18:19:36.108 22 envoy] [22][main][external/envoy/source/server/server.cc:187] flushing stats
D0604 18:19:39.925 22 envoy] [22][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:47] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.service.discovery.v3.DiscoveryRequest) returns (stream .envoy.service.discovery.v3.DiscoveryResponse);

D0604 18:19:39.925 22 envoy] [22][router][external/envoy/source/common/router/router.cc:435] [C0][S12521283791827806850] cluster \'ads_cluster\' match for URL \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'
D0604 18:19:39.925 22 envoy] [22][router][external/envoy/source/common/router/router.cc:590] [C0][S12521283791827806850] router decoding headers:\n\':method\', \'POST\'\n\':path\', \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'\n\':authority\', \'ads_cluster\'\n\':scheme\', \'http\'\n\'te\', \'trailers\'\n\'content-type\', \'application/grpc\'\n\'x-envoy-internal\', \'true\'\n\'x-forwarded-for\', \'172.18.0.8\'

D0604 18:19:39.925 22 envoy] [22][pool][external/envoy/source/common/http/conn_pool_base.cc:346] queueing request due to no available connections
D0604 18:19:39.925 22 envoy] [22][pool][external/envoy/source/common/http/conn_pool_base.cc:47] creating a new connection
D0604 18:19:39.925 22 envoy] [22][client][external/envoy/source/common/http/codec_client.cc:35] [C8] connecting
D0604 18:19:39.925 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:753] [C8] connecting to 127.0.0.1:8790
D0604 18:19:39.926 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:762] [C8] connection in progress
D0604 18:19:39.926 22 envoy] [22][http2][external/envoy/source/common/http/http2/codec_impl.cc:1011] [C8] updating connection-level initial window size to 268435456
D0604 18:19:39.926 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:625] [C8] delayed connection error: 111
D0604 18:19:39.926 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:208] [C8] closing socket: 0
D0604 18:19:39.926 22 envoy] [22][client][external/envoy/source/common/http/codec_client.cc:92] [C8] disconnect. resetting 0 pending requests
D0604 18:19:39.926 22 envoy] [22][pool][external/envoy/source/common/http/conn_pool_base.cc:265] [C8] client disconnected, failure reason: 
D0604 18:19:39.926 22 envoy] [22][router][external/envoy/source/common/router/router.cc:1055] [C0][S12521283791827806850] upstream reset: reset reason connection failure
D0604 18:19:39.926 22 envoy] [22][http][external/envoy/source/common/http/async_client_impl.cc:99] async http request response headers (end_stream=true):\n\':status\', \'200\'\n\'content-type\', \'application/grpc\'\n\'grpc-status\', \'14\'\n\'grpc-message\', \'upstream connect error or disconnect/reset before headers. reset reason: connection failure\'

W0604 18:19:39.926 22 envoy] [22][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
D0604 18:19:39.926 22 envoy] [22][config][external/envoy/source/common/config/grpc_subscription_impl.cc:90] gRPC update for type.googleapis.com/envoy.config.listener.v3.Listener failed
D0604 18:19:39.926 22 envoy] [22][config][external/envoy/source/common/config/grpc_subscription_impl.cc:90] gRPC update for type.googleapis.com/envoy.config.cluster.v3.Cluster failed
D0604 18:19:41.103 22 envoy] [22][upstream][external/envoy/source/common/upstream/upstream_impl.cc:278] transport socket match, socket default selected for host with address 127.0.0.1:8790
D0604 18:19:41.103 22 envoy] [22][upstream][external/envoy/source/common/upstream/strict_dns_cluster.cc:167] DNS refresh rate reset for 127.0.0.1, refresh rate 5000 ms
D0604 18:19:41.109 22 envoy] [22][main][external/envoy/source/server/server.cc:187] flushing stats
D0604 18:19:46.104 22 envoy] [22][upstream][external/envoy/source/common/upstream/upstream_impl.cc:278] transport socket match, socket default selected for host with address 127.0.0.1:8790
D0604 18:19:46.104 22 envoy] [22][upstream][external/envoy/source/common/upstream/strict_dns_cluster.cc:167] DNS refresh rate reset for 127.0.0.1, refresh rate 5000 ms
D0604 18:19:46.109 22 envoy] [22][main][external/envoy/source/server/server.cc:187] flushing stats
D0604 18:19:51.104 22 envoy] [22][upstream][external/envoy/source/common/upstream/upstream_impl.cc:278] transport socket match, socket default selected for host with address 127.0.0.1:8790
D0604 18:19:51.104 22 envoy] [22][upstream][external/envoy/source/common/upstream/strict_dns_cluster.cc:167] DNS refresh rate reset for 127.0.0.1, refresh rate 5000 ms
D0604 18:19:51.109 22 envoy] [22][main][external/envoy/source/server/server.cc:187] flushing stats
D0604 18:19:56.104 22 envoy] [22][upstream][external/envoy/source/common/upstream/upstream_impl.cc:278] transport socket match, socket default selected for host with address 127.0.0.1:8790
D0604 18:19:56.104 22 envoy] [22][upstream][external/envoy/source/common/upstream/strict_dns_cluster.cc:167] DNS refresh rate reset for 127.0.0.1, refresh rate 5000 ms
D0604 18:19:56.109 22 envoy] [22][main][external/envoy/source/server/server.cc:187] flushing stats
D0604 18:20:00.870 22 envoy] [22][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:47] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.service.discovery.v3.DiscoveryRequest) returns (stream .envoy.service.discovery.v3.DiscoveryResponse);

D0604 18:20:00.870 22 envoy] [22][router][external/envoy/source/common/router/router.cc:435] [C0][S11297151677618551664] cluster \'ads_cluster\' match for URL \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'
D0604 18:20:00.870 22 envoy] [22][router][external/envoy/source/common/router/router.cc:590] [C0][S11297151677618551664] router decoding headers:\n\':method\', \'POST\'\n\':path\', \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'\n\':authority\', \'ads_cluster\'\n\':scheme\', \'http\'\n\'te\', \'trailers\'\n\'content-type\', \'application/grpc\'\n\'x-envoy-internal\', \'true\'\n\'x-forwarded-for\', \'172.18.0.8\'

D0604 18:20:00.870 22 envoy] [22][pool][external/envoy/source/common/http/conn_pool_base.cc:346] queueing request due to no available connections
D0604 18:20:00.870 22 envoy] [22][pool][external/envoy/source/common/http/conn_pool_base.cc:47] creating a new connection
D0604 18:20:00.871 22 envoy] [22][client][external/envoy/source/common/http/codec_client.cc:35] [C9] connecting
D0604 18:20:00.871 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:753] [C9] connecting to 127.0.0.1:8790
D0604 18:20:00.871 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:762] [C9] connection in progress
D0604 18:20:00.871 22 envoy] [22][http2][external/envoy/source/common/http/http2/codec_impl.cc:1011] [C9] updating connection-level initial window size to 268435456
D0604 18:20:00.871 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:625] [C9] delayed connection error: 111
D0604 18:20:00.871 22 envoy] [22][connection][external/envoy/source/common/network/connection_impl.cc:208] [C9] closing socket: 0
D0604 18:20:00.871 22 envoy] [22][client][external/envoy/source/common/http/codec_client.cc:92] [C9] disconnect. resetting 0 pending requests
D0604 18:20:00.871 22 envoy] [22][pool][external/envoy/source/common/http/conn_pool_base.cc:265] [C9] client disconnected, failure reason: 
D0604 18:20:00.871 22 envoy] [22][router][external/envoy/source/common/router/router.cc:1055] [C0][S11297151677618551664] upstream reset: reset reason connection failure
D0604 18:20:00.871 22 envoy] [22][http][external/envoy/source/common/http/async_client_impl.cc:99] async http request response headers (end_stream=true):\n\':status\', \'200\'\n\'content-type\', \'application/grpc\'\n\'grpc-status\', \'14\'\n\'grpc-message\', \'upstream connect error or disconnect/reset before headers. reset reason: connection failure\'

W0604 18:20:00.871 22 envoy] [22][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
D0604 18:20:00.871 22 envoy] [22][config][external/envoy/source/common/config/grpc_subscription_impl.cc:90] gRPC update for type.googleapis.com/envoy.config.listener.v3.Listener failed
D0604 18:20:00.871 22 envoy] [22][config][external/envoy/source/common/config/grpc_subscription_impl.cc:90] gRPC update for type.googleapis.com/envoy.config.cluster.v3.Cluster failed
D0604 18:20:01.104 22 envoy] [22][upstream][external/envoy/source/common/upstream/upstream_impl.cc:278] transport socket match, socket default selected for host with address 127.0.0.1:8790
D0604 18:20:01.105 22 envoy] [22][upstream][external/envoy/source/common/upstream/strict_dns_cluster.cc:167] DNS refresh rate reset for 127.0.0.1, refresh rate 5000 ms
D0604 18:20:01.110 22 envoy] [22][main][external/envoy/source/server/server.cc:187] flushing stats

Reducing coldstart for ESP beta 2 on Cloud Run

I am using Google endpoint with Google Cloud functions, with following specifications:

Cloud Run running ESP beta 2 image (service.json configured in the image):
CPU allocated: 1
Memory allocated: 512Mi
Concurrency: 80
Request time-out: 800 seconds

Cloud Function running simple python code with following specification:
Memory allocated: 128 MB
Time-out: 500 seconds

My first request on cloud endpoint(via Cloud run ESp) takes around 25-17 sec to respond (due to cold start), And it takes couple of secs without the cold start.
I want to find if there is any way to reduce this cold start time?

image

Jwt payload is an invalid Base64 or an invalid JSON

My apologies for using an error message as the title, but hopefully this helps reach more people as we have a habit of googling errors when we don't know what they are :P

I've been trying to create a custom authentication provider using the jwcrypto python lib for issuing JWTs. I've tested them on JWT.io and my public key works (valid signature for the returned JWT).

The whole system runs on google cloud run (which at this point in time may be obvious as its the only supported platform, however that may change in the future).

However when I try to send a token through ESPv2, I end up with the following error:

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNAUTHENTICATED
        details = "Jwt payload is an invalid Base64 or an invalid JSON"
        debug_error_string = "{"created":"@1589965852.793650023","description":"Error received from peer ipv4:xxx.xxx.xx.xx:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Jwt payload is an invalid Base64 or an invalid JSON","grpc_status":16}"
>

I went ahead and contacted Google support to which the endpoints specialists recommended I open an issue on github.

This function is used to generate the JWT (it's just a mockup, no actual data is saved or logged, it just generates a JWT).

def SignIn(self,
               request: generated_stub_pb2.UsrLoginRegisterRequest,
               context: grpc.ServicerContext) -> None:
        logging.info("Received request: %s", request)
        # Check if the user has a Refresh token and the expired JWT is valid
        # TODO

        ## Get the PRIVATE key
        # Gets the private key from a cloud storage bucket
        csh = login_cloud_storage_helper()
        privateKey = csh.get_private_jwk()
      
        timestamp = str(datetime.timestamp(datetime.now())).split('.')[0]
        # Generate the JWT
        token = jwt.JWT(header={"alg": "ES256",
                                "typ": "JWT"},
                claims={"iss": "fart",      
                        "sub": "I need a weapon", 
                        "aud": "customers",
                        "exp": "{}".format(int(timestamp) + _TOKEN_LIFETIME),
                        "iat": "{}".format(timestamp), 
                })
        token.make_signed_token(privateKey)
        serial_token = token.serialize()

        # Generate the Refresh token
        return generated_stub_pb2.UsrLoginRegisterResponse(
            status=generated_stub_pb2.UsrLoginRegisterStatus.LOGIN_SUCCESSFUL,
            jwt="{}".format(serial_token))

Which returns a token encoded in Base64URL format like this:

eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9
.
eyJhdWQiOiJjdXN0b21lcnMiLCJleHAiOiIxNTg5OTY4MDI5IiwiaWF0IjoiMTU4OTk2NDQyOSIsImlzcyI6ImZhcnQiLCJzdWIiOiJJIG5lZWQgYSB3ZWFwb24ifQ
.
UwLI3vi5VWKSBNVnrT0QkLsmHKWPGuy0WH9RWxka-mz5Mo9QHuj-RBX7tNmtAbjq91nf9ATt3QgN16zysmNT4Q

you can decode it here and using this public key, you can see it's signature is valid

-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEygbPj3XrVI5XPXkV9rDjuA/yTYTL
IScN+RrmdRBw4UMyy5Swv1nOt73s8Ld9JCCtHvj16nPSe2LyzGqLUVAHHA==
-----END PUBLIC KEY-----

Following the guide google put out here, I created this api_config.yaml file

type: google.api.Service
config_version: 3
name: espv2-abcdefg-an.a.run.app
title: ESPv2 test
apis:
    - name: endpoints.custom.package.name.gRPCService
#
#   API Key usage restrictions (there are none, we don't use API keys)
#
usage:
    rules:
    - selector: "*"
      allow_unregistered_calls: true

#
#   Request authentication (any calls to these methods will require auth)
#
authentication:
    providers:
    - id: a-fancy-id
      issuer: fart
      jwks_uri: "https://storage.googleapis.com/public-bucket/jwk-public-key.pem"
      audiences: "customers"
    rules:
    - selector: endpoints.custom.package.name.gRPCService.Signout
      requirements:
      - provider_id: a-fancy-id

backend:
    rules:
        - selector: "*"
          # This is the grpcs link to an already deployed container
          address: grpcs://login-server-test-abcdefg-an.a.run.app

The config file works, I can send requests to the sign-in method and I get a token back (i.e. the proxy doesn't require authentication.

However when I try and run the Signout method

# This just sends back a status code from a enum and sets the jwt tag to a message
def Signout(self,
                request: generated_stub_pb2.UsrLoginRegisterRequest,
                context: grpc.ServicerContext) -> None:
        return generated_stub_pb2.UsrLoginRegisterResponse(
            status=generated_stub_pb2.UsrLoginRegisterStatus.SIGNOUT_SUCCESSFUL,
            jwt="Signout successful")

with the following client code

host = "espv2-abcdefg-an.a.run.app"
port = "443"
jwt = "eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJh..."
def run2():
    with open('roots.pem', 'rb') as f:
        creds = grpc.ssl_channel_credentials(f.read())
    
    channel = grpc.secure_channel('{}:{}'.format(host, port), creds)
    
    stub = generated_stub_pb2_grpc.LoginStub(channel)
    
    metadata = []
    metadata.append(('authorization', 'Bearer ' + jwt))
    print(metadata)
    
    response = stub.Signout(generated_stub_pb2.UsrLoginRegisterRequest(
            creds=generated_stub_pb2.UsrCreds(
                    username="Firstname", 
                    password="SecretPassword"
                    )
            ), metadata=metadata
    )
    print(response)

I get the dreaded error response.

List of things I've tried

  • Test the backend locally (works)
  • Test the backend uploaded to google cloud run without authentication (works)
  • Test the Signout method with authentication (doesn't work)
  • Generate new keys (doesn't change anything)
  • Generate new tokens (doesn't change anything)
  • I'm aware that endpoints only supports Unary gRPC at the moment, my proto file doesn't use any streaming syntax at all.

I've also got an open stackoverflow post as well.

The google endpoints specialists seemed to have tracked the error to here and since ESPv2 uses Envoy as the proxy, I figured perhaps you guys could help figure out what is causing this authentication error.

Kind regards,

Despicable Bee.

[New Feature] User TLS authentication

What I am trying to do is a End User Certificate Authentication like this:

  1. Request from user comes in to Ingress Gateway
  2. Ingress Gateway checks certificate fingerprint against a whitelist
    2.1. If fingerprint exists in whitelist (populated by a secret), request enters the cluster
    2.2. If fingerprint doesn’t exist, request is rejected

I found this filter on Envoy envoy.filters.network.client_ssl_auth

I will really appreciate to have this envoy configuration option on ESPv2.

I could help if have some guidance

Original authorization token

Is it possible to retrieve an original authorization token from the user?
My use case is log in using hd (hosted domain) parameter, but I'm not able to get to it.
Another use case is, that I'm using firebase auth with custom claims, where I got user's permissions and I need them to control granular access. But again no access to them without an original token.

https://cloud.google.com/endpoints/docs/openapi/authenticating-users-custom, it says that just issuer, id and email is forwarded.

"Jwt is not in the form of Header.Payload.Signature with two dots and 3 sections"

I have a setup similar to the one described here: https://cloud.google.com/endpoints/docs/openapi/get-started-cloud-run.

When I try to generate a JWT from a service account credentials JSON file, I receive an error from ESPv2: "Jwt is not in the form of Header.Payload.Signature with two dots and 3 sections." When I inspect the JWT generated, it is in the correct format, but appears to have extra == signs for padding in the payload section of the token. I'm not sure if this is throwing ESPv2 off, or if it's something else. When I take the JWT and try to decode it using google.auth.jwt.decode, it's able to decode fine, which leads me to believe the JWT is at least in the correct format.

The code for generating a JWT from a service account is below:

...
        signer = google.auth.crypt.RSASigner.from_string(
            service_account_info['private_key'],
            service_account_info['private_key_id'])

        issuer = service_account_info['client_email']

        now = int(time.time())

        payload = {
            'iat': now,
            'exp': now + ONE_HOUR_IN_SECONDS,
            'aud': aud,
            'iss': issuer,
            'sub': issuer,
            'email': issuer
        }

        signed_jwt = google.auth.jwt.encode(signer, payload).decode('utf-8')

Any thoughts?

Local Development with gRPC

The following is a follow-up to comments I shared in #185. I figured with the change of the title to be much more narrow it made more sense opening an issue up to stand on its own.

Some context:

My setup currently works on GCP as expected with the documentation provided in the _. With that out of the way, I would like to work with ESP and the rest of the stack I am experimenting with via docker.

Here is my compose file:

version: '3.7'
services:
  api:
    build:
      context: .
      dockerfile: dev.dockerfile
    volumes:
      - .:/app
      - $GOPATH/pkg/mod/cache:/go/pkg/mod/cache
    working_dir: /app
    env_file:
      - .env
    ports:
      - 50042:8080
  espv2:
    image: gcr.io/endpoints-release/endpoints-runtime:2
    restart: unless-stopped
    ports:
      - "8089:8089"
      - "80:8080"
    volumes:
      - ./.google-devcredentials.json:/.google-devcredentials.json:ro
    env_file:
      # sets GOOGLE_APPLICATION_CREDENTIALS
      - .env
    networks:
      - esp_net
    command:
      - "--service=<SERVICE_NAME>"
      - "--rollout_strategy=managed"
      - "--listener_port=8089"
      - "--backend=grpc://api:50042"
      - "--service_account_key=/.google-devcredentials.json"
      - "--cors_preset=basic"
      # frontend host+port
      - "--cors_allow_origin=http://0.0.0.0:4242"
      # - "--non_gcp"
      - "--enable_debug"
    depends_on:
      - api

networks:
  esp_net:
    driver: bridge

I expected to run the command below successfully, matching my experience on GCP (CloudRun, gRPC):

curl  \                                                                                                                                                                             
  -X POST \
  -d '{"email": "[email protected]", "name": "denis"}' \
  0.0.0.0:80/v1/user/add

However I could not get that far. I was not able to proceed as envoy kept crashing. In short I am looking to:

  1. See a control plane dashboard (cannot view the dashboard)
  2. Send a request to the gateway port I allocated (80) (cannot send requests)
  3. Use the JS binding I created to send CORS requests from an app listening on port 4242, but not in the compose file (unconfirmed)

Here is the error output:

espv2_1      | I0619 04:46:25.981839       9 main.go:29] Output path: /tmp/bootstrap.json
espv2_1      | I0619 04:46:25.981876       9 flags.go:105] Common options: {AdminAddress:0.0.0.0 AdminPort:0 DiscoveryPort:8790 Node:ESPv2 DisableTracing:true TracingProjectId: TracingStackdriverAddress: TracingSamplingRate:0.001 TracingIncomingContext: TracingOutgoingContext: TracingMaxNumAttributes:32 TracingMaxNumAnnotations:32 TracingMaxNumMessageEvents:128 TracingMaxNumLinks:128 NonGCP:false HttpRequestTimeout:5s MetadataURL:http://169.254.169.254/computeMetadata IamURL:https://iamcredentials.googleapis.com ServiceControlCredentials:<nil> BackendAuthCredentials:<nil>}
espv2_1      | I0619 04:46:25.982167       9 flags.go:39] ADS Bootstrapper options: {CommonOptions:{AdminAddress:0.0.0.0 AdminPort:0 DiscoveryPort:8790 Node:ESPv2 DisableTracing:true TracingProjectId: TracingStackdriverAddress: TracingSamplingRate:0.001 TracingIncomingContext: TracingOutgoingContext: TracingMaxNumAttributes:32 TracingMaxNumAnnotations:32 TracingMaxNumMessageEvents:128 TracingMaxNumLinks:128 NonGCP:false HttpRequestTimeout:5s MetadataURL:http://169.254.169.254/computeMetadata IamURL:https://iamcredentials.googleapis.com ServiceControlCredentials:<nil> BackendAuthCredentials:<nil>} AdsConnectTimeout:10s DiscoveryAddress:127.0.0.1:8790}
espv2_1      | Starting Config Manager with args: ['bin/configmanager', '--logtostderr', '--backend_address', 'grpc://api:8080', '--rollout_strategy', 'managed', '--v', '1', '--listener_port', '8089', '--service', 'sndcfegateway', '--disable_tracing', '--cors_preset', 'basic', '--cors_allow_origin', 'http://0.0.0.0:4242', '--cors_allow_origin_regex', '', '--cors_allow_methods', 'GET, POST, PUT, PATCH, DELETE, OPTIONS', '--cors_allow_headers', 'DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization', '--cors_expose_headers', 'Content-Length,Content-Range', '--service_account_key', '/tmp/.google-devcredentials.json', '--non_gcp', '--suppress_envoy_headers=false']
espv2_1      | ['bin/bootstrap', '--logtostderr', '--admin_port', '0', '--disable_tracing', '/tmp/bootstrap.json']
espv2_1      | I0619 04:46:25.981362       7 flags.go:105] Common options: {AdminAddress:0.0.0.0 AdminPort:8001 DiscoveryPort:8790 Node:ESPv2 DisableTracing:true TracingProjectId: TracingStackdriverAddress: TracingSamplingRate:0.001 TracingIncomingContext: TracingOutgoingContext: TracingMaxNumAttributes:32 TracingMaxNumAnnotations:32 TracingMaxNumMessageEvents:128 TracingMaxNumLinks:128 NonGCP:true HttpRequestTimeout:5s MetadataURL:http://169.254.169.254/computeMetadata IamURL:https://iamcredentials.googleapis.com ServiceControlCredentials:<nil> BackendAuthCredentials:<nil>}
espv2_1      | I0619 04:46:25.981675       7 flags.go:173] Config Generator options: {CommonOptions:{AdminAddress:0.0.0.0 AdminPort:8001 DiscoveryPort:8790 Node:ESPv2 DisableTracing:true TracingProjectId: TracingStackdriverAddress: TracingSamplingRate:0.001 TracingIncomingContext: TracingOutgoingContext: TracingMaxNumAttributes:32 TracingMaxNumAnnotations:32 TracingMaxNumMessageEvents:128 TracingMaxNumLinks:128 NonGCP:true HttpRequestTimeout:5s MetadataURL:http://169.254.169.254/computeMetadata IamURL:https://iamcredentials.googleapis.com ServiceControlCredentials:<nil> BackendAuthCredentials:<nil>} CorsAllowCredentials:false CorsAllowHeaders:DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization CorsAllowMethods:GET, POST, PUT, PATCH, DELETE, OPTIONS CorsAllowOrigin:http://0.0.0.0:4242 CorsAllowOriginRegex: CorsExposeHeaders:Content-Length,Content-Range CorsPreset:basic BackendDnsLookupFamily:auto ClusterConnectTimeout:20s BackendAddress:grpc://api:8080 ListenerAddress:0.0.0.0 Healthz: ServiceManagementURL:https://servicemanagement.googleapis.com ServiceControlURL:https://servicecontrol.googleapis.com ListenerPort:8089 SslServerCertPath: SslClientCertPath: SslMinimumProtocol: SslMaximumProtocol: EnableHSTS:false RootCertsPath:/etc/ssl/certs/ca-certificates.crt DnsResolverAddresses: ServiceAccountKey:/tmp/.google-devcredentials.json SkipJwtAuthnFilter:false SkipServiceControlFilter:false AccessLog: AccessLogFormat: EnvoyUseRemoteAddress:false EnvoyXffNumTrustedHops:2 LogJwtPayloads: LogRequestHeaders: LogResponseHeaders: MinStreamReportIntervalMs:0 SuppressEnvoyHeaders:false UnderscoresInHeaders:false ServiceControlNetworkFailOpen:true JwksCacheDurationInS:300 ScCheckTimeoutMs:0 ScQuotaTimeoutMs:0 ScReportTimeoutMs:0 ScCheckRetries:-1 ScQuotaRetries:-1 ScReportRetries:-1 ComputePlatformOverride: TranscodingAlwaysPrintPrimitiveFields:false TranscodingAlwaysPrintEnumsAsInts:false TranscodingPreserveProtoFieldNames:false TranscodingIgnoreQueryParameters: TranscodingIgnoreUnknownQueryParameters:false}
espv2_1      | Starting Envoy with args: ['bin/envoy', '-c', '/tmp/bootstrap.json', '--disable-hot-restart', '--log-format %L%m%d %T.%e %t envoy] [%t][%n]%v', '--log-format-escaped', '-l debug']
espv2_1      | I0619 04:46:26.009 20 envoy] [20][main][external/envoy/source/server/server.cc:293] initializing epoch 0 (hot restart version=disabled)
espv2_1      | I0619 04:46:26.009 20 envoy] [20][main][external/envoy/source/server/server.cc:295] statically linked extensions:
espv2_1      | I0619 04:46:26.009 20 envoy] [20][main][external/envoy/source/server/server.cc:297]   envoy.resource_monitors: envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource
espv2_1      | I0619 04:46:26.009 20 envoy] [20][main][external/envoy/source/server/server.cc:297]   envoy.retry_host_predicates: envoy.retry_host_predicates.previous_hosts
espv2_1      | I0619 04:46:26.009 20 envoy] [20][main][external/envoy/source/server/server.cc:297]   envoy.retry_priorities: envoy.retry_priorities.previous_priorities
espv2_1      | I0619 04:46:26.009 20 envoy] [20][main][external/envoy/source/server/server.cc:297]   envoy.resolvers: envoy.ip
espv2_1      | I0619 04:46:26.009 20 envoy] [20][main][external/envoy/source/server/server.cc:297]   envoy.stats_sinks: envoy.metrics_service, envoy.stat_sinks.metrics_service
espv2_1      | I0619 04:46:26.009 20 envoy] [20][main][external/envoy/source/server/server.cc:297]   envoy.filters.http: envoy.buffer, envoy.cors, envoy.ext_authz, envoy.fault, envoy.filters.http.backend_auth, envoy.filters.http.backend_routing, envoy.filters.http.buffer, envoy.filters.http.cors, envoy.filters.http.ext_authz, envoy.filters.http.fault, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.gzip, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.jwt_authn, envoy.filters.http.path_matcher, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.service_control, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.gzip, envoy.health_check, envoy.ip_tagging, envoy.rate_limit, envoy.router
espv2_1      | I0619 04:46:26.009 20 envoy] [20][main][external/envoy/source/server/server.cc:297]   envoy.filters.listener: envoy.filters.listener.original_dst, envoy.filters.listener.proxy_protocol, envoy.filters.listener.tls_inspector, envoy.listener.original_dst, envoy.listener.proxy_protocol, envoy.listener.tls_inspector
espv2_1      | I0619 04:46:26.009 20 envoy] [20][main][external/envoy/source/server/server.cc:297]   envoy.access_loggers: envoy.access_loggers.file, envoy.access_loggers.http_grpc, envoy.file_access_log, envoy.http_grpc_access_log
espv2_1      | I0619 04:46:26.009 20 envoy] [20][main][external/envoy/source/server/server.cc:297]   envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.tap, envoy.transport_sockets.tls, raw_buffer, tls
espv2_1      | I0619 04:46:26.009 20 envoy] [20][main][external/envoy/source/server/server.cc:297]   envoy.grpc_credentials: envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata
espv2_1      | I0619 04:46:26.009 20 envoy] [20][main][external/envoy/source/server/server.cc:297]   envoy.tracers: envoy.tracers.opencensus
espv2_1      | I0619 04:46:26.009 20 envoy] [20][main][external/envoy/source/server/server.cc:297]   envoy.clusters: envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns
espv2_1      | I0619 04:46:26.009 20 envoy] [20][main][external/envoy/source/server/server.cc:297]   envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.tap, envoy.transport_sockets.tls, raw_buffer, tls
espv2_1      | I0619 04:46:26.009 20 envoy] [20][main][external/envoy/source/server/server.cc:297]   envoy.udp_listeners: raw_udp_listener
espv2_1      | I0619 04:46:26.009 20 envoy] [20][main][external/envoy/source/server/server.cc:297]   envoy.filters.network: envoy.client_ssl_auth, envoy.ext_authz, envoy.filters.network.client_ssl_auth, envoy.filters.network.ext_authz, envoy.filters.network.http_connection_manager, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.sni_cluster, envoy.filters.network.tcp_proxy, envoy.http_connection_manager, envoy.ratelimit, envoy.tcp_proxy
espv2_1      | W0619 04:46:26.014 20 envoy] [20][main][external/envoy/source/server/server.cc:384] No admin address given, so no admin HTTP server started.
espv2_1      | D0619 04:46:26.014 20 envoy] [20][main][external/envoy/source/server/overload_manager_impl.cc:184] No overload action is configured for envoy.overload_actions.shrink_heap.
espv2_1      | D0619 04:46:26.014 20 envoy] [20][main][external/envoy/source/server/overload_manager_impl.cc:184] No overload action is configured for envoy.overload_actions.stop_accepting_connections.
espv2_1      | D0619 04:46:26.014 20 envoy] [20][main][external/envoy/source/server/overload_manager_impl.cc:184] No overload action is configured for envoy.overload_actions.stop_accepting_connections.
espv2_1      | D0619 04:46:26.014 20 envoy] [20][main][external/envoy/source/server/overload_manager_impl.cc:184] No overload action is configured for envoy.overload_actions.stop_accepting_connections.
espv2_1      | D0619 04:46:26.014 20 envoy] [20][main][external/envoy/source/server/overload_manager_impl.cc:184] No overload action is configured for envoy.overload_actions.stop_accepting_connections.
espv2_1      | D0619 04:46:26.014 20 envoy] [20][main][external/envoy/source/server/overload_manager_impl.cc:184] No overload action is configured for envoy.overload_actions.stop_accepting_connections.
espv2_1      | D0619 04:46:26.014 20 envoy] [20][main][external/envoy/source/server/overload_manager_impl.cc:184] No overload action is configured for envoy.overload_actions.stop_accepting_connections.
espv2_1      | D0619 04:46:26.014 20 envoy] [20][main][external/envoy/source/server/overload_manager_impl.cc:184] No overload action is configured for envoy.overload_actions.stop_accepting_connections.
espv2_1      | D0619 04:46:26.014 20 envoy] [20][main][external/envoy/source/server/overload_manager_impl.cc:184] No overload action is configured for envoy.overload_actions.stop_accepting_connections.
espv2_1      | I0619 04:46:26.014 20 envoy] [20][main][external/envoy/source/server/server.cc:525] runtime: layers:\n  - name: base\n    static_layer:\n      {}\n  - name: admin\n    admin_layer:\n      {}
espv2_1      | I0619 04:46:26.014 20 envoy] [20][config][external/envoy/source/server/configuration_impl.cc:103] loading tracing configuration
espv2_1      | I0619 04:46:26.014 20 envoy] [20][config][external/envoy/source/server/configuration_impl.cc:69] loading 0 static secret(s)
espv2_1      | I0619 04:46:26.014 20 envoy] [20][config][external/envoy/source/server/configuration_impl.cc:75] loading 1 cluster(s)
espv2_1      | D0619 04:46:26.014 29 envoy] [29][grpc][external/envoy/source/common/grpc/google_async_client_impl.cc:48] completionThread running
espv2_1      | D0619 04:46:26.015 20 envoy] [20][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:978] adding TLS initial cluster ads_cluster
espv2_1      | D0619 04:46:26.015 20 envoy] [20][upstream][external/envoy/source/common/upstream/upstream_impl.cc:278] transport socket match, socket default selected for host with address 127.0.0.1:8790
espv2_1      | D0619 04:46:26.015 20 envoy] [20][upstream][external/envoy/source/common/upstream/strict_dns_cluster.cc:146] DNS hosts have changed for 127.0.0.1
espv2_1      | D0619 04:46:26.015 20 envoy] [20][upstream][external/envoy/source/common/upstream/strict_dns_cluster.cc:167] DNS refresh rate reset for 127.0.0.1, refresh rate 5000 ms
espv2_1      | D0619 04:46:26.015 20 envoy] [20][upstream][external/envoy/source/common/upstream/upstream_impl.cc:971] initializing Primary cluster ads_cluster completed
espv2_1      | D0619 04:46:26.015 20 envoy] [20][init][external/envoy/source/common/init/manager_impl.cc:45] init manager Cluster ads_cluster contains no targets
espv2_1      | D0619 04:46:26.015 20 envoy] [20][init][external/envoy/source/common/init/watcher_impl.cc:14] init manager Cluster ads_cluster initialized, notifying ClusterImplBase
espv2_1      | D0619 04:46:26.015 20 envoy] [20][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:1137] membership update for TLS cluster ads_cluster added 1 removed 0
espv2_1      | D0619 04:46:26.015 20 envoy] [20][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:105] cm init: init complete: cluster=ads_cluster primary=0 secondary=0
espv2_1      | D0619 04:46:26.015 20 envoy] [20][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:125] maybe finish initialize state: 0
espv2_1      | D0619 04:46:26.015 20 envoy] [20][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:77] cm init: adding: cluster=ads_cluster primary=0 secondary=0
espv2_1      | D0619 04:46:26.015 20 envoy] [20][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:125] maybe finish initialize state: 1
espv2_1      | D0619 04:46:26.015 20 envoy] [20][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:134] maybe finish initialize primary init clusters empty: true
espv2_1      | D0619 04:46:26.015 20 envoy] [20][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:47] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.service.discovery.v3.DiscoveryRequest) returns (stream .envoy.service.discovery.v3.DiscoveryResponse);
espv2_1      | 
espv2_1      | D0619 04:46:26.015 20 envoy] [20][router][external/envoy/source/common/router/router.cc:435] [C0][S13718228616923961177] cluster \'ads_cluster\' match for URL \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'
espv2_1      | D0619 04:46:26.015 20 envoy] [20][router][external/envoy/source/common/router/router.cc:590] [C0][S13718228616923961177] router decoding headers:\n\':method\', \'POST\'\n\':path\', \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'\n\':authority\', \'ads_cluster\'\n\':scheme\', \'http\'\n\'te\', \'trailers\'\n\'content-type\', \'application/grpc\'\n\'x-envoy-internal\', \'true\'\n\'x-forwarded-for\', \'172.20.0.2\'
espv2_1      | 
espv2_1      | D0619 04:46:26.015 20 envoy] [20][pool][external/envoy/source/common/http/conn_pool_base.cc:346] queueing request due to no available connections
espv2_1      | D0619 04:46:26.015 20 envoy] [20][pool][external/envoy/source/common/http/conn_pool_base.cc:47] creating a new connection
espv2_1      | D0619 04:46:26.015 20 envoy] [20][client][external/envoy/source/common/http/codec_client.cc:35] [C0] connecting
espv2_1      | D0619 04:46:26.015 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:753] [C0] connecting to 127.0.0.1:8790
espv2_1      | D0619 04:46:26.015 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:762] [C0] connection in progress
espv2_1      | D0619 04:46:26.015 20 envoy] [20][http2][external/envoy/source/common/http/http2/codec_impl.cc:1011] [C0] updating connection-level initial window size to 268435456
espv2_1      | I0619 04:46:26.015 20 envoy] [20][config][external/envoy/source/server/configuration_impl.cc:79] loading 0 listener(s)
espv2_1      | I0619 04:46:26.015 20 envoy] [20][config][external/envoy/source/server/configuration_impl.cc:129] loading stats sink configuration
espv2_1      | D0619 04:46:26.015 20 envoy] [20][init][external/envoy/source/common/init/manager_impl.cc:20] added target LDS to init manager Server
espv2_1      | D0619 04:46:26.015 20 envoy] [20][init][external/envoy/source/common/init/manager_impl.cc:45] init manager RTDS contains no targets
espv2_1      | D0619 04:46:26.015 20 envoy] [20][init][external/envoy/source/common/init/watcher_impl.cc:14] init manager RTDS initialized, notifying RDTS
espv2_1      | I0619 04:46:26.015 20 envoy] [20][runtime][external/envoy/source/common/runtime/runtime_impl.cc:526] RTDS has finished initialization
espv2_1      | D0619 04:46:26.015 20 envoy] [20][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:196] continue initializing secondary clusters
espv2_1      | D0619 04:46:26.015 20 envoy] [20][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:125] maybe finish initialize state: 2
espv2_1      | D0619 04:46:26.015 20 envoy] [20][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:134] maybe finish initialize primary init clusters empty: true
espv2_1      | D0619 04:46:26.015 20 envoy] [20][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:149] maybe finish initialize secondary init clusters empty: true
espv2_1      | D0619 04:46:26.015 20 envoy] [20][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:172] maybe finish initialize cds api ready: true
espv2_1      | I0619 04:46:26.015 20 envoy] [20][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:174] cm init: initializing cds
espv2_1      | D0619 04:46:26.015 20 envoy] [20][config][external/envoy/source/common/config/grpc_mux_impl.cc:75] gRPC mux addWatch for type.googleapis.com/envoy.config.cluster.v3.Cluster
espv2_1      | I0619 04:46:26.015 20 envoy] [20][main][external/envoy/source/server/server.cc:622] starting main dispatch loop
espv2_1      | D0619 04:46:26.015 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:625] [C0] delayed connection error: 111
espv2_1      | D0619 04:46:26.015 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:208] [C0] closing socket: 0
espv2_1      | D0619 04:46:26.015 20 envoy] [20][client][external/envoy/source/common/http/codec_client.cc:92] [C0] disconnect. resetting 0 pending requests
espv2_1      | D0619 04:46:26.015 20 envoy] [20][pool][external/envoy/source/common/http/conn_pool_base.cc:265] [C0] client disconnected, failure reason: 
espv2_1      | D0619 04:46:26.015 20 envoy] [20][router][external/envoy/source/common/router/router.cc:1055] [C0][S13718228616923961177] upstream reset: reset reason connection failure
espv2_1      | D0619 04:46:26.016 20 envoy] [20][http][external/envoy/source/common/http/async_client_impl.cc:99] async http request response headers (end_stream=true):\n\':status\', \'200\'\n\'content-type\', \'application/grpc\'\n\'grpc-status\', \'14\'\n\'grpc-message\', \'upstream connect error or disconnect/reset before headers. reset reason: connection failure\'
espv2_1      | 
espv2_1      | W0619 04:46:26.016 20 envoy] [20][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
espv2_1      | D0619 04:46:26.016 20 envoy] [20][config][external/envoy/source/common/config/grpc_subscription_impl.cc:90] gRPC update for type.googleapis.com/envoy.config.cluster.v3.Cluster failed
espv2_1      | D0619 04:46:26.385 20 envoy] [20][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:47] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.service.discovery.v3.DiscoveryRequest) returns (stream .envoy.service.discovery.v3.DiscoveryResponse);
espv2_1      | 
espv2_1      | D0619 04:46:26.385 20 envoy] [20][router][external/envoy/source/common/router/router.cc:435] [C0][S703498962821898332] cluster \'ads_cluster\' match for URL \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'
espv2_1      | D0619 04:46:26.385 20 envoy] [20][router][external/envoy/source/common/router/router.cc:590] [C0][S703498962821898332] router decoding headers:\n\':method\', \'POST\'\n\':path\', \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'\n\':authority\', \'ads_cluster\'\n\':scheme\', \'http\'\n\'te\', \'trailers\'\n\'content-type\', \'application/grpc\'\n\'x-envoy-internal\', \'true\'\n\'x-forwarded-for\', \'172.20.0.2\'
espv2_1      | 
espv2_1      | D0619 04:46:26.385 20 envoy] [20][pool][external/envoy/source/common/http/conn_pool_base.cc:346] queueing request due to no available connections
espv2_1      | D0619 04:46:26.385 20 envoy] [20][pool][external/envoy/source/common/http/conn_pool_base.cc:47] creating a new connection
espv2_1      | D0619 04:46:26.385 20 envoy] [20][client][external/envoy/source/common/http/codec_client.cc:35] [C1] connecting
espv2_1      | D0619 04:46:26.385 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:753] [C1] connecting to 127.0.0.1:8790
espv2_1      | D0619 04:46:26.385 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:762] [C1] connection in progress
espv2_1      | D0619 04:46:26.385 20 envoy] [20][http2][external/envoy/source/common/http/http2/codec_impl.cc:1011] [C1] updating connection-level initial window size to 268435456
espv2_1      | D0619 04:46:26.386 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:625] [C1] delayed connection error: 111
espv2_1      | D0619 04:46:26.386 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:208] [C1] closing socket: 0
espv2_1      | D0619 04:46:26.386 20 envoy] [20][client][external/envoy/source/common/http/codec_client.cc:92] [C1] disconnect. resetting 0 pending requests
espv2_1      | D0619 04:46:26.386 20 envoy] [20][pool][external/envoy/source/common/http/conn_pool_base.cc:265] [C1] client disconnected, failure reason: 
espv2_1      | D0619 04:46:26.386 20 envoy] [20][router][external/envoy/source/common/router/router.cc:1055] [C0][S703498962821898332] upstream reset: reset reason connection failure
espv2_1      | D0619 04:46:26.386 20 envoy] [20][http][external/envoy/source/common/http/async_client_impl.cc:99] async http request response headers (end_stream=true):\n\':status\', \'200\'\n\'content-type\', \'application/grpc\'\n\'grpc-status\', \'14\'\n\'grpc-message\', \'upstream connect error or disconnect/reset before headers. reset reason: connection failure\'
espv2_1      | 
espv2_1      | W0619 04:46:26.386 20 envoy] [20][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
espv2_1      | D0619 04:46:26.386 20 envoy] [20][config][external/envoy/source/common/config/grpc_subscription_impl.cc:90] gRPC update for type.googleapis.com/envoy.config.cluster.v3.Cluster failed
espv2_1      | F0619 04:46:26.482986       7 server.go:52] fail to initialize config manager: http call to GET https://servicemanagement.googleapis.com/v1/services/sndcfegateway/rollouts?filter=status=SUCCESS returns not 200 OK: 403 Forbidden
espv2_1      | D0619 04:46:27.121 20 envoy] [20][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:47] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.service.discovery.v3.DiscoveryRequest) returns (stream .envoy.service.discovery.v3.DiscoveryResponse);
espv2_1      | 
espv2_1      | D0619 04:46:27.121 20 envoy] [20][router][external/envoy/source/common/router/router.cc:435] [C0][S5466442642553797511] cluster \'ads_cluster\' match for URL \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'
espv2_1      | D0619 04:46:27.121 20 envoy] [20][router][external/envoy/source/common/router/router.cc:590] [C0][S5466442642553797511] router decoding headers:\n\':method\', \'POST\'\n\':path\', \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'\n\':authority\', \'ads_cluster\'\n\':scheme\', \'http\'\n\'te\', \'trailers\'\n\'content-type\', \'application/grpc\'\n\'x-envoy-internal\', \'true\'\n\'x-forwarded-for\', \'172.20.0.2\'
espv2_1      | 
espv2_1      | D0619 04:46:27.121 20 envoy] [20][pool][external/envoy/source/common/http/conn_pool_base.cc:346] queueing request due to no available connections
espv2_1      | D0619 04:46:27.121 20 envoy] [20][pool][external/envoy/source/common/http/conn_pool_base.cc:47] creating a new connection
espv2_1      | D0619 04:46:27.121 20 envoy] [20][client][external/envoy/source/common/http/codec_client.cc:35] [C2] connecting
espv2_1      | D0619 04:46:27.122 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:753] [C2] connecting to 127.0.0.1:8790
espv2_1      | D0619 04:46:27.122 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:762] [C2] connection in progress
espv2_1      | D0619 04:46:27.122 20 envoy] [20][http2][external/envoy/source/common/http/http2/codec_impl.cc:1011] [C2] updating connection-level initial window size to 268435456
espv2_1      | D0619 04:46:27.122 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:625] [C2] delayed connection error: 111
espv2_1      | D0619 04:46:27.122 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:208] [C2] closing socket: 0
espv2_1      | D0619 04:46:27.122 20 envoy] [20][client][external/envoy/source/common/http/codec_client.cc:92] [C2] disconnect. resetting 0 pending requests
espv2_1      | D0619 04:46:27.122 20 envoy] [20][pool][external/envoy/source/common/http/conn_pool_base.cc:265] [C2] client disconnected, failure reason: 
espv2_1      | D0619 04:46:27.122 20 envoy] [20][router][external/envoy/source/common/router/router.cc:1055] [C0][S5466442642553797511] upstream reset: reset reason connection failure
espv2_1      | D0619 04:46:27.122 20 envoy] [20][http][external/envoy/source/common/http/async_client_impl.cc:99] async http request response headers (end_stream=true):\n\':status\', \'200\'\n\'content-type\', \'application/grpc\'\n\'grpc-status\', \'14\'\n\'grpc-message\', \'upstream connect error or disconnect/reset before headers. reset reason: connection failure\'
espv2_1      | 
espv2_1      | W0619 04:46:27.122 20 envoy] [20][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
espv2_1      | D0619 04:46:27.122 20 envoy] [20][config][external/envoy/source/common/config/grpc_subscription_impl.cc:90] gRPC update for type.googleapis.com/envoy.config.cluster.v3.Cluster failed
espv2_1      | D0619 04:46:27.512 20 envoy] [20][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:47] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.service.discovery.v3.DiscoveryRequest) returns (stream .envoy.service.discovery.v3.DiscoveryResponse);
espv2_1      | 
espv2_1      | D0619 04:46:27.512 20 envoy] [20][router][external/envoy/source/common/router/router.cc:435] [C0][S8658916066719316641] cluster \'ads_cluster\' match for URL \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'
espv2_1      | D0619 04:46:27.512 20 envoy] [20][router][external/envoy/source/common/router/router.cc:590] [C0][S8658916066719316641] router decoding headers:\n\':method\', \'POST\'\n\':path\', \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'\n\':authority\', \'ads_cluster\'\n\':scheme\', \'http\'\n\'te\', \'trailers\'\n\'content-type\', \'application/grpc\'\n\'x-envoy-internal\', \'true\'\n\'x-forwarded-for\', \'172.20.0.2\'
espv2_1      | 
espv2_1      | D0619 04:46:27.512 20 envoy] [20][pool][external/envoy/source/common/http/conn_pool_base.cc:346] queueing request due to no available connections
espv2_1      | D0619 04:46:27.512 20 envoy] [20][pool][external/envoy/source/common/http/conn_pool_base.cc:47] creating a new connection
espv2_1      | D0619 04:46:27.512 20 envoy] [20][client][external/envoy/source/common/http/codec_client.cc:35] [C3] connecting
espv2_1      | D0619 04:46:27.512 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:753] [C3] connecting to 127.0.0.1:8790
espv2_1      | D0619 04:46:27.512 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:762] [C3] connection in progress
espv2_1      | D0619 04:46:27.512 20 envoy] [20][http2][external/envoy/source/common/http/http2/codec_impl.cc:1011] [C3] updating connection-level initial window size to 268435456
espv2_1      | D0619 04:46:27.513 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:625] [C3] delayed connection error: 111
espv2_1      | D0619 04:46:27.513 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:208] [C3] closing socket: 0
espv2_1      | D0619 04:46:27.513 20 envoy] [20][client][external/envoy/source/common/http/codec_client.cc:92] [C3] disconnect. resetting 0 pending requests
espv2_1      | D0619 04:46:27.513 20 envoy] [20][pool][external/envoy/source/common/http/conn_pool_base.cc:265] [C3] client disconnected, failure reason: 
espv2_1      | D0619 04:46:27.513 20 envoy] [20][router][external/envoy/source/common/router/router.cc:1055] [C0][S8658916066719316641] upstream reset: reset reason connection failure
espv2_1      | D0619 04:46:27.513 20 envoy] [20][http][external/envoy/source/common/http/async_client_impl.cc:99] async http request response headers (end_stream=true):\n\':status\', \'200\'\n\'content-type\', \'application/grpc\'\n\'grpc-status\', \'14\'\n\'grpc-message\', \'upstream connect error or disconnect/reset before headers. reset reason: connection failure\'
espv2_1      | 
espv2_1      | W0619 04:46:27.513 20 envoy] [20][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
espv2_1      | D0619 04:46:27.513 20 envoy] [20][config][external/envoy/source/common/config/grpc_subscription_impl.cc:90] gRPC update for type.googleapis.com/envoy.config.cluster.v3.Cluster failed
espv2_1      | D0619 04:46:28.915 20 envoy] [20][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:47] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.service.discovery.v3.DiscoveryRequest) returns (stream .envoy.service.discovery.v3.DiscoveryResponse);
espv2_1      | 
espv2_1      | D0619 04:46:28.915 20 envoy] [20][router][external/envoy/source/common/router/router.cc:435] [C0][S17401446153229744649] cluster \'ads_cluster\' match for URL \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'
espv2_1      | D0619 04:46:28.915 20 envoy] [20][router][external/envoy/source/common/router/router.cc:590] [C0][S17401446153229744649] router decoding headers:\n\':method\', \'POST\'\n\':path\', \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'\n\':authority\', \'ads_cluster\'\n\':scheme\', \'http\'\n\'te\', \'trailers\'\n\'content-type\', \'application/grpc\'\n\'x-envoy-internal\', \'true\'\n\'x-forwarded-for\', \'172.20.0.2\'
espv2_1      | 
espv2_1      | D0619 04:46:28.915 20 envoy] [20][pool][external/envoy/source/common/http/conn_pool_base.cc:346] queueing request due to no available connections
espv2_1      | D0619 04:46:28.915 20 envoy] [20][pool][external/envoy/source/common/http/conn_pool_base.cc:47] creating a new connection
espv2_1      | D0619 04:46:28.915 20 envoy] [20][client][external/envoy/source/common/http/codec_client.cc:35] [C4] connecting
espv2_1      | D0619 04:46:28.915 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:753] [C4] connecting to 127.0.0.1:8790
espv2_1      | D0619 04:46:28.915 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:762] [C4] connection in progress
espv2_1      | D0619 04:46:28.915 20 envoy] [20][http2][external/envoy/source/common/http/http2/codec_impl.cc:1011] [C4] updating connection-level initial window size to 268435456
espv2_1      | D0619 04:46:28.915 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:625] [C4] delayed connection error: 111
espv2_1      | D0619 04:46:28.915 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:208] [C4] closing socket: 0
espv2_1      | D0619 04:46:28.915 20 envoy] [20][client][external/envoy/source/common/http/codec_client.cc:92] [C4] disconnect. resetting 0 pending requests
espv2_1      | D0619 04:46:28.915 20 envoy] [20][pool][external/envoy/source/common/http/conn_pool_base.cc:265] [C4] client disconnected, failure reason: 
espv2_1      | D0619 04:46:28.915 20 envoy] [20][router][external/envoy/source/common/router/router.cc:1055] [C0][S17401446153229744649] upstream reset: reset reason connection failure
espv2_1      | D0619 04:46:28.916 20 envoy] [20][http][external/envoy/source/common/http/async_client_impl.cc:99] async http request response headers (end_stream=true):\n\':status\', \'200\'\n\'content-type\', \'application/grpc\'\n\'grpc-status\', \'14\'\n\'grpc-message\', \'upstream connect error or disconnect/reset before headers. reset reason: connection failure\'
espv2_1      | 
espv2_1      | W0619 04:46:28.916 20 envoy] [20][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
espv2_1      | D0619 04:46:28.916 20 envoy] [20][config][external/envoy/source/common/config/grpc_subscription_impl.cc:90] gRPC update for type.googleapis.com/envoy.config.cluster.v3.Cluster failed
espv2_1      | D0619 04:46:31.018 20 envoy] [20][upstream][external/envoy/source/common/upstream/upstream_impl.cc:278] transport socket match, socket default selected for host with address 127.0.0.1:8790
espv2_1      | D0619 04:46:31.018 20 envoy] [20][upstream][external/envoy/source/common/upstream/strict_dns_cluster.cc:167] DNS refresh rate reset for 127.0.0.1, refresh rate 5000 ms
espv2_1      | D0619 04:46:31.018 20 envoy] [20][main][external/envoy/source/server/server.cc:187] flushing stats
espv2_1      | D0619 04:46:31.018 20 envoy] [20][main][external/envoy/source/server/server.cc:197] Envoy is not fully initialized, skipping histogram merge and flushing stats
espv2_1      | D0619 04:46:34.730 20 envoy] [20][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:47] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.service.discovery.v3.DiscoveryRequest) returns (stream .envoy.service.discovery.v3.DiscoveryResponse);
espv2_1      | 
espv2_1      | D0619 04:46:34.730 20 envoy] [20][router][external/envoy/source/common/router/router.cc:435] [C0][S12299255759467898207] cluster \'ads_cluster\' match for URL \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'
espv2_1      | D0619 04:46:34.730 20 envoy] [20][router][external/envoy/source/common/router/router.cc:590] [C0][S12299255759467898207] router decoding headers:\n\':method\', \'POST\'\n\':path\', \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'\n\':authority\', \'ads_cluster\'\n\':scheme\', \'http\'\n\'te\', \'trailers\'\n\'content-type\', \'application/grpc\'\n\'x-envoy-internal\', \'true\'\n\'x-forwarded-for\', \'172.20.0.2\'
espv2_1      | 
espv2_1      | D0619 04:46:34.730 20 envoy] [20][pool][external/envoy/source/common/http/conn_pool_base.cc:346] queueing request due to no available connections
espv2_1      | D0619 04:46:34.730 20 envoy] [20][pool][external/envoy/source/common/http/conn_pool_base.cc:47] creating a new connection
espv2_1      | D0619 04:46:34.730 20 envoy] [20][client][external/envoy/source/common/http/codec_client.cc:35] [C5] connecting
espv2_1      | D0619 04:46:34.731 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:753] [C5] connecting to 127.0.0.1:8790
espv2_1      | D0619 04:46:34.731 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:762] [C5] connection in progress
espv2_1      | D0619 04:46:34.731 20 envoy] [20][http2][external/envoy/source/common/http/http2/codec_impl.cc:1011] [C5] updating connection-level initial window size to 268435456
espv2_1      | D0619 04:46:34.731 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:625] [C5] delayed connection error: 111
espv2_1      | D0619 04:46:34.731 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:208] [C5] closing socket: 0
espv2_1      | D0619 04:46:34.731 20 envoy] [20][client][external/envoy/source/common/http/codec_client.cc:92] [C5] disconnect. resetting 0 pending requests
espv2_1      | D0619 04:46:34.731 20 envoy] [20][pool][external/envoy/source/common/http/conn_pool_base.cc:265] [C5] client disconnected, failure reason: 
espv2_1      | D0619 04:46:34.731 20 envoy] [20][router][external/envoy/source/common/router/router.cc:1055] [C0][S12299255759467898207] upstream reset: reset reason connection failure
espv2_1      | D0619 04:46:34.731 20 envoy] [20][http][external/envoy/source/common/http/async_client_impl.cc:99] async http request response headers (end_stream=true):\n\':status\', \'200\'\n\'content-type\', \'application/grpc\'\n\'grpc-status\', \'14\'\n\'grpc-message\', \'upstream connect error or disconnect/reset before headers. reset reason: connection failure\'
espv2_1      | 
espv2_1      | W0619 04:46:34.731 20 envoy] [20][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
espv2_1      | D0619 04:46:34.731 20 envoy] [20][config][external/envoy/source/common/config/grpc_subscription_impl.cc:90] gRPC update for type.googleapis.com/envoy.config.cluster.v3.Cluster failed
espv2_1      | D0619 04:46:36.022 20 envoy] [20][main][external/envoy/source/server/server.cc:187] flushing stats
espv2_1      | D0619 04:46:36.022 20 envoy] [20][main][external/envoy/source/server/server.cc:197] Envoy is not fully initialized, skipping histogram merge and flushing stats
espv2_1      | D0619 04:46:36.023 20 envoy] [20][upstream][external/envoy/source/common/upstream/upstream_impl.cc:278] transport socket match, socket default selected for host with address 127.0.0.1:8790
espv2_1      | D0619 04:46:36.023 20 envoy] [20][upstream][external/envoy/source/common/upstream/strict_dns_cluster.cc:167] DNS refresh rate reset for 127.0.0.1, refresh rate 5000 ms
espv2_1      | D0619 04:46:38.306 20 envoy] [20][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:47] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.service.discovery.v3.DiscoveryRequest) returns (stream .envoy.service.discovery.v3.DiscoveryResponse);
espv2_1      | 
espv2_1      | D0619 04:46:38.307 20 envoy] [20][router][external/envoy/source/common/router/router.cc:435] [C0][S18403206789726792937] cluster \'ads_cluster\' match for URL \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'
espv2_1      | D0619 04:46:38.307 20 envoy] [20][router][external/envoy/source/common/router/router.cc:590] [C0][S18403206789726792937] router decoding headers:\n\':method\', \'POST\'\n\':path\', \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'\n\':authority\', \'ads_cluster\'\n\':scheme\', \'http\'\n\'te\', \'trailers\'\n\'content-type\', \'application/grpc\'\n\'x-envoy-internal\', \'true\'\n\'x-forwarded-for\', \'172.20.0.2\'
espv2_1      | 
espv2_1      | D0619 04:46:38.307 20 envoy] [20][pool][external/envoy/source/common/http/conn_pool_base.cc:346] queueing request due to no available connections
espv2_1      | D0619 04:46:38.307 20 envoy] [20][pool][external/envoy/source/common/http/conn_pool_base.cc:47] creating a new connection
espv2_1      | D0619 04:46:38.307 20 envoy] [20][client][external/envoy/source/common/http/codec_client.cc:35] [C6] connecting
espv2_1      | D0619 04:46:38.307 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:753] [C6] connecting to 127.0.0.1:8790
espv2_1      | D0619 04:46:38.307 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:762] [C6] connection in progress
espv2_1      | D0619 04:46:38.307 20 envoy] [20][http2][external/envoy/source/common/http/http2/codec_impl.cc:1011] [C6] updating connection-level initial window size to 268435456
espv2_1      | D0619 04:46:38.307 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:625] [C6] delayed connection error: 111
espv2_1      | D0619 04:46:38.307 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:208] [C6] closing socket: 0
espv2_1      | D0619 04:46:38.307 20 envoy] [20][client][external/envoy/source/common/http/codec_client.cc:92] [C6] disconnect. resetting 0 pending requests
espv2_1      | D0619 04:46:38.307 20 envoy] [20][pool][external/envoy/source/common/http/conn_pool_base.cc:265] [C6] client disconnected, failure reason: 
espv2_1      | D0619 04:46:38.308 20 envoy] [20][router][external/envoy/source/common/router/router.cc:1055] [C0][S18403206789726792937] upstream reset: reset reason connection failure
espv2_1      | D0619 04:46:38.308 20 envoy] [20][http][external/envoy/source/common/http/async_client_impl.cc:99] async http request response headers (end_stream=true):\n\':status\', \'200\'\n\'content-type\', \'application/grpc\'\n\'grpc-status\', \'14\'\n\'grpc-message\', \'upstream connect error or disconnect/reset before headers. reset reason: connection failure\'
espv2_1      | 
espv2_1      | W0619 04:46:38.308 20 envoy] [20][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
espv2_1      | D0619 04:46:38.308 20 envoy] [20][config][external/envoy/source/common/config/grpc_subscription_impl.cc:90] gRPC update for type.googleapis.com/envoy.config.cluster.v3.Cluster failed
espv2_1      | D0619 04:46:41.019 20 envoy] [20][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:125] maybe finish initialize state: 4
espv2_1      | D0619 04:46:41.019 20 envoy] [20][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:134] maybe finish initialize primary init clusters empty: true
espv2_1      | D0619 04:46:41.019 20 envoy] [20][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:149] maybe finish initialize secondary init clusters empty: true
espv2_1      | D0619 04:46:41.019 20 envoy] [20][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:172] maybe finish initialize cds api ready: true
espv2_1      | I0619 04:46:41.019 20 envoy] [20][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:178] cm init: all clusters initialized
espv2_1      | D0619 04:46:41.019 20 envoy] [20][config][external/envoy/source/common/config/grpc_mux_impl.cc:98] Pausing discovery requests for type.googleapis.com/envoy.api.v2.RouteConfiguration
espv2_1      | I0619 04:46:41.019 20 envoy] [20][main][external/envoy/source/server/server.cc:601] all clusters initialized. initializing init manager
espv2_1      | D0619 04:46:41.019 20 envoy] [20][init][external/envoy/source/common/init/manager_impl.cc:49] init manager Server initializing
espv2_1      | D0619 04:46:41.019 20 envoy] [20][init][external/envoy/source/common/init/target_impl.cc:15] init manager Server initializing target LDS
espv2_1      | D0619 04:46:41.019 20 envoy] [20][config][external/envoy/source/common/config/grpc_mux_impl.cc:75] gRPC mux addWatch for type.googleapis.com/envoy.config.listener.v3.Listener
espv2_1      | D0619 04:46:41.019 20 envoy] [20][config][external/envoy/source/common/config/grpc_mux_impl.cc:32] No stream available to sendDiscoveryRequest for type.googleapis.com/envoy.config.listener.v3.Listener
espv2_1      | D0619 04:46:41.019 20 envoy] [20][config][external/envoy/source/common/config/grpc_mux_impl.cc:106] Resuming discovery requests for type.googleapis.com/envoy.api.v2.RouteConfiguration
espv2_1      | D0619 04:46:41.026 20 envoy] [20][upstream][external/envoy/source/common/upstream/upstream_impl.cc:278] transport socket match, socket default selected for host with address 127.0.0.1:8790
espv2_1      | D0619 04:46:41.026 20 envoy] [20][upstream][external/envoy/source/common/upstream/strict_dns_cluster.cc:167] DNS refresh rate reset for 127.0.0.1, refresh rate 5000 ms
espv2_1      | D0619 04:46:41.026 20 envoy] [20][main][external/envoy/source/server/server.cc:187] flushing stats
espv2_1      | D0619 04:46:41.026 20 envoy] [20][main][external/envoy/source/server/server.cc:197] Envoy is not fully initialized, skipping histogram merge and flushing stats
espv2_1      | D0619 04:46:46.030 20 envoy] [20][upstream][external/envoy/source/common/upstream/upstream_impl.cc:278] transport socket match, socket default selected for host with address 127.0.0.1:8790
espv2_1      | D0619 04:46:46.030 20 envoy] [20][upstream][external/envoy/source/common/upstream/strict_dns_cluster.cc:167] DNS refresh rate reset for 127.0.0.1, refresh rate 5000 ms
espv2_1      | D0619 04:46:46.030 20 envoy] [20][main][external/envoy/source/server/server.cc:187] flushing stats
espv2_1      | D0619 04:46:46.030 20 envoy] [20][main][external/envoy/source/server/server.cc:197] Envoy is not fully initialized, skipping histogram merge and flushing stats
espv2_1      | D0619 04:46:48.639 20 envoy] [20][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:47] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.service.discovery.v3.DiscoveryRequest) returns (stream .envoy.service.discovery.v3.DiscoveryResponse);
espv2_1      | 
espv2_1      | D0619 04:46:48.639 20 envoy] [20][router][external/envoy/source/common/router/router.cc:435] [C0][S15554854657023217580] cluster \'ads_cluster\' match for URL \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'
espv2_1      | D0619 04:46:48.639 20 envoy] [20][router][external/envoy/source/common/router/router.cc:590] [C0][S15554854657023217580] router decoding headers:\n\':method\', \'POST\'\n\':path\', \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'\n\':authority\', \'ads_cluster\'\n\':scheme\', \'http\'\n\'te\', \'trailers\'\n\'content-type\', \'application/grpc\'\n\'x-envoy-internal\', \'true\'\n\'x-forwarded-for\', \'172.20.0.2\'
espv2_1      | 
espv2_1      | D0619 04:46:48.639 20 envoy] [20][pool][external/envoy/source/common/http/conn_pool_base.cc:346] queueing request due to no available connections
espv2_1      | D0619 04:46:48.639 20 envoy] [20][pool][external/envoy/source/common/http/conn_pool_base.cc:47] creating a new connection
espv2_1      | D0619 04:46:48.640 20 envoy] [20][client][external/envoy/source/common/http/codec_client.cc:35] [C7] connecting
espv2_1      | D0619 04:46:48.640 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:753] [C7] connecting to 127.0.0.1:8790
espv2_1      | D0619 04:46:48.640 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:762] [C7] connection in progress
espv2_1      | D0619 04:46:48.640 20 envoy] [20][http2][external/envoy/source/common/http/http2/codec_impl.cc:1011] [C7] updating connection-level initial window size to 268435456
espv2_1      | D0619 04:46:48.640 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:625] [C7] delayed connection error: 111
espv2_1      | D0619 04:46:48.640 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:208] [C7] closing socket: 0
espv2_1      | D0619 04:46:48.640 20 envoy] [20][client][external/envoy/source/common/http/codec_client.cc:92] [C7] disconnect. resetting 0 pending requests
espv2_1      | D0619 04:46:48.640 20 envoy] [20][pool][external/envoy/source/common/http/conn_pool_base.cc:265] [C7] client disconnected, failure reason: 
espv2_1      | D0619 04:46:48.640 20 envoy] [20][router][external/envoy/source/common/router/router.cc:1055] [C0][S15554854657023217580] upstream reset: reset reason connection failure
espv2_1      | D0619 04:46:48.641 20 envoy] [20][http][external/envoy/source/common/http/async_client_impl.cc:99] async http request response headers (end_stream=true):\n\':status\', \'200\'\n\'content-type\', \'application/grpc\'\n\'grpc-status\', \'14\'\n\'grpc-message\', \'upstream connect error or disconnect/reset before headers. reset reason: connection failure\'
espv2_1      | 
espv2_1      | W0619 04:46:48.641 20 envoy] [20][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
espv2_1      | D0619 04:46:48.641 20 envoy] [20][config][external/envoy/source/common/config/grpc_subscription_impl.cc:90] gRPC update for type.googleapis.com/envoy.config.listener.v3.Listener failed
espv2_1      | D0619 04:46:48.641 20 envoy] [20][config][external/envoy/source/common/config/grpc_subscription_impl.cc:90] gRPC update for type.googleapis.com/envoy.config.cluster.v3.Cluster failed
espv2_1      | D0619 04:46:51.035 20 envoy] [20][main][external/envoy/source/server/server.cc:187] flushing stats
espv2_1      | D0619 04:46:51.035 20 envoy] [20][main][external/envoy/source/server/server.cc:197] Envoy is not fully initialized, skipping histogram merge and flushing stats
espv2_1      | D0619 04:46:51.035 20 envoy] [20][upstream][external/envoy/source/common/upstream/upstream_impl.cc:278] transport socket match, socket default selected for host with address 127.0.0.1:8790
espv2_1      | D0619 04:46:51.035 20 envoy] [20][upstream][external/envoy/source/common/upstream/strict_dns_cluster.cc:167] DNS refresh rate reset for 127.0.0.1, refresh rate 5000 ms
espv2_1      | D0619 04:46:56.023 20 envoy] [20][init][external/envoy/source/common/init/watcher_impl.cc:14] target LDS initialized, notifying init manager Server
espv2_1      | D0619 04:46:56.023 20 envoy] [20][init][external/envoy/source/common/init/watcher_impl.cc:14] init manager Server initialized, notifying RunHelper
espv2_1      | I0619 04:46:56.023 20 envoy] [20][config][external/envoy/source/server/listener_manager_impl.cc:836] all dependencies initialized. starting workers
espv2_1      | D0619 04:46:56.023 20 envoy] [20][config][external/envoy/source/server/listener_manager_impl.cc:847] starting worker 0
espv2_1      | D0619 04:46:56.023 20 envoy] [20][config][external/envoy/source/server/listener_manager_impl.cc:847] starting worker 1
espv2_1      | D0619 04:46:56.023 37 envoy] [37][main][external/envoy/source/server/worker_impl.cc:111] worker entering dispatch loop
espv2_1      | D0619 04:46:56.023 20 envoy] [20][config][external/envoy/source/server/listener_manager_impl.cc:847] starting worker 2
espv2_1      | D0619 04:46:56.023 38 envoy] [38][main][external/envoy/source/server/worker_impl.cc:111] worker entering dispatch loop
espv2_1      | D0619 04:46:56.024 38 envoy] [38][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:978] adding TLS initial cluster ads_cluster
espv2_1      | D0619 04:46:56.024 41 envoy] [41][grpc][external/envoy/source/common/grpc/google_async_client_impl.cc:48] completionThread running
espv2_1      | D0619 04:46:56.024 39 envoy] [39][grpc][external/envoy/source/common/grpc/google_async_client_impl.cc:48] completionThread running
espv2_1      | D0619 04:46:56.024 38 envoy] [38][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:1137] membership update for TLS cluster ads_cluster added 1 removed 0
espv2_1      | D0619 04:46:56.024 20 envoy] [20][config][external/envoy/source/server/listener_manager_impl.cc:847] starting worker 3
espv2_1      | D0619 04:46:56.024 40 envoy] [40][main][external/envoy/source/server/worker_impl.cc:111] worker entering dispatch loop
espv2_1      | D0619 04:46:56.024 37 envoy] [37][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:978] adding TLS initial cluster ads_cluster
espv2_1      | D0619 04:46:56.028 20 envoy] [20][config][external/envoy/source/server/listener_manager_impl.cc:847] starting worker 4
espv2_1      | D0619 04:46:56.028 42 envoy] [42][main][external/envoy/source/server/worker_impl.cc:111] worker entering dispatch loop
espv2_1      | D0619 04:46:56.028 37 envoy] [37][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:1137] membership update for TLS cluster ads_cluster added 1 removed 0
espv2_1      | D0619 04:46:56.028 40 envoy] [40][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:978] adding TLS initial cluster ads_cluster
espv2_1      | D0619 04:46:56.028 43 envoy] [43][grpc][external/envoy/source/common/grpc/google_async_client_impl.cc:48] completionThread running
espv2_1      | D0619 04:46:56.028 42 envoy] [42][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:978] adding TLS initial cluster ads_cluster
espv2_1      | D0619 04:46:56.028 20 envoy] [20][config][external/envoy/source/server/listener_manager_impl.cc:847] starting worker 5
espv2_1      | D0619 04:46:56.028 44 envoy] [44][grpc][external/envoy/source/common/grpc/google_async_client_impl.cc:48] completionThread running
espv2_1      | D0619 04:46:56.028 40 envoy] [40][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:1137] membership update for TLS cluster ads_cluster added 1 removed 0
espv2_1      | D0619 04:46:56.028 45 envoy] [45][main][external/envoy/source/server/worker_impl.cc:111] worker entering dispatch loop
espv2_1      | D0619 04:46:56.028 42 envoy] [42][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:1137] membership update for TLS cluster ads_cluster added 1 removed 0
espv2_1      | D0619 04:46:56.028 20 envoy] [20][config][external/envoy/source/server/listener_manager_impl.cc:847] starting worker 6
espv2_1      | D0619 04:46:56.028 46 envoy] [46][main][external/envoy/source/server/worker_impl.cc:111] worker entering dispatch loop
espv2_1      | D0619 04:46:56.029 45 envoy] [45][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:978] adding TLS initial cluster ads_cluster
espv2_1      | D0619 04:46:56.029 48 envoy] [48][main][external/envoy/source/server/worker_impl.cc:111] worker entering dispatch loop
espv2_1      | D0619 04:46:56.029 47 envoy] [47][grpc][external/envoy/source/common/grpc/google_async_client_impl.cc:48] completionThread running
espv2_1      | D0619 04:46:56.029 20 envoy] [20][config][external/envoy/source/server/listener_manager_impl.cc:847] starting worker 7
espv2_1      | D0619 04:46:56.029 49 envoy] [49][grpc][external/envoy/source/common/grpc/google_async_client_impl.cc:48] completionThread running
espv2_1      | D0619 04:46:56.029 45 envoy] [45][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:1137] membership update for TLS cluster ads_cluster added 1 removed 0
espv2_1      | D0619 04:46:56.029 46 envoy] [46][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:978] adding TLS initial cluster ads_cluster
espv2_1      | D0619 04:46:56.029 48 envoy] [48][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:978] adding TLS initial cluster ads_cluster
espv2_1      | D0619 04:46:56.029 50 envoy] [50][grpc][external/envoy/source/common/grpc/google_async_client_impl.cc:48] completionThread running
espv2_1      | D0619 04:46:56.029 51 envoy] [51][main][external/envoy/source/server/worker_impl.cc:111] worker entering dispatch loop
espv2_1      | D0619 04:46:56.029 46 envoy] [46][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:1137] membership update for TLS cluster ads_cluster added 1 removed 0
espv2_1      | D0619 04:46:56.029 48 envoy] [48][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:1137] membership update for TLS cluster ads_cluster added 1 removed 0
espv2_1      | D0619 04:46:56.030 51 envoy] [51][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:978] adding TLS initial cluster ads_cluster
espv2_1      | D0619 04:46:56.030 51 envoy] [51][upstream][external/envoy/source/common/upstream/cluster_manager_impl.cc:1137] membership update for TLS cluster ads_cluster added 1 removed 0
espv2_1      | D0619 04:46:56.030 52 envoy] [52][grpc][external/envoy/source/common/grpc/google_async_client_impl.cc:48] completionThread running
espv2_1      | D0619 04:46:56.041 20 envoy] [20][upstream][external/envoy/source/common/upstream/upstream_impl.cc:278] transport socket match, socket default selected for host with address 127.0.0.1:8790
espv2_1      | D0619 04:46:56.041 20 envoy] [20][upstream][external/envoy/source/common/upstream/strict_dns_cluster.cc:167] DNS refresh rate reset for 127.0.0.1, refresh rate 5000 ms
espv2_1      | D0619 04:46:56.041 20 envoy] [20][main][external/envoy/source/server/server.cc:187] flushing stats
espv2_1      | D0619 04:46:57.983 20 envoy] [20][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:47] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.service.discovery.v3.DiscoveryRequest) returns (stream .envoy.service.discovery.v3.DiscoveryResponse);
espv2_1      | 
espv2_1      | D0619 04:46:57.983 20 envoy] [20][router][external/envoy/source/common/router/router.cc:435] [C0][S6067856308410903096] cluster \'ads_cluster\' match for URL \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'
espv2_1      | D0619 04:46:57.983 20 envoy] [20][router][external/envoy/source/common/router/router.cc:590] [C0][S6067856308410903096] router decoding headers:\n\':method\', \'POST\'\n\':path\', \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'\n\':authority\', \'ads_cluster\'\n\':scheme\', \'http\'\n\'te\', \'trailers\'\n\'content-type\', \'application/grpc\'\n\'x-envoy-internal\', \'true\'\n\'x-forwarded-for\', \'172.20.0.2\'
espv2_1      | 
espv2_1      | D0619 04:46:57.983 20 envoy] [20][pool][external/envoy/source/common/http/conn_pool_base.cc:346] queueing request due to no available connections
espv2_1      | D0619 04:46:57.983 20 envoy] [20][pool][external/envoy/source/common/http/conn_pool_base.cc:47] creating a new connection
espv2_1      | D0619 04:46:57.983 20 envoy] [20][client][external/envoy/source/common/http/codec_client.cc:35] [C8] connecting
espv2_1      | D0619 04:46:57.983 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:753] [C8] connecting to 127.0.0.1:8790
espv2_1      | D0619 04:46:57.984 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:762] [C8] connection in progress
espv2_1      | D0619 04:46:57.984 20 envoy] [20][http2][external/envoy/source/common/http/http2/codec_impl.cc:1011] [C8] updating connection-level initial window size to 268435456
espv2_1      | D0619 04:46:57.984 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:625] [C8] delayed connection error: 111
espv2_1      | D0619 04:46:57.984 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:208] [C8] closing socket: 0
espv2_1      | D0619 04:46:57.984 20 envoy] [20][client][external/envoy/source/common/http/codec_client.cc:92] [C8] disconnect. resetting 0 pending requests
espv2_1      | D0619 04:46:57.984 20 envoy] [20][pool][external/envoy/source/common/http/conn_pool_base.cc:265] [C8] client disconnected, failure reason: 
espv2_1      | D0619 04:46:57.984 20 envoy] [20][router][external/envoy/source/common/router/router.cc:1055] [C0][S6067856308410903096] upstream reset: reset reason connection failure
espv2_1      | D0619 04:46:57.984 20 envoy] [20][http][external/envoy/source/common/http/async_client_impl.cc:99] async http request response headers (end_stream=true):\n\':status\', \'200\'\n\'content-type\', \'application/grpc\'\n\'grpc-status\', \'14\'\n\'grpc-message\', \'upstream connect error or disconnect/reset before headers. reset reason: connection failure\'
espv2_1      | 
espv2_1      | W0619 04:46:57.984 20 envoy] [20][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
espv2_1      | D0619 04:46:57.984 20 envoy] [20][config][external/envoy/source/common/config/grpc_subscription_impl.cc:90] gRPC update for type.googleapis.com/envoy.config.listener.v3.Listener failed
espv2_1      | D0619 04:46:57.984 20 envoy] [20][config][external/envoy/source/common/config/grpc_subscription_impl.cc:90] gRPC update for type.googleapis.com/envoy.config.cluster.v3.Cluster failed
espv2_1      | D0619 04:47:01.040 20 envoy] [20][main][external/envoy/source/server/server.cc:187] flushing stats
espv2_1      | D0619 04:47:01.041 20 envoy] [20][upstream][external/envoy/source/common/upstream/upstream_impl.cc:278] transport socket match, socket default selected for host with address 127.0.0.1:8790
espv2_1      | D0619 04:47:01.041 20 envoy] [20][upstream][external/envoy/source/common/upstream/strict_dns_cluster.cc:167] DNS refresh rate reset for 127.0.0.1, refresh rate 5000 ms
espv2_1      | CRITICAL: Config Manager is down, killing all processes.
espv2_1      | D0619 04:47:06.040 20 envoy] [20][upstream][external/envoy/source/common/upstream/upstream_impl.cc:278] transport socket match, socket default selected for host with address 127.0.0.1:8790
espv2_1      | D0619 04:47:06.040 20 envoy] [20][upstream][external/envoy/source/common/upstream/strict_dns_cluster.cc:167] DNS refresh rate reset for 127.0.0.1, refresh rate 5000 ms
espv2_1      | D0619 04:47:06.040 20 envoy] [20][main][external/envoy/source/server/server.cc:187] flushing stats
espv2_1      | D0619 04:47:07.452 20 envoy] [20][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:47] Establishing new gRPC bidi stream for rpc StreamAggregatedResources(stream .envoy.service.discovery.v3.DiscoveryRequest) returns (stream .envoy.service.discovery.v3.DiscoveryResponse);
espv2_1      | 
espv2_1      | D0619 04:47:07.452 20 envoy] [20][router][external/envoy/source/common/router/router.cc:435] [C0][S314116016824065774] cluster \'ads_cluster\' match for URL \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'
espv2_1      | D0619 04:47:07.452 20 envoy] [20][router][external/envoy/source/common/router/router.cc:590] [C0][S314116016824065774] router decoding headers:\n\':method\', \'POST\'\n\':path\', \'/envoy.service.discovery.v3.AggregatedDiscoveryService/StreamAggregatedResources\'\n\':authority\', \'ads_cluster\'\n\':scheme\', \'http\'\n\'te\', \'trailers\'\n\'content-type\', \'application/grpc\'\n\'x-envoy-internal\', \'true\'\n\'x-forwarded-for\', \'172.20.0.2\'
espv2_1      | 
espv2_1      | D0619 04:47:07.452 20 envoy] [20][pool][external/envoy/source/common/http/conn_pool_base.cc:346] queueing request due to no available connections
espv2_1      | D0619 04:47:07.452 20 envoy] [20][pool][external/envoy/source/common/http/conn_pool_base.cc:47] creating a new connection
espv2_1      | D0619 04:47:07.452 20 envoy] [20][client][external/envoy/source/common/http/codec_client.cc:35] [C9] connecting
espv2_1      | D0619 04:47:07.453 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:753] [C9] connecting to 127.0.0.1:8790
espv2_1      | D0619 04:47:07.453 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:762] [C9] connection in progress
espv2_1      | D0619 04:47:07.453 20 envoy] [20][http2][external/envoy/source/common/http/http2/codec_impl.cc:1011] [C9] updating connection-level initial window size to 268435456
espv2_1      | D0619 04:47:07.453 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:625] [C9] delayed connection error: 111
espv2_1      | D0619 04:47:07.453 20 envoy] [20][connection][external/envoy/source/common/network/connection_impl.cc:208] [C9] closing socket: 0
espv2_1      | D0619 04:47:07.453 20 envoy] [20][client][external/envoy/source/common/http/codec_client.cc:92] [C9] disconnect. resetting 0 pending requests
espv2_1      | D0619 04:47:07.453 20 envoy] [20][pool][external/envoy/source/common/http/conn_pool_base.cc:265] [C9] client disconnected, failure reason: 
espv2_1      | D0619 04:47:07.453 20 envoy] [20][router][external/envoy/source/common/router/router.cc:1055] [C0][S314116016824065774] upstream reset: reset reason connection failure
espv2_1      | D0619 04:47:07.453 20 envoy] [20][http][external/envoy/source/common/http/async_client_impl.cc:99] async http request response headers (end_stream=true):\n\':status\', \'200\'\n\'content-type\', \'application/grpc\'\n\'grpc-status\', \'14\'\n\'grpc-message\', \'upstream connect error or disconnect/reset before headers. reset reason: connection failure\'
espv2_1      | 
espv2_1      | W0619 04:47:07.453 20 envoy] [20][config][bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamAggregatedResources gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
espv2_1      | D0619 04:47:07.453 20 envoy] [20][config][external/envoy/source/common/config/grpc_subscription_impl.cc:90] gRPC update for type.googleapis.com/envoy.config.listener.v3.Listener failed
espv2_1      | D0619 04:47:07.453 20 envoy] [20][config][external/envoy/source/common/config/grpc_subscription_impl.cc:90] gRPC update for type.googleapis.com/envoy.config.cluster.v3.Cluster failed
espv2_1      | D0619 04:47:11.038 20 envoy] [20][upstream][external/envoy/source/common/upstream/upstream_impl.cc:278] transport socket match, socket default selected for host with address 127.0.0.1:8790
espv2_1      | D0619 04:47:11.038 20 envoy] [20][upstream][external/envoy/source/common/upstream/strict_dns_cluster.cc:167] DNS refresh rate reset for 127.0.0.1, refresh rate 5000 ms
espv2_1      | D0619 04:47:11.038 20 envoy] [20][main][external/envoy/source/server/server.cc:187] flushing stats
espv2_1      | D0619 04:47:16.038 20 envoy] [20][upstream][external/envoy/source/common/upstream/upstream_impl.cc:278] transport socket match, socket default selected for host with address 127.0.0.1:8790
espv2_1      | D0619 04:47:16.039 20 envoy] [20][upstream][external/envoy/source/common/upstream/strict_dns_cluster.cc:167] DNS refresh rate reset for 127.0.0.1, refresh rate 5000 ms
espv2_1      | D0619 04:47:16.039 20 envoy] [20][main][external/envoy/source/server/server.cc:187] flushing stats
espv2_1      | D0619 04:47:21.042 20 envoy] [20][main][external/envoy/source/server/server.cc:187] flushing stats
espv2_1      | D0619 04:47:21.043 20 envoy] [20][upstream][external/envoy/source/common/upstream/upstream_impl.cc:278] transport socket match, socket default selected for host with address 127.0.0.1:8790
espv2_1      | D0619 04:47:21.043 20 envoy] [20][upstream][external/envoy/source/common/upstream/strict_dns_cluster.cc:167] DNS refresh rate reset for 127.0.0.1, refresh rate 5000 ms
espv2_1      | D0619 04:47:26.043 20 envoy] [20][main][external/envoy/source/server/server.cc:187] flushing stats
espv2_1      | D0619 04:47:26.043 20 envoy] [20][upstream][external/envoy/source/common/upstream/upstream_impl.cc:278] transport socket match, socket default selected for host with address 127.0.0.1:8790
espv2_1      | D0619 04:47:26.043 20 envoy] [20][upstream][external/envoy/source/common/upstream/strict_dns_cluster.cc:167] DNS refresh rate reset for 127.0.0.1, refresh rate 5000 ms

ESPv2 not forwarding original Authorization header when an x-google-backend is specified

Noticed this behaviour recently.

I'm using ESPv2 as a gateway to connect multiple cloud run services. As such, I have multiple x-google-backends specified in my API spec.

As per the docs, if ESP needs to overwrite the original Authorization header, it will forward the original one in the X-Forwarded-Authorization header.

However, when I log the headers received by my cloud run services this forwarded header is missing.

Any ideas?

Thanks :)

Latest versions of ESPv2 do not work with manually specified service account key file

Originally posted by @someone1 in #185 (comment)

@nareddyt - should this be working now with 2.14? It works when run on Cloud Run but not locally. It seems to get a lot further, actually downloading the config and listening to requests and the previous snake_case error message no longer appears. However, I still only get a single request before the entire container locks up, and it never reaches my backend:

HTTP Response (Status Code 500):

{"code":500,"message":"INTERNAL:Calling Google Service Control API failed with: 401 and body: \b\u0010\u0012Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project."}

Relevant logs around the request:

D0723 02:03:33.267 53 envoy] [53][http][external/envoy/source/common/http/conn_manager_impl.cc:261] [C3] new stream
D0723 02:03:33.267 53 envoy] [53][http][external/envoy/source/common/http/conn_manager_impl.cc:808] [C3][S14536850401322214170] request headers complete (end_stream=true):\n\':authority\', \'localhost:8089\'\n\':path\', \'/v1alpha/accounts?key=<removed>\'\n\':method\', \'GET\'\n\'connection\', \'keep-alive\'\n\'pragma\', \'no-cache\'\n\'cache-control\', \'no-cache\'\n\'authorization\', \'Bearer <removed>\'

D0723 02:03:33.267 53 envoy] [53][http][external/envoy/source/common/http/conn_manager_impl.cc:1377] [C3][S14536850401322214170] request end stream
D0723 02:03:33.267 53 envoy] [53][jwt][external/envoy/source/extensions/filters/http/jwt_authn/filter.cc:127] Called Filter : setDecoderFilterCallbacks
D0723 02:03:33.267 53 envoy] [53][filter][src/envoy/http/path_matcher/filter.cc:81] matched operation: v1alpha.web_api_dot_project_id_appspot_com.Web_ListAccounts
D0723 02:03:33.267 53 envoy] [53][jwt][external/envoy/source/extensions/filters/http/jwt_authn/filter.cc:49] Called Filter : decodeHeaders
D0723 02:03:33.267 53 envoy] [53][jwt][bazel-out/k8-opt/bin/external/envoy/source/extensions/filters/http/jwt_authn/_virtual_includes/filter_config_interface/extensions/filters/http/jwt_authn/filter_config.h:117] use filter state value v1alpha.web_api_dot_project_id_appspot_com.Web_ListAccounts to find verifier.
D0723 02:03:33.267 53 envoy] [53][jwt][external/envoy/source/extensions/filters/http/jwt_authn/extractor.cc:188] extract authorizationBearer 
D0723 02:03:33.267 53 envoy] [53][jwt][external/envoy/source/extensions/filters/http/jwt_authn/extractor.cc:188] extract x-goog-iap-jwt-assertion
D0723 02:03:33.267 53 envoy] [53][jwt][external/envoy/source/extensions/filters/http/jwt_authn/authenticator.cc:127] auth0_jwk: JWT authentication starts (allow_failed=false), tokens size=1
D0723 02:03:33.267 53 envoy] [53][jwt][external/envoy/source/extensions/filters/http/jwt_authn/authenticator.cc:138] auth0_jwk: startVerify: tokens size 1
D0723 02:03:33.267 53 envoy] [53][jwt][external/envoy/source/extensions/filters/http/jwt_authn/authenticator.cc:149] auth0_jwk: Verifying JWT token of issuer https://<project-id>.auth0.com/
D0723 02:03:33.267 53 envoy] [53][filter][external/envoy/source/extensions/filters/http/common/jwks_fetcher.cc:55] fetch pubkey from [uri = https://<project-id>.auth0.com/.well-known/jwks.json]: start
D0723 02:03:33.267 53 envoy] [53][router][external/envoy/source/common/router/router.cc:426] [C0][S12502690874350086838] cluster \'<project-id>.auth0.com:443\' match for URL \'/.well-known/jwks.json\'
D0723 02:03:33.267 53 envoy] [53][router][external/envoy/source/common/router/router.cc:583] [C0][S12502690874350086838] router decoding headers:\n\':path\', \'/.well-known/jwks.json\'\n\':authority\', \'<project-id>.auth0.com\'\n\':method\', \'GET\'\n\':scheme\', \'https\'\n\'x-envoy-internal\', \'true\'\n\'x-forwarded-for\', \'172.18.0.8\'\n\'x-envoy-expected-rq-timeout-ms\', \'30000\'

D0723 02:03:33.267 53 envoy] [53][pool][external/envoy/source/common/http/conn_pool_base.cc:71] queueing request due to no available connections
D0723 02:03:33.267 53 envoy] [53][pool][external/envoy/source/common/conn_pool/conn_pool_base.cc:53] creating a new connection
D0723 02:03:33.268 53 envoy] [53][client][external/envoy/source/common/http/codec_client.cc:35] [C4] connecting
D0723 02:03:33.268 53 envoy] [53][connection][external/envoy/source/common/network/connection_impl.cc:753] [C4] connecting to 35.165.199.195:443
D0723 02:03:33.268 53 envoy] [53][connection][external/envoy/source/common/network/connection_impl.cc:769] [C4] connection in progress
D0723 02:03:33.268 53 envoy] [53][jwt][external/envoy/source/extensions/filters/http/jwt_authn/filter.cc:76] Called Filter : decodeHeaders Stop
D0723 02:03:33.333 53 envoy] [53][connection][external/envoy/source/common/network/connection_impl.cc:616] [C4] connected
D0723 02:03:33.333 53 envoy] [53][connection][external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:191] [C4] handshake expecting read
D0723 02:03:33.399 53 envoy] [53][connection][external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:191] [C4] handshake expecting read
D0723 02:03:33.399 53 envoy] [53][connection][external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:191] [C4] handshake expecting read
D0723 02:03:33.399 53 envoy] [53][connection][external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:191] [C4] handshake expecting read
D0723 02:03:33.401 53 envoy] [53][connection][external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:191] [C4] handshake expecting read
D0723 02:03:33.401 53 envoy] [53][connection][external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:191] [C4] handshake expecting read
D0723 02:03:33.466 53 envoy] [53][connection][external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:176] [C4] handshake complete
D0723 02:03:33.466 53 envoy] [53][client][external/envoy/source/common/http/codec_client.cc:73] [C4] connected
D0723 02:03:33.466 53 envoy] [53][pool][external/envoy/source/common/conn_pool/conn_pool_base.cc:146] [C4] attaching to next request
D0723 02:03:33.466 53 envoy] [53][pool][external/envoy/source/common/conn_pool/conn_pool_base.cc:73] [C4] creating stream
D0723 02:03:33.466 53 envoy] [53][router][external/envoy/source/common/router/upstream_request.cc:342] [C0][S12502690874350086838] pool ready
D0723 02:03:33.513 53 envoy] [53][conn_handler][external/envoy/source/server/connection_handler_impl.cc:422] [C5] new connection
D0723 02:03:33.545 53 envoy] [53][router][external/envoy/source/common/router/router.cc:1153] [C0][S12502690874350086838] upstream headers complete: end_stream=false
D0723 02:03:33.545 53 envoy] [53][http][external/envoy/source/common/http/async_client_impl.cc:99] async http request response headers (end_stream=false):\n\':status\', \'200\'\n\'date\', \'Thu, 23 Jul 2020 02:03:33 GMT\'\n\'content-type\', \'application/json; charset=utf-8\'\n\'content-length\', \'3160\'\n\'connection\', \'keep-alive\'\n\'server\', \'nginx\'\n\'vary\', \'Accept-Encoding,Accept-Encoding\'\n\'ot-tracer-spanid\', \'7ce79021366dfbbf\'\n\'ot-tracer-traceid\', \'072b59c46a854402\'\n\'ot-tracer-sampled\', \'true\'\n\'ot-baggage-auth0-request-id\', \'8b8f630012aa4e2c4e692a80\'\n\'x-auth0-requestid\', \'85ce479a57d43747ed67\'\n\'access-control-allow-origin\', \'*\'\n\'access-control-allow-credentials\', \'false\'\n\'x-ratelimit-limit\', \'20\'\n\'x-ratelimit-remaining\', \'19\'\n\'x-ratelimit-reset\', \'1595469814\'\n\'cache-control\', \'public, max-age=15, stale-while-revalidate=15, stale-if-error=15\'\n\'etag\', \'W/\"c58-WlyAZ0fSEWSGQDEN8saHRcVzjpw\"\'\n\'x-cache-status\', \'EXPIRED\'\n\'x-envoy-upstream-service-time\', \'277\'

D0723 02:03:33.545 53 envoy] [53][client][external/envoy/source/common/http/codec_client.cc:105] [C4] response complete
D0723 02:03:33.545 53 envoy] [53][filter][external/envoy/source/extensions/filters/http/common/jwks_fetcher.cc:71] onSuccess: fetch pubkey [uri = https://<project-id>.auth0.com/.well-known/jwks.json]: success
D0723 02:03:33.545 53 envoy] [53][filter][external/envoy/source/extensions/filters/http/common/jwks_fetcher.cc:78] onSuccess: fetch pubkey [uri = https://<project-id>.auth0.com/.well-known/jwks.json]: succeeded
D0723 02:03:33.546 53 envoy] [53][jwt][external/envoy/source/extensions/filters/http/jwt_authn/authenticator.cc:267] auth0_jwk: JWT token verification completed with: OK
D0723 02:03:33.546 53 envoy] [53][jwt][external/envoy/source/extensions/filters/http/jwt_authn/filter.cc:87] Called Filter : check complete OK
D0723 02:03:33.546 53 envoy] [53][filter][src/envoy/http/service_control/filter.cc:46] Called ServiceControl Filter : decodeHeaders
D0723 02:03:33.546 53 envoy] [53][filter][src/envoy/http/service_control/service_control_call_impl.cc:182] Sending check : service_name: \"web-api-dot-<project-id>.appspot.com\"\noperation {\n  operation_id: \"a50ce11c-62fc-471c-8183-358ed0284212\"\n  operation_name: \"v1alpha.web_api_dot_project_id_appspot_com.Web_ListAccounts\"\n  consumer_id: \"api_key:<removed>\"\n  start_time {\n    seconds: 1595469813\n    nanos: 546096908\n  }\n  end_time {\n    seconds: 1595469813\n    nanos: 546096908\n  }\n  labels {\n    key: \"servicecontrol.googleapis.com/caller_ip\"\n    value: \"172.18.0.1\"\n  }\n  labels {\n    key: \"servicecontrol.googleapis.com/referer\"\n    value: \"http://localhost:8080/accounts/manage\"\n  }\n  labels {\n    key: \"servicecontrol.googleapis.com/service_agent\"\n    value: \"ESPv2/2.14.0\"\n  }\n  labels {\n    key: \"servicecontrol.googleapis.com/user_agent\"\n    value: \"ESPv2\"\n  }\n}\nservice_config_id: \"2020-07-16r1\"

D0723 02:03:33.546 53 envoy] [53][filter][src/envoy/http/service_control/http_call.cc:322] Service Control remote call: Check is created
D0723 02:03:33.546 53 envoy] [53][filter][src/envoy/http/service_control/http_call.cc:201] http call from [uri = https://servicecontrol.googleapis.com/v1/services/web-api-dot-<project-id>.appspot.com:check]: start
D0723 02:03:33.546 53 envoy] [53][router][external/envoy/source/common/router/router.cc:426] [C0][S312826532313055775] cluster \'service-control-cluster\' match for URL \'/v1/services/web-api-dot-<project-id>.appspot.com:check\'
D0723 02:03:33.546 53 envoy] [53][router][external/envoy/source/common/router/router.cc:583] [C0][S312826532313055775] router decoding headers:\n\':path\', \'/v1/services/web-api-dot-<project-id>.appspot.com:check\'\n\':authority\', \'servicecontrol.googleapis.com\'\n\':method\', \'POST\'\n\':scheme\', \'https\'\n\'content-length\', \'505\'\n\'authorization\', \'Bearer <removed>\'

D0723 02:03:33.546 53 envoy] [53][pool][external/envoy/source/common/http/conn_pool_base.cc:71] queueing request due to no available connections
D0723 02:03:33.546 53 envoy] [53][pool][external/envoy/source/common/conn_pool/conn_pool_base.cc:53] creating a new connection
D0723 02:03:33.546 53 envoy] [53][client][external/envoy/source/common/http/codec_client.cc:35] [C6] connecting
D0723 02:03:33.546 53 envoy] [53][connection][external/envoy/source/common/network/connection_impl.cc:753] [C6] connecting to 172.217.10.234:443
D0723 02:03:33.546 53 envoy] [53][connection][external/envoy/source/common/network/connection_impl.cc:769] [C6] connection in progress
D0723 02:03:33.546 53 envoy] [53][filter][src/envoy/http/service_control/filter.cc:64] Called ServiceControl filter : Stop
D0723 02:03:33.546 53 envoy] [53][pool][external/envoy/source/common/http/http1/conn_pool.cc:49] [C4] response complete
D0723 02:03:33.546 53 envoy] [53][pool][external/envoy/source/common/conn_pool/conn_pool_base.cc:98] [C4] destroying stream: 0 remaining
D0723 02:03:33.564 53 envoy] [53][connection][external/envoy/source/common/network/connection_impl.cc:616] [C6] connected
D0723 02:03:33.564 53 envoy] [53][connection][external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:191] [C6] handshake expecting read
D0723 02:03:33.589 53 envoy] [53][connection][external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:191] [C6] handshake expecting read
D0723 02:03:33.589 53 envoy] [53][connection][external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:191] [C6] handshake expecting read
D0723 02:03:33.589 53 envoy] [53][connection][external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:191] [C6] handshake expecting read
D0723 02:03:33.600 53 envoy] [53][connection][external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:176] [C6] handshake complete
D0723 02:03:33.600 53 envoy] [53][client][external/envoy/source/common/http/codec_client.cc:73] [C6] connected
D0723 02:03:33.600 53 envoy] [53][pool][external/envoy/source/common/conn_pool/conn_pool_base.cc:146] [C6] attaching to next request
D0723 02:03:33.600 53 envoy] [53][pool][external/envoy/source/common/conn_pool/conn_pool_base.cc:73] [C6] creating stream
D0723 02:03:33.600 53 envoy] [53][router][external/envoy/source/common/router/upstream_request.cc:342] [C0][S312826532313055775] pool ready
D0723 02:03:33.629 53 envoy] [53][router][external/envoy/source/common/router/router.cc:1153] [C0][S312826532313055775] upstream headers complete: end_stream=false
D0723 02:03:33.629 53 envoy] [53][http][external/envoy/source/common/http/async_client_impl.cc:99] async http request response headers (end_stream=false):\n\':status\', \'401\'\n\'www-authenticate\', \'Bearer <removed>\'

D0723 02:03:33.629 53 envoy] [53][client][external/envoy/source/common/http/codec_client.cc:105] [C6] response complete
D0723 02:03:33.630 53 envoy] [53][filter][src/envoy/http/service_control/http_call.cc:101] http call response status code: 401, body: \010\020\022\320\001Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
E0723 02:03:33.630 53 envoy] [53][filter][src/envoy/http/service_control/client_cache.cc:144] Failed to call check, error: UNAUTHENTICATED:Calling Google Service Control API failed with: 401 and body: \010\020\022\320\001Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project., str body: \010\020\022\320\001Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.
E0723 02:03:43.942451       6 rollout_id_change_detector.go:65] error occurred when checking new rollout id, fail to fetch new rollout id, http call to POST https://servicecontrol.googleapis.com/v1/services/web-api-dot-<project-id>.appspot.com:report returns not 200 OK: 403 Forbidden

I can open a new issue if this is unrelated - and I'd be happy to share my entire log privately if it helps any. I'm using the same ESPv2 startup options as mentioned earlier in this issue. It continues to work fine with ESPv2.7.0.

I should mention, the 401/403 error messages may be symptomatic of an error earlier in the stack. The following works fine with the same service account file so I'm pretty sure it has permissions:

curl -s \
    -H "Authorization: Bearer $(gcloud auth application-default print-access-token)" \
    -H "Content-Type: application/json" \
    https://servicecontrol.googleapis.com/v1/services/web-api-dot-<project-id>.appspot.com:check \
    --data '{
        "serviceConfigId": "2020-07-16r1", 
        operation: {
            "operationId": "<removed>",
            "consumerId":  "api_key:<removed>",
            "operationName": "<removed>",
            "startTime": "2020-07-22T22:01:23.045123456Z",
            "endTime": "2020-07-22T22:01:23.045123456Z",
        }
    }'

503 error when deploying to App Engine

Hi

I'm trying to deploy esp in the following configuration:

ESPv2 - gcr.io/endpoints-release/endpoints-runtime-serverless:2.10.0 deployed to App Engine flex with env:

ESPv2_ARGS  = "^++^--cors_preset=cors_with_regex++--cors_allow_origin_regex=...++--enable_debug"
ENDPOINTS_SERVICE_NAME = "<service>-dot-<project>.appspot.com"

Endpoints config: Copy of sample with these changes:

-host: "YOUR-PROJECT-ID.appspot.com"
+host: "<service>-dot-<project>.appspot.com"

-    x-google-audiences: "YOUR-CLIENT-ID"
+    x-google-audiences: "<actual IAP client id>"

+  auth0_jwk:
+    authorizationUrl: ""
+    flow: "implicit"
+    type: "oauth2"
+    x-google-issuer: "https://<tenant>.auth0.com/"
+    x-google-jwks_uri: "https://<tenant>.auth0.com/.well-known/jwks.json"
+    x-google-audiences: "https://<esp service>-dot-<project>.appspot.com/"
+
+x-google-backend:
+  address: "https://<service>-dot-<project>.appspot.com"
+  jwt_audience: "<IAP oauth client id>"
+  protocol: h2
+
+
+x-google-endpoints:
+  - name: <service>-dot-<project>.appspot.com

When I hit the ESP service with appropriate JWT token, it fails with a 503 error:

HTTP/2 503 
date: Fri, 29 May 2020 23:15:26 GMT
content-type: text/plain
content-length: 91
server: envoy
via: 1.1 google
alt-svc: h3-27=":443"; ma=2592000,h3-25=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"

upstream connect error or disconnect/reset before headers. reset reason: connection failure

In ESP logs, I see:

D0529 22:30:59.101 27 envoy] [27][connection][external/envoy/source/common/network/connection_impl.cc:504] [C21] raising immediate error
...
D0529 22:30:59.101 27 envoy] [27][router][external/envoy/source/common/router/router.cc:1018] [C19][S9973334829757176186] upstream reset: reset reason connection failure
...
D0529 22:30:59.101 27 envoy] [27][http][external/envoy/source/common/http/conn_manager_impl.cc:1475] [C19][S9973334829757176186] Sending local reply with details upstream_reset_before_response_started{connection failure}

In Produced API logs, I see:

jsonPayload: {
  api_method: "1.<service>_dot_<project>_appspot_com.Echo"   
  api_name: "1.<service>_dot_<project>_appspot_com"   
  api_version: "1.0.0"   
  client_ip: "172.17.0.5"   
  http_method: "POST"   
  http_response_code: 503   
  location: "us-central1-b"   
  log_message: "1.<service>_dot_<project>_appspot_com.Echo is called"   
  producer_project_id: "<project>"   
  request_latency_in_ms: 246   
  request_size_in_bytes: 2072   
  response_size_in_bytes: 183   
  service_agent: "ESPv2/2.10.0"   
  service_config_id: "2020-05-29r2"   
  timestamp: 1590794126.0151885   
  url: "/echo"   
 }

I'm not sure what's going on here or if this is an unsupported pattern of deployment, and ESP v2 cannot work in App Engine and has to be Cloud Run.

Support for `requires_any` and `allow_missing` in auth filter

Is there a way to mimic this configuration for jwt? More specifically I want to allow a call with JWT or not.
Is requires_any and allow_missing supported?

- name: envoy.filters.http.jwt_authn
  typed_config:
    "@type": type.googleapis.com/envoy.config.filter.http.jwt_authn.v2alpha.JwtAuthentication
    providers:
      firebase:
        issuer: https://securetoken.google.com/comic-run-app
        audiences:
          - "comic-run-app"
        local_jwks:
          inline_string: '{"5e9ee97c840f97e0253688a3b7e94473e528a7b5":"-----BEGIN CERTIFICATE-----\nMIIDHDCCAgSgAwIBAgIIZC7uPq+FdkIwDQYJKoZIhvcNAQEFBQAwMTEvMC0GA1UE\nAxMmc2VjdXJldG9rZW4uc3lzdGVtLmdzZXJ2aWNlYWNjb3VudC5jb20wHhcNMjAw\nNDE5MDkxOTU2WhcNMjAwNTA1MjEzNDU2WjAxMS8wLQYDVQQDEyZzZWN1cmV0b2tl\nbi5zeXN0ZW0uZ3NlcnZpY2VhY2NvdW50LmNvbTCCASIwDQYJKoZIhvcNAQEBBQAD\nggEPADCCAQoCggEBAN4QmiJxlOTvbw/wwJPYISofcxRXb/I4+K8noG7fFhim+RxU\n/f84uYyDssopXt6jiUGeBKMvm65fi108EfGZXCYPZVBX1dgkddRkgNA2afhvrgdF\n7BG9U1e1SPlcovJH4upn5bQb0kOr7yTg6LfihA30kgZ3RyekrSx4VJP+UNb38f+J\ntOhiROEwUOS/0J35+8jZtO5FqVfp/hxfGLMdsi+l7kA65ogW/4uQCaD8V54Ncf6D\nn0qX3qW3ze2kO1W4NDmOhLhpef8nBORs1Mt7dvKxK3QNMJQwtqqO1wQC4oKzvhNY\neNsfw+nFiDtAyUFwZOVbYP2Vsp3p3oYu1impJu0CAwEAAaM4MDYwDAYDVR0TAQH/\nBAIwADAOBgNVHQ8BAf8EBAMCB4AwFgYDVR0lAQH/BAwwCgYIKwYBBQUHAwIwDQYJ\nKoZIhvcNAQEFBQADggEBAE7NhWSHuaikAo6EVt7Q4F+eOCAywQbGJgVE/xdK8AA+\nmEbU6Ybi8wc4yCE2QW2NYMH822MPiuiTKgBRPdOPQa1YrOfmvLQt0ZHGADl/d2U8\ny0wPZJWjCWHRkufPOMr2EFhwlA5Pj1mBTKn9PZAQf9rWiJuYhkb4jm6hsABmL0HY\ngGZkLcXsnzIWfNj0IXU7YRbY7ko5NoqfXe3aOoNIysgF59wInPUPnYyKZjrRS0Jj\nU93x28x7EC2clk59tRylyoJnRDbs4WbLMpzS8Fq6APa0ukHRV9cimjqmQbEJ5v6y\nTBtNG1gPXsUJQldCKMYWmXh0hAdq+ZjcRVZnOngS35Q=\n-----END CERTIFICATE-----\n","0babb22440da032c5d0042cdaa9d285cf8d202c4":"-----BEGIN CERTIFICATE-----\nMIIDHDCCAgSgAwIBAgIIN3zFmJoUjoYwDQYJKoZIhvcNAQEFBQAwMTEvMC0GA1UE\nAxMmc2VjdXJldG9rZW4uc3lzdGVtLmdzZXJ2aWNlYWNjb3VudC5jb20wHhcNMjAw\nNDExMDkxOTU2WhcNMjAwNDI3MjEzNDU2WjAxMS8wLQYDVQQDEyZzZWN1cmV0b2tl\nbi5zeXN0ZW0uZ3NlcnZpY2VhY2NvdW50LmNvbTCCASIwDQYJKoZIhvcNAQEBBQAD\nggEPADCCAQoCggEBAKjiZSt/6eRBHhSAWZBzfnG3oCx5DhPKqYLCOVONGkWRbJ1K\nRRbww77SH4PbRRP7kz/1jADgg9D0klW7QlD04ZXm/sUDPYwguTta5IRnOao+vyJp\nvk/XPCX2deQn+whblrhQJ5eAIos6Fd4KUNnhMQjWgzXp0k4Xnx5vFVNexp4GP1Bv\nvPppQQB+V/SBsqeL/oDpjmJiw5NSwFvexs54x9+K9vHVMrdjYSHBMEVNzVcZSHzn\n6y1a0Q3I52FUwAaqydcV+fzCW+gP/LlM9NNpio9GhotmjDZw+4BsgSU+8xyfcyR/\nowjcyG1fbgu9uQL95BaeQoPsfMRfdSXMsLZoOOcCAwEAAaM4MDYwDAYDVR0TAQH/\nBAIwADAOBgNVHQ8BAf8EBAMCB4AwFgYDVR0lAQH/BAwwCgYIKwYBBQUHAwIwDQYJ\nKoZIhvcNAQEFBQADggEBAGzj/Cu2T7fHT9wvJorxV5vy0BJ+w0/H4jQET5URXcva\nK5DnPO46fbRI+ri8OhYn4X4/tOMF+HumOp+bo/me6eJYTfTIBiIrX3yoTcag/tkz\nTMKeFW7kcyptHTWOp+qzz2hezR44+JwmE4WN3yV5hoQRXpar9Hzln+rcuUi3hlRZ\nhQk2qgcBkQ/EwGYqPh7JFYriwbEOpHNJFJUMiBPVYO9MEdI0Yrs875cAF6FI8iFj\nNA1A2Mvn7rQvRtPpRQY4/NDABDvkcjPD+imyG/RKXjY+XXRjYY/Jw8YnMxcsFh81\n1N5z2VeA+n03vLqll5vgecVfjn2d+GpN4fONUQtFLN4=\n-----END CERTIFICATE-----\n"}'
        forward_payload_header: "x-endpoint-api-userinfo"
        payload_in_metadata: "jwt_payloads"
    rules:
      - match:
          prefix: "/comicrun.v1.ComicMetadata/CreateKeyInfo"
        requires:
          provider_name: firebase
      - match:
          prefix: "/comicrun.v1.ComicMetadata/UpdateKeyInfo"
        requires:
          provider_name: firebase
      - match:
          prefix: "/comicrun.v1.ComicMetadata/ListKeyInfo"
        requires:
          requires_any:
            requirements:
              - provider_name: firebase
              - allow_missing: {}
      - match:
          prefix: /

Windows-compatible gcloud_build_image script

When following this tutorial, it currently requires the rebuild of the ESPv2 image. What surprised me is the note that said gcloud_build_image doesn't support Windows. Fortunately, a workaround is given (use Cloud Shell), but that didn't work for me ("Permission Denied" error when executing the uploaded script).

It would be great if you could port this script to PowerShell, or make a batch file, so Windows users can complete the tutorial steps locally.

And maybe, if the docs team lurks here, add extra instructions explaining how to correctly execute the script in Cloud Shell for those who have never used it before (like me😬).

Issues trying to use ESPv2 with gRPC-web with CORS

I've recently been trying to ping my backend from the browser using the client stub generated from the following commands:

Generate client stubs

protoc -I=. \
    --proto_path=. \
    --js_out=import_style=commonjs:. example.proto 

protoc -I=. \
    --grpc-web_out=import_style=commonjs,mode=grpcwebtext:. example.proto

Client code

import { Injectable } from '@angular/core';

import { ApplicationProcessorClient } from './stub/pluto-website_grpc_web_pb';
import { DetailsRequest, DetailsResponse } from './stub/pluto-website_pb';

@Injectable({
  providedIn: 'root'
})
export class ApplicationClientService {

  constructor() {
    this.ApplicationService = new ApplicationProcessorClient('https://ESPv2_container-abcdefg-an.a.run.app');
  }

public send_details() {
let request = new DetailsRequest()
    request.setTitle("Mr");
    request.setFirstName("firstName");
    request.setLastName("lastName");
   
    const metadata = {};
    this.ApplicationService.confirmDetails(request, metadata, function(err, response) {
      console.log(response);
      console.log(err);
    })
}


private ApplicationService;
}

I've tested my server code with a separate python script and it works as expected.

I've deployed my ESPv2 container with the following commands (in order)

Update the endpoints definition

gcloud endpoints services \
    deploy api_descriptor.pb \
    api_config.yaml

Build the image

./gcloud_build_image -s ESPv2_container-abcdefg-an.a.run.app \
        -c 2020-xx-xxr0 \
        -p gcloud-project-id

Deploy the image

gcloud run deploy espv2-container \
    --image="gcr.io/gcloud-project-id/endpoints-runtime-serverless:ESPv2_container-abcdefg-an.a.run.app-2020-xx-xxr0"\
    --set-env-vars=ESPv2_ARGS=--cors_preset=basic,--cors_allow_headers="User-Agent"\
    --allow-unauthenticated --platform managed\
    --project gcloud-project-id

When I call the client code from the browser, I get the following error

Access to XMLHttpRequest at 'https://ESPv2_container-abcdefg-an.a.run.app/endpoints.application.ApplicationProcessor/ConfirmDetails' from origin 'http://localhost:4200' has been blocked by CORS policy: Request header field x-user-agent is not allowed by Access-Control-Allow-Headers in preflight response.

I'm not quite sure why this is the case. The google docs specify that the --cors=basic config should include User-Agent. I tried adding the flag manually to the response header with the --cors_allow_headers="User-Agent' option, doesn't make a difference.

I'm really stumped as to what's gone wrong, I have a basic understanding of CORS, so if its something dumb can someone please point it out for me?

Kind regards,

Despicable-b

Support open API v3

It would be great if support for Open API v3 was there. (Polymorphism is a lot better there for instance.)

Are there any plans to support this soon?

gcloud_build_image bug "line 73: TAGS_BY_LENGTH: bad array subscript"

Following the Building a new ESPv2 Beta image instructions I got the following error (omitted some exact details for privacy):

chmod +x gcloud_build_image.sh
./gcloud_build_image.sh -s <CLOUD_RUN_HOSTNAME> -c <CONFIG_ID> -p <ESP_PROJECT_ID>
Using base image: gcr.io/endpoints-release/endpoints-runtime-serverless:2
Determining fully-qualified ESP version for tag: 2
./gcloud_build_image.sh: line 73: TAGS_BY_LENGTH: bad array subscript

I was able to use the -i flag, but I'm not exactly clear on the side effects of using a "custom" ESP version, so ideally I could use the defaults.

./gcloud_build_image.sh  -s <CLOUD_RUN_HOSTNAME> -c <CONFIG_ID> -p <ESP_PROJECT_ID> -i gcr.io/endpoints-release/endpoints-runtime-serverless:2
Using base image: gcr.io/endpoints-release/endpoints-runtime-serverless:2
Building image for ESP version: custom
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 22454    0 22454    0     0  13886      0 --:--:--  0:00:01 --:--:-- 13886
[...]

Using custom domain mapping as backend address in api config results in "invalid token" from backend service.

Summary

Using custom domain mapping as backend address in api config results in "invalid token" error from backend service. Workaround it to use the domain generated by Cloud Run as backend address to make it work.

Background

We are just getting started with Cloud Run and we wished to use our own .run domain for all our Cloud Run services. Using our custom domain makes deployment easier, because we have already planned our domain names in advance and it also simplifies service discovery since the domain names are static and predetermined.

We deployed our gRPC service, as per instructions here: https://cloud.google.com/endpoints/docs/grpc/get-started-cloud-run, but we used our custom domains instead of the auto-generated.
After deployment we got 401 from our backend service:
The request was not authorized to invoke this service. Read more at https://cloud.google.com/run/docs/securing/authenticating.

After some debugging we realized that the esp service was not authorized to call
the backend service because the backend address was set to our
custom domain, see api_config.yaml. When we enabled debugging for the espv2
container the error, "invalid token", from the backend service was visible in
the logs, which are included below.

Workaround

Use the automatically assigned domain as backend address

backend:
  rules:
    - selector: "*"
      address: {{cloud run assigned domain}}

Config

The esp container used was:
gcr.io/endpoints-release/endpoints-runtime-serverless:2.5.0

api_config.yaml

type: google.api.Service
config_version: 3

name: {our domain}.dev

title: Service API
apis:
- name: service.Service

usage:
  rules:
  - selector: "service.path"
    allow_unregistered_calls: true
backend:
  rules:
    - selector: "*"
      address: grpcs://{our domain}.run

Error message from esp

Error message that the esp container got from the backend service:

 "D0221 11:58:51.089 28 envoy] [28][http][external/envoy/source/common/http/conn_manager_impl.cc:1569] [C5][S5014636426089945857] encoding headers via codec (end_stream=false):
':status', '401'
'www-authenticate', 'Bearer error=\"invalid_token\" error_description=\"The access token could not be verified\"'
'date', 'Fri, 21 Feb 2020 11:58:50 GMT'
'content-type', 'text/html; charset=UTF-8'
'server', 'envoy'
'content-length', '337'
'x-envoy-upstream-service-time', '45'"

How to reproduce

Follow the instructions in https://cloud.google.com/endpoints/docs/grpc/get-started-cloud-run to setup the service. Use custom domain instead of the auto generated ones.
Enable debugging for the esp container.
The logs will contain the error message described above.

ESPv2 Beta does NOT pass the CORS request to my backend Cloud Run

I have a Cloud Run backend service and I'm integrating to ESPv2 Beta following the official doc. It looks like CORS is not passed to the backend even if I have this configuration:

x-google-endpoints:
- name: <my-endpoint-service>.run.app
  allowCors: True

When I call it with my reactJS web app, I have the following error:

[..] has been blocked by CORS policy: Request header field x-api-key is not allowed by Access-Control-Allow-Headers in preflight response

because I'm using api key in the header x-api-key and OPTIONS response has only these headers allowed:

DNT,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Range,Authorization

I've already tried using the flags --cors_preset=basic++--cors_allow_headers but it looks ignored and I continue having the same issue.

(If I call directly my backend, it works so it looks an issue with ESP)

How to automate `gcloud_build_image` now that the output image is prefixed with `ESP_VERSION`?

Originally posted by @vorobievalex in #229 (comment)

@nareddyt, what is a possible way for a script, that uses gcloud_build_image, to refer to the container later in 'gcloud run deploy'?
(A script, that performs steps described in https://cloud.google.com/endpoints/docs/openapi/get-started-cloud-functions)
I am fixing such a script, that was previously able to calculate the name as "gcr.io/ESP_PROJECT_ID/endpoints-runtime-serverless:CLOUD_RUN_HOSTNAME-CONFIG_ID" from the variables known from the appropriate gcloud commands output.
Adding "ESP_VERSION-" to the name adds a variable, that is calculated inside gcloud_build_image and never exposed outside.
I can yet only think of temporary option of patching gcloud_build_image to expose the variable or repeating ESP_VERSION calculation outside of the script...

The requested URL returned error 404. Failed to download service config

After upload service configuration, I tried to build ESP2 with command:

./gcloud_build_image -s my-cloud-run-host-name -c 2020-09-25r2 -p my-project-id

Then it return error:

Using base image: gcr.io/endpoints-release/endpoints-runtime-serverless:2
Determining fully-qualified ESP version for tag: 2
Building image for ESP version: 2.18.0
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
curl: (22) The requested URL returned error: 404
./gcloud_build_image: line 84: Failed to download service config (exit 1)

Retrying for backend requests

Hello,

Currently the dynamic Envoy configuration for backend requests generated by ESPv2's Go code doesn't contain any retry policy. This is a problem because now transient 5xx errors get directly forwarded to end-user as 500s.

The generated route configuration for our backend is now something like this, according to ESPv2 debug logs:

"route": {
    "cluster": "europe-west1-<our-project>.cloudfunctions.net:443",
    "host_rewrite": "europe-west1-<our-project>.cloudfunctions.net",
    "timeout": "900s"
}

Could we possibly have a configuration for ESPv2 so that we would end up with a configuration like this for every routes:

"route": {
    "cluster": "europe-west1-<our-project>.cloudfunctions.net:443",
    "host_rewrite": "europe-west1-<our-project>.cloudfunctions.net",
    "timeout": "900s",
    "retry_policy": {
        "retry_on": "5xx",
        "num_retries": 2
    }
}

This would hopefully be relatively small addition but bring significant value for many. For example, using a Cloud Function backend is really cool and scalable and fulfills many needs but due to occasional transient 500s that we cannot handle in any way we cannot provide a reliable service. But of course this would be great for all other kinds of backends as well!

At first I started to hack route_generator.go (without any Go skills) but then I realized I have to build the serverless ESPv2 locally to use it in Cloud Run. I tried to find some sensible Dockerfile or build command but didn't make it.

But I think this would be useful for other users as well!

Thank you!

Edit: Envoy's retry_policy documentation: https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/route/v3/route_components.proto#envoy-v3-api-msg-config-route-v3-retrypolicy

Support AppEngine Flex in custom image mode based on ESPv2 image

Hi

I'm trying to deploy esp in the following configuration:

ESPv2 - gcr.io/endpoints-release/endpoints-runtime-serverless:2.10.0 deployed to App Engine flex with env:

GOOGLE_APPLICATION_CREDENTIALS = "<service account JSON path>"
ENDPOINTS_SERVICE_NAME = "<service>-dot-<project>.appspot.com"

Endpoints config: Copy of sample with these changes:

-host: "YOUR-PROJECT-ID.appspot.com"
+host: "<service>-dot-<project>.appspot.com"

-    x-google-audiences: "YOUR-CLIENT-ID"
+    x-google-audiences: "<actual IAP client id>"

+  auth0_jwk:
+    authorizationUrl: ""
+    flow: "implicit"
+    type: "oauth2"
+    x-google-issuer: "https://<tenant>.auth0.com/"
+    x-google-jwks_uri: "https://<tenant>.auth0.com/.well-known/jwks.json"
+    x-google-audiences: "https://<esp service>-dot-<project>.appspot.com/"
+
+x-google-backend:
+  address: "https://<service>-dot-<project>.appspot.com"
+  jwt_audience: "<IAP oauth client id>"
+  protocol: h2
+
+
+x-google-endpoints:
+  - name: <service>-dot-<project>.appspot.com

The app is fronted by IAP to restrict access only to the ESP user. However, the requests fail with this error:

HTTP/2 401 
date: Sat, 30 May 2020 22:38:06 GMT
content-type: text/html; charset=UTF-8
content-length: 57
x-goog-iap-generated-response: true
x-envoy-upstream-service-time: 27
server: envoy
via: 1.1 google
alt-svc: h3-27=":443"; ma=2592000,h3-25=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q049=":443"; ma=2592000,h3-Q048=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"

Invalid IAP credentials: JWT 'email' claim isn't a string

Upon debugging, I found the token that ESP sends its requests with looks like this:

{
  "aud": "<IAP oauth client id>",
  "azp": "103551354333791234484",
  "exp": 1590879029,
  "iat": 1590875429,
  "iss": "https://accounts.google.com",
  "sub": "103551354333791234484"
}

As seen here, sub isn't the email of the service account I specified in ESP configuration.

Please advise.

Cannot disable debug logging

Steps to reproduce:

  1. Enable debug logging with this flag: --set-env-vars=ESPv2_ARGS=--enable_debug
  2. Deploy, enjoy debug logs
  3. Fix bugs
  4. Try to disable debug logging by reverting the commit that enabled the debug logging, i.e. remove this line: --set-env-vars=ESPv2_ARGS=--enable_debug, deploy again
  5. Does not work, the debug flag is still in the yaml:
        env:
        - name: ESPv2_ARGS
          value: --enable_debug

Apparently by not passing any --set-env-vars=ESPv2_ARGS args it interprets that OK, no changes to these args, no need to change the yaml in that section

I tried also this: --set-env-vars=ESPv2_ARGS= (empty ESPv2_ARGS) but then it says: Cloud Run error: Container failed to start. Failed to start and then listen on the port defined by the PORT environment variable. Logs for this revision might contain more information.

I'm not sure if root bug is in Cloud Run or GKE or ESPv2. But I think the --set-env-vars=ESPv2_ARGS= shouldn't complain about PORT if this didn't either: --set-env-vars=ESPv2_ARGS=--enable_debug

Change --service_control_network_fail_open flag

This flag default is true already, by specifying it, it is also true. So it is confusing.

Please see this cloudendpoints/esp#816 for detail

Suggested changes: keep this flag for backward compatible, but mark it as deprecated
add a new flag
--service_control_network_fail_policy=[open|close] default is open.

Added Latency

I have ESPv2 deployed as a Cloud Run service, and the backend is another Cloud Run service hosting my simple API.
Both are in us-central1.
Doing a quick TTFB test on my API through the ESPv2 Cloud Run URL yields an average of 68ms.
Doing the same test on the API directly (essentially bypassing ESPv2) yields an average of 56ms.

Is this amount of latency normal?
I recognize this project is still in beta - is there ongoing work being done to reduce latency before the GA release?

If you need more details from me, let me know. I'd be interested in seeing your results as well.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.