Giter Club home page Giter Club logo

Comments (3)

lbenguigui avatar lbenguigui commented on July 2, 2024

Hello @caiuskong ,

Thanks for your interest in Traefik.
It appears that your profiles are related to 2 different setups. To conduct a more accurate comparison, I recommend generating pprof profiles on the same setup (e.g., with consulcatalog enabled) at different intervals (e.g., 4 hours and 72 hours).
After analyzing your profiles it seems that the issue may be related to plugins.
To be able to dig further into this issue, can you send us a minimal reproducible setup?

from traefik.

caiuskong avatar caiuskong commented on July 2, 2024

Thanks for your(@lbenguigui) reply.

I tried setting up the environment locally and testing, unfortunately I couldn't reproduce the memory leak.
In the local environment, I either file provider or consul provider for API stress testing (router using middlewares), and the memory changes were similar.

I enabled both consul/consulcatalog provider and did the following tests on the stag environment, potential memory leaks have occurred.
By analyzing the pprof, I suspect a goroutine leak and it is caused by the plugin (yaegi)?

Testing Data

time action pprof
5m do nothing, only start traefik goroutine-profile.pb.gz , heap-profile.pb.gz
25m API stress testing (106540 requests in 5.00m). Router from consul provider. goroutine-profile.pb.gz, heap-profile.pb.gz
45m API stress testing (288594 requests in 5.00m). Router from consulcatalog. provider goroutine-profile.pb.gz, heap-profile.pb.gz
22h Regular requests, about 20,000+ per hour goroutine-profile.pb.gz, heap-profile.pb.gz

Traefik Static Configration

global:
  checkNewVersion: false

serversTransport:
  maxIdleConnsPerHost: 1000
  forwardingTimeouts:
    dialTimeout: 60s
    responseHeaderTimeout: 60s
    idleConnTimeout: 90s

entryPoints:
  web:
    address: :80
    transport:
      respondingTimeouts:
        idleTimeout: 120s
      lifeCycle:
        graceTimeOut: 30s
    http2:
      maxConcurrentStreams: 1000
    forwardedHeaders:
      insecure: true
    proxyProtocol:
      insecure: true
  # websecure:
  #   address: ":443"

api:
  dashboard: true
  insecure: true
  debug: true

ping:
  entryPoint: traefik

log:
  level: DEBUG
  # filePath: /var/log/traefik.log
  format: json

accessLog:
  # filePath: /var/log/access.log
  bufferingSize: 100
  format: json
  filters:
    minDuration: "3ms"
  fields:
    defaultMode: keep
    headers:
      defaultMode: keep
      names:
        Lares-Authorization: redact
        Authorization: redact
        token: redact

metrics:
  prometheus:
    entryPoint: web

tracing:
  serviceName: traefik
  spanNameLimit: 150

  jaeger:
    localAgentHostPort: ""
    collector:
      endpoint: https://jaeger-collector-stg.***.com/api/traces

providers:
  file:
    directory: "/etc/traefik/config"
    watch: true
  consul:
    endpoints:
      - "consul-stag.***.com"
    token: "${CONSUL_HTTP_TOKEN}"
  consulCatalog:
    refreshInterval: 30s
    watch: false
    endpoint:
      address: consul-stag.***.com
      scheme: "https"
      datacenter: aws1
      token: "${CONSUL_HTTP_TOKEN}"
    exposedByDefault: false
    defaultRule: PathPrefix(`/stag/{{ normalize .Name }}`)
    connectAware: true
    connectByDefault: true

experimental:
  localPlugins:
    jwt-auth:
      moduleName: github.com/traefik/jwt-auth
    validate-request-schema:
      moduleName: github.com/traefik/validate-request-schema

Traefik Dynamic Configuration Raw Data (Part)

{
    "routers": {
        "caius-ex-service@consul": {
            "entryPoints": [
                "web"
            ],
            "middlewares": [
                "secured-chain@consul"
            ],
            "service": "caius-ex-service",
            "rule": "PathPrefix(`/stag/caius-ex-service`)",
            "status": "enabled",
            "using": [
                "web"
            ]
        },
        "caius-in-service@consulcatalog": {
            "entryPoints": [
                "web"
            ],
            "middlewares": [
                "secured-chain@consul"
            ],
            "service": "caius-in-service",
            "rule": "PathPrefix(`/stag/caius-in-service`)",
            "status": "enabled",
            "using": [
                "web"
            ]
        },
    },
    "middlewares": {
        "auth@consul": {
            "plugin": {
                "jwt-auth": {
                    "redisUrl": "***",
                    "secret": "***"
                }
            },
            "status": "enabled"
        },
        "validateRequestSchema@consul": {
            "plugin": {
                "validate-request-schema": {
                    "extension": {
                        "name": "x-amazon-apigateway-integration",
                        "replacePathServices": "***"
                    },
                    "redisUrl": "***"
                }
            },
            "status": "enabled"
        },
        "replacepathregex-gw-prefix@consul": {
            "replacePathRegex": {
                "regex": "^/(stag|prod)/[^/]+/?(.*?)/?$",
                "replacement": "/${2}"
            },
            "status": "enabled"
        },
        "secured-chain@consul": {
            "chain": {
                "middlewares": [
                    "auth@consul",
                    "validateRequestSchema@consul",
                    "replacepathregex-gw-prefix@consul"
                ]
            },
            "status": "enabled",
            "usedBy": [
                "caius-ex-service@consul",
            ]
        },
    },
    "services": {
        "caius-ex-service@consul": {
            "loadBalancer": {
                "servers": [
                    {
                        "url": "***"
                    }
                ],
                "passHostHeader": false
            },
            "status": "enabled",
            "usedBy": [
                "caius-ex-service@consul"
            ],
            "serverStatus": {
                "***": "UP"
            }
        },
        "caius-in-service@consulcatalog": {
            "loadBalancer": {
                "servers": [
                    {
                        "url": "***"
                    }
                ],
                "passHostHeader": true,
                "serversTransport": "tls-default-aws1-caius-in-service@consulcatalog"
            },
            "status": "enabled",
            "usedBy": [
                "caius-in-service@consulcatalog"
            ],
            "serverStatus": {
                "***": "UP"
            }
        },
    }
}

from traefik.

caiuskong avatar caiuskong commented on July 2, 2024

[4d18h] - goroutine pprof

      File: traefik
Type: goroutine
Time: Feb 26, 2024 at 10:48am (CST)
Showing nodes accounting for 21546, 100% of 21546 total
----------------------------------------------------------+-------------
      flat  flat%   sum%        cum   cum%   calls calls% + context 	 	 
----------------------------------------------------------+-------------
                                             21531 99.94% |   runtime.selectgo runtime/select.go:327
                                                 8 0.037% |   runtime.netpollblock runtime/netpoll.go:564
                                                 4 0.019% |   runtime.chanrecv runtime/chan.go:583
     21543   100%   100%      21543   100%                | runtime.gopark runtime/proc.go:398
----------------------------------------------------------+-------------
...
...
----------------------------------------------------------+-------------
                                             21492 99.82% |   reflect.rselect runtime/select.go:589
                                                 6 0.028% |   io.(*pipe).read io/pipe.go:57
                                                 5 0.023% |   github.com/patrickmn/go-cache.(*janitor).Run github.com/patrickmn/[email protected]+incompatible/cache.go:1079
                                                 4 0.019% |   github.com/traefik/traefik/v2/pkg/server.(*httpForwarder).Accept github.com/traefik/traefik/v2/pkg/server/server_entrypoint_tcp.go:58
                                                 3 0.014% |   github.com/traefik/traefik/v2/pkg/provider/aggregator.(*RingChannel).ringBuffer github.com/traefik/traefik/v2/pkg/provider/aggregator/ring_channel.go:48
                                                 3 0.014% |   github.com/traefik/traefik/v2/pkg/provider/aggregator.maybeThrottledProvide.func1.1 github.com/traefik/traefik/v2/pkg/provider/aggregator/aggregator.go:47
                                                 3 0.014% |   net/http.(*http2ClientConn).RoundTrip net/http/h2_bundle.go:8381
                                                 3 0.014% |   net/http.(*http2clientStream).writeRequest net/http/h2_bundle.go:8549
                                                 1 0.0046% |   github.com/traefik/traefik/v2/pkg/provider/consulcatalog.(*Provider).Provide.func1.1 github.com/traefik/traefik/v2/pkg/provider/consulcatalog/consul_catalog.go:227
                                                 1 0.0046% |   github.com/traefik/traefik/v2/pkg/provider/consulcatalog.(*Provider).watchConnectTLS github.com/traefik/traefik/v2/pkg/provider/consulcatalog/consul_catalog.go:560
                                                 1 0.0046% |   github.com/traefik/traefik/v2/pkg/provider/consulcatalog.repeatSend github.com/traefik/traefik/v2/pkg/provider/consulcatalog/consul_catalog.go:623
                                                 1 0.0046% |   github.com/traefik/traefik/v2/pkg/provider/file.(*Provider).addWatcher.func1 github.com/traefik/traefik/v2/pkg/provider/file/file.go:110
                                                 1 0.0046% |   github.com/traefik/traefik/v2/pkg/provider/kv.(*Provider).watchKv.func1 github.com/traefik/traefik/v2/pkg/provider/kv/kv.go:101
                                                 1 0.0046% |   github.com/traefik/traefik/v2/pkg/server.(*ConfigurationWatcher).applyConfigurations github.com/traefik/traefik/v2/pkg/server/configurationwatcher.go:151
                                                 1 0.0046% |   github.com/traefik/traefik/v2/pkg/server.(*ConfigurationWatcher).receiveConfigurations github.com/traefik/traefik/v2/pkg/server/configurationwatcher.go:103
                                                 1 0.0046% |   github.com/traefik/traefik/v2/pkg/server.(*Server).listenSignals github.com/traefik/traefik/v2/pkg/server/server_signals.go:20
                                                 1 0.0046% |   github.com/uber/jaeger-client-go.(*remoteReporter).processQueue github.com/uber/[email protected]+incompatible/reporter.go:296
                                                 1 0.0046% |   go.opencensus.io/stats/view.(*worker).start [email protected]/stats/view/worker.go:292
                                                 1 0.0046% |   net/http.(*persistConn).writeLoop net/http/transport.go:2421
                                                 1 0.0046% |   os/signal.NotifyContext.func1 os/signal/signal.go:288
         0     0%   100%      21531 99.93%                | runtime.selectgo runtime/select.go:327
                                             21531   100% |   runtime.gopark runtime/proc.go:398
----------------------------------------------------------+-------------

In context, I still suspect a consul catalog provider issue ❓

from traefik.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.