Giter Club home page Giter Club logo

wavefront-proxy's Introduction

Security Advisories

Wavefront proxy version 10.10 and earlier is impacted by a Log4j vulnerability — CVE-2021-44228. VMware Security Advisory (VMSA): CVE-2021-44228 – VMSA-2021-0028 discusses this vulnerability and its impact on VMware products.

Patches

Wavefront proxy version 10.11 and later use a version of Log4j that addresses this vulnerability.


Wavefront Proxy Project

Wavefront is a high-performance streaming analytics platform for monitoring and optimizing your environment and applications.

The Wavefront Proxy is a light-weight Java application that you send your metrics, histograms, logs, and trace data to. It handles batching and transmission of your data to the Wavefront service in a secure, fast, and reliable manner.

Requirements

  • Java 8, 9, 10 or 11 (11 recommended)
  • Maven

Overview

  • pkg: Build and runtime packaging for the Wavefront proxy.
  • proxy: Wavefront Proxy source code.

Please refer to the project page for further details.

To start developing

git clone https://github.com/wavefronthq/wavefront-proxy
cd wavefront-proxy
mvn -f proxy clean install -DskipTests

If you want to skip code formatting during build use:
mvn -f proxy clean install -DskipTests -DskipFormatCode

Contributing

Public contributions are always welcome. Please feel free to report issues or submit pull requests.

wavefront-proxy's People

Contributors

ajayj89 avatar akodali18 avatar alchen1218 avatar basilisk487 avatar conorbev avatar deosu avatar dependabot[bot] avatar djia-vm-wf avatar ezeev avatar hanwavefront avatar haosong avatar howardyoo avatar jbau avatar joannatk avatar kentwang929 avatar laullon avatar narawf avatar oppegard avatar panghy avatar sbhakta-vmware avatar sliu2013 avatar spramanik4667 avatar srujann avatar sushantdewan123 avatar thepeterstone avatar timcschmidt avatar vikramraman avatar wavefront-mike avatar wf-jenkins avatar xuranchen avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

wavefront-proxy's Issues

Support Groking against line breaks

Currently I have a HTTP client sending logs to wavefront as the following format

INFO:root:POST request,
Path: /
Headers:
Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3
Accept: */*
User-Agent: Ruby
Content-Type: text/csv
Host: 10.193.245.4:8080
Content-Length: 8088230



Body:
b'"stderr","fluentbit-control-plane-pkb94","vsphere-csi-controller-66b875d646-nf6z7","vsphere-csi-controller","vmware-system-csi"\n"stderr","fluentbit-control-plane-pkb94","vsphere-csi-controller-66b875d646-nf6z7","vsphere-csi-controller","vmware-system-csi"\n

This is actually 2 lines of entries with "¥n" as the line break.
However looking at the results of the metrics being sent it seems (i hate to use this, but it is really hard to test/prove this) wavefront proxy is handling as a single line.
Would want a feature that can support handling this as a multiline.

NPE on bad configuration

https://github.com/wavefrontHQ/java/blob/ac619e175577aa1e8c0ad24e99ce3ef65f785d19/proxy/src/main/java/com/wavefront/agent/preprocessor/AgentPreprocessorConfiguration.java#L69

2019-03-18 10:34:35,541 WARN  [agent:loadListenerConfigurationFile] Loaded configuration file /var/vcap/jobs/wavefront_proxy/conf/wavefront.conf
2019-03-18 10:34:35,590 ERROR [agent:start] Aborting start-up
java.lang.NullPointerException: null
	at com.wavefront.agent.preprocessor.AgentPreprocessorConfiguration.loadFromStream(AgentPreprocessorConfiguration.java:69) ~[wavefront-proxy.jar:?]
	at com.wavefront.agent.AbstractAgent.initPreprocessors(AbstractAgent.java:849) ~[wavefront-proxy.jar:?]
	at com.wavefront.agent.AbstractAgent.loadListenerConfigurationFile(AbstractAgent.java:1107) ~[wavefront-proxy.jar:?]
	at com.wavefront.agent.AbstractAgent.start(AbstractAgent.java:1180) [wavefront-proxy.jar:?]
	at com.wavefront.agent.PushAgentDaemon.start(PushAgentDaemon.java:23) [wavefront-proxy.jar:?]
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_121]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_121]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_121]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121]

Ability to limit TLS version(s) supported by wavefront proxy

We're going to deploy the wavefront proxy into our environment and are going to enable TLS/SSL connections to the proxy. In our environment, we only allow TLS v1.2. Would like to submit a feature enhancement to be able to control which versions of TLS the proxy allows for incoming connections.

Service Dashboard uses incorrect tags

When using preprocessor rules to change tags (like cluster and shard), the service dashboard does not respect the changes. The traces show the modified tags and they are filterable on the main application status page, but you must choose "none" on the service dashboard if the jaeger app didn't set "cluster" or "shard". Even if the preprocessor rules change it, the dropdowns are broken.

Pre-built binaries for Amazon Linux 2023

Hi team,

I want to deploy amazonlinux:2023.3.20240108.0 image under my wf-proxy service Dockerfile.
As for now, we used atlas-base-photon3-jdk11 image and we used tdnf package manager for installing wavefront.
Not the aim is to deploy ''skyscraper-docker-local.artifactory.eng.vmware.com/amazonlinux:2023.3.20240108.0'',
Can you help me understand what pre-built binaries Wavefront offers for Amazon Linux 2023 that I can download and place within the container.
Are dnf packages available for Wavefront?

Thanks!

Docker container has a huge layer because of chown of /var

There is

RUN chown -R wavefront:wavefront /var

done when building Docker container - https://github.com/wavefrontHQ/wavefront-proxy/blob/master/proxy/docker/Dockerfile#L20
As docker does copy-on-write, this operation creates a huge layer - docker image history shows

<missing>      2 days ago   /bin/sh -c chown -R wavefront:wavefront /var    180MB     

which is quite a lot compared to seemingly innocent operation.

I suspect you do not need to change ownership to the entire /var hierarchy...

Latest releases not showing up in OSS repositories

I'm not seeing any releases since 3.10 in the OSS maven repositories. It would be good if the recent releases could get published.

Also, it would be nice if the git repo is tagged per release so git history doesn't have to be used to determine what sha corresponds to a release.

java-client depends on deprecated commons-lang:commons-lang:2.6

commons-lang:commons-lang was moved to org.apache.commons:commons-lang3.

Also, the only use of commons-lang is for StringUtils.isBlank(), which could be easily re-implemented in Wavefront.java rather than taking an external dependency.

This causes conflicting JAR problems for anyone who depends on commons-lang3.

config files should be in more canonical places

Linux convention is to have config files under /etc, but we have ours in /opt/wavefront/wavefront-proxy/wavefront.conf.

We should move to the more conventional thing without breaking backwrards compatibility.

Error in logs "cannot be cast to class java.util.List" when configuring the proxy preprocessor

Issue Description

When I pass the wavefront-proxy a preprocessor configuration file, I see this error in the logs:

2023-04-27 10:01:13,854 WARN  [proxy:processConfiguration] Error during configuration update
java.lang.RuntimeException: Can't parse preprocessor configuration
	at com.wavefront.agent.preprocessor.PreprocessorConfigManager.loadFromStream(PreprocessorConfigManager.java:994) ~[classes!/:?]
	at com.wavefront.agent.preprocessor.PreprocessorConfigManager.processRemoteRules(PreprocessorConfigManager.java:200) ~[classes!/:?]
	at com.wavefront.agent.PushAgent.processConfiguration(PushAgent.java:2042) ~[classes!/:?]
	at com.wavefront.agent.ProxyCheckInScheduler.updateConfiguration(ProxyCheckInScheduler.java:350) ~[classes!/:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
	at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: java.lang.ClassCastException: class java.lang.String cannot be cast to class java.util.List (java.lang.String and java.util.List are in module java.base of loader 'bootstrap')
	at com.wavefront.agent.preprocessor.PreprocessorConfigManager.loadFromStream(PreprocessorConfigManager.java:241) ~[classes!/:?]
	... 9 more

Despite the above error, the preprocessor still seems to be working as intended, so I'm not sure whether this error is meaningful or it should be ignored.

Steps to replicate

Deploy the Wavefront proxy into a Kubernetes cluster with the following configuration (I am using VMware TKGI):

---
apiVersion: v1
kind: Namespace
metadata:
  name: wavefront-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: proxy-preprocessor-config
  namespace: wavefront-system
data:
  preprocessor_rules.yaml: |
    ---
    '2878':
    # Drop Telegraf k8s metrics because they are
    # duplicates of the metrics sent by the wavefront-collector
    - rule    : drop-telegraf-kubernetes-metrics
      action  : block
      scope   : metricName
      match   : 'telegraf.kubernetes.*'
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: wavefront-proxy
    name: wavefront-proxy
  name: wavefront-proxy
  namespace: wavefront-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: wavefront-proxy
  strategy:
    type: "Recreate"
  template:
    metadata:
      labels:
        app: wavefront-proxy
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - wavefront-proxy
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: wavefront-proxy
        image: wavefronthq/proxy:12.3
        imagePullPolicy: Always
        envFrom:
        - secretRef:
            name: wavefront-token
        env:
        - name: WAVEFRONT_URL
          value: <Your Wavefront URL>
        - name: WAVEFRONT_PROXY_ARGS
          value: --preprocessorConfigFile /preprocessor-config/preprocessor_rules.yaml
        ports:
        - containerPort: 2878
          protocol: TCP
        volumeMounts:
        - name: proxy-preprocessor-config
          mountPath: /preprocessor-config/
          readOnly: true
        securityContext:
          privileged: false
      volumes:
      - name: proxy-preprocessor-config
        configMap:
          name: proxy-preprocessor-config
---
apiVersion: v1
kind: Service
metadata:
  name: wavefront-proxy
  labels:
    app: wavefront-proxy
  namespace: wavefront-system
spec:
  type: LoadBalancer
  ports:
  - name: wavefront
    port: 2878
    protocol: TCP
  selector:
    app: wavefront-proxy

Also, create a secret to contain your Wavefront token:

kubectl -n wavefront-system create secret generic wavefront-token \
  --from-literal=WAVEFRONT_TOKEN='<YOUR_WAVEFRONT_TOKEN>'

no main manifest attribute for proxy-11.3-jar-with-dependencies.jar

I am upgarding to wavefront-proxy-11.3. Succesfully build the project locally, but for proxy-11.3-jar-with-dependencies.jar I am getting following error

$java -jar proxy-11.3-jar-with-dependencies.jar
$no main manifest attribute, in proxy-11.3-jar-with-dependencies.jar

Manifest file of proxy-11.3-jar-with-dependencies.jar has following content
Manifest-Version: 1.0 Created-By: Plexus Archiver 4.2.1

In wavefront-proxy 11.1 version we have proxy-11.1-uber.jar instead of proxy-11.3-jar-with-dependencies.jar
Manifest file of proxy-11.1-uber.jar has following content
Manifest-Version: 1.0 Archiver-Version: Plexus Archiver Created-By: Apache Maven 3.6.3 Built-By: root Build-Jdk: 11.0.12-internal Main-Class: com.wavefront.agent.PushAgent Multi-Release: true

Due to the proxy-11.3-jar-with-dependencies.jar Main class error, not able to consume wavefront-proxy-11.3.

Push artifacts to PackageCloud within Docker

https://packagecloud.io/docs#cli

package_cloud push wavefront/proxy-next/ubuntu/xenial wavefront-proxy_3.20-1_amd64.deb
package_cloud push wavefront/proxy-next/ubuntu/trusty wavefront-proxy_3.20-1_amd64.deb
package_cloud push wavefront/proxy-next/ubuntu/precise wavefront-proxy_3.20-1_amd64.deb
package_cloud push wavefront/proxy-next/debian/wheezy wavefront-proxy_3.20-1_amd64.deb
package_cloud push wavefront/proxy-next/debian/jessie wavefront-proxy_3.20-1_amd64.deb
package_cloud push wavefront/proxy-next/debian/buster wavefront-proxy_3.20-1_amd64.deb
package_cloud push wavefront/proxy-next/debian/stretch wavefront-proxy_3.20-1_amd64.deb

package_cloud push wavefront/proxy-next/el/6 wavefront-proxy-3.20-1.x86_64.rpm
package_cloud push wavefront/proxy-next/el/7 wavefront-proxy-3.20-1.x86_64.rpm

Port 5044 becomes inactive after deploying 12.3 version of wavefront-proxy

We have wavefront-proxy deployed in AWS instance. After deploying the latest version (12.3) on the instance, port 5044 becomes inactive. We use this port to send some metrics from a log file via this filebeat port to wavefront dashboard.
Also tried the 12.4 version and same failure.
Reverted to 12.2 to get the metrics correctly on wavefront dashboard.

Can't find custom metrics on wavefront sent by wavefront client

I am able to send metrics from command prompt with nc -c command but while sending metrics with API, I could debug that API connects to server with isConnected() method of Wavefront class but can't locate metrics on wavefront sent by API using send function. I also got failure count as 0.

Prometheus Histogram Data not recognized as histogram type in Wvavefront

Hi,
I have set up the Wavefront proxy via the wavefront Kubernetes operator and enabled the histogram configs for the proxy however I am still having an issue with my histogram data types being recognized as such in Wavefront once they are ingested.

I am using Prometheus in my node application that is running in my Kubernetes cluster and it seems as though the Prometheus histogram type and the wavefront histogram type are not compatible.

Is there additional tooling that can be used that can help make them compatible? I am running into some issues trying to parse out the data to use in charts in a meaningful way as it is and hoping to try to get it into wavefront in a histogram type that it can recognize.

Thanks,
Dylan

wavefront proxy should bundle JRE with the RPM

When installing Wavefront proxy on disconnected server, the installation fails since the JRE download fails.

Wavefront should bundle a dedicated JRE for the proxy.

problems with current situation:

  • installation with "Limited Network Access" always fail since the installer tries to download the JRE
    • in most cases going via the http proxy will not help since that proxy will be open only to push metrics, not more than that (in particular not to downloading)
  • detection of existing JDK\JRE is not accurate here is an example:
[root@wf-poc-01 ~]# java -version
openjdk version "1.8.0_102"
OpenJDK Runtime Environment (build 1.8.0_102-b14)
OpenJDK 64-Bit Server VM (build 25.102-b14, mixed mode)
[root@wf-poc-01 ~]# which java
/usr/bin/java

[root@wf-poc-01 ~]# rpm -U /installers/wavefront-proxy-4.36-2.x86_64.rpm
/etc/wavefront/wavefront-proxy/wavefront.conf already exists
JRE not found - trying to download and install
Checking /etc/wavefront/wavefront-proxy/wavefront.conf for HTTP proxy settings
No HTTP proxy configuration detected - attempting direct download
tar: /tmp/jre.tar.gz: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
rm: cannot remove '/tmp/jre.tar.gz': No such file or directory

Dropwizard plugin does not respect proxy's DNS TTL

The dropwizard plugin seems to grab the address of the proxy only at startup and is neither respecting the TTL of the proxy's address (mentioning the TTL in specific as it is 15s in our case) or doing a fresh lookup on failures to send.

Support UUID trace/span ids in zipkin format

Currently the code tries to support them (here), but the problem is for zipkin payloads they go through zipkin2 library first, and zipkin2 library will reject them (here)

So in order to actually support UUIDs in zipkin payloads, wavefront-proxy would need to switch to a more lenient parser for zipkin payloads.

WavefrontReporter supports additional point tags for specific metrics

Currently, the WavefrontReporter implementation allows me to include pointTags for all metrics that get exported to my Wavefront instance.

I'd like to add additional point tags to specific metrics based off their name.

I'm interested in tracking response codes for a certain Health API. I might have the following Meter names in my MetricRegistry tracking the counts/rates of the various return codes for this health API:

  • health_api[status=200]
  • health_api[status=500]
  • health_api[status=503]

Right now, the WavefrontReporter "sanitizes" a metric name like this. I'd like to see it be able to parse out key/value pairs within [%s=%s] and add them as additional point tags.

If they were point tags, I could create a Query for ts(health_api) and see a breakdown of all the different response codes. It would also roll up a "total" API calls metric automatically without me having to track it separately.

Is there a better way to accomplish this? Or is there a reason why this approach would be a bad idea? I'm admittedly no expert when it comes to Wavefront or Dropwizards.

Setting for adding a proxy level point tag in wavefront.conf

I am looking at the use-case where I will have to repeat the point tag addition rule for all the ports (3 for now we have) in the preprocessor_rules.html file.

For me this is a proxy level rule, not an collector agent level rule.
Hence I started looking into wavefront.conf, tried out few things. and finally came to know that I need to actually apply this point tag rule in preprocessor_rules.

It is instinctive for this proxy level rule to be in wavefront.conf

Adam agreed to this logic over slack channel.
Adam Johnson [3:08 PM]
I agree, for a global thing like that wavefront.conf would be good, but I’m not sure the main conf file has any point tag controls.

And then as per Conor's suggestion - opening this PR

NPE occurred

Discovered below NPE this morning. I got 2 proxies and 2 prometheus adapters running in k8s. Setup is 1 prom adapter and 1 proxy per pod (2 containers per pod). An AWS ALB is round robining to both prom adapters and each prom adapter sends its data to the 1 proxy in its pod.

It seems the proxy is still sending the metrics to wavefront even though it's no longer logging the points delivery rate.

021-05-28 13:58:44,571 INFO [AbstractReportableEntityHandler:printStats] [2878] Points delivered rate: 142 pps (1 min), 145 pps (5 min)
2021-05-28 13:58:54,571 INFO [AbstractReportableEntityHandler:printStats] [2878] Points received rate: 138 pps (1 min), 144 pps (5 min), 0 pps (current).
Exception in thread "stats-output-points.2878" java.lang.NullPointerException
at com.yammer.metrics.core.BurstRateTrackingCounter.lambda$getFiveMinuteCount$2(BurstRateTrackingCounter.java:128)
at java.base/java.util.stream.ReferencePipeline$5$1.accept(ReferencePipeline.java:229)
at java.base/java.util.ArrayList$SubList$2.forEachRemaining(ArrayList.java:1511)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.LongPipeline.reduce(LongPipeline.java:474)
at java.base/java.util.stream.LongPipeline.sum(LongPipeline.java:432)
at com.yammer.metrics.core.BurstRateTrackingCounter.getFiveMinuteCount(BurstRateTrackingCounter.java:128)
at com.yammer.metrics.core.BurstRateTrackingCounter.getFiveMinutePrintableRate(BurstRateTrackingCounter.java:109)
at com.wavefront.agent.handlers.AbstractReportableEntityHandler.printStats(AbstractReportableEntityHandler.java:232)
at com.wavefront.agent.handlers.AbstractReportableEntityHandler$3.run(AbstractReportableEntityHandler.java:117)
at java.base/java.util.TimerThread.mainLoop(Timer.java:556)
at java.base/java.util.TimerThread.run(Timer.java:506)
2021-05-28 13:58:55,010 INFO [proxy:checkin] Checking in: https://longboard.wavefront.com/api/
2021-05-28 13:59:55,191 INFO [proxy:checkin] Checking in: https://longboard.wavefront.com/api/
2021-05-28 14:00:55,243 INFO [proxy:checkin] Checking in: https://longboard.wavefront.com/api/
2021-05-28 14:01:55,302 INFO [proxy:checkin] Checking in: https://longboard.wavefront.com/api/
2021-05-28 14:02:55,360 INFO [proxy:checkin] Checking in: https://longboard.wavefront.com/api/

Question about project goals

"Hi, i am trying to understand what is the project about?
The README for this project, it is difficult for me to understand. Could you please improve the file? I would be happy to help if you need it.

Prefix prepended to Internal SDK metrics resulting in invalid point metric name

When the prefix value is set on the proxy, these errors are continuously logged:
2023-03-17 11:08:37,149 INFO [AbstractReportableEntityHandler:reject] [2878] blocked input: [WF-400: Point metric has illegal character(s): prefix.~sdk.go.core.sender.proxy.points.queue.remaining_capacity ("prefix.~sdk.go.core.sender.proxy.points.queue.remaining_capacity" 10187.0 1679051317 source="ip-192-168-0-2" "pid"="1567411" "version"="0.11.0")]
These appear to be Internal SDK Metrics being reported from Telegraf agents sending metrics through the proxy. Unfortunately, it is not (yet) possible to disable the Internal SDK Metrics to work around this issue.

Wavefront-proxy 12.2

Hi,

when I try to create a container y run the docker-compose.yaml I get the following error:

proxy-2_1  | 2023-03-16 08:57:47,681 ERROR [proxy:start] RESTEASY004655: Unable to invoke request: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:80 [/127.0.0.1] failed: Connection refused (Connection refused)
proxy-2_1  | javax.ws.rs.ProcessingException: RESTEASY004655: Unable to invoke request: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:80 [/127.0.0.1] failed: Connection refused (Connection refused)
proxy-2_1  | 	at org.jboss.resteasy.client.jaxrs.engines.ApacheHttpClient4Engine.invoke(ApacheHttpClient4Engine.java:341) ~[resteasy-client-3.15.3.Final.jar!/:3.15.3.Final]
proxy-2_1  | 	at org.jboss.resteasy.client.jaxrs.internal.ClientInvocation.invoke(ClientInvocation.java:464) ~[resteasy-client-3.15.3.Final.jar!/:3.15.3.Final]
proxy-2_1  | 	at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invokeSync(ClientInvoker.java:152) ~[resteasy-client-3.15.3.Final.jar!/:3.15.3.Final]
proxy-2_1  | 	at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientInvoker.invoke(ClientInvoker.java:115) ~[resteasy-client-3.15.3.Final.jar!/:3.15.3.Final]
proxy-2_1  | 	at org.jboss.resteasy.client.jaxrs.internal.proxy.ClientProxy.invoke(ClientProxy.java:76) ~[resteasy-client-3.15.3.Final.jar!/:3.15.3.Final]
proxy-2_1  | 	at com.sun.proxy.$Proxy40.proxySaveConfig(Unknown Source) ~[?:?]
proxy-2_1  | 	at com.wavefront.agent.ProxyCheckInScheduler.sendConfig(ProxyCheckInScheduler.java:136) ~[classes!/:?]
proxy-2_1  | 	at com.wavefront.agent.ProxyCheckInScheduler.<init>(ProxyCheckInScheduler.java:93) ~[classes!/:?]
proxy-2_1  | 	at com.wavefront.agent.AbstractAgent.start(AbstractAgent.java:320) ~[classes!/:?]
proxy-2_1  | 	at com.wavefront.agent.PushAgent.main(PushAgent.java:220) ~[classes!/:?]
proxy-2_1  | 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
proxy-2_1  | 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
proxy-2_1  | 	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
proxy-2_1  | 	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
proxy-2_1  | 	at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) ~[wavefront-proxy.jar:?]
proxy-2_1  | 	at org.springframework.boot.loader.Launcher.launch(Launcher.java:108) ~[wavefront-proxy.jar:?]
proxy-2_1  | 	at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) ~[wavefront-proxy.jar:?]
proxy-2_1  | 	at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65) ~[wavefront-proxy.jar:?]
proxy-2_1  | Caused by: org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:80 [/127.0.0.1] failed: Connection refused (Connection refused)
...

Wavefront Proxy 9.7 Docker Container getting crashed due to OOM

We have lately been observing WF Proxy Version 9.7 getting OOM'd. This is causing a back pressure on telegraf container and it sometimes results in telegraf container not being able to send metrics to WF Proxy. This results in missing metrics and alerts being triggered in Production. This was earlier not observed on WF Proxy version 9.2.

Captured heap dump(using jmap) from WF Proxy running 9.7 from Production.
heapdump_9.7_prod.bin.zip

Happy to provide further information as required since this is a frequent occurrence now in Production.

WF Proxy is running with 2GB Memory in Production with 1024 CPU.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.