lukemonahan / splunk_modinput_prometheus Goto Github PK

I was testing out the released version on Splunk 7.3.1 (deployed with Docker, Splunk Developer License).

When trying to create a static prometheus exporter setup through the UI I found the "Index" drop down box would only show me indexes of type "Event". I couldn't see the "Metrics" type index I had created for the data.

I was able to manually edit /etc/splunk/etc/search/local/inputs.conf to point to the right index and the metrics worked. After changing this I could then see the "metrics" index I chose through the UI.

Is there a way to ensure the drop down box presents only the Metrics type indexes that are available?

I've made this change on my local installation, but would love to see it added to the official app. I've added these two input config settings, similar to prometheusrw.go, and it works great.

I'm trying to connect splunk via this mod_input to our Prometheus server.

The Prometheus api is as below and works fine:
curl https://usr:pwd@hostname/prometheus/ciams/api/v1/label/job/values

However the config in the inputs.conf file should be in a different format according to the examples.
But all I try doesn't work, as there is nothing listening on /metrics
example:

[prometheus://example]
URI = https://usr:pwd@hostname:443/metrics
index = prometheus
sourcetype = prometheus:metric
interval = 30
disabled = 0

How can I pull the metrics? Should something be reconfigured on Prometheus?

We've configured prometheus remote-write to write data into metrics type index in Splunk however seeing the following error on the indexer

Metric value=<unset> is not valid for source=prometheusrw, sourcetype=metric,
host=$decideOnStartup, index=prometheus_metrics. 
Metric event data with an invalid metric value would not be indexed. 
Ensure the input metric data is not malformed.

Our configs

Splunk

[prometheusrw://test]
bearerToken = XXXXXX
index = prometheus_metrics
whitelist = *
sourcetype = prometheus:metric
disabled = 0

Prometheus

  remoteWrite:
  - bearerToken: XXXXXX
    url: http://<splunk url>:8098
    writeRelabelConfigs:
    - action: keep
      regex: node_load[0-9]*
      sourceLabels:
      - __name__

Same configuration works if we try to write in the regular event based index in Splunk

Is this targeting a normal Splunk index, or does it target the newer metrics indexes in Splunk?

Hello,

We are trying to use the remote-write feature, but we keep getting this ERROR below on our HF.

We opened firewalls, and tcp port on the local CentOS server. Also, enabled receiving on the heavy forwarder on a specified port. The inputs.conf is also configured with the proper bearer token to collect metrics from Prometheus, and we specified the port inside the inputs.conf global settings as well.

Message rejected. Received unexpected message of size=369295616 bytes from src=ip:port in streaming mode. Maximum message size allowed=67108864. (::) Possible invalid source sending data to splunktcp port or valid source sending unsupported payload.

Hi,

would it be possible to extend the input to provide credential based authentication?

Working on to integrate metric forwarding via Prometheus remote-write to Splunk HF.

Current Setup:- Splunk HF is hosted on HTTPS. As below document suggest for HTTP, Than how to use HTTPS for remote-write?

Using Below configuration in Splunk-HF as reference to https://splunkbase.splunk.com/app/4077

[prometheusrw]
port = 8098
maxClients = 10
disabled = 0

[prometheusrw://testing]
bearerToken = ABC123
index = prometheus
whitelist = *
sourcetype = prometheus:metric
disabled = 0

And not able to justify the team that how to configure bearer-token in prometheus, As for this no document is available.
And also how to use HTTPS in url as splunk hosted over HTTPS

remote_write:
- url: "http://myhost:8098"
bearer_token: "ABC123"
write_relabel_configs:
- source_labels: [name]
regex: expensive.*
action: drop

Hello,
I'm new for Prometheus and Splunk. In README, it mentioned prometheusrw has been designed to mimic HEC but much simpler.

Then I'm wondering could prometheusrw support SSL communication?
If yes, how should we setup SSL?

It appears to be a couple years since the last commit but says its not quite 1.0 ready yet.

I'm just wondering what the status of this plugin is? Is there an alternate way to get prometheus metrics into splunk?

Hello,

Did you updated the splunk app on splunk base : https://splunkbase.splunk.com/app/4077/
Seems not based on the last update from github vs Splunk base.

Thanks.

@lukemonahan,

Is it possible to send prometheus instance name as hostname to splunk. I have two instances prometheus running in my openshift cluster and I want to distinguish them both in splunk. How do I do that ?

Thank you
Charan

I am planning on add the capacity for setting up a new dynamic dimension via stanza configuration. By new dimension I mean the hability to setup for example datacenter dimension so you may split data in the dashboard by datacenter if that dimensions is to set (for me a dimension is a property like the job property coming by default from Prometheus)

Originally posted by @sky-philipalmeida in #4 (comment)

Hi, I'm trying to install this on an openshift cluster. I face some issues with the fact that It’s an operator. Can someone please help me?

Prometheus metric namespaces are separated with underscores (ie, process_virtual_memory_bytes). In the Splunk metrics explorer, metric names separated with periods automatically create a nested hierarchy (making metric navigation much easier). It would be helpful to convert the underscores in the prometheus metric names to periods to take advantage of this automatic nesting.

@lukemonahan, We observed that splunk received 350G+ of data for one day via remote write but when I checked the disk size of prometheus instance, it's not even 100G for that day. How is this possible ?

Thanks,
Charan

Greetings.I installed v1.0.0 in Splunk Enterprise v9.0.0 today, but I'm afraid I can't get past this issue.

My config in <splunk_dir>/etc/apps/modinput_prometheus/local/inputs.conf is

[prometheus://kfk-akira-1]
URI = http://hostabc.my.domain:8297/metrics
index = prometheus
sourcetype = prometheus:metric
interval = 10
disabled = 0

The index "prometheus" has already been created, as "metrics" type.

Unfortunately nothing is being collected. Instead the following message appears in <splunk_dir>/var/log/splunkd.log, once per interval period.

06-27-2022 08:08:04.049 +0000 ERROR ExecProcessor [2378 ExecProcessor] - message from "/opt/splunk/etc/apps/modinput_prometheus/linux_x86_64/bin/prometheus"
  2022/06/27 08:08:04 Get http://hostabc.my.domain:8297/metrics: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

If I use curl to fetch http://hostabc.my.domain:8297/metrics from the shell on the same server there is no problem though.

root@splunkserver:/opt/splunk/etc/apps/modinput_prometheus# curl http://hostabc.my.domain:8297/metrics | head
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 387.12
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.656312303304E9
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
...

Would anyone have any clues?

Hi Luke,

Thanks for producing this great addon. We're attempting to use the scraper input against our TLS protected metrics endpoint. We're not having any luck pulling in metrics yet. At first, we were getting:

x509: certificate signed by unknown authority

We put the root cert into the root CA store on the machine, and that made the error go away, but it still won't pull the metrics in and there are no other errors. I assume we need to specify the client cert, but there doesn't seem to be any way to do that?

I see following error in splunk, even after updating file descriptor limit to much higher number.

http: Accept error: accept tcp 0.0.0.0:8098: accept4: too many open files; retrying in 20ms.

Prometheus data in splunk is not continuous, which I think is due to above problem. And there are several gaps and seeing data at some intervals.