Giter Club home page Giter Club logo

pure-exporter's Introduction

Current version

NOTE.... This exporter is deprecated in favor of the individual OpenMetrics exporters for FlashArray and FlashBlade

Pure Storage Prometheus exporter

Prometheus exporter for Pure Storage FlashArrays and FlashBlades.

Support Statement

These exporters are provided under Best Efforts support by the Pure Portfolio Solutions Group, Open Source Integrations team. For feature requests and bugs please use GitHub Issues. We will address these as soon as we can, but there are no specific SLAs.

Overview

This applications aims to help monitor Pure Storage FlashArrays and FlashBlades by providing an "exporter", which means it extracts data from the Purity API and converts it to a format which is easily readable by Prometheus.

The stateless design of the exporter allows for easy configuration management as well as scalability for a whole fleet of Pure Storage systems. Each time Prometheus scrapes metrics for a specific system, it should provide the hostname via GET parameter and the API token as Authorization token to this exporter.


Note: The previous method to provide the Pure API token via a GET parameter is now deprecated and will be removed in the next major version.


To monitor your Pure Storage appliances, you will need to create a new dedicated user on your array, and assign read-only permissions to it. Afterwards, you also have to create a new API key. The exporter is provided as three different options:

  • pure-exporter. Full exporter for both FlashArray and FlashBlade in a single bundle
  • pure-fa-exporter. FlashArray exporter
  • pure-fb-exporter. FlashBlade exporter

Building and Deploying

The exporter is preferably built and launched via Docker. You can also scale the exporter deployment to multiple containers on Kubernetes thanks to the stateless nature of the application.


The official docker images are available at Quay.io

docker pull quay.io/purestorage/pure-exporter:1.2.5-a

or for the FlashArray exporter

docker pull quay.io/purestorage/pure-fa-exporter:1.2.5-a

or for the FlashBlade exporter

docker pull quay.io/purestorage/pure-fb-exporter:1.2.5-a

To build and deploy the application via Docker, your local linux user should be added to the docker group in order to be able to communicate with the Docker daemon. (If this is not possible, you can still use sudo)

The detailed description on how to do that can be found on the Docker official documentation for your operating systemm To run a simple instance of the exporter, run:

make -f Makefile.fa test
make -f Makefile.fb test
make -f Makefile.mk test

The Makefile currently features these targets:

  • build - builds the docker image with preconfigured tags.
  • test - spins up a new docker container with all required parameters.

Local development

The application is usually not run by itself, but rather with the gunicorn WSGI server. If you want to contribute to the development, you can run the exporter locally without a WSGI server, by executing the application directly.

The following commands are required for a development setup:

# it is recommended to use virtual python environments!
python -m venv env
source ./env/bin/activate

# install dependencies
python -m pip install -r requirements.txt

# run the application in debug mode
python pure_exporter.py

Use the same approach to modify the FlashArray and/or the FlashBlade exporter, by simply using the related requitements file.

Scraping endpoints

The exporter uses a RESTful API schema to provide Prometheus scraping endpoints.

Authentication

Authentication is used by the exporter as the mechanism to cross authenticate to the scraped appliance, therefore for each array it is required to provide the REST API token for an account that has a 'readonly' role. The api-token must be provided in the http request using the HTTP Authorization header of type 'Bearer'. This is achieved by specifying the api-token value as the authorization parameter of the specific job in the Prometheus configuration file. As an alternative, it is possible to provide the api-token as a request argument, using the apitoken key. Note this option is deprecated and will be removed from the next releases.

The full exporter understands the following requests:

System URL GET parameters description
FlashArray http://<exporter-host>:<port>/metrics/flasharray endpoint Full array metrics
FlashArray http://<exporter-host>:<port>/metrics/flasharray/array endpoint Array only metrics
FlashArray http://<exporter-host>:<port>/metrics/flasharray/volumes endpoint Volumes only metrics
FlashArray http://<exporter-host>:<port>/metrics/flasharray/hosts endpoint Hosts only metrics
FlashArray http://<exporter-host>:<port>/metrics/flasharray/pods endpoint Pods only metrics
FlashBlade http://<exporter-host>:<port>/metrics/flashblade endpoint Full array metrics
FlashBlade http://<exporter-host>:<port>/metrics/flashblade/array endpoint Array only metrics
FlashBlade http://<exporter-host>:<port>/metrics/flashblade/clients endpoint Clients only metrics
FlashBlade http://<exporter-host>:<port>/metrics/flashblade/quotas endpoint Quotas only metrics

The FlashArray-only and FlashBlade only exporters use a slightly different schema, which consists of the removal of the flasharray|flashblade string from the path.

FlashArray

URL GET parameters description
http://<exporter-host>:<port>/metrics endpoint Full array metrics
http://<exporter-host>:<port>/metrics/array endpoint Array only metrics
http://<exporter-host>:<port>/metrics/volumes endpoint Volumes only metrics
http://<exporter-host>:<port>/metrics/hosts endpoint Hosts only metrics
http://<exporter-host>:<port>/metrics/pods endpoint Pods only metrics

FlashBlade

URL GET parameters description
http://<exporter-host>:<port>/metrics endpoint Full array metrics
http://<exporter-host>:<port>/metrics/array endpoint Array only metrics
http://<exporter-host>:<port>/metrics/clients endpoint Clients only metrics
http://<exporter-host>:<port>/metrics/quotas endpoint Quotas only metrics

Depending on the target array, scraping for the whole set of metrics could result into timeout issues, in which case it is suggested either to increase the scraping timeout or to scrape each single endpoint instead.

Prometheus configuration examples

The config directory provides a couple of Prometheus configuration examples that can be used as the starting point to build your own solution.

Usage example

In a typical production scenario, it is recommended to use a visual frontend for your metrics, such as Grafana. Grafana allows you to use your Prometheus instance as a datasource, and create Graphs and other visualizations from PromQL queries. Grafana, Prometheus, are all easy to run as docker containers.

To spin up a very basic set of those containers, use the following commands:

# Pure exporter
docker run -d -p 9491:9491 --name pure-exporter quay.io/purestorage/pure-exporter:<version>

# Prometheus with config via bind-volume (create config first!)
docker run -d -p 9090:9090 --name=prometheus -v /tmp/prometheus-pure.yml:/etc/prometheus/prometheus.yml -v /tmp/prometheus-data:/prometheus prom/prometheus:latest

# Grafana
docker run -d -p 3000:3000 --name=grafana -v /tmp/grafana-data:/var/lib/grafana grafana/grafana

Please have a look at the documentation of each image/application for adequate configuration examples.

Bugs and Limitations

  • Pure FlashBlade REST APIs are not designed for efficiently reporting on full clients and objects quota KPIs, therefrore it is suggested to scrape the "array" metrics preferably and use the "clients" and "quotas" metrics individually and with a lower frequency than the other.. In any case, as a general rule, it is advisable to do not lower the scraping interval down to less than 30 sec. In case you experience timeout issues, you may want to increase the internal Gunicorn timeout by specifically setting the --timeout variable and appropriately reduce the scraping intervall as well.

  • By default the number of workers spawn by Gunicorn is set to 2 and this is not optimal when monitoring a relatively large amount of arrays. The suggested approach is therefore to run the exporter with a number of workers that approximately matches the number of arrays to be scraped.

License

This project is licensed under the Apache 2.0 License - see the LICENSE.md file for details

pure-exporter's People

Contributors

emily-potyraj avatar genegr avatar knack-roque avatar rickhlx avatar rlondner avatar sdodsley avatar zleinweber avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pure-exporter's Issues

Export Flashblade Object Replica Link Metrics

In version 3.x of the Flashblade Firmware there is now an 'object replica link' feature. This feature can be used to replicate objects that are in a bucket on the flashblade to a remote target. The remote target can be either another flashblade or an alternative S3 target (like AWS S3).

Our organization uses object replica links to replicate data to offsite storage. We've already had several scenarios where these replica links have gotten into an unhealthy state and required manual intervention to fix. It would be nice to have metrics around this exported so that we can proactively alert in these types of scenarios.

I propose adding the following metrics for each object replica link:

  • purefb_bucket_replica_links_replicating - (Gauge) 1 if replicating 0 otherwise
  • purefb_bucket_replica_links_paused - (Gauge) 1 if paused 0 otherwise
  • purefb_bucket_replica_links_unhealthy - (Gauge) 1 if unhealthy 0 otherwise
  • purefb_bucket_replica_links_lag - (Gauge) Time in MS that the replication is behind
  • purefb_bucket_replica_links_recovery_point - (Gauge) Unix time stamp to last full recovery point for the replica link

The relevant purity-fb documentation for this is: https://purity-fb.readthedocs.io/en/latest/BucketReplicaLinksApi/

Mediator status

Hi,
Is it possible to add a "metric" which give us mediator status by array.

Panel plugin not found: michaeldmoore-annunciator-panel

This looks to be a third party plugin from https://github.com/michaeldmoore/michaeldmoore-annunciator-panel/blob/master/package.json

If required this should be packaged with this project (following appropriate licensing) but if possible change to a built in panel type to make the dashboards more portable.

For now I'm changing the panel to a single stat (json below) since my version of grafana is too old (7.0.6) for that third party panel (requires 7.3.7+):

    {
      "cacheTimeout": null,                                                                                                                                                      
      "datasource": "$datasource",                                                                                                                                               
      "fieldConfig": {
        "defaults": {
          "custom": {},                                                                                                                                                          
          "decimals": 2,                                                                                                                                                         
          "mappings": [
            {
              "id": 0,                                                                                                                                                           
              "op": "=",                                                                                                                                                         
              "text": "N/A",                                                                                                                                                     
              "type": 1,
              "value": "null"                                                                                                                                                    
            } 
          ],
          "nullValueMode": "connected",                                                                                                                                          
          "thresholds": {
            "mode": "absolute",                                                                                                                                                  
            "steps": [
              {
                "color": "#d44a3a",                                                                                                                                              
                "value": null
              },
              {
                "color": "FF9830",                                                                                                                                               
                "value": 1.5
              },
              {
                "color": "#299c46",                                                                                                                                              
                "value": 2.5
              } 
            ] 
          },
          "unit": "none"                                                                                                                                                         
        },
        "overrides": []                                                                                                                                                          
      },
      "gridPos": {                                                                                                                                                               
        "h": 3,
        "w": 3,                                                                                                                                                                  
        "x": 21,                                                                                                                                                                 
        "y": 0
      },
      "id": 14,                                                                                                                                                                  
      "interval": null,                                                                                                                                                          
      "links": [],
      "maxDataPoints": 100,                                                                                                                                                      
      "options": {
        "colorMode": "background",                                                                                                                                               
        "fieldOptions": {
          "calcs": [
            "lastNotNull"                                                                                                                                                        
          ] 
        },
        "graphMode": "none",                                                                                                                                                     
        "justifyMode": "auto",                                                                                                                                                   
        "orientation": "horizontal",                                                                                                                                             
        "reduceOptions": {
          "calcs": [
            "mean" 
          ],
          "values": false                                                                                                                                                        
        } 
      },
      "pluginVersion": "7.0.1",                                                                                                                                                  
      "targets": [
        {
          "expr": "purefa_volume_space_datareduction_ratio{job=~\"$job\",instance=\"$instance\",volume=\"$volume\"}",                                                            
          "format": "time_series",
          "instant": true,
          "intervalFactor": 1,                                                                                                                                                   
          "refId": "A"
        } 
      ],
      "timeFrom": null,                                                                                                                                                          
      "timeShift": null,                                                                                                                                                         
      "title": "data reduction ratio",                                                                                                                                           
      "type": "stat"
    },

Suggestion to put the "info" metric into each endpoint for tagging purposes

On the FlashArray exporter - Currently only the /array endpoint contains "purefa_info"

HELP purefa_info FlashArray system information

TYPE purefa_info gauge

purefa_info{array_name="xxx",hostname="xxx",system_id="xxx",version="5.3.17"} 1.0

This would be very useful to use as a header on all endpoints for version tagging purposes.

pure-exporter - [CRITICAL] WORKER TIMEOUT

Hi, I'm having some issues while trying to run the (pure-exporter) container:

genegatpure/pure-exporter "gunicorn pure_expor…" 18 minutes ago Up 9 minutes 0.0.0.0:9491->9491/tcp, :::9491->9491/tcp pure-exporter

And the problem is that Prometheus is not logging any metrics from the arrays.

docker logs pure-exporter

[2021-06-02 21:48:28 +0000] [1] [INFO] Starting gunicorn 20.1.0
[2021-06-02 21:48:28 +0000] [1] [INFO] Listening at: http://0.0.0.0:9491 (1)
[2021-06-02 21:48:28 +0000] [1] [INFO] Using worker: sync
[2021-06-02 21:48:28 +0000] [7] [INFO] Booting worker with pid: 7
[2021-06-02 21:48:28 +0000] [8] [INFO] Booting worker with pid: 8
[2021-06-02 21:57:45 +0000] [1] [INFO] Handling signal: term
[2021-06-02 21:57:45 +0000] [8] [INFO] Worker exiting (pid: 8)
[2021-06-02 21:57:45 +0000] [7] [INFO] Worker exiting (pid: 7)
[2021-06-02 21:57:45 +0000] [1] [INFO] Shutting down: Master
[2021-06-02 21:57:55 +0000] [1] [INFO] Starting gunicorn 20.1.0
[2021-06-02 21:57:55 +0000] [1] [INFO] Listening at: http://0.0.0.0:9491 (1)
[2021-06-02 21:57:55 +0000] [1] [INFO] Using worker: sync
[2021-06-02 21:57:55 +0000] [7] [INFO] Booting worker with pid: 7
[2021-06-02 21:57:55 +0000] [8] [INFO] Booting worker with pid: 8
[2021-06-02 21:58:40 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:8)
...
[2021-06-02 22:04:05 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:21)
[2021-06-02 22:04:05 +0000] [21] [INFO] Worker exiting (pid: 21)
[2021-06-02 22:04:05 +0000] [23] [INFO] Booting worker with pid: 23
[2021-06-02 22:04:35 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:22)
[2021-06-02 22:04:35 +0000] [22] [INFO] Worker exiting (pid: 22)
[2021-06-02 22:04:35 +0000] [24] [INFO] Booting worker with pid: 24

Any ideas? Thanks in advance

FlashArray dashboards use non-existent metric for average throughput

The average throughput panels on the dashboards show N/A because they're calculated on a non-existent metric of purefa_array_performance_throughput_bytes. If it's supposed to be similar to the average volume throughput panels then I'd expect the query to be sum (purefa_array_performance_bandwidth_bytes{job=~"$job", instance="$instance", volume="$volume"} ) or the metric name needs to change.

For now in my local copy I'm just going to change the dashboard to use the metric that exists since that's less effort for me.

get_network_interfaces KeyError

File "/home/lk130f/pure_exporter/flasharray_collector/flasharray_metrics/network_interface_metrics.py", line 32, in _performance
self.fa.get_network_interfaces(),
File "/home/lk130f/pure_exporter/flasharray_collector/flasharray_metrics/flasharray.py", line 185, in get_network_interfaces
nicdict[n['name']].update(n)
KeyError: 'ct0.eth18'

The function get_network_interfaces fails for flasharray configured with bond interfaces. Bond interface is defined with this dict:
{'subnet': None, 'name': 'bond3', 'address': None, 'enabled': True, 'mtu': 9000, 'hwaddr': '92:7a:9f:23:3c:4d', 'netmask': None, 'slaves': ['ct0.eth18', 'ct0.eth19'], 'services': ['iscsi'], 'speed': 50000000000, 'gateway': None}

nicdict key (n['name']) is bond3 but the performance metric are reported under ct0.eth18/19 names, therefore an error is returned when nicdict update is called.

Missing Metric from metrics/flasharray/hosts endpoint

The purefa_host_volumes_info metric is only available from most expensive /metrics/flasharray/ endpoint, I would expect it to also be available in /metrics/flasharray/hosts as well.

Without this metric, it is hard to do correlation without the host, naaid table. It may also be helpful to add the volume name.

Replication follow in prometheus

Hi, I look for a solution to get replication activity on flasharray like on graphical vue. I'd like to put it in prometheus to get alert when we have problem
image

Export the exporter's metrics

It would be nice expose metrics about the exporter itself (cpu consumption, memory)
and expose some metrics about the collection itself (time spent collecting each kind of stuffs)

FlashBlade Array Space Metrics only shows used space and does not show total capacity

The exported FlashBlade array space metrics shows currently used space for:

  • unique (Unsure what this is)
  • virtual (space before compression)
  • total_physical (total physical space used)
  • snapshots (total space used by snapshots)

This is good. But, if we want to calculate total utilization as a percentage of available capacity we also need to know the raw capacity of the array. However, as far as I can tell the array capacity is not currently exported. It would be real nice to have this as an exported metric.

I can submit a PR for this!

Incorporate array_total_load function

In Pure1 manage portal --> "Analytics" --> Performance has a performance metrics which shows "Load" in percentage. Can the pure-exporter calculate or scrape this metric directly from the FA? I am aware that there is an API to pull this metrics from Pure1 portal itself but we would like to see this in real time.

Export quota metrics

Hi guys,

Was wondering if it would be possible to have quota metrics added to the exporter? It seems to already exist in the purity-fb python client but I don't see anything exported from the exporter. Right now this is one of the few things blocking us from being able to fully cutover from this old exporter (https://github.com/man-group/prometheus-flashblade-exporter)

Thanks!

Sending the API token as http Authorization header rather than a request parameter

Hello,

We are trying to set this up as a part of a k8s cluster. What bothers us is the security issues around Pure API token since currently the exporter exposes it as a GET request parameter. The ideal way would have been to use the HTTP Authorization header since the Pure API also supports that.
With the above changes, Prometheus could use the "bearer_token" configuration option https://prometheus.io/docs/prometheus/latest/configuration/configuration/ for embedding this http header.

Let me know your opinions/suggestion around this.

Non-docker setup

I'm looking to deploy this manually on RHEL 8.4.1. I was hoping I could use packages from the Redhat and EPEL repo and then put this into a systemd service. Is this possible?

After upgrade to 1.2.4, many array_space* metrics becomes zero

Hi
Thank you for fix #26
but after upgrade to 1.2.4, many array_space* metrics becomes zero

curl "http://localhost:9491/metrics/flasharray?endpoint=192.168.1.xx&apitoken=xxx" | grep array_space

# HELP purefa_array_space_datareduction_ratio FlashArray overall data reduction
# TYPE purefa_array_space_datareduction_ratio gauge
purefa_array_space_datareduction_ratio{dimension=""} 0.0
# HELP purefa_array_space_capacity_bytes FlashArray overall space capacity
# TYPE purefa_array_space_capacity_bytes gauge
purefa_array_space_capacity_bytes{dimension=""} 0.0
# HELP purefa_array_space_provisioned_bytes FlashArray overall provisioned space
# TYPE purefa_array_space_provisioned_bytes gauge
purefa_array_space_provisioned_bytes{dimension=""} 0.0
# HELP purefa_array_space_used_bytes FlashArray overall used space
# TYPE purefa_array_space_used_bytes gauge
purefa_array_space_used_bytes{dimension="shared"} 0.0
purefa_array_space_used_bytes{dimension="system"} 0.0
purefa_array_space_used_bytes{dimension="volumes"} 0.0
purefa_array_space_used_bytes{dimension="snapshots"} 0.0
purefa_array_space_used_bytes{dimension="replication"} 0.0

FlasharrayCollector: 'ConnectionError' object has no attribute 'message'

The exporter occasionally fails with the following error:

WARNING in pure_exporter: FlasharrayCollector: 'ConnectionError' object has no attribute 'message' 

Version : 1.0.3

Once it happens, the scape duration times out forever, until I restart the exporter.
The failure happens every few weeks.

My assumption so far is that there's a connection Error (connecting with http), but then we have this error: PureStorage-OpenConnect/rest-client@3023163#r40240923

So we may need rest-client v1.19

Unable to scrape "purefa_array_performance_throughput_bytes" metrics

I have installed the (quay.io/purestorage/pure-exporter:1.2.5-a) pure-exporter docker image. In this image it unable to scrape "purefa_array_performance_throughput_bytes" metrics. From the promql prompt it errors with "No datapoints found."

On the promql prompt the search shows only avg_block_bytes, bandwidth_bytes, latency_usec, qdepth as shown in the below screenshot.

Screen Shot 2022-10-04 at 10 33 16 AM

Config file example not available

Hello,

I am trying to configure the exporter, but am not able to make it work.
I am using the pure-exporter container (version 1.2.5-a).
This is my Prometheus configuration file :

scrape_configs:
  - job_name: "pure-exporter"
    params:
      endpoint: ['array.domain.intra']
      #user: ['centreon']
      api_token: ['376db02f-249e-d056-ed91-a9df394aea48']
    metrics_path: '/metrics/flasharray'
    static_configs:
      - targets: ["localhost:9491"]

Both containers (pure-exporter and Prometheus) both starts correctly but I get an "Unauthorized access" when trying to access the exporter metrics.

The resulting URL is : http://localhost:9491/metrics/flasharray?api_token=376db02f-249e-d056-ed91-a9df394aea48&endpoint=array.domain.intra

The thing is, the specified token is created for the user 'centreon'. When I uncomment the line, the result is the same.

The config directory (https://github.com/PureStorage-OpenConnect/pure-exporter/tree/master/config) being empty, I was not able to find a solution.

What am I missing here ?
Simon

Export network interface metrics

Would be nice to obtain metrics for all network interfaces.

From the REST API doc:

received_bytes_per_sec - number of bytes received per second.
received_packets_per_sec - number of packets received per second.
transmitted_bytes_per_sec - number of bytes transmitted per second.
transmitted_packets_per_sec - number of packets transmitted per second.
total_errors_per_sec - total number of received packets and transmitted packets per second with errors.

flashblade exporter: AttributeError: 'ArraysApi' object has no attribute 'list_arrays_nfs_specific_performance'

Hi,

i am trying to set up flashblade exporter and prometheus to scrape the metrics. While setting that up, for /metrics , /metrics/array, /metrics/quota the exporter is failing with the error message
AttributeError: 'ArraysApi' object has no attribute 'list_arrays_nfs_specific_performance'
whereas for the endpoint /metrics/clients, i don't see this error.
can you tell me if this is a bug and if yes, can this be fixed?

Thanks
Eswar

ERROR in app: Exception on /metrics/flasharray [GET]

Hi
I'm using docker quay.io/purestorage/pure-fa-exporter:1.2.3
to collect pure storage flash array(6.0.4) metrics
it works fine before setting pods
after creater pod to test active-dr feature
pure-exporter return error

[2021-03-04 03:40:56,834] ERROR in app: Exception on /metrics/flasharray [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.6/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.6/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/app/pure_fa_exporter.py", line 70, in route_flasharray
    resp = make_response(generate_latest(registry), 200)
  File "/usr/local/lib/python3.6/site-packages/prometheus_client/exposition.py", line 107, in generate_latest
    for metric in registry.collect():
  File "/usr/local/lib/python3.6/site-packages/prometheus_client/registry.py", line 83, in collect
    for metric in collector.collect():
  File "/app/flasharray_collector/flasharray_collector.py", line 48, in collect
    yield from PodStatusMetrics(self.fa).get_metrics()
  File "/app/flasharray_collector/flasharray_metrics/pod_status_metrics.py", line 43, in get_metrics
    self._status()
  File "/app/flasharray_collector/flasharray_metrics/pod_status_metrics.py", line 29, in _status
    for p in self.fa.get_pods():
  File "/app/flasharray_collector/flasharray_metrics/flasharray.py", line 125, in get_pods
    pdict[p['name']].update(p)
KeyError: 'm20-db2-testpod.undo-demote'

Thanks for your reply!

Labels are not being passed as tuples

In the metrics registration for at least array_space_metrics, the labels (of which there is a single one, "dimension"), the label value is not being passed as a list or tuple, but rather the string rep. This causes the metrics to come in with a single letter for dimension instead of the whole dimension name, which causes overwriting in the case of:

  • shared
  • system
  • snapshot
    for purefa_space_used_bytes metric amonst others. Should be an easy fix and improve accuracy of metrics being shipped.

Array name, location and firmware showing as blank

Hey,

I've having issues getting the array name, location and firmware showing on the graphs, but everything else looks to be working. I've tried updating the datasource to show $datasource and still seeing the same issue. Any advice?

Cheers

FlashArray without Volume Groups fails to retrieve volume metrics

When a FlashArray has no any volume groups, even an empty one is enough, the get_volumes() returns without 'vgroup' field included. I think this is something to do with the actual Pure API.

The exporter drops the following error:

[2021-04-14 05:45:16,694] ERROR in app: Exception on /metrics/flasharray [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.9/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.9/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/app/pure_exporter.py", line 135, in route_flasharray_all
    return route_flasharray('all')
  File "/app/pure_exporter.py", line 131, in route_flasharray
    return route_array('flasharray', m_type)
  File "/usr/local/lib/python3.9/site-packages/flask_httpauth.py", line 164, in decorated
    return f(*args, **kwargs)
  File "/app/pure_exporter.py", line 125, in route_array
    resp = make_response(generate_latest(registry), 200)
  File "/usr/local/lib/python3.9/site-packages/prometheus_client/exposition.py", line 177, in generate_latest
    for metric in registry.collect():
  File "/usr/local/lib/python3.9/site-packages/prometheus_client/registry.py", line 83, in collect
    for metric in collector.collect():
  File "/app/flasharray_collector/flasharray_collector.py", line 46, in collect
    yield from VolumeSpaceMetrics(self.fa).get_metrics()
  File "/app/flasharray_collector/flasharray_metrics/volume_space_metrics.py", line 53, in get_metrics
    self._data_reduction()
  File "/app/flasharray_collector/flasharray_metrics/volume_space_metrics.py", line 37, in _data_reduction
    self.data_reduction.add_metric([v_name[1], v['naaid'], v_name[0], v['vgroup']], v['data_reduction'] if v['data_reduction'] is not None else 0)
KeyError: 'vgroup'

To fix it we just need to add 'vgroup' where it is required.

Pure Exporter doesn't report discards

We only see writes and reads. If you're able to tell the difference between write throughput for replicas, mirrored vs normal writes, surely you'd be able to report on discard traffic.

Configuration error

Hello, i'm trying to scrape metrics from my arrays but I get an error and I don't know why.
I think it's due to an error in my configuration.

I run pure-exporter and prometheus as docker services.
Here is my configuration file :

`

  • job_name: 'pure_flasharray'
    scrape_timeout: 15s
    metrics_path: /metrics/flasharray
    relabel_configs:
    • source_labels: [address]
      target_label: __param_endpoint
    • source_labels: [__pure_apitoken]
      target_label: __param_apitoken
    • source_labels: [address]
      target_label: instance
    • target_label: address
      replacement: pure-exporter:9491
      static_configs:
    • targets: [ 172.xx.xx.x ]
      labels:
      __pure_apitoken: << token >>
    • targets: [ 172.xx.xx.x ]
      labels:
      __pure_apitoken: << token >>
    • targets: [ 172.xx.xx.x ]
      labels:
      __pure_apitoken: << token >>
    • targets: [ 172.xx.xx.x ]
      labels:
      __pure_apitoken: << token >>
      `

And here is the error I get in all prometheus targets :

Get "http://172.xx.xx.x:80/metrics/flasharray?apitoken=<< token >>&endpoint=172.xx.xx.x": context deadline exceeded

Can anyone help me with this issue ?
Thanks in advance

Active cluster writes

In an active cluster the mirror write metrics for hosts and volumes doesn't exist .Is it possible to add them.
Another thing, the read metric is not OK on hosts. We have the value * 2 .

Missing Metric

I have configured Prometheus server to scrape Pure exporter end-points for blade server. I see the metrics but I do not see this particular metric. Any idea why?

Missing Metric: purefb_array_capacity_bytes

Available Metrics:

purefb_filesystem_performance_throughput_bytes
purefb_array_space_bytes
purefb_array_specific_performance_latency_usec
purefb_filesystem_user_usage_bytes
purefb_hw_status
purefb_info
purefb_array_performance_latency_usec
purefb_array_specific_performance_iops
purefb_filesystem_performance_iops
purefb_filesystems_data_reduction
purefb_filesystems_space_bytes
purefb_array_performance_throughput_bytes
purefb_array_performance_opns_bytes
purefb_array_space_data_reduction
purefb_filesystem_group_usage_bytes
purefb_filesystem_performance_latency_usec
purefb_filesystem_performance_opns_bytes
purefb_open_events_total
purefb_array_performance_iops

I have configured these 3 metric path. Do I need to configure any other metrics path?

/metrics/flashblade/array
/metrics/flashblade/clients
/metrics/flashblade/usage

Broken Docker build due to purity_fb's code

The Dockerfile uses python:3-alpine as the base, which currently uses Python3.7

However, purity_fb is not compatible with Python3.7. The issue I encountered was:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
    worker.init_process()
  File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 119, in init_process
    self.load_wsgi()
  File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 144, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/usr/local/lib/python3.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/usr/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 49, in load
    return self.load_wsgiapp()
  File "/usr/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 39, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/usr/local/lib/python3.7/site-packages/gunicorn/util.py", line 358, in import_app
    mod = importlib.import_module(module)
  File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/app/pure_exporter.py", line 8, in <module>
    from collectors import FlasharrayCollector
  File "/app/collectors/__init__.py", line 2, in <module>
    from .flashblade import FlashbladeCollector
  File "/app/collectors/flashblade.py", line 3, in <module>
    from purity_fb import PurityFb
  File "/usr/local/lib/python3.7/site-packages/purity_fb/__init__.py", line 1, in <module>
    from .fb_client import PurityFb
  File "/usr/local/lib/python3.7/site-packages/purity_fb/fb_client.py", line 21, in <module>
    from . import purity_fb_1dot9
  File "/usr/local/lib/python3.7/site-packages/purity_fb/purity_fb_1dot9/__init__.py", line 17, in <module>
    from .models.admin import Admin
  File "/usr/local/lib/python3.7/site-packages/purity_fb/purity_fb_1dot9/models/__init__.py", line 128, in <module>
    from .performance_replication import PerformanceReplication
  File "/usr/local/lib/python3.7/site-packages/purity_fb/purity_fb_1dot9/models/performance_replication.py", line 47
    def __init__(self, id=None, name=None, async=None, time=None):
                                               ^
SyntaxError: invalid syntax

https://github.com/purestorage/purity_fb_python_client/blob/master/purity_fb/purity_fb_1dot9/models/performance_replication.py#L47

In Python 3.7, async is a reserved keyword.

I am not familiar with purity_fb's code base, so not sure if this could be an easy fix. Can we peg the base image of the exporter to be python:3.6-alpine for now? Happy to make a PR if this is an acceptable temporary solution.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.