collect() costs a lot of time about nginx-lua-prometheus HOT 8 CLOSED

This looks like https://github.com/knyar/nginx-lua-prometheus#caveats
How many metrics do you have? You can run curl -s localhost:9145/metrics | grep -c -v '^#' to check.

from nginx-lua-prometheus.

thanks for reply,more than 60,000 records,most time,two or three seconds can get all metrics. exceptional cases cost a lot; any advice ?

from nginx-lua-prometheus.

Yes, 60000 sounds like a lot. You'll need to reduce the number of metrics, or add support for external metric storage (like redis) instead of using the lua dictionary.

Keep in mind, that the metric dictionary is locked while metrics are exposed via /metrics, and it's likely that other HTTP requests for nginx are being blocked while that happens (i.e. every time you fetch metrics, other requests might experience higher latency).

from nginx-lua-prometheus.

https://groups.google.com/forum/#!topic/openresty/8MX33Dw-E38
hi,following this page,i add ngx.flush(true) to collect() long-time wait reduced greatly.
Unfortunately, this page is Chinese...

from nginx-lua-prometheus.

That's a useful data point, thanks. Could you please try applying the following patch (instead of adding ngx.flush) and check if it makes things faster for you?

diff --git a/prometheus.lua b/prometheus.lua
index 0a62166..e6e8194 100644
--- a/prometheus.lua
+++ b/prometheus.lua
@@ -504,25 +504,30 @@ function Prometheus:collect()
   table.sort(keys)
 
   local seen_metrics = {}
+  local output = {}
   for _, key in ipairs(keys) do
     local value, err = self.dict:get(key)
     if value then
       local short_name = short_metric_name(key)
       if not seen_metrics[short_name] then
         if self.help[short_name] then
-          ngx.say("# HELP " .. self.prefix .. short_name .. " " .. self.help[short_name])
+          table.insert(output, string.format("# HELP %s%s %s\n",
+            self.prefix, short_name, self.help[short_name]))
         end
         if self.type[short_name] then
-          ngx.say("# TYPE " .. self.prefix .. short_name .. " " .. self.type[short_name])
+          table.insert(output, string.format("# TYPE %s%s %s\n",
+            self.prefix, short_name, self.type[short_name]))
         end
         seen_metrics[short_name] = true
       end
       -- Replace "Inf" with "+Inf" in each metric's last bucket 'le' label.
-      ngx.say(self.prefix .. key:gsub('le="Inf"', 'le="+Inf"'), " ", value)
+      table.insert(output, string.format("%s%s %d\n",
+        self.prefix, key:gsub('le="Inf"', 'le="+Inf"'), value))
     else
       self:log_error("Error getting '", key, "': ", err)
     end
   end
+  ngx.print(output)
 end
 
 return Prometheus

from nginx-lua-prometheus.

Great,~10s(ngx.say+ngx.flush) to ~2s(table),Not seen long-time wait yet .
A little different: an type error for "value" here,so i use "%s%s %s\n",

table.insert(output, string.format("%s%s %d\n",self.prefix, key:gsub('le="Inf"', 'le="+Inf"'), value))

from nginx-lua-prometheus.

I am glad it helped. I will merge this into the master branch soon.

an type error for "value" here,so i use "%s%s %s\n",

Good catch, you probably have metric values which are floats rather than integers.

from nginx-lua-prometheus.

many thanks for your help👻

from nginx-lua-prometheus.