cpu usage metrics about origin-metrics HOT 18 CLOSED

With bucketed data, you are taking multiple data points within a timeframe and performing calculating on them.

So lets say you are asking for a bucket which corresponds to an hour of time. For all of the data points between that time frame:
min: the data point with the lowest value
max: the data point with the highest value
avg: the average of all the data points
mean: the mean of all the data point

For information over what each metric contains, please see https://github.com/kubernetes/heapster/blob/master/docs/storage-schema.md Each metric also has this information stored as a tag within the metric itself.

So from that heapster storage schema, CPU is "The number of millisecond since the container was started".

It may seem like a weird metric at first, but combining it with the 'uptime' metric and the cpu limit metrics, you can determine things millicore usage as well as percentage usage based on specified limit (if a limit was created).

If you want to graph something with the CPU usage metric, its probably better to request the rate of change between buckets instead of the absolute value of the bucket.

If you can let me know what exactly you are trying to do from the CPU usage, I may be able to help.

from origin-metrics.

@mwringe : Thanks for the reply.

From https://github.com/kubernetes/heapster/blob/master/docs/storage-schema.md
the "The number of millisecond since the container was started" is the uptime.

I had read earlier from cadvisor docs, that the CPU usage value(in nanoseconds) is the delta of cumulative CPU usage from the beginning of the minute to the end of the minute - https://github.com/catalyzeio/cadvisor-metrics

So as per your explanation above if my bucketDuration is specified as 8hours, hawkular collects datapoints between this duration to then provide me with min, max, avg etc of cpu usage. However if cpu usage is calculated from beginning of a minute to end of the minute(for every datapoint), the min/max/avg collected from all the datapoints in the duration of 8hrs, should never exceed "a minute"; which was not the case when I verified it using the hawkular metrics.

If my understanding above is incorrect, could you kindly give me some insight on how cpu usage is calculated for a datapoint

Regarding your question :what exactly am I trying to do:

1. I need to aggregate the metrics data over a period of 1 month to implement a charging solution
2. I also need to use the metrics data to trigger some actions (like scaling up/down) etc..

Thanks in advance.

from origin-metrics.

I had read earlier from cadvisor docs, that the CPU usage value(in nanoseconds) is the delta of cumulative CPU usage from the beginning of the minute to the end of the minute - https://github.com/catalyzeio/cadvisor-metrics

Those instructions are specific to the catalyzeio implementation and should not be used as a reference for origin-metrics. We don't have this minute by minute concept that you are referencing.

/hawkular/metrics/counters/data gives you the raw data, this does not include any deltas.

Lets say you are dealing with the raw data and data which is always increasing. For a bucket of 1 hour, the min should correspond to the value at the start of the time window and the max should correspond to the value at the end of the window (since the data always increases, if the data decreases this is no longer the case). The value of the max minus the min should give you the usage for that particular time window.

The same can be done for the uptime. the max minus the min should give you the uptime for the container during that time window.

NOTE: the above assumes that the container wasn't restarted, in which case things may be reset back to zero.

/hawkular/metrics/counters/rate will return you deltas.

Using rate data, or deltas, will give you a different result. For a rate bucket, the max will give you the greatest rate of change between any two points, not the greatest value of any given point.

Hopefully that helps.

from origin-metrics.

@mwringe Thanks again for the details.

When I tried to get the rate, it does not return me any data. Could you kindly see if the query is ok?

curl -H "Authorization: Bearer tyLZLyH66S3PiA00kgZXqhN5VeMCscwB7vVcRBbg-nQ" -H "Hawkular-tenant: ametrics" -H "Accept: application/json" -X GET https://172.30.231.219/hawkular/metrics/counters/rate?'tags=descriptor_name:cpu/usage&bucketDuration=2h' --insecure

Thanks in advance.

from origin-metrics.

I have updated the docs a bit to explain how the CPU metric can be used to extract more useful information: https://github.com/openshift/origin-metrics/blob/master/docs/hawkular_metrics.adoc#calcuating-percentage-cpu-usage

Unfortunately, this requires the client to calculate some calculations. I am going to see if we can be done to support this in a better way in Hawkular Metrics.

I realize docs are not the clearest there in terms of what it is doing or how to use it, I will see if I can clean it up a bit and perhaps add in a script which will show a more concrete example implementation.

I think the reason your last curl command is failing is due to https://issues.jboss.org/browse/HWKMETRICS-332 which should be fixed in the next Hawkular Metrics release.

The work around for now would be to first do the query on /hawkular/metrics/metrics?tags=....
That should return the metrics you want, you will then need to grab the metric id from that query and fetch the data directly.

from origin-metrics.

Is there anything else we can help here? Or has the previous comments helped to resolve the issues?

from origin-metrics.

We are still thinking as to how we can use this... We are currently collecting the metrics at the namespace level and we only have uptime property when we collect metrics at the pod level and not at namespace level

from origin-metrics.

@shilpapadgaonkar : hope you are doing good :) , I read conversation here , and found this matches with my usecase I have for openshift v3. So wanted to check were you able to get cpu/usage for defined timeperiod (lets say for a week or month).

I also tried using hawkular REST API , output is not very much clear. As explained by mwringe "The value of the max minus the min should give you the usage for that particular time window." I am assuming your time period here is 8 hrs.

could you help me understand that.

Thanks a lot
Yash

from origin-metrics.

@yashu5 : Since the documentation refers to using uptime to get the % usage, we are still trying to see how we can use it, as we are trying to get namespace/project level metrics where we do not have an uptime property. I will let you know when I come up with something..

from origin-metrics.

@shilpapadgaonkar : Thanks a lot for your quick response, appreciate a lot ! I was able to get uptime metrics for a namespace, may be recent changes

also need to know about filesystem/usage metrics , could you please tell me whats "resource_id" here https://github.com/kubernetes/heapster/blob/master/docs/storage-schema.md

from origin-metrics.

@yashu5 : You can use this docu reference to get node metrics
https://github.com/openshift/origin-metrics/blob/master/docs/hawkular_metrics.adoc#accessing-node-metrics

from origin-metrics.

@shilpapadgaonkar thanks for response, I am looking for pod or project level network/filesystem metrics, is there way we can get this?

from origin-metrics.

@yashu5: Filesystem usage AFAIK are node level metrics.
For network @pod level, this is coming soon -- kubernetes-retired/heapster#852

To see what metrics in general are available at project/namespace/tenant level just use as below (where hawkular-tenant = project name)

curl -H "Authorization: Bearer XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
-H "Hawkular-tenant: test"
-X GET https://hawkular-metrics.example.com/hawkular/metrics/metrics

From this you can find out individual pod ids which you can query for to find out more details at individual pod level

from origin-metrics.

@shilpapadgaonkar : thanks a lot :) very helpful

from origin-metrics.

@shilpapadgaonkar : was curious to know , were you able to implement charging solution for you infrastructure? did you use any available tools for this? if yes could you please tell me which tool?

thanku so much!

from origin-metrics.

@yashu5 i did not find anything at least a couple of months ago.. i haven't searched for it recently.. so we decided to write one.. do let me know if you come across one later :)

from origin-metrics.

@shilpapadgaonkar sure! same here :( I just came across Appdirect but not sure how this can be used with openshift.

from origin-metrics.

Housekeeping to close older issues. If you think this issue is not resolved, please reopen it.

from origin-metrics.