Comments (18)
With bucketed data, you are taking multiple data points within a timeframe and performing calculating on them.
So lets say you are asking for a bucket which corresponds to an hour of time. For all of the data points between that time frame:
min: the data point with the lowest value
max: the data point with the highest value
avg: the average of all the data points
mean: the mean of all the data point
For information over what each metric contains, please see https://github.com/kubernetes/heapster/blob/master/docs/storage-schema.md Each metric also has this information stored as a tag within the metric itself.
So from that heapster storage schema, CPU is "The number of millisecond since the container was started".
It may seem like a weird metric at first, but combining it with the 'uptime' metric and the cpu limit metrics, you can determine things millicore usage as well as percentage usage based on specified limit (if a limit was created).
If you want to graph something with the CPU usage metric, its probably better to request the rate of change between buckets instead of the absolute value of the bucket.
If you can let me know what exactly you are trying to do from the CPU usage, I may be able to help.
from origin-metrics.
@mwringe : Thanks for the reply.
From https://github.com/kubernetes/heapster/blob/master/docs/storage-schema.md
the "The number of millisecond since the container was started" is the uptime.
I had read earlier from cadvisor docs, that the CPU usage value(in nanoseconds) is the delta of cumulative CPU usage from the beginning of the minute to the end of the minute - https://github.com/catalyzeio/cadvisor-metrics
So as per your explanation above if my bucketDuration is specified as 8hours, hawkular collects datapoints between this duration to then provide me with min, max, avg etc of cpu usage. However if cpu usage is calculated from beginning of a minute to end of the minute(for every datapoint), the min/max/avg collected from all the datapoints in the duration of 8hrs, should never exceed "a minute"; which was not the case when I verified it using the hawkular metrics.
If my understanding above is incorrect, could you kindly give me some insight on how cpu usage is calculated for a datapoint
Regarding your question :what exactly am I trying to do:
- I need to aggregate the metrics data over a period of 1 month to implement a charging solution
- I also need to use the metrics data to trigger some actions (like scaling up/down) etc..
Thanks in advance.
from origin-metrics.
I had read earlier from cadvisor docs, that the CPU usage value(in nanoseconds) is the delta of cumulative CPU usage from the beginning of the minute to the end of the minute - https://github.com/catalyzeio/cadvisor-metrics
Those instructions are specific to the catalyzeio implementation and should not be used as a reference for origin-metrics. We don't have this minute by minute concept that you are referencing.
/hawkular/metrics/counters/data
gives you the raw data, this does not include any deltas.
Lets say you are dealing with the raw data and data which is always increasing. For a bucket of 1 hour, the min should correspond to the value at the start of the time window and the max should correspond to the value at the end of the window (since the data always increases, if the data decreases this is no longer the case). The value of the max minus the min should give you the usage for that particular time window.
The same can be done for the uptime. the max minus the min should give you the uptime for the container during that time window.
NOTE: the above assumes that the container wasn't restarted, in which case things may be reset back to zero.
/hawkular/metrics/counters/rate
will return you deltas.
Using rate data, or deltas, will give you a different result. For a rate bucket, the max will give you the greatest rate of change between any two points, not the greatest value of any given point.
Hopefully that helps.
from origin-metrics.
@mwringe Thanks again for the details.
When I tried to get the rate, it does not return me any data. Could you kindly see if the query is ok?
curl -H "Authorization: Bearer tyLZLyH66S3PiA00kgZXqhN5VeMCscwB7vVcRBbg-nQ" -H "Hawkular-tenant: ametrics" -H "Accept: application/json" -X GET https://172.30.231.219/hawkular/metrics/counters/rate?'tags=descriptor_name:cpu/usage&bucketDuration=2h' --insecure
Thanks in advance.
from origin-metrics.
I have updated the docs a bit to explain how the CPU metric can be used to extract more useful information: https://github.com/openshift/origin-metrics/blob/master/docs/hawkular_metrics.adoc#calcuating-percentage-cpu-usage
Unfortunately, this requires the client to calculate some calculations. I am going to see if we can be done to support this in a better way in Hawkular Metrics.
I realize docs are not the clearest there in terms of what it is doing or how to use it, I will see if I can clean it up a bit and perhaps add in a script which will show a more concrete example implementation.
I think the reason your last curl command is failing is due to https://issues.jboss.org/browse/HWKMETRICS-332 which should be fixed in the next Hawkular Metrics release.
The work around for now would be to first do the query on /hawkular/metrics/metrics?tags=....
That should return the metrics you want, you will then need to grab the metric id from that query and fetch the data directly.
from origin-metrics.
Is there anything else we can help here? Or has the previous comments helped to resolve the issues?
from origin-metrics.
We are still thinking as to how we can use this... We are currently collecting the metrics at the namespace level and we only have uptime property when we collect metrics at the pod level and not at namespace level
from origin-metrics.
@shilpapadgaonkar : hope you are doing good :) , I read conversation here , and found this matches with my usecase I have for openshift v3. So wanted to check were you able to get cpu/usage for defined timeperiod (lets say for a week or month).
I also tried using hawkular REST API , output is not very much clear. As explained by mwringe "The value of the max minus the min should give you the usage for that particular time window." I am assuming your time period here is 8 hrs.
could you help me understand that.
Thanks a lot
Yash
from origin-metrics.
@yashu5 : Since the documentation refers to using uptime to get the % usage, we are still trying to see how we can use it, as we are trying to get namespace/project level metrics where we do not have an uptime property. I will let you know when I come up with something..
from origin-metrics.
@shilpapadgaonkar : Thanks a lot for your quick response, appreciate a lot ! I was able to get uptime metrics for a namespace, may be recent changes
also need to know about filesystem/usage metrics , could you please tell me whats "resource_id" here https://github.com/kubernetes/heapster/blob/master/docs/storage-schema.md
from origin-metrics.
@yashu5 : You can use this docu reference to get node metrics
https://github.com/openshift/origin-metrics/blob/master/docs/hawkular_metrics.adoc#accessing-node-metrics
from origin-metrics.
@shilpapadgaonkar thanks for response, I am looking for pod or project level network/filesystem metrics, is there way we can get this?
from origin-metrics.
@yashu5: Filesystem usage AFAIK are node level metrics.
For network @pod level, this is coming soon -- kubernetes-retired/heapster#852
To see what metrics in general are available at project/namespace/tenant level just use as below (where hawkular-tenant = project name)
curl -H "Authorization: Bearer XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
-H "Hawkular-tenant: test"
-X GET https://hawkular-metrics.example.com/hawkular/metrics/metrics
From this you can find out individual pod ids which you can query for to find out more details at individual pod level
from origin-metrics.
@shilpapadgaonkar : thanks a lot :) very helpful
from origin-metrics.
@shilpapadgaonkar : was curious to know , were you able to implement charging solution for you infrastructure? did you use any available tools for this? if yes could you please tell me which tool?
thanku so much!
from origin-metrics.
@yashu5 i did not find anything at least a couple of months ago.. i haven't searched for it recently.. so we decided to write one.. do let me know if you come across one later :)
from origin-metrics.
@shilpapadgaonkar sure! same here :( I just came across Appdirect but not sure how this can be used with openshift.
from origin-metrics.
Housekeeping to close older issues. If you think this issue is not resolved, please reopen it.
from origin-metrics.
Related Issues (20)
- cassandra stops cluster - Digest mismatch exception HOT 5
- set certificate for url of openshift HOT 4
- Upgrade to Cassandra 3.0.15 HOT 1
- Is there a way to curl metrics without Authorization token in a pod? HOT 5
- How do to deploy metrics components manually in Openshift origin? HOT 8
- Hawkular-Cassandra pod consuming all memory assigned to it HOT 5
- Hawkular-metrics pod in CrashLoopBackOff HOT 5
- Cassandra Request timeout in Hawkular cannot be changed from default value. HOT 6
- Documentation Issue: Ansible openshift_metrics_cassandra_storage_type values HOT 4
- Origin-metrics Docker tags: No tag for v3.9.0 HOT 6
- Is this code supposed to do anything? As far as I can tell, this JVM_OPTS env var is not being leveraged. HOT 5
- Exporting data in cassandra HOT 6
- Access metrics from Hawkular-cassandra pod HOT 4
- metrics integration with log HOT 6
- Metrics startup fails with v3.10 because of missing corresponding images at the docker.io registry HOT 19
- hawkular-alerts not in openshift 3.9 HOT 5
- Document Old Cassandra Snapshots Clearing Process HOT 3
- Deployer container based on origin:latest HOT 4
- Readiness probe failing in pod: invalid literal for int() with base 10 HOT 4
- Got an error messages from hawkular-metrics HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from origin-metrics.