Libraries and utilities for testing, profiling, benchmarking, and debugging performance bottlenecks on the ManageIQ application. Includes, but not limited to:
- Middleware for injecting performance monitoring per request by header
- Executables for testing endpoint performance
- Railties for convenience including portions of this into the project
- Automated performance reporting
The goal of this project is to aid in pro actively determining performance issues and provide a mechanism for easier debugging when issues arise out in the field.
Add this line to your ManageIQ/manageiq Gemfile.dev.rb:
gem 'manageiq-performance', :github => 'ManageIQ/manageiq-performance'
Since this is meant to be modular, you can include specific parts of the gem
that you need in the ManageIQ application by modifying the :require
statement
in the above gem
invocation:
gem 'manageiq-performance', :require => [ 'manageiq_performance/railtie/rake_tasks',
'manageiq_performance/railtie/middleware' ],
:github => 'ManageIQ/manageiq-performance'
This will only then load the railties for the rake_tasks
and the
middleware
, but nothing else (currently, this is all there is...)
To make use of some of the middleware components and other gem specific
features, you will also need to include their corresponding gems. For example,
the stackprof
middleware requires the
stackprof
gem, so include it by
including the following in your Gemfile.dev.rb
:
gem 'stackprof', :require => false
The middleware included in this gem is a modular middleware, and will add components based on if the gem required by the component is present.
The three middleware components currently available are:
active_support_timers
active_record_queries
stackprof
Each will write their data to a directory that is configured on application
boot, something like tmp/manageiq_performance/run_123456789
, where 123456789
is
a timestamp of when the application was booted.
From there, any request with the header WITH_PERFORMANCE_MONITORING
, will
then run the component reporters and output the results to that directory.
Results are grouped by path, so requests to /
will show up in a subdirectory
of root
, and requests to /vm_infa/explorer
will be put in
vm_infra-explorer
:
tmp/
├── manageiq_performance
│ └── run_1472612126
│ ├── root
│ │ ├── request_1472612151930416.info
│ │ ├── request_1472612151930416.queries
│ │ └── request_1472612151930416.stackprof
│ └── vm_infra-explorer
│ ├── request_1472612151930416.info
│ ├── request_1472612151930416.queries
│ └── request_1472612151930416.stackprof
This acts similarly to the Started
/Completed
timers found in the log, and
is implemented in a similar fashion to this blog
post
by Basecamp. This will write an .info
file that will include the controller,
action, path, status, and timers (total, views, and db) of the request.
Also making use of ActiveSupport::Notifications
, these currently log the
following specifics in regards to the queries made by your application on a
request:
- Total number of sql queries made in during the request
- Data about each query fired, which includes the following:
- SQL generated for the query
- Time taken for the query to be executed in the DB
params
passed into the query
- Total number of rows returned
- Number of rows returned per class
This will run the stackprof profiler for the request, and output it to a
.stackprof
file.
The Stackprof
middleware component can currently be configured using HTTP
headers on a per request basis:
MIQ_PERF_STACKPROF_RAW
will configured theraw
output, which is require if you want to use the stackprof output to generate a flamegraph (defaults totrue
)MIQ_PERF_STACKPROF_MODE
will change the profiling mode. You can choose between 'wall', 'cpu', and 'object'. Default is 'wall'.MIQ_PERF_STACKPROF_INTERVAL
will change how often the stack is profiled. Default is1000
(in microseconds).
More info can be found on the github page
manageiq-performance
comes with 3 rake tasks:
rake manageiq_performance:build_request_file
rake manageiq_performance:benchmark
rake manageiq_performance:benchmark_url["<REQUEST_PATH>"]
This will build a plaintext Requestfile
in the following format:
GET => '/'
POST => '/vm/tree_select
GET => '/vm/show/12345'
To do it, it will connect to the database for your rails application to fill in IDs for requests, and other params necessary to successfully make a request (currently only fills in IDs, but more options are planned).
This will generate or load a previously generated Requestfile
, and make
requests in serial against the application, using the
WITH_PERFORMANCE_MONITORING
and the MIQ_PERF_TIMESTAMP
headers for each
request.
You can configure the endpoint for the host
using the MIQ_HOST
or
CFME_HOST
environment variables. Defaults to http://localhost:3000
.
Similar to the benchmark
task, but is used for one off requests. Can be used
like so:
$ rake manageiq_performance:benchmark_url["/vm/show/123456"]
Currently this only supports GET
requests, but will also accept the
MIQ_HOST
and CFME_HOST
variables.
- Some tests.. hey, what a thought!
- Add configuration file of some sort
- Add Profiling around MiqQueue
- Better Requestfile generation
- Handle CSRF Token better in
Requestor
- Include support for headless browser requests through Phantom.js/Poltergeist
- Add more middleware components (
statsd
,rake-mini-profiler
, etc.) - Human readable report generation
- Tooling around data analysis and debugging
Bug reports and pull requests are welcome on GitHub at https://github.com/ManageIQ/manageiq-performance.