performanceci / performanceci-core Goto Github PK

child of #36

Support entry of external URL and name of repo

With minimal effort, it should be easy for a developer or customer to have a environment tailored for contributions or demonstration

Child of: #32

We now have all supporting infrastructure configuration now driven by pillar data and external formulas. We also have all relevant bits of configuration for the main application represented (or representable) in pillar data. As such, we should be able to represent the functionality of the core application as a salt formula.

As a perfci user, I should be able to use projects that deployed externally and not within perfci. I should be able to trigger a build via a webhook and see build results.

Acceptance Criteria

• be able to configure external projects
• test is run against an external base URL
• webhook that supports build of project

Sub-Tasks

• #37 | UI support to configure an external project
• #38 | Backend to skip git checkout / docker builds and test
• #39 | Provide webhook to trigger builds when new version is deployed

General blog article on continuous performance testing. Blurb about how PerfCI could fit in to this flow. Maybe screencast as well.

As a perfci user, I should be able to run performance tests against authenticated endpoints. This requires ability to track sessions or authentication tokens after issuing log in request

See http://jmeter.apache.org/usermanual/component_reference.html#HTTP_Cookie_Manager for information on how JMeter handles this

Perhaps we can support something like this? Note sure what the best naming is for the save_cookies option.

  -
    name: login
    uri: '/login'
    method: post
    body: 
      username: john
      password: super_secret
    save_cookies: true

Acceptance Criteria

• Support for cookie / session capture after login, and passing these to subsequent requests

Use case

Sub-tasks

• Support for an option to save cookies using an initial url endpoint
• Apply 'set-cookie' header to future requests as a 'Cookie' header
• Supporting token based authentication - how do we extract a token from the login request?

As a perfci user, I should be able to update a project configuration through the UI.

Acceptance Criteria

• UI element for project configuration
• project configuration page displays current config as editable text
• save UI element updates project configuration

Builds are failing because garbage is getting into benchmark results. This is happening because of recent changes to VegetaDriver

As a perfci contributor, I should be able to focus on the main application without concern for underlying infrastructure. I should still be able to vagrant up and get what I do today, without having to bother with salt states or formula management.

Acceptance Criteria

• vagrant up continues to behave as it does today (easy to contribute)
• no salt states or formulas in the core repository

Sub-Tasks

• #33 | Use a salt master
• #34 | Use GitFS for formulas
• #35 | perfci-formula

Web apps typically need additional services as a database, redis, etc.

Some options are:

• Have users define these in their single docker container
• Allow multiple docker containers and use support something like fig / compose
• Provide these services and allow configuration via environment variable / swapping out database.yml file

Child of: #32

In a distributed production environment, one would most likely use a salt-master. We should be able to represent this in our local development environment.

In order to make benchmarks more realistic, it would be great to support production-like datasets in your database. We need to think of what the best way is to support this. A possible feature could be to provide these containers which we store and allow people to use without having to recreate them every time.

We should support running the target containers and http load testing containers (currently vegeta) on different machines.

Child of: #32
Blocked by: #33

We currently have a number of formulas subtree'd into the main repository. These should be available on the salt-master a la gitfs remotes rather than maintaining these subtrees in the main repo.

Make sure vegeta args are valid, otherwise may get JSON error:

> docker run --rm -e TARGET="http://www.yahoo.com/" -e DURATION=1s -e RATE=1 perfci/vegeta
> 2015/07/15 03:07:59 json: unsupported value: +Inf

If can't be fixed, make sure error is understandable and gets propagated to front end

child of #49

VegetaDriver should create a file based on it's ProjectConfigration which will serve as the vegeta attack command's -targets flag file argument.

We should manage simple nohup scripts on the VMs to start core and worker services with salt.

This is more of a reminder to adhere to salt best practices as much as possible to provide more flexibility and portability of infrastructure.

Our current implimentation of pg can easily be replicated by adding a little pillar data and including the community maintained postgres-formula

Currently application can be built using Docker. Fill out HerokuBuilder class to push code to Heroku and instantiate application, and tear it down on cleanup. Would require some changes in UI as well to capture Heroku credentials.

child of #36

Backend worker support for testing external projects. Skip git checkout / docker and test perfci against an external project with provided url.

child of #36

Add a webhook to support triggering of external project

We should pull our vegeta docker files and scripts out of the core repo. We should ultimately submit a patch the main vegeta repo for these changes

As a perfci user, I should be able to create endpoints that are PUT / POST / DELETE with options to specify headers and request boy.

Acceptance Criteria

• Endpoint configuration to support options above (I don't believe any work is needed for this).
• Vegeta driver to support options above
• Vegeta docker image to support options above

Use case

• I would like to use it to test ElasticSearch query behavior - usually ES can have some pretty long queries which go beyond the URL limit.

Sub-tasks

• #49 new configuration options
• cdarwin/vegeta#1 vegeta docker image support for targets file
• #50 VegetaDriver support for targets file

Currently we are using "vegeta" as the load testing tool. Add support for JMeter as well.

We should have some cleanup for images and containers otherwise it will run out of space as new docker images are pulled and run.

• Average response times for each configured endpoint within kallabeez worker
• Record information about undesirable responses from endpoints
• The hash returned to the main worker should include the following for each endpoint:
  • Avg response time
  • Total iterations
  • Successful iterations
  • Response codes of unsuccessful hits

As a user, I should be able to see logs of docker building components, my container starting up / running, and responding to HTTP requests

child of #48

ProjectConfiguration should recognize tunable options for endpoint['method'] (request method) and endpoint['body'] (HTTP message body)