Copyright (C) 2018 The Open Library Foundation

This software is distributed under the terms of the Apache License, Version 2.0. See the file "LICENSE" for more information.

• Jenkins on Linux with default plugins installed

• Extra Jenkins plugins: Pipeline Utility Steps, HTTP Request Plugin, SSH Agent Plugin, CloudBees AWS Credentials Plugin, Performance

• Install aws-cli and JMeter on Jenkins

• Jenkins access to AWS

This is a list of metrics that are gathered during this experiment:

• Average response times (ART) for each transaction
• Min and Max response times
• Median
• Failure rate and errors/warnings in the logs
• Throughtput

These metrics are collected as part of Performance Report which are generated as build artifact

Tools used for these test cases is JMeter - https://jmeter.apache.org/ Utilized non-GUI JMeter.

For example, non-GUI commandline to generate report:

jmeter -Jjmeter.save.saveservice.output_format=xml -n -l jmeter_perf.jtl -t Folio-Test-Plans/mod-inventory-storage/instance-storage/instance-storageTestPlan.jmx -j jmeter_46_instance-storageTestPlan.jmx.log

Reports are generating in Jenkins using JMeter Performance plugin - https://wiki.jenkins.io/display/JENKINS/Performance+Plugin

Tests are scheduled to run in Jenkins pipeline - https://jenkins-aws.indexdata.com/job/Automation/job/folio-perf-test/

• The average response time (AVG RT) for the JMeter captured transaction should not be more than 1000 milliseconds.
• The percent of CPU utilization on any module should not be more than 50%.
• JMeter tests running nightly in Jenkins pipeline as a job will fail if even a single test fails

• Engine: PostgreSQL 9.6.8
• Entire Stack(environment) is created fresh from scratch everyday by populating dataset in database then running JMeter on top of it and once tests complete running, tear down the environment.
• JMeter scripts are running against ~3 million Harvard dataset

• Create new data by doing POST HTTP request, run JMeter tests and clean it by doing DELETE HTTP request once test completes.

• Import in Jenkins as standard pipeline project

• Pick a Jenkinsfile and adjust build parameters as needed

See project FOLIO at the FOLIO issue tracker.

Other infrastructure projects are described, with further FOLIO Developer documentation at dev.folio.org