Peel is a framework that assists you in the execution and results analysis of experiments evaluating massively-parallel algorithms and systems.

• Specify and maintain a collection of reusable experiments
• Use default parameters or configure your systems with your favorite settings
• Run different experiments on a variety of systems
• Evaluate and compare the performance of your systems

Peel automatically sets up the systems that you specify while taking care of dependency-relations. If you want to change parameters between experiments in your suite, peel updates the corresponding systems and their dependants automatically. All you need to do is to specify your experiments and let peel do the rest!

Supported Systems

• Flink 0.5.1 & 0.6
• Hadoop 1.2.1 (HDFS and MapReduce)
• Spark 1.1.0

The peel command line interface offers a comprehensive help of the available commands:

./peel -h

Please refer to the help screens for more detailed information.

If you are not sure if the configuration and the experiment jars work fine, you can run a single experiment with the following sequence of commands:

# 1. set-up all systems systems 
./peel exp:setup ${SUITE} ${EXPERIMENT} --experiments ${EXPFILE}
# 2. just run the experiment, skip system set-up and tear-down
./peel exp:run ${SUITE} ${EXPERIMENT} --experiments ${EXPFILE} --run 1 --just
# 3. run the experiment without system set-up or tear-down
./peel exp:teardown ${SUITE} ${EXPERIMENT} --experiments ${EXPFILE}

Halt and repeat the second step if your algorithm does not terminate or is too slow. When you are done, make sure you execute the third step to shut everything down.

To run a you use the suite:run command and specify the experiments file and the id of the suite in your fixture:

./peel suite:run --experiments ${EXPFILE} ${SUITE}

Results from the suite experiments are written into the ${app.path.results}/${suite}/${experiment.name} folders. The state.json file in each folder contains information about the exit state of the experiment. Per default, running the same suite again will skip experiments that returned a zero exit code. If you want to rerun a particular experiment, delete the corresponding experiment folder before you rerun the suite.

In experiments.wordcount.xml you can see a minimal example that runs the example Stratosphere wordcount job with one, two, three, and four slaves. The following table provides a short descriptions of the beans configured in this file.

To setup your own experiment for your algorithm, you just have to create a fixture describing the experiment. The experiments.template.xml contains preconfigured beans for all data sets as well as inline pointers for adapting the file to your algorithm. You can use this file as a starting point for your configuration. Overall, the required steps are:

1. Introduce a new suffix for your experiment.
2. Replace the command string with the right path to your jar, input and output file.
3. Replace the referenced data set with the one suitable for your experiment.

Placeholders in the fixture files (e.g. ${app.path.datasets}) are references to values specified in configuration files. For each experiment, Peel will load and resolve the following hierarchy of configuration files:

1. Reference application configuration: resource:reference.conf.
2. For each system with bean id systemID:
   1. Reference system configuration reference.${system}.conf.
   2. Custom bean configuration located at ${app.path.config}/${systemID}.conf (optional).
   3. Host-specific custom bean configuration located at ${app.path.config}/${hostname}/${systemID}.conf (optional).
3. Custom application configuration located at ${app.path.config}/application.conf (optional).
4. Custom application configuration located at ${app.path.config}/${hostname}/application.conf (optional).
5. Experiment configuration defined in the config argument of the current experiment.
6. The Java system properties.

This section contains some examples based on the example wordcount experiments file.

To execute run #1 from the wc.single-run experiment in the wc.default suite, type:

./peel exp:setup wc.default wc.single-run --experiments ./config/experiments.wordcount.xml
./peel exp:run wc.default wc.single-run --experiments ./config/experiments.wordcount.xml --run 1 --just
./peel exp:teardown wc.default wc.single-run --experiments ./config/experiments.wordcount.xml

To execute all runs from all experiments in the suite, type:

./peel run-suite --experiments ./config/experiments.wordcount.xml <suit-name>