The data and analysis scripts for case studies on Confidence Composition.
Full paper: arXiv. Feel free to contact the first author with any questions.
- coco-case-studies/: source code for the analysis of the case studies.
- coco-case-studies/mountaincar-logs: the mountain car case study from OpenAI Gym.
- coco-case-studies/uuv-logs: the unmanned underwater vehicle (UUV) pipeline-scanning case study based on the ROS UUV simulator.
The goal of the analysis is to reproduce the performance measurements of confidence monitors as reported in Tables 1 and 2 in the paper references above.
To run the analysis scripts, you would need to dowload and install a free version of the Wolfram Engine:
- Download and install the Wolfram Engine for your operating system.
- Create a Wolfram ID account (if you do not have one).
- Accept the terms of the free license
- Run "wolframscript" in your command line
- Input your Wolfram ID and password into the prompt
You can follow the Docker-based repeatability instructions in the reproduction-instruction.pdf. Follow the build-image-from-scratch route or download the built docker image.
To run the analysis scripts:
- UUV case study
- Neutral calibration: wolframscript <path_to>/uuv-analysis-neutral.wls
- Conservative calibration: wolframscript <path_to>/uuv-analysis-conservative.wls
- Mountain car case study
- Neutral calibration: wolframscript <path_to>/mountaincar-analysis-neutral.wls
- Conservative calibration: wolframscript <path_to>/mountaincar-analysis-conservative.wls
"wolframscript" is equivalent to and can be replaced with "wolfram -script".
The notebook-based analysis allows interactive processing and visualization of the data. You would need to install Wolfram Mathematica, version 12.1+. You can obtain a free trial or you may be at an institution with an academic license. For read-only access, you can download a free Wolfram Player.
The following notebooks are provided:
- composition-utils.nb: utility functions to analyze compositions of confidence monitors. It contains the functionality analogous to the composition-utils-package.wl package.
- mountaincar-analysis.nb: a notebook with the analysis and exploration of the data from the mountain car case study.
- uuv-analysis.nb: a notebook with the analysis and exploration of the data from the underwater vehicle case study.