maintainable-security's Introduction

security-maintainability

Pre-print available here: https://arxiv.org/abs/2106.03271. Published at the Empirical Software Engineering Journal.

Installation

Requirements installation:

virtualenv --python=python3.7 venv
source venv/bin/activate
pip install -r requirements.txt

Merge different caches in scripts/maintainability/caches/ folder:

source venv/bin/activate
cd scripts
python -m maintainability.merge_cache -cache maintainability/cache -output maintainability/bch_cache.zip

Data Analysis

How to collect maintainability results from BCH cache:

source venv/bin/activate
cd scripts
python report.py --report export -secdb ../dataset/db_security_changes.csv -regdb ../dataset/db_regular_changes_random.csv -baseline random -results ../results -cache maintainability/bch_cache.zip

Comparison between security and regular commits:

source venv/bin/activate
cd scripts
python report.py --report comparison -results ../results/ -reports ../reports

Get security maintainability report per guideline:

source venv/bin/activate
cd scripts
python report.py --report guideline -secdb ../results/maintainability_release_security_fixes.csv -reports ../reports

Get security maintainability report per language:

source venv/bin/activate
cd scripts
python report.py --report language -secdb ../results/maintainability_release_security_fixes.csv -reports ../reports

Get security maintainability report per severity:

source venv/bin/activate
cd scripts
python report.py --report severity -secdb ../results/maintainability_release_security_fixes.csv -reports ../reports

Get security maintainability report per cwe:

source venv/bin/activate
cd scripts
python report.py --report cwe -secdb ../results/maintainability_release_security_fixes.csv -reports ../reports

Get security maintainability report per specific cwe (available for CWE_664 and CWE_707):

source venv/bin/activate
cd scripts
python report.py --report cwe-spec -secdb ../results/maintainability_release_security_fixes.csv -cwe CWE_664 -reports ../reports

Experiments

How to collect maintainability reports from BCH:

Create a config file in scripts/maintainability/config.json based on the example in scripts/maintainability/config-template.json.

virtualenv --python=python3.7 venv
source venv/bin/activate
pip install -r requirements.txt
cd scripts
python -m maintainability.eval_maintainability

maintainable-security's People

Contributors

Stargazers

Watchers

maintainable-security's Issues

Improve RQ3. Compare sec with 2 baselines (randomly picked vs same size).

Minor issues: Section 4

Section 4:

Page 13, Line 3 - 969 security commits and 969 baseline commits? It's hard to know how to parse that.
Page 13, Line 31 "overall patches …" rather than having us rely on looking at the Figure, tell us the number or percentage of when the impact is positive like you do with the negative in the next sentence. [You do provide the specifics starting on page 15 line 49. I wanted it here.]
Page 19, Line 7 - for the unindoctrinated - I'd explain the concept of Research Concepts. Also explain your choice of 7a and 7b and how representative they are from the several hundred choices. Similar for your explanations in the second two paragraphs on this page. How far down the concept/vulnerability hierarchy tree are these CWE choices or are they lower level vulnerabilities chosen from the 700+ CWE types?
Page 20, Line 42 - how many is "a considerable number of cases"?

Minor issues: Abstract + Info

Abstract:

Throughout the paper, "hypothesize" is probably a better word than "suspect" for sounding more scientific.

Intro:

First sentence - quality is not ONLY related to cost but also to security and safety.
Provide a URL to Software Improvement Group and Better Code Hub.
Page 3, Line 1 - Application Security Verification Standard (ASVS).
Page 3, Line 7 - instead of a "broad number of code metrics" - tell us exactly the number of metrics.
Page 3, Line 13 - "suggest" sounds more scientific than "hint at"
Page 3, Line 24 - "we intend to highlight the need …". Is that the broad goal of your paper? The goal should be explicitly stated in the abstract and intro.