nikivanstein / gsareport Goto Github PK

View Code? Open in Web Editor NEW

5.0 5.0 0.0 70.7 MB

Global Sensitivity reporting for Explainable AI

License: MIT License

Python 47.95% Jupyter Notebook 4.74% HTML 40.76% CSS 6.55%

explainable-ai machine-learning python sensitivity-analysis

gsareport's People

Contributors

Stargazers

Watchers

gsareport's Issues

Unbalanced number of samples generated

When generating samples through the command

docker run --rm -v "$(pwd)"/data:/data \ ghcr.io/basvanstein/gsareport:main -p /data/problem.json -d /data --sample --samplesize 500

Only the Latin hypercube sampling generates the exact number of samples specified. Sobol and Morris use that input differently and generate many more samples.

Add community guidelines

JOSS requires some community guidelines. Add guidelines for third parties wishing to:

Contribute to the software
Report issues or problems with the software
Seek support

Unable to build container using Dockerfile

I attempted to build the Dockerfile using the command docker build . and got the following error while pip installing the dependencies:

#13 [ 9/10] RUN pip install -r requirements.txt
#13 sha256:90bf66eac24407c6165fc93b5c6371f577684790a59e67c4585a7d2f107480a1
#13 0.797 Requirement already satisfied: appdirs==1.4.4 in /usr/lib/python3.10/site-packages (from -r requirements.txt (line 1)) (1.4.4)
#13 0.914 Collecting argon2-cffi==21.1.0
#13 1.018   Downloading argon2_cffi-21.1.0-cp35-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.whl (96 kB)
#13 1.049 Requirement already satisfied: async-generator==1.10 in /usr/lib/python3.10/site-packages (from -r requirements.txt (line 3)) (1.10)
#13 1.086 Collecting attrs==21.3.0
#13 1.102   Downloading attrs-21.3.0-py2.py3-none-any.whl (61 kB)
#13 1.110 Requirement already satisfied: backcall==0.2.0 in /usr/lib/python3.10/site-packages (from -r requirements.txt (line 5)) (0.2.0)
#13 1.163 Collecting black==22.3.0
#13 1.217   Downloading black-22.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.5 MB)
#13 1.356 Requirement already satisfied: bleach==4.1.0 in /usr/lib/python3.10/site-packages (from -r requirements.txt (line 7)) (4.1.0)
#13 1.408 Collecting bokeh==2.4.2
#13 1.433   Downloading bokeh-2.4.2-py3-none-any.whl (18.5 MB)
#13 2.245 Requirement already satisfied: CacheControl==0.12.6 in /usr/lib/python3.10/site-packages (from -r requirements.txt (line 9)) (0.12.6)
#13 2.246 Requirement already satisfied: cairocffi==1.3.0 in /usr/lib/python3.10/site-packages (from -r requirements.txt (line 10)) (1.3.0)
#13 2.246 Requirement already satisfied: cffi==1.15.0 in /usr/lib/python3.10/site-packages (from -r requirements.txt (line 11)) (1.15.0)
#13 2.246 Requirement already satisfied: chardet==4.0.0 in /usr/lib/python3.10/site-packages (from -r requirements.txt (line 12)) (4.0.0)
#13 2.277 Collecting cloudpickle==2.1.0
#13 2.297   Downloading cloudpickle-2.1.0-py3-none-any.whl (25 kB)
#13 2.303 Requirement already satisfied: colorama==0.4.4 in /usr/lib/python3.10/site-packages (from -r requirements.txt (line 14)) (0.4.4)
#13 2.303 Requirement already satisfied: contextlib2==0.6.0.post1 in /usr/lib/python3.10/site-packages (from -r requirements.txt (line 15)) (0.6.0.post1)
#13 2.601 Collecting cryptography==36.0.1
#13 2.629   Downloading cryptography-36.0.1-cp36-abi3-manylinux_2_24_x86_64.whl (3.6 MB)
#13 2.783 Collecting cycler==0.10.0
#13 2.802   Downloading cycler-0.10.0-py2.py3-none-any.whl (6.5 kB)
#13 2.911 Collecting deap==1.3.1
#13 2.931   Downloading deap-1.3.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (160 kB)
#13 3.126 Collecting debugpy==1.5.1
#13 3.148   Downloading debugpy-1.5.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.9 MB)
#13 3.243 Collecting decorator==5.1.0
#13 3.258   Downloading decorator-5.1.0-py3-none-any.whl (9.1 kB)
#13 3.262 Requirement already satisfied: defusedxml==0.7.1 in /usr/lib/python3.10/site-packages (from -r requirements.txt (line 21)) (0.7.1)
#13 3.314 Collecting dill==0.3.4
#13 3.330   Downloading dill-0.3.4-py2.py3-none-any.whl (86 kB)
#13 3.365 Collecting distlib==0.3.3
#13 3.387   Downloading distlib-0.3.3-py2.py3-none-any.whl (496 kB)
#13 3.435 Collecting distro==1.6.0
#13 3.451   Downloading distro-1.6.0-py2.py3-none-any.whl (19 kB)
#13 3.483 Collecting entrypoints==0.3
#13 3.498   Downloading entrypoints-0.3-py2.py3-none-any.whl (11 kB)
#13 3.624 Collecting fonttools==4.28.5
#13 3.642   Downloading fonttools-4.28.5-py3-none-any.whl (890 kB)
#13 3.677 Requirement already satisfied: html5lib==1.1 in /usr/lib/python3.10/site-packages (from -r requirements.txt (line 27)) (1.1)
#13 3.677 Requirement already satisfied: idna==3.3 in /usr/lib/python3.10/site-packages (from -r requirements.txt (line 28)) (3.3)
#13 3.678 Requirement already satisfied: importlib-metadata==4.8.1 in /usr/lib/python3.10/site-packages (from -r requirements.txt (line 29)) (4.8.1)
#13 3.733 Collecting ipykernel==6.6.0
#13 3.750   Downloading ipykernel-6.6.0-py3-none-any.whl (126 kB)
#13 3.835 Collecting ipython==7.30.1
#13 3.853   Downloading ipython-7.30.1-py3-none-any.whl (791 kB)
#13 3.911 Collecting ipython-genutils==0.2.0
#13 3.925   Downloading ipython_genutils-0.2.0-py2.py3-none-any.whl (26 kB)
#13 3.982 Collecting ipywidgets==7.6.5
#13 4.028   Downloading ipywidgets-7.6.5-py2.py3-none-any.whl (121 kB)
#13 4.100 Collecting isort==5.10.1
#13 4.115   Downloading isort-5.10.1-py3-none-any.whl (103 kB)
#13 4.124 Requirement already satisfied: jedi==0.18.1 in /usr/lib/python3.10/site-packages (from -r requirements.txt (line 35)) (0.18.1)
#13 4.124 Requirement already satisfied: Jinja2==3.0.3 in /usr/lib/python3.10/site-packages (from -r requirements.txt (line 36)) (3.0.3)
#13 4.172 Collecting joblib==1.1.0
#13 4.190   Downloading joblib-1.1.0-py2.py3-none-any.whl (306 kB)
#13 4.206 Requirement already satisfied: jsonschema==3.2.0 in /usr/lib/python3.10/site-packages (from -r requirements.txt (line 38)) (3.2.0)
#13 4.257 Collecting jupyter-client==7.1.0
#13 4.276   Downloading jupyter_client-7.1.0-py3-none-any.whl (129 kB)
#13 4.362 Collecting jupyter-console==6.4.0
#13 4.379   Downloading jupyter_console-6.4.0-py3-none-any.whl (22 kB)
#13 4.429 Collecting jupyter-core==4.9.1
#13 4.443   Downloading jupyter_core-4.9.1-py3-none-any.whl (86 kB)
#13 4.478 Collecting jupyterlab-pygments==0.1.2
#13 4.499   Downloading jupyterlab_pygments-0.1.2-py2.py3-none-any.whl (4.6 kB)
#13 4.503 Requirement already satisfied: kiwisolver==1.3.2 in /usr/lib/python3.10/site-packages (from -r requirements.txt (line 43)) (1.3.2)
#13 4.602 Collecting llvmlite==0.38.1
#13 4.617   Downloading llvmlite-0.38.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
#13 5.768 Collecting MarkupSafe==2.0.1
#13 5.785   Downloading MarkupSafe-2.0.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (30 kB)
#13 5.992 Collecting matplotlib==3.5.1
#13 6.012   Downloading matplotlib-3.5.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.9 MB)
#13 6.373 Requirement already satisfied: matplotlib-inline==0.1.3 in /usr/lib/python3.10/site-packages (from -r requirements.txt (line 47)) (0.1.3)
#13 6.373 Requirement already satisfied: mistune==0.8.4 in /usr/lib/python3.10/site-packages (from -r requirements.txt (line 48)) (0.8.4)
#13 6.407 Collecting more-itertools==8.10.0
#13 6.422   Downloading more_itertools-8.10.0-py3-none-any.whl (51 kB)
#13 6.429 Requirement already satisfied: msgpack==1.0.3 in /usr/lib/python3.10/site-packages (from -r requirements.txt (line 50)) (1.0.3)
#13 6.472 Collecting multiprocess==0.70.12.2
#13 6.508   Downloading multiprocess-0.70.12.2-py39-none-any.whl (128 kB)
#13 6.556 Collecting nbclient==0.5.4
#13 6.574   Downloading nbclient-0.5.4-py3-none-any.whl (66 kB)
#13 6.625 Collecting nbconvert==6.1.0
#13 6.643   Downloading nbconvert-6.1.0-py3-none-any.whl (551 kB)
#13 6.701 Collecting nbformat==5.1.3
#13 6.717   Downloading nbformat-5.1.3-py3-none-any.whl (178 kB)
#13 6.728 Requirement already satisfied: nest-asyncio==1.5.4 in /usr/lib/python3.10/site-packages (from -r requirements.txt (line 55)) (1.5.4)
#13 6.786 Collecting notebook==6.4.4
#13 6.809   Downloading notebook-6.4.4-py3-none-any.whl (9.9 MB)
#13 7.331 Collecting numba==0.55.2
#13 7.448   Downloading numba-0.55.2-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.4 MB)
#13 7.955 Collecting numpy==1.21.5
#13 8.223   Downloading numpy-1.21.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (15.9 MB)
#13 8.712 Requirement already satisfied: ordered-set==4.0.2 in /usr/lib/python3.10/site-packages (from -r requirements.txt (line 59)) (4.0.2)
#13 8.713 Requirement already satisfied: packaging==21.3 in /usr/lib/python3.10/site-packages (from -r requirements.txt (line 60)) (21.3)
#13 8.920 Collecting pandas==1.3.5
#13 8.939   Downloading pandas-1.3.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (11.5 MB)
#13 9.445 Collecting pandas-bokeh==0.5.5
#13 9.489   Downloading pandas_bokeh-0.5.5-py2.py3-none-any.whl (29 kB)
#13 9.521 Collecting pandocfilters==1.4.3
#13 9.534   Downloading pandocfilters-1.4.3.tar.gz (16 kB)
#13 9.677 Requirement already satisfied: parso==0.8.2 in /usr/lib/python3.10/site-packages (from -r requirements.txt (line 64)) (0.8.2)
#13 9.702 Collecting pathos==0.2.8
#13 9.724   Downloading pathos-0.2.8-py2.py3-none-any.whl (81 kB)
#13 9.733 Requirement already satisfied: pep517==0.12.0 in /usr/lib/python3.10/site-packages (from -r requirements.txt (line 66)) (0.12.0)
#13 9.734 Requirement already satisfied: pexpect==4.8.0 in /usr/lib/python3.10/site-packages (from -r requirements.txt (line 67)) (4.8.0)
#13 9.734 Requirement already satisfied: pickleshare==0.7.5 in /usr/lib/python3.10/site-packages (from -r requirements.txt (line 68)) (0.7.5)
#13 10.06 Collecting Pillow==8.4.0
#13 10.08   Downloading Pillow-8.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.1 MB)
#13 10.18 Requirement already satisfied: ply==3.11 in /usr/lib/python3.10/site-packages (from -r requirements.txt (line 70)) (3.11)
#13 10.21 Collecting pox==0.3.0
#13 10.23   Downloading pox-0.3.0-py2.py3-none-any.whl (30 kB)
#13 10.30 Collecting ppft==1.6.6.4
#13 10.32   Downloading ppft-1.6.6.4-py3-none-any.whl (65 kB)
#13 10.32 Requirement already satisfied: progress==1.6 in /usr/lib/python3.10/site-packages (from -r requirements.txt (line 73)) (1.6)
#13 10.36 Collecting prometheus-client==0.11.0
#13 10.37   Downloading prometheus_client-0.11.0-py2.py3-none-any.whl (56 kB)
#13 10.44 Collecting prompt-toolkit==3.0.24
#13 10.45   Downloading prompt_toolkit-3.0.24-py3-none-any.whl (374 kB)
#13 10.47 Requirement already satisfied: ptyprocess==0.7.0 in /usr/lib/python3.10/site-packages (from -r requirements.txt (line 76)) (0.7.0)
#13 10.50 Collecting pycairo==1.20.1
#13 10.51   Downloading pycairo-1.20.1.tar.gz (344 kB)
#13 10.55   Installing build dependencies: started
#13 12.66   Installing build dependencies: finished with status 'done'
#13 12.66   Getting requirements to build wheel: started
#13 12.83   Getting requirements to build wheel: finished with status 'done'
#13 12.83     Preparing wheel metadata: started
#13 13.00     Preparing wheel metadata: finished with status 'done'
#13 13.00 Requirement already satisfied: pycparser==2.21 in /usr/lib/python3.10/site-packages (from -r requirements.txt (line 78)) (2.21)
#13 13.04 Collecting Pygments==2.11.1
#13 13.10   Downloading Pygments-2.11.1-py3-none-any.whl (1.1 MB)
#13 13.17 Collecting PyGObject==3.42.0
#13 13.19   Downloading PyGObject-3.42.0.tar.gz (716 kB)
#13 13.28   Installing build dependencies: started
#13 16.38   Installing build dependencies: finished with status 'error'
#13 16.38   ERROR: Command errored out with exit status 1:
#13 16.38    command: /usr/bin/python /usr/lib/python3.10/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-dh3zzgid/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel pycairo
#13 16.38        cwd: None
#13 16.38   Complete output (29 lines):
#13 16.38   Collecting setuptools
#13 16.38     Using cached setuptools-65.3.0-py3-none-any.whl (1.2 MB)
#13 16.38   Collecting wheel
#13 16.38     Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
#13 16.38   Collecting pycairo
#13 16.38     Downloading pycairo-1.21.0.tar.gz (340 kB)
#13 16.38     Installing build dependencies: started
#13 16.38     Installing build dependencies: finished with status 'done'
#13 16.38     Getting requirements to build wheel: started
#13 16.38     Getting requirements to build wheel: finished with status 'done'
#13 16.38       Preparing wheel metadata: started
#13 16.38       Preparing wheel metadata: finished with status 'done'
#13 16.38   Building wheels for collected packages: pycairo
#13 16.38     Building wheel for pycairo (PEP 517): started
#13 16.38     Building wheel for pycairo (PEP 517): finished with status 'error'
#13 16.38     ERROR: Command errored out with exit status 1:
#13 16.38      command: /usr/bin/python /usr/lib/python3.10/site-packages/pep517/in_process/_in_process.py build_wheel /tmp/tmp_62d2zbe
#13 16.38          cwd: /tmp/pip-install-zztmw9q1/pycairo_09b188e4764c49b3bf3f8e7c6a7f34a3
#13 16.38     Complete output (6 lines):
#13 16.38     running bdist_wheel
#13 16.38     running build
#13 16.38     running build_py
#13 16.38     running build_ext
#13 16.38     'pkg-config' not found.
#13 16.38     Command ['pkg-config', '--print-errors', '--exists', 'cairo >= 1.15.10']
#13 16.38     ----------------------------------------
#13 16.38     ERROR: Failed building wheel for pycairo
#13 16.38   Failed to build pycairo
#13 16.38   ERROR: Could not build wheels for pycairo which use PEP 517 and cannot be installed directly
#13 16.38   ----------------------------------------
#13 16.38 WARNING: Discarding https://files.pythonhosted.org/packages/cc/72/48cfdd7a0caf3c27f392d2657731ac6f7c3c1c0a60bfeba3e1ba9ffa7ba9/PyGObject-3.42.0.tar.gz#sha256=b9803991ec0b0b4175e81fee0ad46090fa7af438fe169348a9b18ae53447afcd (from https://pypi.org/simple/pygobject/) (requires-python:>=3.6, <4). Command errored out with exit status 1: /usr/bin/python /usr/lib/python3.10/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-dh3zzgid/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel pycairo Check the logs for full command output.
#13 16.38 ERROR: Could not find a version that satisfies the requirement PyGObject==3.42.0
#13 16.38 ERROR: No matching distribution found for PyGObject==3.42.0
#13 ERROR: executor failed running [/bin/sh -c pip install -r requirements.txt]: exit code: 1
------
 > [ 9/10] RUN pip install -r requirements.txt:
------
executor failed running [/bin/sh -c pip install -r requirements.txt]: exit code: 1

[REVIEW]: GSAreport: Easy to Use Global Sensitivity Reporting

Hi @basvanstein and sorry for the late review,

Here are few suggestions and requests,

Numpy version conflict between gsareport and shap (Numpy version should be < = 1.20), may need to update requirements.txt at the src.
Readme is too informative. The "installation instructions" and the " usage" sections are the same in github readme and in the documentation. It would be more convenient to follow the readme if it contains the highly recommended installation/usage and hyperlinks(documentation) for the others, including the 'Building binaries (for developers)' section.
How about some example analysis images/gifs (e.g. headmaps, correlation analysis) in the introduction section of the readme.
References in the paper and in the readme have different citation styles. Citation syntax
It is not required, but I suggest test coverage in the actions. E.g. the plot functions are not included in the automated test.

Let me know what you think of these suggestions.

JOSS Review - References on sampling schme adapation

Reading through the JOSS paper and the Readme of your project I find myself wondering what exactly the added value of your application is. I am not saying it is not there I just don't quite understand why I would not just be using SAlib directly. Also I think you should point out other toolboxes like SAFE.

To be more precise: Please add a little bit more description of what the user explicitly gains from using this software. Be a bit more explicit on how it fits into the world of existing toolboxes out there.

Furthermore, you state that "each method in the original SALib package requires a different sampling scheme and different sample size". Well the sample size is related to convergence and the scheme very much to the underlying methods and has nothing to do with SAlib. Please elaborate what your your tool does exactly. For me this is currently rather confusing.

JOSS review: notes on example usage

The demo mode on the docker image doesn't seem to work
Please provide the files necessary to run the example at https://basvanstein.github.io/GSAreport/1.0/usecases It is nicely described but providing the first json file would make it much easier to get started.
I am a bit confused by the example: you state that the user does not necessary need any python knowledge. But for running the SA in the quickstart you really need to know what you are doing even if it is a simple example. So GSA report is not running the SA for you? It is simplifying the sampling and the analysis but it is not running the experiment. I think that should be clarified in the project and paper description.
the last line in the quickstart as improper line breaks - a simple copy and paste does not work
After running the Qucikstart it provides me with the output: "Created a graph with 3 vertices and 1 edges.
Vertices are the top 5 ST values greater than 0.01.
Only S2 values (edges) greater than 0.0 are included." ok well, what does that tell me? The output folder is empty. How do I get the graphical report?

Update README Docker commands to handle cases where working directory has spaces

The Docker commands will fail if the users local working directory has a space. I recommend changing all instances of:

`pwd` to "$(pwd)"

In addition, specifying docker volumes on their own lines will also present issues. I recommend changing the following command from:

docker run --rm -v `pwd`/output:/output \ 
    -v `pwd`/data:/data \
    ghcr.io/basvanstein/gsareport:main -p /data/problem.json -d /data -o /output

to:

docker run --rm -v "$(pwd)"/output:/output -v "$(pwd)"/data:/data \
    ghcr.io/basvanstein/gsareport:main -p /data/problem.json -d /data -o /output

Include sample CSV files in repository

The example documentation refers to several CSV files in order to conduct a test run of the application. However, there are no samples in the repository. It would be helpful to include some samples so an end user could try out the software without having their own data.

From README.md:

x.csv, y.csv #optional, in case you use an existing design of experiments
x_sobol.csv, y_sobol.csv
x_morris.csv, y_morris.csv
x_lhs.csv, y_lhs.csv

Permission denied when generating samples through python command

Command: python GSAreport.py -p problem.json -d pwd/data --sample --samplesize 1000

Output:

loaded problem definition
  0%|                                                                                                                                                                               | 0/100 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "GSAreport.py", line 756, in <module>
    report = SAReport(
  File "GSAreport.py", line 138, in __init__
    os.makedirs(output_dir)
  File "/usr/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/output/'

Solution: Specify output folder path as last instruction: python GSAreport.py -p problem.json -d pwd/data --sample --samplesize 1000 -o pwd/output

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.