rkkautsar / reprobench Goto Github PK

View Code? Open in Web Editor NEW

1.0 3.0 6.0 9.82 MB

📊 Reproducible Benchmark for Everyone

License: MIT License

Python 94.60% Jupyter Notebook 5.40%

research-tool benchmarking benchmark-framework reproducible-research reproducibility reproducible-experiments

reprobench's Introduction

reprobench

Development Guide

Installing locally

We recommend using Anaconda.

$ conda env create -f environment.yml
$ conda activate reprobench

reprobench's People

Contributors

Stargazers

Watchers

Forkers

daajoe trellixvulnteam

reprobench's Issues

Reserve -h cli option for help

Rename the -h host option into -J or something (similar to ssh).

Missing handling of pstree killing

When the timeout hits and the subprocess is killed not all subprocesses (children of the subprocess) are send the signal as well, which potentially results in zombies.

SLURM manager is broken after run id changes

Previously job array is used because run id is integers.

Number of used HT cores

Set default no of HT cores to HT_cores/2
Allow to change the number of used cores in the benchmark config.

Drop SSH Tunneling and implement end-to-end encryption

Related: #42

Combine bootstrap and update

The bootstrap extra step is unnecessary. Whenever a manager starts a connection with the server, a bootstrap can always be done to update the state in the server.

Insecure TCP connection between client and server

Opening a 0.0.0.0:... port is bad security practice. Consider using ssh tunnel or implement key exchange and end-to-end encryption.

Dependabot couldn't fetch all your path-based dependencies

Dependabot couldn't fetch one or more of your project's path-based Python dependencies. The affected dependencies were psmon/pyproject.toml.

To use path-based dependencies with Dependabot the paths must be relative and resolve to a directory in this project's source code.

You can mention @dependabot in the comments below to contact the Dependabot team.

Exceptions when trying to benchmark sudoku-sat solver

I'm getting a crash when trying to run the benchmark or the analyzer after that. Installation and starting the server went fine before.

2019-06-05 00:02:08.061 | INFO     | reprobench.core.bootstrap.client:bootstrap_tasks:10 - Bootstrapping tasks...
2019-06-05 00:02:08.077 | INFO     | reprobench.core.bootstrap.client:bootstrap_tools:33 - Setting up tools...
2019-06-05 00:02:08.079 | INFO     | reprobench.core.bootstrap:cli:39 - Sending bootstrap event to server
Executing runs:   0%|                                                                                                                                                    | 0/32 [00:00<?, ?it/s]2019-06-05 00:02:08.440 | INFO     | reprobench.core.worker:run:48 - Processing task: output/my_solver.MySudokuSolver/clasp/sudoku/table16-2.txt
2019-06-05 00:02:08.444 | INFO     | reprobench.core.worker:run:48 - Processing task: output/my_solver.MySudokuSolver/clasp/sudoku/table25-2.txt
2019-06-05 00:02:08.447 | INFO     | reprobench.core.worker:run:48 - Processing task: output/my_solver.MySudokuSolver/clasp/sudoku/table49-1.txt
2019-06-05 00:02:08.452 | INFO     | reprobench.core.worker:run:48 - Processing task: output/my_solver.MySudokuSolver/clasp/sudoku/table9-4.txt
2019-06-05 00:02:08.457 | INFO     | reprobench.core.worker:run:48 - Processing task: output/my_solver.MySudokuSolver/clasp/sudoku/table36-4.txt
2019-06-05 00:02:08.467 | INFO     | reprobench.core.worker:run:48 - Processing task: output/my_solver.MySudokuSolver/clasp/sudoku/table36-5.txt
2019-06-05 00:02:08.483 | INFO     | reprobench.core.worker:run:48 - Processing task: output/my_solver.MySudokuSolver/clasp/sudoku/table16-1.txt
2019-06-05 00:02:08.491 | INFO     | reprobench.core.worker:run:48 - Processing task: output/my_solver.MySudokuSolver/clasp/sudoku/table16-5.txt
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/home/rubydragon/Programming/sudokusat-example/env/lib/python3.7/site-packages/reprobench/managers/local/manager.py", line 34, in spawn_worker
    worker.run()
  File "/home/rubydragon/Programming/sudokusat-example/env/lib/python3.7/site-packages/reprobench/core/worker.py", line 60, in run
    step.execute(context, config)
  File "/home/rubydragon/Programming/sudokusat-example/env/lib/python3.7/site-packages/reprobench/executors/base.py", line 39, in execute
    tool(context).run(executor)
  File "/home/rubydragon/Programming/sudokusat-example/env/lib/python3.7/site-packages/reprobench/tools/executable.py", line 41, in run
    err_path=self.get_err_path(),
  File "/home/rubydragon/Programming/sudokusat-example/env/lib/python3.7/site-packages/reprobench/executors/psmon.py", line 84, in run
    stats = monitor.run()
  File "/home/rubydragon/Programming/sudokusat-example/env/lib/python3.7/site-packages/psmon/main.py", line 100, in run
    with Popen(*self.popenargs, preexec_fn=os.setpgrp, **self.kwargs) as process:
  File "/usr/lib/python3.7/subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.7/subprocess.py", line 1453, in _execute_child
    restore_signals, start_new_session, preexec_fn)
TypeError: expected str, bytes or os.PathLike object, not NoneType
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/rubydragon/Programming/sudokusat-example/env/bin/reprobench", line 10, in <module>
    sys.exit(cli())
  File "/home/rubydragon/Programming/sudokusat-example/env/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/rubydragon/Programming/sudokusat-example/env/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/rubydragon/Programming/sudokusat-example/env/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/rubydragon/Programming/sudokusat-example/env/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/rubydragon/Programming/sudokusat-example/env/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/rubydragon/Programming/sudokusat-example/env/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/rubydragon/Programming/sudokusat-example/env/lib/python3.7/site-packages/reprobench/console/decorators.py", line 37, in wrapper
    return func(server_address=server_address, *args, **kwargs)
  File "/home/rubydragon/Programming/sudokusat-example/env/lib/python3.7/site-packages/reprobench/console/decorators.py", line 27, in wrapper
    return func(*args, **kwargs)
  File "/home/rubydragon/Programming/sudokusat-example/env/lib/python3.7/site-packages/reprobench/managers/local/__init__.py", line 21, in cli
    manager.run()
  File "/home/rubydragon/Programming/sudokusat-example/env/lib/python3.7/site-packages/reprobench/managers/base.py", line 34, in run
    self.wait()
  File "/home/rubydragon/Programming/sudokusat-example/env/lib/python3.7/site-packages/reprobench/managers/local/manager.py", line 44, in wait
    for _ in self.pool_iterator:
  File "/usr/lib/python3.7/multiprocessing/pool.py", line 748, in next
    raise value
TypeError: expected str, bytes or os.PathLike object, not NoneType
2019-06-05 00:02:08.541 | INFO     | reprobench.core.worker:run:48 - Processing task: output/my_solver.MySudokuSolver/clasp/sudoku/table25-1.txt
2019-06-05 00:02:08.545 | INFO     | reprobench.core.worker:run:48 - Processing task: output/my_solver.MySudokuSolver/clasp/sudoku/table36-1.txt
2019-06-05 00:02:08.549 | INFO     | reprobench.core.worker:run:48 - Processing task: output/my_solver.MySudokuSolver/clasp/sudoku/table144-1.txt
2019-06-05 00:02:08.553 | INFO     | reprobench.core.worker:run:48 - Processing task: output/my_solver.MySudokuSolver/clasp/sudoku/table9-5.txt
2019-06-05 00:02:08.558 | INFO     | reprobench.core.worker:run:48 - Processing task: output/my_solver.MySudokuSolver/clasp/sudoku/table16-3.txt
2019-06-05 00:02:08.638 | INFO     | reprobench.managers.local.manager:exit:24 - Total time elapsed: 0.23797832299896982

Here the analyzer

$ reprobench analyze
Traceback (most recent call last):
  File "/home/rubydragon/Programming/sudokusat-example/env/bin/reprobench", line 10, in <module>
    sys.exit(cli())
  File "/home/rubydragon/Programming/sudokusat-example/env/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/rubydragon/Programming/sudokusat-example/env/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/rubydragon/Programming/sudokusat-example/env/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/rubydragon/Programming/sudokusat-example/env/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/rubydragon/Programming/sudokusat-example/env/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/rubydragon/Programming/sudokusat-example/env/lib/python3.7/site-packages/reprobench/console/decorators.py", line 27, in wrapper
    return func(*args, **kwargs)
  File "/home/rubydragon/Programming/sudokusat-example/env/lib/python3.7/site-packages/reprobench/core/analyzer.py", line 35, in cli
    analyzer.run()
  File "/home/rubydragon/Programming/sudokusat-example/env/lib/python3.7/site-packages/reprobench/core/analyzer.py", line 25, in run
    module.execute(context, config)
  File "/home/rubydragon/Programming/sudokusat-example/env/lib/python3.7/site-packages/reprobench/statistics/tables/base.py", line 29, in execute
    df = cls.get_dataframe(config)
  File "/home/rubydragon/Programming/sudokusat-example/env/lib/python3.7/site-packages/reprobench/statistics/tables/run.py", line 51, in get_dataframe
    tool_df = pd.read_sql(sql, db, params=params)
  File "/home/rubydragon/Programming/sudokusat-example/env/lib/python3.7/site-packages/pandas/io/sql.py", line 380, in read_sql
    chunksize=chunksize)
  File "/home/rubydragon/Programming/sudokusat-example/env/lib/python3.7/site-packages/pandas/io/sql.py", line 1469, in read_query
    columns = [col_desc[0] for col_desc in cursor.description]
apsw.ExecutionCompleteError: Can't get description for statements that have completed execution

I don't know what causes this.

Example SAT benchmark verdict is wrong

Different from SATLIB, ACLib benchmark instances doesn't have a comment saying it's sat/unsat in the cnf file.

Notes: Circuit Fuzz benchmark has a classify_100.txt file, a dump of solver attempts on the instances.

Running reprobench without setup

It would be nice to be able to run reprobench without an additional setup step. mainly for easy development and quick bugfixing on a cluster for example.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.