Giter Club home page Giter Club logo

basil's Introduction

Basil

Benchmarking And Statistical Inspection Library

โš ๏ธ This library is still in development. Breaking changes may occur at any time.

Install

Use pip to install the package:

pip install basil-benchmark

Example

Define a benchmark to solve, a solver and the instances, which are made of a pair of a benchmark and a solver.

from time import time

import jax
import numpy as np
from jax import numpy as jnp

from basil import Benchmark, Solver, parametrize, register_benchmark

class SimpleBenchmark(Benchmark):    # Define the benchmark
    def __init__(self, N):
        self.N = N

    @property
    def parameters(self):
        return {"num_values": self.N}

    @property
    def inputs(self):
        matrix = np.random.rand(self.N, self.N)
        ref_value = np.sum(matrix)
        return {"inputs": {"matrix": matrix}, 'aux': ref_value}

    def evaluate(self, outputs, ref_value):
        return {
            "error": abs(outputs["result"] - ref_value),
            "execution_time": outputs["execution_time"]
        }


class JaxSolver(Solver):    # Define benchmark solver
    def run(self, inputs):
        func = jax.jit(lambda x: jnp.sum(x))

        # Compile the function
        mock_inputs = jnp.ones_like(inputs["matrix"])
        _ = func(mock_inputs)

        # Run the function
        x = jnp.asarray(inputs["matrix"])
        start = time()
        result = func(x).block_until_ready()
        execution_time = time() - start
        return {"result": float(result), "execution_time": execution_time}

# Define the benchmark instances
@register_benchmark
@parametrize("N", [100, 1000])
def sum_test(N: int):
    return {"benchmark": SimpleBenchmark(N), "solver": JaxSolver()}

After saving this in a file that follows the pattern benchmark_*.py into some BENCHMARKS_PATH subfolder, invoke basil to run the benchmark instances.

basil BENCHMARKS_PATH

It should print something like this:

Adding simple_benchmark with jax_solver to register
Adding simple_benchmark with jax_solver to register
Running all 2 runs
Benchmark: simple_benchmark
Results: {'system_info': {'OS': 'Linux', 'OS_version': '#1 SMP Fri Jan 27 02:56:13 UTC 2023', 'CPU': 'x86_64', 'RAM': 16714588160, 'GPU': 'No GPU info'}, 'solver': 'jax_solver', 'options': {}, 'parameters': {'num_values': 100}, 'results': {'error': 0.0001370069894619519, 'execution_time': 0.00036644935607910156}}

Benchmark: simple_benchmark
Results: {'system_info': {'OS': 'Linux', 'OS_version': '#1 SMP Fri Jan 27 02:56:13 UTC 2023', 'CPU': 'x86_64', 'RAM': 16714588160, 'GPU': 'No GPU info'}, 'solver': 'jax_solver', 'options': {}, 'parameters': {'num_values': 1000}, 'results': {'error': 0.025644428096711636, 'execution_time': 0.0019795894622802734}}

All runs finished

The results are also saved in a json file, with the same name as the benchmark (in this case, it would be simple_benchmark.json).

Visualize results

A prototype visualization via streamlit is provided, but it is not yet complete. To preview it in development mode, use

streamlit run src/basil/_dashboard.py -- --folder RESULTS_FOLDER

basil's People

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.