Giter Club home page Giter Club logo

Comments (5)

matthewfeickert avatar matthewfeickert commented on May 16, 2024

@lukasheinrich Hm, strange. When I check it in a similar manner to how I did with the timing benchmark

def check_runOnePoint_qmu(n_bins):
    pdf, data = generate_pdf_and_data(n_bins)

    q_mu = pyhf.runOnePoint(1.0, data, pdf,
                            pdf.config.suggested_init(),
                            pdf.config.suggested_bounds())[0]

    # Reset the TensorFlow graph and session for each run
    if isinstance(pyhf.tensorlib, tensorflow_backend):
        tf.reset_default_graph()
        pyhf.tensorlib.session = tf.Session()

    return q_mu

def test_backend(backend, n_bins):
    default_backend = pyhf.tensorlib
    pyhf.set_backend(backend)

    test_statistics = [check_runOnePoint_qmu(bins) for bins in n_bins]

    # Reset backend
    pyhf.set_backend(default_backend)

    return test_statistics

I now see that both TensorFlow and PyTorch are deviating from SciPy (or the other way around).

diff_qmu

When I get time I'll check the unconstrained_bestfit() and constrained_bestfit() outputs.

from pyhf.

lukasheinrich avatar lukasheinrich commented on May 16, 2024

ok I would attribute this to double vs single precision (can you do a ratio e.g. relative to scipy/numpy)

from pyhf.

matthewfeickert avatar matthewfeickert commented on May 16, 2024

@lukasheinrich sure
ratio_qmu

from pyhf.

lukasheinrich avatar lukasheinrich commented on May 16, 2024

ok so it's not getting worse so i wouldn't worry about this too much. perhaps we could unit test

pytorch / tensorflow < 1e-3 (does that pass?
pytorch / scipy < 1e-2
tensorflow / scipy < 1e-2

from pyhf.

matthewfeickert avatar matthewfeickert commented on May 16, 2024

abs((tensorflow/pytorch)-1) < 5e-03 passes (but not 1e-03)

from pyhf.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.