astropy / astropy-benchmarks Goto Github PK

View Code? Open in Web Editor NEW

7.0 7.0 26.0 110.86 MB

Benchmarks for the astropy project

Home Page: https://spacetelescope.github.io/bench/astropy-benchmarks/

License: BSD 3-Clause "New" or "Revised" License

Python 98.01% Shell 1.99%

astropy-benchmarks's Introduction

Astropy

The Astropy Project (https://astropy.org/) is a community effort to develop a single core package for Astronomy in Python and foster interoperability between Python astronomy packages. This repository contains the core package which is intended to contain much of the core functionality and some common tools needed for performing astronomy and astrophysics with Python.

Installation
Contributing
Getting Started with GitHub Codespaces
Supporting the Project
License

Installation

Releases are registered on PyPI, and development is occurring at the project's GitHub page.

For detailed installation instructions, see the online documentation or docs/install.rst in this source distribution.

To install astropy from PyPI, use:

pip install astropy

Contributing

The Astropy Project is made both by and for its users, so we welcome and encourage contributions of many kinds. Our goal is to keep this a positive, inclusive, successful, and growing community by abiding with the Astropy Community Code of Conduct.

More detailed information on contributing to the project or submitting feedback can be found on the contributions page. A summary of contribution guidelines can also be used as a quick reference when you are ready to start writing or validating code for submission.

Getting Started with GitHub Codespaces

Codespaces is a cloud development environment supported by GitHub. None of the Astropy build machinery depends on it, but it is a convenient way to quickly get started doing development on Astropy.

To get started, create a codespace for this repository by clicking this:

A codespace will open in a web-based version of Visual Studio Code. The dev container is fully configured with software needed for this project. For help, see the GitHub Codespaces Support page.

Note: Dev containers is an open spec which is supported by GitHub Codespaces and other tools.

Supporting the Project

The Astropy Project is sponsored by NumFOCUS, a 501(c)(3) nonprofit in the United States. You can donate to the project by using the link above, and this donation will support our mission to promote sustainable, high-level code base for the astronomy community, open code development, educational materials, and reproducible scientific research.

License

Astropy is licensed under a 3-clause BSD style license - see the LICENSE.rst file.

astropy-benchmarks's People

Contributors

Stargazers

Watchers

astropy-benchmarks's Issues

bug in benchmark - but would fix break history?

Looking at the setup for the modelling branchmarks, it look to me that the "medium" array has 2000 entries and the "large" array has only one (or indeed would fail if we used the most current numpy, since the step number is 5e-6, which is << 1 :
https://github.com/astropy/astropy-benchmarks/blob/master/benchmarks/modeling/model.py#L13
(same for the arrays with units). And indeed the tests with the "large" array run faster than the tests with the medium array...

The fix is easy, just change from 5e-6 to 5e+6 (or better 5000000 because that's an int and not a float) but that would probably make a regression appear? What to do? Fix and rename the test so that it's not compared to the old one?

List all the packages and their versions in benchmark log

I cannot find an easy way to see in the log what packages were installed for a given benchmark run and what versions. Would be nice to have this info. A simple pip freeze would suffice for now.

xref astropy/astropy#16296 (comment)

Modeling benchmarks failing

@nden - it looks like some of the modeling benchmarks are currently failing:

[ 22.46%] ···· Traceback (most recent call last):
                 File "/home/travis/miniconda/envs/test/lib/python3.6/site-packages/asv/benchmark.py", line 1039, in main_run_server
                   main_run(run_args)
                 File "/home/travis/miniconda/envs/test/lib/python3.6/site-packages/asv/benchmark.py", line 913, in main_run
                   result = benchmark.do_run()
                 File "/home/travis/miniconda/envs/test/lib/python3.6/site-packages/asv/benchmark.py", line 412, in do_run
                   return self.run(*self._current_params)
                 File "/home/travis/miniconda/envs/test/lib/python3.6/site-packages/asv/benchmark.py", line 506, in run
                   min_run_count=self.min_run_count)
                 File "/home/travis/miniconda/envs/test/lib/python3.6/site-packages/asv/benchmark.py", line 569, in benchmark_timing
                   timing = timer.timeit(number)
                 File "/home/travis/miniconda/envs/test/lib/python3.6/timeit.py", line 178, in timeit
                   timing = self.inner(it, self.timer)
                 File "<timeit-src>", line 6, in inner
                 File "/home/travis/build/astropy/astropy-benchmarks/benchmarks/modeling/compound.py", line 77, in time_large
                   r, d, = self.model(x_no_units_large * u.pix, x_no_units_large * u.pix)
                 File "/home/travis/miniconda/envs/test/lib/python3.6/site-packages/astropy/modeling/core.py", line 386, in __call__
                   __call__, args, [('model_set_axis', None)])
                 File "/home/travis/miniconda/envs/test/l/travis/miniconda/envs/test/lib/python3.6/site-packages/astropy/modeling/core.py", line 2349, in <lambda>
                   (evaluate(*chain(inputs, islice(params, n_params))),)
                 File "/home/travis/miniconda/envs/test/lib/python3.6/site-packages/astropy/modeling/functional_models.py", line 479, in evaluate
                   return x + offset
                 File "/home/travis/miniconda/envs/test/lib/python3.6/site-packages/astropy/units/quantity.py", line 418, in __array_prepare__
                   .format(function.__name__))
               astropy.units.core.UnitsError: Can only apply 'add' function to dimensionless quantities when other argument is not a quantity (unless the latter is all zero/infinity/nan)

Would you have any time to look into this? (in case it is a regression in astropy)

How to see which code is covered or not by benchmarks?

Recently @eerovaher brought up this good question. Would be nice to see what code is actually benchmarked (like codecov style) without having to dig through each benchmark module.

Not sure how hard this is to do. Running coverage during actual benchmarking would give you inaccurate timing, so the coverage measurement would have to be a separate job just for coverage, not timing. 💭

Website not loading

https://www.astropy.org/astropy-benchmarks/ shows an error message currently (2018-09-09).
Maybe missing git submodule update --init if you installed asv from git?

Add benchmark of table masked column

@taldcroft -- From astropy/astropy#7399 and astropy/astropy#7422, e.g.

dat = np.arange(1e7)  # setup
col = MaskedColumn(dat)  # Time this

@astrofrog -- Yep, this should definitely go in astropy-benchmarks

Moved from astropy/astropy#7423

Copy units benchmarks from unyt paper

For details see astropy/astropy#7546

Profile/document the details of the performance enhancement from astropy#7324

Ported from astropy/astropy#7521

astropy/astropy#7324 created a substantial speedup in the coordinate-matching functionality. In astropy/astropy#7324 (comment) @ideasrule gave a good set of tests (suggested by @adrn) on this enhancement. We should probably find someplace to record the results of these tests. These could be put in as benchmarks on airspeed velocity? Or possibly just documented in the coordinates docs? Open to ideas...

Add time benchmarks

Looks like we don't have any Time benchmarks at all.

xref astropy/astropy#10587 and https://github.com/astropy/astropy/issues?q=is%3Aopen+is%3Aissue+label%3Atime+label%3APerformance in general.

Update JSON to display new benchmarks

#49 is not appearing in render results page.

@astrofrog , "I think the issue is that my update script is not updating the benchmarks.json file."

README.rst: broken link to ST ASV doc

The link to http://spacetelescope.github.io/asv yields a 404.

MNT: Rename default branch from master to main

Please rename your default branch from master to main as part of astropy/astropy-project#151 , preferably by 2021-03-22. Also a friendly reminder to check documentation, workflows, etc., and update them accordingly. Please don't forget to communicate this change to your users and stakeholders. To summarize:

Rename branch from master to main, preferably by 2021-03-22.
Update documentation, workflows, etc., accordingly. -- See #104
Communicate this change to your users and stakeholders. -- N/A

Once this is taken care of, you may close this issue.

This is an automated issue. If this is opened in error, please let @pllim know!

Do we need oneesk benchmark with Numpy 1.16?

I notice that we only have up to Numpy 1.14, which is pretty old by now?

Move benchmark codes to astropy core lib and only keep results here

As asked by @Cadair , is there any good reason why the benchmarking code cannot be in astropy repo? Can we move the code to astropy and only keep the results here?

@astrofrog , do you remember why this is a separate repo?

LICENSE missing

We would like to use your cron.sh script for MDAnalysis/benchmarks#2 but there is no license.

Would you be willing to add an open source license that would make it clear how your code can be used by other projects?

Many thanks!

Remove large io.ascii files from master branch

It would be better to generate these on-the-fly in the benchmark setup functions/methods

Add a benchmark of drawing a contour in WCSAxes

This can be notoriously slow

Add WCSAxes benchmarks

WCSAxes has some performance issues in places, so we should add benchmarks. There are already a few here:

https://github.com/astrofrog/wcsaxes-benchmarks/blob/master/benchmarks/benchmarks.py

leftover from when WCSAxes was a separate package.

IPAC benchmark failing

The following benchmarks are currently failing:

 62.50%] ··· Running io_ascii.time_ipac.IPACSuite.time_data_str_vals                                                                                                              failed
[ 62.50%] ····· Traceback (most recent call last):
                  File "/Users/tom/miniconda3/envs/dev35/lib/python3.5/site-packages/asv-0.2.dev966+211cc811-py3.5.egg/asv/benchmark.py", line 792, in <module>
                    commands[mode](args)
                  File "/Users/tom/miniconda3/envs/dev35/lib/python3.5/site-packages/asv-0.2.dev966+211cc811-py3.5.egg/asv/benchmark.py", line 769, in main_run
                    result = benchmark.do_run()
                  File "/Users/tom/miniconda3/envs/dev35/lib/python3.5/site-packages/asv-0.2.dev966+211cc811-py3.5.egg/asv/benchmark.py", line 453, in do_run
                    return self.run(*self._current_params)
                  File "/Users/tom/miniconda3/envs/dev35/lib/python3.5/site-packages/asv-0.2.dev966+211cc811-py3.5.egg/asv/benchmark.py", line 528, in run
                    timing = timer.timeit(number)
                  File "/Users/tom/tmp/astropy-benchmarks/env/705fbb2b350520526769cb8754181ec0/lib/python3.4/timeit.py", line 186, in timeit
                    timing = self.inner(it, self.timer)
                  File "/Users/tom/tmp/astropy-benchmarks/env/705fbb2b350520526769cb8754181ec0/lib/python3.4/timeit.py", line 87, in inner
                    _func()
                  File "/Users/tom/tmp/astropy-benchmarks/benchmarks/io_ascii/time_ipac.py", line 38, in time_data_str_vals
                    data.str_vals()
                  File "/Users/tom/tmp/astropy-benchmarks/env/705fbb2b350520526769cb8754181ec0/lib/python3.4/site-packages/astropy/io/ascii/core.py", line 801, in str_vals
                    self._set_fill_values(self.cols)
                  File "/Users/tom/tmp/astropy-benchmarks/env/705fbb2b350520526769cb8754181ec0/lib/python3.4/site-packages/astropy/io/ascii/core.py", line 755, in _set_fill_values
                    colnames = set(self.header.colnames)
                AttributeError: 'IpacData' object has no attribute 'header'

[ 75.00%] ··· Running io_ascii.time_ipac.IPACSuite.time_get_cols                                                                                                                   failed
[ 75.00%] ····· Traceback (most recent call last):
                  File "/Users/tom/miniconda3/envs/dev35/lib/python3.5/site-packages/asv-0.2.dev966+211cc811-py3.5.egg/asv/benchmark.py", line 792, in <module>
                    commands[mode](args)
                  File "/Users/tom/miniconda3/envs/dev35/lib/python3.5/site-packages/asv-0.2.dev966+211cc811-py3.5.egg/asv/benchmark.py", line 769, in main_run
                    result = benchmark.do_run()
                  File "/Users/tom/miniconda3/envs/dev35/lib/python3.5/site-packages/asv-0.2.dev966+211cc811-py3.5.egg/asv/benchmark.py", line 453, in do_run
                    return self.run(*self._current_params)
                  File "/Users/tom/miniconda3/envs/dev35/lib/python3.5/site-packages/asv-0.2.dev966+211cc811-py3.5.egg/asv/benchmark.py", line 528, in run
                    timing = timer.timeit(number)
                  File "/Users/tom/tmp/astropy-benchmarks/env/705fbb2b350520526769cb8754181ec0/lib/python3.4/timeit.py", line 186, in timeit
                    timing = self.inner(it, self.timer)
                  File "/Users/tom/tmp/astropy-benchmarks/env/705fbb2b350520526769cb8754181ec0/lib/python3.4/timeit.py", line 87, in inner
                    _func()
                  File "/Users/tom/tmp/astropy-benchmarks/benchmarks/io_ascii/time_ipac.py", line 27, in time_get_cols
                    self.header.get_cols(self.lines)
                  File "/Users/tom/tmp/astropy-benchmarks/env/705fbb2b350520526769cb8754181ec0/lib/python3.4/site-packages/astropy/io/ascii/ipac.py", line 212, in get_cols
                    if self.ipac_definition == 'right':
                AttributeError: 'IpacHeader' object has no attribute 'ipac_definition'

[ 87.50%] ··· Running io_ascii.time_ipac.IPACSuite.time_header_str_vals                                                                                                            failed
[ 87.50%] ····· Traceback (most recent call last):
                  File "/Users/tom/miniconda3/envs/dev35/lib/python3.5/site-packages/asv-0.2.dev966+211cc811-py3.5.egg/asv/benchmark.py", line 792, in <module>
                    commands[mode](args)
                  File "/Users/tom/miniconda3/envs/dev35/lib/python3.5/site-packages/asv-0.2.dev966+211cc811-py3.5.egg/asv/benchmark.py", line 769, in main_run
                    result = benchmark.do_run()
                  File "/Users/tom/miniconda3/envs/dev35/lib/python3.5/site-packages/asv-0.2.dev966+211cc811-py3.5.egg/asv/benchmark.py", line 453, in do_run
                    return self.run(*self._current_params)
                  File "/Users/tom/miniconda3/envs/dev35/lib/python3.5/site-packages/asv-0.2.dev966+211cc811-py3.5.egg/asv/benchmark.py", line 528, in run
                    timing = timer.timeit(number)
                  File "/Users/tom/tmp/astropy-benchmarks/env/705fbb2b350520526769cb8754181ec0/lib/python3.4/timeit.py", line 186, in timeit
                    timing = self.inner(it, self.timer)
                  File "/Users/tom/tmp/astropy-benchmarks/env/705fbb2b350520526769cb8754181ec0/lib/python3.4/timeit.py", line 87, in inner
                    _func()
                  File "/Users/tom/tmp/astropy-benchmarks/benchmarks/io_ascii/time_ipac.py", line 33, in time_header_str_vals
                    header.str_vals()
                  File "/Users/tom/tmp/astropy-benchmarks/env/705fbb2b350520526769cb8754181ec0/lib/python3.4/site-packages/astropy/io/ascii/ipac.py", line 278, in str_vals
                    null = col.fill_values[core.masked]
                AttributeError: 'Column' object has no attribute 'fill_values'

[100.00%] ··· Running io_ascii.time_ipac.IPACSuite.time_splitter

@taldcroft @hamogu - is this a real regression, or should the benchmarks be updated? (if the latter, could you open a PR to update them?)

DOC: Add user guide in readme and contributing.md

As discussed on June 22, 2018 telecon. cc @eteq

Add benchmarks for FITS Header parsing

EDIT: Add benchmarks for astropy/astropy#8428

Add a benchmark for io.ascii with wide files that have thousands of columns

For example the issue solved by astropy/astropy#10306.

See also https://github.com/astropy/astropy-notebooks/tree/master/io/ascii. I think that at least some of those made it into the benchmarks but not sure.

cc: @dhomeier

Add compose benchmark for complex unit

This can be quite slow:

(u.kg / u.s ** 3 * u.au ** 2.5 / u.yr ** 0.5 / u.sr ** 2).compose()

so we should add a benchmark

Some table memory benchmarks failing

I haven't had a chance to investigate yet:

[ 19.68%] ··· io_ascii.table.TableSuite.mem_table_init                                                                                                   failed
[ 19.68%] ···· Traceback (most recent call last):
                 File "/home/tom/python/dev/lib/python3.7/site-packages/asv/benchmark.py", line 1184, in main_run_server
                   main_run(run_args)
                 File "/home/tom/python/dev/lib/python3.7/site-packages/asv/benchmark.py", line 1058, in main_run
                   result = benchmark.do_run()
                 File "/home/tom/python/dev/lib/python3.7/site-packages/asv/benchmark.py", line 537, in do_run
                   return self.run(*self._current_params)
                 File "/home/tom/python/dev/lib/python3.7/site-packages/asv/benchmark.py", line 729, in run
                   sizeof2 = asizeof.asizeof([obj, obj])
                 File "/home/tom/python/dev/lib/python3.7/site-packages/asv/extern/asizeof.py", line 2708, in asizeof
                   s = _asizer.asizeof(*t)
                 File "/home/tom/python/dev/lib/python3.7/site-packages/asv/extern/asizeof.py", line 2137, in asizeof
                   return sum(self._sizer(o, 0, 0, None) for o in objs)
                 File "/home/tom/python/dev/lib/python3.7/site-packages/asv/extern/asizeof.py", line 2137, in <genexpr>
                   return sum(self._sizer(o, 0, 0, None) for o in objs)
                 File "/home/tom/python/dev/lib/python3.7/site-packages/asv/extern/asizeof.py", line 2075, in _sizer
                   s += z(o, i, d, None)
                 File "/home/tom/python/dev/lib/python3.7/site-packages/asv/extern/asizeof.py", line 2075, in _sizer
                   s += z(o, i, d, None)
                 File "/home/tom/python/dev/lib/python3.7/site-packages/asv/extern/asizeof.py", line 2075, in _sizer
                   s += z(o, i, d, None)
                 [Previous line repeated 1 more time]
                 File "/home/tom/python/dev/lib/python3.7/site-packages/asv/extern/asizeof.py", line 2050, in _sizer
                   infer=self._infer_)
                 File "/home/tom/python/dev/lib/python3.7/site-packages/asv/extern/asizeof.py", line 1711, in _typedef
                   v.set(**_numpy_kwds(obj))
                 File "/home/tom/python/dev/lib/python3.7/site-packages/asv/extern/asizeof.py", line 1269, in set
                   self.reset(**d)
                 File "/home/tom/python/dev/lib/python3.7/site-packages/asv/extern/asizeof.py", line 1278, in reset
                   raise ValueError('invalid option: %s=%r' % ('base', base))
               ValueError: invalid option: base=-7880

[ 19.84%] ··· io_ascii.table.TableSuite.mem_table_outputter                                                                                              failed
[ 19.84%] ···· Traceback (most recent call last):
                 File "/home/tom/python/dev/lib/python3.7/site-packages/asv/benchmark.py", line 1184, in main_run_server
                   main_run(run_args)
                 File "/home/tom/python/dev/lib/python3.7/site-packages/asv/benchmark.py", line 1058, in main_run
                   result = benchmark.do_run()
                 File "/home/tom/python/dev/lib/python3.7/site-packages/asv/benchmark.py", line 537, in do_run
                   return self.run(*self._current_params)
                 File "/home/tom/python/dev/lib/python3.7/site-packages/asv/benchmark.py", line 729, in run
                   sizeof2 = asizeof.asizeof([obj, obj])
                 File "/home/tom/python/dev/lib/python3.7/site-packages/asv/extern/asizeof.py", line 2708, in asizeof
                   s = _asizer.asizeof(*t)
                 File "/home/tom/python/dev/lib/python3.7/site-packages/asv/extern/asizeof.py", line 2137, in asizeof
                   return sum(self._sizer(o, 0, 0, None) for o in objs)
                 File "/home/tom/python/dev/lib/python3.7/site-packages/asv/extern/asizeof.py", line 2137, in <genexpr>
                   return sum(self._sizer(o, 0, 0, None) for o in objs)
                 File "/home/tom/python/dev/lib/python3.7/site-packages/asv/extern/asizeof.py", line 2075, in _sizer
                   s += z(o, i, d, None)
                 File "/home/tom/python/dev/lib/python3.7/site-packages/asv/extern/asizeof.py", line 2075, in _sizer
                   s += z(o, i, d, None)
                 File "/home/tom/python/dev/lib/python3.7/site-packages/asv/extern/asizeof.py", line 2075, in _sizer
                   s += z(o, i, d, None)
                 [Previous line repeated 1 more time]
                 File "/home/tom/python/dev/lib/python3.7/site-packages/asv/extern/asizeof.py", line 2050, in _sizer
                   infer=self._infer_)
                 File "/home/tom/python/dev/lib/python3.7/site-packages/asv/extern/asizeof.py", line 1711, in _typedef
                   v.set(**_numpy_kwds(obj))
                 File "/home/tom/python/dev/lib/python3.7/site-packages/asv/extern/asizeof.py", line 1269, in set
                   self.reset(**d)
                 File "/home/tom/python/dev/lib/python3.7/site-packages/asv/extern/asizeof.py", line 1278, in reset
                   raise ValueError('invalid option: %s=%r' % ('base', base))
               ValueError: invalid option: base=-7880

Add stats benchmarks

E.g. sigma clip was being reported very slow here:

astropy/ccdproc#624

Add nddata benchmarks

We've got none except the raw import timing.

Refactor benchmarking process to run relative benchmark for PR

Perhaps we can be inpired by https://labs.quansight.org/blog/2021/08/github-actions-benchmarks/ .

Looks like it has been deployed in scikit-image:

Also see:

Change from Python 3.3 to 3.4

@mdboom Could you please change from Python 3.3 to Python 3.4?

There' s the issue that everything is slower on Python 3, but it only really makes sense to spend time on this if it's still the case with 3.4.

Also, it looks like the Python 3.3 tests stopped running a month ago?
http://www.astropy.org/astropy-benchmarks/#time_coordinates.FrameBenchmarks.time_init_array

Rename some confusing benchmark test names

For example: time_fits has nothing to do with astropy.time. Suggested naming to follow actual astropy sub-packages. c/c @bsipocz @astrofrog

NOTE: This needs to wait until current benchmarking is done. Renaming needs to be done also in the results across multiple branches.

numpy scimath ImportError on Mac

@mdboom I wanted to try running astropy-benchmarks locally on my Mac and ran into this ImportError for scimath in numpy:
https://gist.github.com/cdeil/8400139

How we Visualize Benchmarks

Throwing out a suggestion. Rather than ASV I recommend https://docs.codspeed.io/.
codspeed is really good, free for open-source, and integrates deeply with pytest (see pytest-benchmark). With codspeed + pytest ecosystem it's easy for use to have a specific benchmark test suite and also have benchmarked tests in the normal test suite.