sambit-giri / tools21cm Goto Github PK

License: MIT License

Python 95.67% TeX 3.55% Shell 0.19% Cython 0.60%

reionization 21-cm-cosmology 21-cm-powerspectrum epoch-of-reionization cosmic-dawn

tools21cm's Introduction

Tools21cm

A python package for analysing simulated 21-cm signals from the Epoch of Reionization (EoR) and Cosmic Dawn (CD). Full documentation (with examples, installation instructions and complete module description) can be found at readthedocs.

Package details

The package provides tools to analyse cosmological simulations of EoR and CD. It contains modules to create mock 21-cm observations for current and upcoming radio telescopes, such as LOFAR, MWA and SKA, and to construct statistical measures.

Input

Currently, Tools21cm supports the following simulation codes:

Outputs

There are various manipulation and analysis moduled in Tools21cm.

Angular coordinates: methods to convert data between physical (cMpc) coordinates and observational (angular-frequency) coordinates
Bubble Statistics: methods to calcluate the sizes of the regions of interest and estimate the size distributions
Cosmological calculators: various functions for calculating some cosmological stuff
Identifying regions: methods to identify regions of interest in images
Lightcone: methods to construct lightcones
Power spectrum: contains functions to estimate various two point statistics
Reading simuation outputs: methods to read simulations outputs
Smoothing: methods to smooth or reduce resolution of the data to reduce noise
Point statistics: contains various useful statistical methods
Temperature: methods to estimate the brightness temperature

For detail documentation and how to use them, see here.

Under Developement

Foreground model: methods to simulate and analyse the foreground signal for 21 cm signal
Primary beam: methods to simulate the primary beam of radio telescope
Telescope noise:
- simulate the radio telescope observation strategy
- simulate telescope noise
Topology: methods to estimate the topology of the region of interest

INSTALLATION

One can install a stable version of this package using pip by running the following command::

pip install tools21cm

This package is being actively under-development, which involves addition of new modules and bug fixes. In order to use the latest version, one can clone this package.

To install the package from source, one should clone this package running the following::

git clone https://github.com/sambit-giri/tools21cm.git

To install the package in the standard location, run the following in the root directory::

python setup.py install

In order to install it in a separate directory::

python setup.py install --home=directory

One can also install the latest version using pip by running the following command::

pip install git+https://github.com/sambit-giri/tools21cm.git

The dependencies should be installed automatically during the installation process. If they fail for some reason, you can install them manually before installing tools21cm. The list of required packages can be found in the requirements.txt file present in the root directory.

Tests

For testing, one can use pytest or nosetests. Both packages can be installed using pip. To run all the test script, run the either of the following::

python -m pytest tests

nosetests -v

CONTRIBUTING

If you find any bugs or unexpected behavior in the code, please feel free to open a Github issue. The issue page is also good if you seek help or have suggestions for us. For more details, please see here.

tools21cm's People

Contributors

Stargazers

Watchers

Forkers

boondein dprelogo lizhangscience garrelt tommychinwj micbia ngillet jlewis47 haricash abinashkumarshaw simonp2207 joohyun-lee lconaboy gracospa andresfgomez970 liambx leonnoble

tools21cm's Issues

tools21cm numpy version

In the current version of tools21cm (master branch), Numba needs NumPy 1.21 or less. This can give some dependency contrast to users with packages (e.g., Astropy) requiring newer versions of NumPy.

indexing error in resample_slice

the line
idx = np.linspace(0, slice_smoothed.shape[0], n_output_cells)
should be modified to
idx = np.linspace(0, slice_smoothed.shape[0]-1, n_output_cells)

otherwise the idx is longer than output_slice, and when executing
output_slice = smoothing.interpolate2d(slice_smoothed, idx, idx, order=order)
output slice repeats the end values at the beginning

Installation in editable/development mode does not work properly

The documentation suggests installing the package as pip install -e /path/to/tools21cm [--user] to allow doing development work on it. However, if you do this (or any other type of development install such as python setup.py develop or conda develop /path/to/tools21cm, you cannot run import tools21cm from within python. The reason is that the source directory is called t2c and not tools21cm. If you change the name of the source directory to tools21cm the import command works again. If you do not, you can also run import t2c which does work. However, I am not sure if we want to have a different import command for the development version.

If there are good reasons not to rename the directory we could perhaps add a soft link?

Failure during test

Hi,

I am getting the following error in Smoothing.py while testing the installation.

tests/test_BubbleStatistics.py ... [ 12%]
tests/test_CosmoCalc.py . [ 16%]
tests/test_IdentifyRegions.py ... [ 28%]
tests/test_PointStatistics.py ..... [ 48%]
tests/test_PowerSpectrum.py ..... [ 68%]
tests/test_Smoothing.py ...F [ 84%]
tests/test_Temperature.py .... [100%]

=================================== FAILURES ===================================
____________________________ test_tophat_kernel_3d _____________________________

def test_tophat_kernel_3d():

  kernel = t2c.tophat_kernel_3d(5)

E TypeError: tophat_kernel_3d() missing 1 required positional argument: 'tophat_width'

tests/test_Smoothing.py:30: TypeError
=============================== warnings summary ===============================
tests/test_IdentifyRegions.py::test_bubbles_from_kmeans
tests/test_IdentifyRegions.py::test_bubbles_from_kmeans
/home/aritra/anaconda3/envs/21cmTools/lib/python3.8/site-packages/sklearn/cluster/_kmeans.py:792: FutureWarning: 'n_jobs' was deprecated in version 0.23 and will be removed in 1.0 (renaming of 0.25).
warnings.warn("'n_jobs' was deprecated in version 0.23 and will be"

-- Docs: https://docs.pytest.org/en/stable/warnings.html
=========================== short test summary info ============================
FAILED tests/test_Smoothing.py::test_tophat_kernel_3d - TypeError: tophat_ker...
=================== 1 failed, 24 passed, 2 warnings in 5.09s ===================

What could be the possible solution to this problem ?

Use of matrix_rank() in helper_functions.py

The function fftconvolve in helper_functions.py has two inputs. These are turned into arrays with asarray()
and next a test is done on them:

if matrix_rank(in1) == matrix_rank(in2) == 0: # scalar inputs
return in1 * in2
I am not sure what this test is but if the inputs are 3D arrays python complains that matrix_rank(in1) and matrix_rank(in2) are arrays and therefore cannot be compared like this ("The truth value of an array with more than one element is ambiguous."). Therefore the routine does not work.

The intended meaning of this test seems to catch the case where the inputs are not arrays but scalars. However, I am not sure this is the way to test this. matrix_rank(asarray([5]) returns 1, not 0. Perhaps the intended use was different??

ModuleNotFound Error during test

Hi,

I'm a beginner and was getting the error when running 'python3 -m pytest -tests'. This is not resolved when I change 'find_package()' to find_namespace_package() in the setup.py file. Could you give me some advice? I really appreciate it.

Cheers

cdist_to_z is not independant of the specified cosmology

Because t2c.cdist_to_z use a precalculated table it is not valid when changing the cosmology parameters to other values than the default, e.g. when setting H0 to 100 for testing purpose.

conflicting naming with astropy.cosmology

Conflicting naming between the cosmology module from astropy and the t2c cosmology method. A quick fix would be to rename the t2c module comology.py to cosm.py

angular resolution when computing uv coverage

In function telescope_functions.get_uv_coverage, could you explain why is Nb = Nbase*theta_max/2 and not only Nbase*theta_max?

I'm using your code for applying thermal noise to the 21cmfast lightcones, found this as final-checking, not sure if it is a mistake or I don't see something...

Thanks :)

Mistake in setup.py?

The setup.py file has 'git+https://github.com/sambit-giri/cosmospectra.git' in the list of requirements. This generates an error when I try to install it with pip 20.3.4. It seems this string should instead be 'cosmospectra @ git+https://github.com/sambit-giri/cosmospectra.git'.
See https://stackoverflow.com/questions/60414753/how-to-install-githttps-from-setup-py-using-install-requires

Main Module Documentation Missing?

Hi,

The main module documentation is missing, and it seems only the part of smoothing is there.

Tests do not run with the command in README

Running pytest -v as stated in the README causes the following errors:

$ pytest -v
===================================================================================== test session starts ======================================================================================
platform linux -- Python 3.7.6, pytest-5.4.3, py-1.9.0, pluggy-0.13.1 -- /home/tomasi/miniconda3/bin/python
cachedir: .pytest_cache
rootdir: /home/tomasi/Documents/work/articles/referee/JOSS/tools21cm/tools21cm
collected 0 items / 7 errors                                                                                                                                                                   

============================================================================================ ERRORS ============================================================================================
_______________________________________________________________________ ERROR collecting tests/test_BubbleStatistics.py ________________________________________________________________________
ImportError while importing test module '/home/tomasi/Documents/work/articles/referee/JOSS/tools21cm/tools21cm/tests/test_BubbleStatistics.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_BubbleStatistics.py:2: in <module>
    from skimage.morphology import ball
E   ModuleNotFoundError: No module named 'skimage'
___________________________________________________________________________ ERROR collecting tests/test_CosmoCalc.py ___________________________________________________________________________
ImportError while importing test module '/home/tomasi/Documents/work/articles/referee/JOSS/tools21cm/tools21cm/tests/test_CosmoCalc.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_CosmoCalc.py:2: in <module>
    import tools21cm as t2c
E   ModuleNotFoundError: No module named 'tools21cm'
________________________________________________________________________ ERROR collecting tests/test_IdentifyRegions.py ________________________________________________________________________
ImportError while importing test module '/home/tomasi/Documents/work/articles/referee/JOSS/tools21cm/tools21cm/tests/test_IdentifyRegions.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_IdentifyRegions.py:2: in <module>
    from skimage.morphology import ball
E   ModuleNotFoundError: No module named 'skimage'
________________________________________________________________________ ERROR collecting tests/test_PointStatistics.py ________________________________________________________________________
ImportError while importing test module '/home/tomasi/Documents/work/articles/referee/JOSS/tools21cm/tools21cm/tests/test_PointStatistics.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_PointStatistics.py:2: in <module>
    import tools21cm as t2c
E   ModuleNotFoundError: No module named 'tools21cm'
_________________________________________________________________________ ERROR collecting tests/test_PowerSpectrum.py _________________________________________________________________________
ImportError while importing test module '/home/tomasi/Documents/work/articles/referee/JOSS/tools21cm/tools21cm/tests/test_PowerSpectrum.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_PowerSpectrum.py:2: in <module>
    import tools21cm as t2c
E   ModuleNotFoundError: No module named 'tools21cm'
___________________________________________________________________________ ERROR collecting tests/test_Smoothing.py ___________________________________________________________________________
ImportError while importing test module '/home/tomasi/Documents/work/articles/referee/JOSS/tools21cm/tools21cm/tests/test_Smoothing.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_Smoothing.py:2: in <module>
    import tools21cm as t2c
E   ModuleNotFoundError: No module named 'tools21cm'
__________________________________________________________________________ ERROR collecting tests/test_Temperature.py __________________________________________________________________________
ImportError while importing test module '/home/tomasi/Documents/work/articles/referee/JOSS/tools21cm/tools21cm/tests/test_Temperature.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_Temperature.py:2: in <module>
    from skimage.morphology import ball
E   ModuleNotFoundError: No module named 'skimage'
=================================================================================== short test summary info ====================================================================================
ERROR tests/test_BubbleStatistics.py
ERROR tests/test_CosmoCalc.py
ERROR tests/test_IdentifyRegions.py
ERROR tests/test_PointStatistics.py
ERROR tests/test_PowerSpectrum.py
ERROR tests/test_Smoothing.py
ERROR tests/test_Temperature.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 7 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
====================================================================================== 7 errors in 0.17s =======================================================================================

This can be fixed using the trick explained in a StackOverflow answer, i.e., using the command python -m pytest tests/:

$ python -m pytest tests/
===================================================================================== test session starts ======================================================================================
platform linux -- Python 3.7.6, pytest-5.4.3, py-1.9.0, pluggy-0.13.1
rootdir: /home/tomasi/Documents/work/articles/referee/JOSS/tools21cm/tools21cm
collected 25 items                                                                                                                                                                             

tests/test_BubbleStatistics.py ...                                                                                                                                                       [ 12%]
tests/test_CosmoCalc.py .                                                                                                                                                                [ 16%]
tests/test_IdentifyRegions.py ...                                                                                                                                                        [ 28%]
tests/test_PointStatistics.py .....                                                                                                                                                      [ 48%]
tests/test_PowerSpectrum.py .....                                                                                                                                                        [ 68%]
tests/test_Smoothing.py ....                                                                                                                                                             [ 84%]
tests/test_Temperature.py ....                                                                                                                                                           [100%]

======================================================================================= warnings summary =======================================================================================
tests/test_IdentifyRegions.py::test_bubbles_from_slic
  /home/tomasi/Documents/work/articles/referee/JOSS/tools21cm/venv/lib/python3.7/site-packages/skimage/util/dtype.py:226: DeprecationWarning: Converting `np.inexact` or `np.floating` to a dtype is deprecated. The current result is `float64` which is not strictly correct.
    dtypeobj_out = np.dtype(dtype)

tests/test_IdentifyRegions.py::test_bubbles_from_slic
  /home/tomasi/Documents/work/articles/referee/JOSS/tools21cm/venv/lib/python3.7/site-packages/tools21cm/superpixels.py:16: FutureWarning: skimage.measure.label's indexing starts from 0. In future version it will start from 1. To disable this warning, explicitely set the `start_label` parameter to 1.
    labels = slic(cube, n_segments=n_segments, compactness=compactness, max_iter=max_iter, sigma=sigma, max_size_factor=max_size_factor, slic_zero=True, multichannel=multichannel)

tests/test_IdentifyRegions.py::test_bubbles_from_slic
  /home/tomasi/Documents/work/articles/referee/JOSS/tools21cm/venv/lib/python3.7/site-packages/skimage/segmentation/slic_superpixels.py:262: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
    if not isinstance(sigma, coll.Iterable):

tests/test_IdentifyRegions.py::test_bubbles_from_kmeans
tests/test_IdentifyRegions.py::test_bubbles_from_kmeans
  /home/tomasi/Documents/work/articles/referee/JOSS/tools21cm/venv/lib/python3.7/site-packages/sklearn/cluster/_kmeans.py:974: FutureWarning: 'n_jobs' was deprecated in version 0.23 and will be removed in 0.25.
    " removed in 0.25.", FutureWarning)

-- Docs: https://docs.pytest.org/en/latest/warnings.html
================================================================================ 25 passed, 5 warnings in 3.15s ================================================================================

Perhaps it would be good to update the README.

(This issue was triggered while reviewing the JOSS paper.)

Non-periodic data when using subvolumes from light-cones

The power spectrum estimators use FFT to transform data from real to Fourier space. If the data is not periodic or has a sharp change at the boundary, it can introduce ripples during the transformation.

Temperature correction in _dt_full buggy

in the function _dt_full :

firstly there is the parameter 'correct' that can not be set to false.
then the formula to correct the temperature is wrong I think, just a typo of missing parenthesis :
Ts_new = Ts-xi * T_HII / (1.-xi)
should be
Ts_new = ( Ts-xi * T_HII ) / (1.-xi)

Non-manifold points

In topology.py, how do you handle non-manifold points when computing the Euler characteristic?

set H0

I just noticed that when you set little h with t2c.set_hubble_h(1) and then print the H0 value t2c.H0 the result is still the standard setup 70 km/s/Mpc. Need further investigation in the conv.py script.