Giter Club home page Giter Club logo

toast's People

Contributors

giuspugl avatar ickc avatar jrmadsen avatar jsavarit avatar keskitalo avatar mhasself avatar triskadecaepyon avatar tskisner avatar ziotom78 avatar zonca avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

toast's Issues

Installation requires MPI

Currently, running setup.py instantiates the MPI-aware unit test runner. This effectively means that in order to install the package (python3 setup.py install), it must be possible to initialize MPI. However, on some systems (NERSC), MPI cannot be initialized on the login nodes. This is inconvenient, but when installing from a compute node, the global/common filesystem is mounted read-only. So we effectively cannot install pytoast to global/common. We should be able to optionally enable MPI somehow to bypass this.

Common mode noise

We should add support for generating common node noise between detectors

Add moving-slab atmosphere model

Following Josquin Errard's approach, construct realizations of the atmospheric emission and pass them across the field of view.

Noise simulations will have discontinuities at data distribution boundaries.

Since current noise sims have each process simulate a piece of timestream independently, there will be discontinuities at the process boundaries. The solution is to either redistribute the data so that one process has the whole timestream, OR have every process simulate the whole dataset up to their local start sample, OR to use a python wrapper around fftw-mpi.

Implement data redistribution

Currently we only use time-based data distribution within the communicator that handles a given observation. There are some cases where we want to have a timestream assigned to a single process, to facilitate simulation. Add functions to redistribute and communicate data within an observation.

Memory usage is high in TOD base class

The TOD base class is used to cache pointing, pointing matrices, etc. These vectors are numpy arrays stored in a dictionary. Even when these dictionary keys are explicitly deleted, the memory does not seem to be garbage collected (based on memory_profiler output).

Native timestream data format

We should implement a TOD derived class which reads / writes to a tidas volume (HDF5 backend). This is pending several things, in particular a few remaining issues holding up the initial tidas release. This functionality would be useful for dumping timestreams to disk for both debugging and external validation by other software tools.

When passing data to libmadam, do not copy

change the interface to libmadam to take an array of pointers to each detector's data. This would allow us to avoid duplicating all the memory when passing it to libmadam

TOD flag methods

To make the interface more consistent, the read_flags() method should be the only one returning flags. The read(), read_pntg(), etc methods should just return data. For some experiment-specific TOD classes reading some data formats (e.g. FITS binary tables), it is most efficient to read flags at the same time as the data and cache them. That is a detail left to those classes.

Design Roadmap Notes

This issue is for documenting design discussions held offline concerning the use of compiled code in toast. There are a variety of tradeoffs in play, but broadly:

  1. Python is great for prototyping and for use in workflow management. However, it can't use threads.
  2. Manycore systems are coming online right now. They require the use of threads for best performance.
  3. In toast, we will need to move performance critical code into optimized, compiled libraries which can use threads.
  4. In a project that mixes python and compiled libraries the 2 build systems that might work are autotools and python distutils. Any build system will have to be able to compile and link the non-python code with external libraries (FFTW, etc).
  5. It would be nice from a usability perspective to be able to build / install parts of toast while ignoring the rest.
  6. For compiled libraries, we need to have effective unit tests, just as we do for the python code.

All of these thoughts / requirements boil down to some design changes that should improve performance, development workflow, and usability:

  1. Switch to autotools for a build system rather than distutils. We have extensive experience with autotools, including custom macros for detection of the external libraries we need. We have also installed python packages with autotools in the past, so we know it works.
  2. There will be a single "libtoast" library which contains multiple internal libraries according to what features are available at configure time. The internal compiled code in libtoast will be C++ with a C interface for easy calling from python. It will be possible to build libtoast and use the math functions without installing the python package.
  3. Unit tests in the compiled code will use the gtest framework. "make check" can run both the unit tests for the compiled code and also the python unit tests.
  4. The tools building pipelines, doing I/O, and operations that are not performance critical (or are at the prototyping stage) will remain in python.

In the coming weeks we'll be adding a feature branch to work on this reorganization.

Potential issue with example toast_satellite_sim.py

This problem was actually observed with the LiteBIRD simulation script, which is very similar to the example script. The simulation configuration was using observation lengths of 23 hours, with a one hour gap in between. The data was distributed in chunks of 1 hour. The symptom is that when running with one simulated day per node, things seem to usually work fine. When running with different concurrencies (including having more nodes than days). These jobs fail in the middle of libmadam execution. Initially this was thought to be an out-of-memory condition. This is no longer clear. More investigation is needed on smaller jobs that explore the parameter space of one hour chunks to find the source of this problem.

Consistent treatment of base class methods

The TOD base class is currently acting as both an interface definition for internal methods called by the high-level read/write functions and as a memory-backed generic TOD container. We should split this functionality into the actual base class where the internal methods (prefixed with underscore) raise exceptions when called and a separate derived class which has a memory-based backend.

Reorganize caching in TOD class

Currently the read/write methods of the TOD base class are being abused to implement timestream caching. This prevents descendant classes from using this cache. This should be moved into an explicit, separate object. The TOD base class can still contain an instance of this "cache" object, but this will then be available to all descendents. It also opens the possibility for us to explicitly manage the memory of this cache outside of python, which may be needed to address things like issue #13.

Noise model PSD spacing

The current noise PSD model is not being generated at high enough resolution for the interpolation. Also, we probably remove the built-in high pass in this model.

Operators should support subsets of detectors

Currently, toast operators have an "exec()" method which takes as an argument the high-level distributed data object. This contains one or more observations, each with some number of locally-stored detector TODs. These operators then loop over all detectors in each observation and "do stuff", potentially storing the results of some calculation in the TOD's cache. For complex workflows, this can lead to a huge number of intermediate temporary vectors in memory. For certain types of workflows (that don't require collective operations between detectors), it would be better to do an outer loop over detectors and run a chain of operators on each detector up to some point.

After some offline discussion with @keskitalo , an easy way to support this would be to pass a list of detector names as an optional argument to the exec() method. An operator could then support working with only a subset of detectors (in particular a single detector). At the top pipeline level, the chain of operators could be called one detector at a time, purging the cached products after each one. This type of loop could co-exist with other top-level operators that worked with the full detector set in one go.

This change would be backwards compatible with the current pipelines.

Update Documentation

Add docstrings for all public classes and functions. Expand examples. Update generated docs on gh-pages.

TOD telescope position and velocity

The TOD class represents data from a telescope. There should be general methods to return the position and velocity vectors of the telescope.

Elemental Library is required for building from source; not optional

When building TOAST from source, Elemental is a strict requirement and not optional, as the documentation alludes to.

In toast/src/libtoast/toast.cpp with HAVE_ELEMENTAL, the IFDEF will break the namespace of toast when compiling the c++ dependencies.

Configure, make, and the documentation should be modified to reflect this.

Set of Examples

We should clean up and update the set of example scripts in toast. This is loosely coupled to updates to the documentation. These scripts can serve as both use case examples and for use with scaling tests on Knights Landing. We should have a set of scripts of different complexity that can then also be run at different concurrency.

Pipeline Scripts:

  • Satellite scanning with noise simulation
  • Ground scanning with noise + atmosphere + sky signal

Configurations:

  • "Laptop": tiny detectors, minimum time range
  • Single node: smaller number of detectors and time range
  • Moderate node count: all detectors, as much time as reasonable

For each pipeline and configuration, we should provide a shell script to generate any auxiliary data, and then a slurm script for running it at NERSC (if relevant). For the satellite sims, we can keep the time length fixed and scale the number of detectors. For the ground based sims, we can keep the detector count fixed and scale the time range.

Generated noise TOD has wrong PSD

It pains me to open this issue, but it looks like there is a problem with a few lines of code which results in generated noise timestreams having the incorrect PSD. The problem is that the Fourier domain vector that is input to scipy.fft.irfft expects complex values packed as pairs of real and imaginary values, and we have instead been passing in data in the half-complex fftw storage ordering. The result is that the PSD amplitude is applied in the Fourier domain to the wrong elements, resulting in some low frequency power being shifted to higher frequency. This failure mode actually still allows our unit tests (based on overall normalization) to pass.

This was discovered when looking at pull request #67 in more detail. That pull request fixes the issue as a side effect of switching the numpy fft package.

I'm working to fix this now, but wanted to open this issue for the sake of transparency.

Use cache for DistPixels

Move the Cache class out of tod and up one level. Use an instance of Cache for local data storage in the DistPixels class. This will make it easier to manage memory used by map domain objects.

Improve install docs

Some of our optional dependencies (Elemental) can be quite challenging to install. We should give some more details here. Also, we are almost completely independent of healpy in the core toast package, so we might consider finishing that transition.

Pointing manipulation functions

We will need a set of common routines for various pointing operations: aberration calculation, positional astronomy operations, coordinate transforms, etc. These are free functions that we should put into a new source file in toast.tod

Dipole timestream generation

We should have an operator in toast.tod.sim_detdata that generates a dipole signal for each detector. This would be computed based on the detector pointing and telescope velocity.

MKL FFT failures

When building with gcc and linking to modern MKL versions, the MKL unit tests fail. This happens for both single and batched FFTs. This problem is not seen when using FFTW. I will disable MKL FFTs in the mean time.

Map domain distribution is tedious

The use of the DistPixels class and externally calculating the locally hit pixels is "clunky". We should provide a more intuitive way of just getting the already-distributed hits and noise covariance without lots of boilerplate.

Noise simulations do not use streamed RNG

Currently, OpSimNoise just implements different seeds for each detector in an observation. This does not guarantee statistical independence. Wrap rng123 with python so we can use it.

Implement toast.tod.noise

Create a base class which represents the noise properties for all detectors for a given observation. For the first implementation, this is just a list of PSDs, one per detector. The interface will include methods for applying ntt and ntt^-1.

Fix build system annoyances

There are several automake / libtool features that prevent us from cleanly doing parallel builds. These have been fixed in other packages, so we should port those changes to toast.

Test suite does not work with installed modules

When running the test suite from within the source tree, but using an installed version of toast, there are errors. The installed code is usable outside of the source tree, and the tests work when running within the source tree using the develop script.

Add documentation about internals

We need some "mid-level" documentation covering the overall design philosophy to help people in writing custom pipeline scripts and new operators. Something higher-level than just the source code.

Implement or find PCG solver

Scipy has generic solvers, so perhaps we can use them. The critical feature is that we need to be able to perform MPI communication inside the left hand side evaluation. This solver is needed for both the full GLS mapmaking ported from TOAST, and also for the destriper. This issue is to track search results on this topic.

Noise timestream normalization

I am raising this issue even though I am close to fixing it, because it impacts simulations which are underway. In our current formalism for generating noise timestreams from a PSD, we usually express Parseval's theorem as: the variance of the generated noise timestream should equal the integral of the PSD from zero to Nyquist. There was actually not a unit test for this requirement, but I just added one in a branch.

The code in the master branch curently generates timestreams which are off by a factor of 2 from the normalization requirement above. However, this is not the whole story. After correcting the factor of 2, the TOD variance is also consistently slightly lower than the integral of the PSD. Recall that since we have a finite number of samples, we know that our estimated variance has a standard error given by

err(variance) = variance * sqrt(2/(N-1))

However, the TOD variance differs from the integral of the PSD by slightly more than this amount. I am still trying to understand this slight discrepancy.

Noise generation seed broken

OpSimNoise is missing functionality to set the random seeds based on the global stream index, the observation index, the number of chunks in a single observation, and the channel number.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.