hpc4cmb / toast Goto Github PK
View Code? Open in Web Editor NEWTime Ordered Astrophysics Scalable Tools
License: Other
Time Ordered Astrophysics Scalable Tools
License: Other
Quaternions come in two branches: quat and -quat describe the same final orientation. When interpolating between two quaternions, it is imperative to ensure the two are from the same branch.
Currently, running setup.py instantiates the MPI-aware unit test runner. This effectively means that in order to install the package (python3 setup.py install), it must be possible to initialize MPI. However, on some systems (NERSC), MPI cannot be initialized on the login nodes. This is inconvenient, but when installing from a compute node, the global/common filesystem is mounted read-only. So we effectively cannot install pytoast to global/common. We should be able to optionally enable MPI somehow to bypass this.
We should add support for generating common node noise between detectors
Add a free function to toast.tod.tod.py that takes a sparsely sampled set of gain factors, interpolates them to detector timestamps (using some interpolation order), and applies them element-wise to a data vector.
It would be good to modify the healpix pointing operator to add arguments for the input and desired coordinate systems.
Following Josquin Errard's approach, construct realizations of the atmospheric emission and pass them across the field of view.
Since current noise sims have each process simulate a piece of timestream independently, there will be discontinuities at the process boundaries. The solution is to either redistribute the data so that one process has the whole timestream, OR have every process simulate the whole dataset up to their local start sample, OR to use a python wrapper around fftw-mpi.
In addition to the seed, we should pass the stream index.
Currently we only use time-based data distribution within the communicator that handles a given observation. There are some cases where we want to have a timestream assigned to a single process, to facilitate simulation. Add functions to redistribute and communicate data within an observation.
The TOD base class is used to cache pointing, pointing matrices, etc. These vectors are numpy arrays stored in a dictionary. Even when these dictionary keys are explicitly deleted, the memory does not seem to be garbage collected (based on memory_profiler output).
Many recent functions that are in the litebird software stack are more general and should be moved to pytoast proper.
Hi guys:
https://github.com/hpc4cmb/toast/blob/master/toast/map/madam.py#L318
if self._apply_flag is False and self._purge is True, then the variable cacheflagname is accessed prior to initialization.
Should be easy to fix!
Cheers,
Brendan
We should implement a TOD derived class which reads / writes to a tidas volume (HDF5 backend). This is pending several things, in particular a few remaining issues holding up the initial tidas release. This functionality would be useful for dumping timestreams to disk for both debugging and external validation by other software tools.
change the interface to libmadam to take an array of pointers to each detector's data. This would allow us to avoid duplicating all the memory when passing it to libmadam
The current unit tests that call libmadam do not include a list of valid intervals. We should add that feature to improve our test coverage.
To make the interface more consistent, the read_flags() method should be the only one returning flags. The read(), read_pntg(), etc methods should just return data. For some experiment-specific TOD classes reading some data formats (e.g. FITS binary tables), it is most efficient to read flags at the same time as the data and cache them. That is a detail left to those classes.
This issue is for documenting design discussions held offline concerning the use of compiled code in toast. There are a variety of tradeoffs in play, but broadly:
All of these thoughts / requirements boil down to some design changes that should improve performance, development workflow, and usability:
In the coming weeks we'll be adding a feature branch to work on this reorganization.
This problem was actually observed with the LiteBIRD simulation script, which is very similar to the example script. The simulation configuration was using observation lengths of 23 hours, with a one hour gap in between. The data was distributed in chunks of 1 hour. The symptom is that when running with one simulated day per node, things seem to usually work fine. When running with different concurrencies (including having more nodes than days). These jobs fail in the middle of libmadam execution. Initially this was thought to be an out-of-memory condition. This is no longer clear. More investigation is needed on smaller jobs that explore the parameter space of one hour chunks to find the source of this problem.
We often just want all the samples, specifying local_start and n is tedious.
The TOD base class is currently acting as both an interface definition for internal methods called by the high-level read/write functions and as a memory-backed generic TOD container. We should split this functionality into the actual base class where the internal methods (prefixed with underscore) raise exceptions when called and a separate derived class which has a memory-based backend.
Currently the read/write methods of the TOD base class are being abused to implement timestream caching. This prevents descendant classes from using this cache. This should be moved into an explicit, separate object. The TOD base class can still contain an instance of this "cache" object, but this will then be available to all descendents. It also opens the possibility for us to explicitly manage the memory of this cache outside of python, which may be needed to address things like issue #13.
The current noise PSD model is not being generated at high enough resolution for the interpolation. Also, we probably remove the built-in high pass in this model.
Currently, toast operators have an "exec()" method which takes as an argument the high-level distributed data object. This contains one or more observations, each with some number of locally-stored detector TODs. These operators then loop over all detectors in each observation and "do stuff", potentially storing the results of some calculation in the TOD's cache. For complex workflows, this can lead to a huge number of intermediate temporary vectors in memory. For certain types of workflows (that don't require collective operations between detectors), it would be better to do an outer loop over detectors and run a chain of operators on each detector up to some point.
After some offline discussion with @keskitalo , an easy way to support this would be to pass a list of detector names as an optional argument to the exec() method. An operator could then support working with only a subset of detectors (in particular a single detector). At the top pipeline level, the chain of operators could be called one detector at a time, purging the cached products after each one. This type of loop could co-exist with other top-level operators that worked with the full detector set in one go.
This change would be backwards compatible with the current pipelines.
Add docstrings for all public classes and functions. Expand examples. Update generated docs on gh-pages.
In order to load a map for scanning or to write out a distributed pixel covariance, we need to implement read-and-broadcast and reduce-and-write.
The TOD class represents data from a telescope. There should be general methods to return the position and velocity vectors of the telescope.
All planck extensions should go to new pytoast-planck repo.
Implement an operator which uses libconviqt to to simulate data for all detectors in an observation.
When building TOAST from source, Elemental is a strict requirement and not optional, as the documentation alludes to.
In toast/src/libtoast/toast.cpp with HAVE_ELEMENTAL, the IFDEF will break the namespace of toast when compiling the c++ dependencies.
Configure, make, and the documentation should be modified to reflect this.
We should clean up and update the set of example scripts in toast. This is loosely coupled to updates to the documentation. These scripts can serve as both use case examples and for use with scaling tests on Knights Landing. We should have a set of scripts of different complexity that can then also be run at different concurrency.
Pipeline Scripts:
Configurations:
For each pipeline and configuration, we should provide a shell script to generate any auxiliary data, and then a slurm script for running it at NERSC (if relevant). For the satellite sims, we can keep the time length fixed and scale the number of detectors. For the ground based sims, we can keep the detector count fixed and scale the time range.
It pains me to open this issue, but it looks like there is a problem with a few lines of code which results in generated noise timestreams having the incorrect PSD. The problem is that the Fourier domain vector that is input to scipy.fft.irfft expects complex values packed as pairs of real and imaginary values, and we have instead been passing in data in the half-complex fftw storage ordering. The result is that the PSD amplitude is applied in the Fourier domain to the wrong elements, resulting in some low frequency power being shifted to higher frequency. This failure mode actually still allows our unit tests (based on overall normalization) to pass.
This was discovered when looking at pull request #67 in more detail. That pull request fixes the issue as a side effect of switching the numpy fft package.
I'm working to fix this now, but wanted to open this issue for the sake of transparency.
Move the Cache class out of tod and up one level. Use an instance of Cache for local data storage in the DistPixels class. This will make it easier to manage memory used by map domain objects.
Some of our optional dependencies (Elemental) can be quite challenging to install. We should give some more details here. Also, we are almost completely independent of healpy in the core toast package, so we might consider finishing that transition.
Currently, pointing constructed with OpPointingPlanck passing 'mode = "IQU"' to the constructor results in OpMadam not solving for any pixels.
We will need a set of common routines for various pointing operations: aberration calculation, positional astronomy operations, coordinate transforms, etc. These are free functions that we should put into a new source file in toast.tod
Add a class that implements a exclusive lock using MPI-2 one sided communication. Similar to what is done in HARP.
It is fine for the base class to provide its list of internal PSDs. However, we should allow derived classes to provide generic methods to multiply by the time domain noise covariance. This would allow (for example) inclusion of a filter in the noise model.
We should have an operator in toast.tod.sim_detdata that generates a dipole signal for each detector. This would be computed based on the detector pointing and telescope velocity.
When building with gcc and linking to modern MKL versions, the MKL unit tests fail. This happens for both single and batched FFTs. This problem is not seen when using FFTW. I will disable MKL FFTs in the mean time.
The use of the DistPixels class and externally calculating the locally hit pixels is "clunky". We should provide a more intuitive way of just getting the already-distributed hits and noise covariance without lots of boilerplate.
Currently, OpSimNoise just implements different seeds for each detector in an observation. This does not guarantee statistical independence. Wrap rng123 with python so we can use it.
We should construct a generic class for simulated constant elevation scans from the ground. The PALPY package is actively maintained successor to pyslalib that we can use.
Create a base class which represents the noise properties for all detectors for a given observation. For the first implementation, this is just a list of PSDs, one per detector. The interface will include methods for applying ntt and ntt^-1.
There are several automake / libtool features that prevent us from cleanly doing parallel builds. These have been fixed in other packages, so we should port those changes to toast.
When running the test suite from within the source tree, but using an installed version of toast, there are errors. The installed code is usable outside of the source tree, and the tests work when running within the source tree using the develop script.
We need some "mid-level" documentation covering the overall design philosophy to help people in writing custom pipeline scripts and new operators. Something higher-level than just the source code.
Scipy has generic solvers, so perhaps we can use them. The critical feature is that we need to be able to perform MPI communication inside the left hand side evaluation. This solver is needed for both the full GLS mapmaking ported from TOAST, and also for the destriper. This issue is to track search results on this topic.
I am raising this issue even though I am close to fixing it, because it impacts simulations which are underway. In our current formalism for generating noise timestreams from a PSD, we usually express Parseval's theorem as: the variance of the generated noise timestream should equal the integral of the PSD from zero to Nyquist. There was actually not a unit test for this requirement, but I just added one in a branch.
The code in the master branch curently generates timestreams which are off by a factor of 2 from the normalization requirement above. However, this is not the whole story. After correcting the factor of 2, the TOD variance is also consistently slightly lower than the integral of the PSD. Recall that since we have a finite number of samples, we know that our estimated variance has a standard error given by
err(variance) = variance * sqrt(2/(N-1))
However, the TOD variance differs from the integral of the PSD by slightly more than this amount. I am still trying to understand this slight discrepancy.
OpSimNoise is missing functionality to set the random seeds based on the global stream index, the observation index, the number of chunks in a single observation, and the channel number.
Some files did not make it into the EXTRA_DIST automake variable. Same problem was just fixed in another project: hpc4cmb/tidas@5b0823d
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.