This module already exists in the old version of the pipeline (not public), but I need to port it here.

Note.
I am not a big fan of demodulation (as opposed to pair-differencing), because it adds a lot of complexity for small gain in return. For example the sampling frequency of the detector needs to be quite big for the demodulation to be efficient so the memory increases a lot. Another example is that for demodulation to work, there are few filtering steps to perform (low-pass, band-pass), and this slows down a lot the code.

Currently, TOD weights are set to one (i.e. no noise weighting while performing mapmaking). While this is OK for noiseless runs, one would like to weight properly the TOD if one uses noise. There are several ways to weight the timestream, but one could simply use the PSD of the timestream (routines already exist). The problem currently is that it adds a bit to the runtime (precomputation and store on disk for MC runs?).

The difficulty here is to get something publicly available and easy/fast to run.
Reijo suggested using PolSpice.
I will give a try and report.

With recent versions of numpy, pyslalib does not seem to be happy...

The fact that we are currently using inifiles to define parameters to be used in App is only historical.
There is no real advantage using inifiles over python if the main script is in python.
First, it makes things more complex (needed to introduce an extra step using NormaliseParser to decode it), and second I think that if one can do something in python without suffering much, one must do.

Need to inspect that!

Does this include Neil's latest updates on scan strategy here?
What happens to subscan information? Do we have the turnarounds? How do you identify the subscans in the TOD?

I recently tried to run the doctests with python 3.7 on my machine with latest versions of dependencies, and I got many failures... Inspecting further, it seems that there are two separate issues: python 3.7 vs others, and dependency versions. I need to fix that...

The figure is pretty but the display is very poor with the dark mode activated.
A white background may be necessary (for once! :)

The code quality is very bad... and it should change (especially that we could have avoided many bugs). We should add a linter check in the GH actions.

I was trying to get the so_MC_app.py running and noticed I got an error that I didn't get with simple_app.py.
I traced it down to this difference in code between the two example apps:

== in simple_app.py ==

           d = []
           for det in tqdm(range(inst.focal_plane.nbolometer)):
                d.append(tod.map2tod(det))
           ## Project TOD to maps
           tod.tod2map(np.array(d), sky_out_tot)

== in so_MC_app.py ==

         for pair in tod.pair_list:
                d = np.array([tod.map2tod(det) for det in pair])
                ## Project TOD to maps
                tod.tod2map(d, sky_out_tot)

When I use the "so_MC_app" way in the simple app, I get the same error.
The error is:

Traceback (most recent call last):
File "examples/test/simple_app.py", line 179, in
tod.tod2map(d, sky_out_tot)
File "/data/home/joy/simons_observatory/code/s4cmb/s4cmb/tod.py", line 635, in tod2map
assert npixfp == self.point_matrix.shape[0]
AssertionError

Do you understand what's going on? Does the use of one code vs the other has to match the "perpair" parameter?
Thanks!!

Hi Julien !

Is it normal to have pixels falling outside the list of observed pixels (obspix) with the "deep_patch" scanning strategy, when we set width = 130 (default value) ?

Below is the inverse condition number of each pixel (1 CES, nside = 128), I realized with mollview that some pixels fall clearly out of the main sky patch defined by the scan.

The two maps are equal to within 2.5% on my machine hence the test suite fails when running ./coverage_and_test.sh

Following https://joss.readthedocs.io/en/latest/submitting.html, we can open a branch and push the paper.

s4cmb paper has been just accepted in JOSS! DOI: https://joss.theoj.org/papers/10.21105/joss.03022

Action item: update the README with the badge

1. It would be convenient to have a hardware input file in YAML that can be shared between TOAST an s4cmb, so that we can just import an experiment (SO for example).
2. We're also wondering if the following are implemented/easily implemented into the instrument setup

• square focal plane vs. hex focal plane layout
• dichroic detectors

3. Is there room for a hierarchy of detector info (wafer, physical location, electrical info) versus just detector index? For example, a bolometer class with these attributes.

The CI is only triggered when code is pushed to any branch in a repository. We would like instead to trigger the workflow on push or pull request events (eventually from forks).

Action item: add the pull_request event in the workflow.

I encountered this error while installing s4cmb on MacOS High Sierra.
Doing

export PATH=/usr/bin:$PATH

before installing worked for me.

It was an issue with my environment rather then the repo. I report it here in case someone else has the same problem.

Hi Julien,

I would like to try the dichroich example:
https://github.com/JulienPeloton/s4cmb-resources/blob/master/Part1/s4cmb_dichroic_06.ipynb

for which I need to checkout the origin/dichroic branch'.
I have never used before git, so I am a very new user. I tried looking online/forums but I still have the issue.

Basically, I do not see the 'origin/dichroich' branch in my local copy.
I tried several commands (fetch, pull origin master etc etc)....
I did:

git pull origin master

From https://github.com/MariaJK/s4cmb

• branch master -> FETCH_HEAD
  Already up-to-date.

git branch -r

origin/HEAD -> origin/master
origin/beam_ellipticity
origin/master

Moreover, looking at the github repo pag, https://github.com/JulienPeloton/s4cmb,
I only see 'master' and 'correlated_noise'. Joy found the same.

Could you help me?
Thanks,
Maria

When running with any ini file that is not simple_parameter.py, this error comes up:

python examples/test/simple_app.py -inifile examples/inifiles/so_sac_parameters.py -tag test

Traceback (most recent call last):
File "examples/test/simple_app.py", line 155, in
mapping_perpair=params.mapping_perpair)
File "/data/home/joy/simons_observatory/code/s4cmb/s4cmb/tod.py", line 127, in init
self.get_boresightpointing()
File "/data/home/joy/simons_observatory/code/s4cmb/s4cmb/tod.py", line 426, in get_boresightpointing
lat=lat, ra_src=ra_src, dec_src=dec_src)
File "/data/home/joy/simons_observatory/code/s4cmb/s4cmb/detector_pointing.py", line 138, in init
self.ut1utc = get_ut1utc(self.ut1utc_fn, self.time[0])
File "/data/home/joy/simons_observatory/code/s4cmb/s4cmb/detector_pointing.py", line 53, in get_ut1utc
ut1utc = ut1utcs[uindex]
IndexError: index 2193 is out of bounds for axis 0 with size 2193

Is there a way to deal with beams without analytical solutions? For example, if someone provided a beam map, could this be used in the sims? If so, what format of beam map would be easy to implement?

Vast topic. The first things that come to my mind are

• Modifying the TOD model, by including templates that will be filtered out in the mapmaking stage.
• Deprojecting the signal at the TOD level directly (e.g. a la SPTPol for dealing with the detector crosstalk).

I might give a try soon at the first point.

From the test suites newly setup in GH actions:
https://github.com/JulienPeloton/s4cmb/runs/1761872495?check_suite_focus=true

This issue has been brought by Maria S.
By default on Cori, an intel environment is loaded.
NERSC recently switched on using a new default version of ifort (the fortran intel compiler) which doesn’t support anymore the option -openmp (but -qopenmp).

Unfortunately, f2py (which is used for pyslalib or s4cmb for example) didn’t implement this change (it uses -openmp by default for Intel), hence there will be a problem at compilation.
You check that just typing the following in your terminal:

$ f2py -c --help-fcompiler

A quick workaround if you really want to use Intel is to unload the current module, and load a previous version:

$ module unload intel
$ module load intel/17.0.2.174

You see a deprecation warning, but that should work.
Another option (which I do) is to use GNU compilers only.

I leave this issue open in case someone still has a problem.

I have been trying to do a recent reinstall with a fresh environment using conda and MacOS Ventura. The following command installs an environment that still works.

conda create -n s4cmb-dev -c conda-forge python=3.7 camb openmpi ephem pyslalib jupyter healpy coverage coveralls mpi4py

However switching just the python version to 3.9 the compilation seems ok but there are multiple failures on the test level. The difference between the 2 environment in dumpy/f2py is the version that moves from 1.21 to 1.24.

**********************************************************************
File "/Users/gfabbian/Software/s4cmb-dev/s4cmb/s4cmb/tod.py", line 1321, in __main__.TimeOrderedDataPairDiff.tod2map
Failed example:
    for pair in tod.pair_list:
      d = np.array([tod.map2tod(det) for det in pair])
      tod.tod2map(d, m, gdeprojection=True)
Exception raised:
    Traceback (most recent call last):
      File "/Users/gfabbian/opt/anaconda3/envs/s4cmb-dev/lib/python3.9/doctest.py", line 1334, in __run
        exec(compile(example.source, filename, "single",
      File "<doctest __main__.TimeOrderedDataPairDiff.tod2map[31]>", line 3, in <module>
        tod.tod2map(d, m, gdeprojection=True)
      File "/Users/gfabbian/Software/s4cmb-dev/s4cmb/s4cmb/tod.py", line 1406, in tod2map
        tod_f.tod2map_pair_f(
    tod_f.error: (shape(diff_weight, 0) == npix) failed for 14th argument npix: tod2map_pair_f:npix=139992

The behavior behavior seems connected to how f2py/numpy interpret data types with the current fortran code written in the repo. Suggestions welcome.

We need a realistic atmospheric model and common mode subtraction for many systematics. We need a realistic model that we can use with some level of 1/f noise and correlation across focal plane. We need it to have both temporal and spatial variations. Also need to have some random noise after common noise subtraction. It may be that we can model this in TOAST with a given focal plane and import a per pixel TOD. (Let's discuss this)

Hi Julien,

I am sorry, I cannot load s4cmb from NERSC jupyter.
Here is the sequence of commands I did:

(from a putty terminal)
I did:

$ module unload intel
$ module load gcc/4.9.3

$ module load python/2.7-anaconda
$ pip install s4cmb --user
$ pip install -r requirements.txt
$ make

(I didn't find errors in running up to 'make' command).

and then I open the jupyter notebook from

https://jupyter-dev.nersc.gov

The notebook does not find the module 'instrument' and the following ones (see attached snapshots of my account).

I have tried also loading
$ module load python/2.7-anaconda-4.4
without any difference.

Thanks,
Maria

In scanning multiple CESs, the observed pixels are fixed from the first CES and fixed for the rest, but these may change slightly with each CES (since the array is extended).

In tod.py, compute_simpolangle function.