It would be good to use the work from @yoachim and @dkirkby.

As discussed in PR #4.

I think I will try to make a single source at the center of the focal plane and see what happens.

Currently, attributes like .PSF and .noise_and_background for the ImSimStars and ImSimGalaxies class are instance-level variables, though they should likely be the same for all object types. Making these class-level variables would enforce that, but the GalSimBase class in sims_GalSimInterface would need to be modified.

The sims_GalSimInterface code expects radians for the position angle, but the units in the instance catalogs are degrees.

Currently, there will be no vignetting in produced images across the focal plane. There are several ways this could be addressed.

In a sky model (issue #5) it could manually be added or, if in that model, the light from the sky was actually raytraced through the atmosphere (issue #8) and into the optics (issue #10) it could happen naturally.

I think an appropriate obscuration of the pupil in the optics FFT convolution might also naturally handle this (an expert should comment).

This issue is just a place holder to remind us that it is an issue.

We should at least add the option to turn this on and perhaps use it in DC2. It exists already as an option in GalSim as implemented by @jmeyers314. The code is here.

https://github.com/GalSim-developers/GalSim/blob/releases/1.4/galsim/dcr.py

The final columns in the object lines of an instance catalog contain the parameters for the spatial model and for the rest frame and Galactic extinction. According to the phosim documentation, the extinction entries can have four forms:

1. <internal dust model> iA_v iR_v <Galactic dust model> gA_v gR_v
2. none <Galactic dust model> gA_v gR_v
3. <internal dust model> iA_v iR_v none
4. none none

where the choices for dust model are CCM or calzetti and the A_vs and R_vs are the model parameters, and none means that no dust model is to be applied. In practice, only 1. and 2. will appear, but owing to a bug, CatSim currently writes the Galactic entry before the internal one; and the code now in imSim only handles cases that look like 1. and 3. and assumes that ordering. Since CatSim will presumably soon be fixed, the parser code should be updated to handle all four cases with the correct ordering. Twinkles instance catalogs for Run3 already have the fix applied.

The spatial_pars columns can have 0, 1, 2, 4, or 6 columns depending on the source type, so those columns should also be handled correctly. The current imSim code only handles point and sersic2d models.

Add the electronics readout to the e-image files. Must understand how this interfaces with GalSim-developers/GalSim#799 (mid-layer API for drawImage).

Issue #5 implements the ESO sky model.

We should replace the function that converts the sky level electron counts (from @dkirkby and the WL deblending package) with calls to the system utilities in photUtils. @danielsf has summarized how these functions work in a recent DESC SSim meeting.

The function to be replaced is:

def skyCountsPerSec(surface_brightness=21, filter_band='r',
effective_area=32.4u.m**2, pixel_size=0.2u.arcsec):

The zero points in this function now are hardcoded from the calculations by @dkirkby and @fjaviersanchez in the "[skyModel_params]" section of data/default_imsim_configs.

This is related to the fixes discussed at LSSTDESC/SSim_DC1#36

On OS X SIP strips out LIBRARY_PATH related environment variables. This destroys the EUPS based setup system.

In practice for imSim this means this works:

python bin/imsim.py tests/tiny_instcat.txt

but this

bin/imsim.py tests/tiny_instcat.txt

does not.

This is due to the fact that the script starts with

#!/usr/bin/env python

Instead of the actual relevant python binary we want to use.

@timj has written a DM tech note describing this issue and solutions here:

http://dmtn-001.lsst.io/en/master/

We can either just always run explicitly with python on Mac and make sure this is well documented, or we could apply the DM solution which is to start with a bin.src directory and then have an installation procedure which rewrites the shebang and copies the new binary to "bin".

See discussion at #2 (comment) . I'll also refactor the command dictionary produced by parsePhoSimInstanceFile to make it a dictionary instead of a dataframe.

This may belong as a GalSim issue (does one already exist?) @aaronroodman was thinking about this problem.

PhoSim has a useful feature described as:

'''
Centroid files contain the true numbers of photons that were detector for every source, and the average x and y position. This is useful for crowded fields where the photons from each source can be tagged internally in the simulator.
'''

This may actually be better handled as a GalSim feature as it could be part of the raytracing in either/both the atmosphere and the sensor.

One of the initial driving development philosophies for imSim was to make it a drop in for PhoSim which could read the same instance files. This was very successful for allowing us to use it quickly and effectively.

However, as we have used it we have found we wanted to do somethings differently (like issues related to proper motion and nutation) and now the formats are in fact not exactly equivalent. Now that we have more of an infrastructure built up around the imSim ecosystem @jchiang87 and others have suggested we start from scratch and build a new parser that has all of the features we really want and need. Previous discussion about this topic has happened in email and also can also be found in LSSTDESC/DC2-production#16.

This parser could use binary data input files with several related pandas dataframes that would be compact and would also serve to store all of the related truth information in a way we can't now.

This format needs to serve several roles:

1. Production runs generated from CatSim for the data challenges
2. R&D work on the small scale
3. Sensor only or calibration simulations for validation.

Having the ability to also have text based input (especially for 2 and 3) is also very desirable.

In the short term we should discuss whether this is a DC2 or DC3 era job. This format should be clearly defined so that if could also be used by other tools (including PhoSim) if desired.

• Enable the package to build and run on OS X El Capitan. #18
• Connect repo to CI systems. #20
• Add command/configuration file capability. #11

We have at least one cloud model in the project that could be used for simulating observations that have some cloud cover. I don't know exactly how to parameterize this. Perhaps Peter Yoachim or Lynne Jones would have ideas. To first order it could just be a parameter indicating how thick a uniform, grey layer of clouds is. Taking it a bit further, one could potentially specify a structure function.

Currently, some routines in the interface library are monkey patched here in order to separate the source input mechanism from the GalSim interface code itself.

We should consider the class layout more generally and the relationship between this driving code and the library. We can also drive other sorts of lookups (e.g. CatSim directly) but perhaps other things might use the library too.

This is a placeholder just to remind us this is an issue.

After the fix to #62, if two snaps are requested the sky and sources will all be normalized by the total exposure time.

To better simulate the actual telescope output if multiple snaps are requested then a fits file should be produced for each snap.

Seeing GalSim GitHub email today reminded me of this...

@craiglagegit Has shown that he can put tree rings into this Poisson simulation and they add linearly to the BF distortions. However, we cannot pre-calculate all possible tree-ring orientations. We need to have a template that can oriented (randomly?) and be added to the BF area distortions in the GalSim module.

It has been suggested we could just do this with a WCS (as apparently DES does). However, our correction will rely on WCS and we would rather implement this in the model itself.

This is one of the items on the checklist for a validated sensor model in #7.

Now that V12.1 based sims releases are available it should be possible to remove the backward compatible calls to the pexlogger and galsim interface measurement libraries from the code base.

We can make Sims tag 2.3.1 a minimum requirement.

On DC1 scale time frames we are using e-image files with only saturation. We are assuming everything else is corrected as DM does not yet have tools to handle all of the other sensor effects.

There is also the related issue of non-linearity but I think that should show up in the electronics module. Saturation is really an electron effect in the sensor.

There are (at least?) three (?) ways saturation can be included:

• (Best) The new framework that @rmjarvis is preparing in GalSim-developers/GalSim#799 could include by hosting the new sensor models which include it. Currently the models are really focused on BF and don't yet implement this.

• In the shorter term can Mike's framework apply this at one of the steps?

• Is there an option to already to this in the photon shooting in GalSim? I looked at issues etc and I don't see anything. But, somehow I thought I remembered seeing this option before being used in the context of WFIRST.

I think we need this before we can do meaningful comparisons with PhoSim . Can @rmjarvis or @rmandelb comment on they best way to do this?

In the lsst.sims.photUtils.PhotometricParameters class, nsnap is the nexp attribute.

In this issue I am making a list of features that should be configurable (on in some cases implemented) in the command file described in issue #11.

Options associated with issues:

• - Set exposure time variables (issue #22)
• - Set atmospheric handling (raytraced vs FFT issue #8)
• - Set sky model (ESO vs GalSim default etc issue #5)

Not yet associated with issues:

• Add the ability to drive different types of inputs with different input files in addition to PhoSim instance files.
  • I imagine direct CatSim queries might be possible.
  • The ability to drive a setup of CCD sensor level simulations to compare with test stand data for validating the sensor model. The goal will be to generate a set of data vs simulation reference plots tied to our requirements. Any implemented model should be able to show adequate agreement (as defined by requirements) between the simulation and test stand data in all of these plots and be callable by the GalSim interface being developed by @rmjarvis. Specification of what these plots should contain should be make in consultation with the camera team (Note: @smr456) simulators, test stand owners and SAWG.

I have (finally) started trying to address the position angle problem in imSim. Unfortunately, I seem to have confused myself.

As a first step, I generated an image of one extremely ellipsoidal galaxy with known position angle (in this case 45 degrees) in both GalSim and PhoSim so that I could verify that they do, indeed disagree. I have attached

1. The InstanceCatalog I fed to PhoSim (the *.txt file)
2. The e-image produced by PhoSim v 3.5.3
3. The FITS image produced by sims_GalSimInterface

(the images are screenshots of what I saw when I opened the FITS files with ds9). Looking at the two images in ds9 I notice three things.

1. The WCSs don't agree on where North is. This could be an artifact of the disagreement between how DM and the camera team define the x, y axes on the detector. sims_GalSimInterface defines x, y the way DM does.

2. The WCSs don't agree on the relationship between North and East. I can rationalize this by saying the PhoSim image is an e-image, which means it should appear as if you were looking down on it from above, rather than up at the sky (though I do not know that this is true).

3. Both images actually agree on what a position angle of 45 degrees means, but both define a positive position angle as rotating from North towards West. I thought position angle was supposed to be defined form North towards East.

Probably some of these are bugs and some of these are expected behaviors. I'm not 100% sure which are which, though. Thoughts?

@cwwalter @jchiang87 @johnrpeterson

phosim_rotsky0.txt

Issue #5 implemented the ESO sky model. However, now the level is set solely by the pointing at the center of the focal plane.

The code should be modified to vary per chip and then per pixel. This probably means refactoring the code to recalculate the sky level in a healpix map once outside of the chip loop.

The problem is that when phoSimStarCatalog.get_fitsFiles() is called, there are no objects to be simulated and no bandPassDict is created because of
https://github.com/lsst/sims_GalSimInterface/blob/master/python/lsst/sims/GalSimInterface/galSimCatalogs.py#L358
and this line in imsim.py
https://github.com/LSSTDESC/imSim/blob/master/bin.src/imsim.py#L153
fails with

Traceback (most recent call last):
  File "/gpfs/slac/kipac/fs1/g/desc/imsim_deep/imSim/bin/imsim.py", line 179, in <module>
    main()
  File "/gpfs/slac/kipac/fs1/g/desc/imsim_deep/imSim/bin/imsim.py", line 165, in main
    phoSimGalaxyCatalog.copyGalSimInterpreter(phoSimStarCatalog)
  File "/gpfs/slac/kipac/fs1/g/desc/imsim_deep/PRs/sims_GalSimInterface/python/lsst/sims/GalSimInterface/galSimCatalogs.py", line 429, in copyGalSimInterpreter
    self.bandpassDict = otherCatalog.bandpassDict
AttributeError: 'ImSimStars' object has no attribute 'bandpassDict'

The commands at issue are vistime, nsnap, and possibly exptime. Some relevant discussion is at #15 (comment)

For cases where the --sensor option is specified for imsim.py, we should set the allowed_chips attribute to a list containing the chip name: https://github.com/lsst/sims_GalSimInterface/blob/master/python/lsst/sims/GalSimInterface/galSimCatalogs.py#L168

The following features need to be added/validated to the GalSim module before we can use it for DC2.

• Leveraging the interface in GalSim-developers/GalSim#799 (mid-layer API for drawImage)
• Validate the brighter-fatter code. Being done by @craiglagegit and Kristen (no Github?)
• Implement adding per-sensor orientation tree rings (#59).
• The model does saturation, and blooming. We need to add bleed trails (in @jchiang87 readout code in #6?)
• Add cosmic rays #76
• Performance tests #86

We should start to further document imSim on the wiki page here. We should document the format of the command and input files along with references to software packages and physics models we are using either here or in GalSim.

More detailed installation with links to the sims stack would also be appropriate.

Currently the parser and interface only handle stars and 2d Sersics. These were the only examples in the instance catalog I saw.

However, there are other types in GalSim (like Gaussians and the new inclined Sersic) which could handle more PhoSim morphologies.

The full list of PhoSim spatial models is below. Currently, all modes other than point and sersic2D are ignored.

• point (No parameters This is a model primarily used for stars, but also unresolved objects.)
• gauss (1 parameter: sigma in arcseconds This is a model for a gaussian-shaped object.)
• movingpoint (2 parameters: the derivative of the velocity arcseconds per second along the ra direction, the derivative of the velocity in arcseconds per second along the dec direction)
• sersic2D (4 parameters: half-light radius of semi-major axis in arcseconds, half-light radius of semi-minor axis in arcseconds, position angle in radians, sersic index)
• sersic (6 parameters: size of axis 1 in arcseconds, size of axis 2 in arcseconds, size of axis 3 in arcseconds, sersic index, polar angle in radians, position angle in radians)
• If the SOURCE_TYPE contains the word fits or fit,it will look for that file in the image directory; (2 parameters: pixel size (in arcseconds) and rotation angle (in degrees). RA goes to the left and DEC goes up in physical coordinates Note that PhoSim will not use the header information, because you may want to use the same image in multiple field locations)

I think the twinkles catalogs currently only contain the cases handled but others could be added in the future to this interface. Alternatively, other types could only be handled in other interfaces.

This issue tracks the integration of @jmeyers314 's Batoid raytracing code with imsim.

Here we list important effects that we should be able to now implement.

• Vignetting
• Diffraction Spikes from Spider
• Ghosts and scattered light.

imsim.py responds ungracefully when certain instance catalog edge cases are encountered:

• No object entries at all.
• objects with type sersic2d but no point entries

During the Weak Lensing meeting today it was mentioned it would be good to know the actual true PSF for each object drawn on the focal plane when we were measuring and interpolating the PSFs.

By making sure we keep the random # seeds, or even storing the PSFs themselves it would be possible to do this either on the fly or by lookup.

Is there a GalSim issue for this yet? @jmeyers314 will work on this at the hack day.

@dkirkby showed me examples from DESI where units are specified for everything including in the configuration files.

It looked very useful/powerful. We could consider this too. Doing it early would be better as it would get harder and harder to do as more gets added to the program. David may point to examples.

CatSim code would then be used to apply Earth precession and nutation to convert to geocentric apparent RA, Dec. We should retain the ability to read in PhoSim compatible instance catalogs.

Issue #5 implemented the ESO sky model. The absolute value of the dark sky in the red band seems reasonable. With version 0.1.0 it is 438 electrons for a 15 second exposure.

This can be compared with https://github.com/lsst-pst/syseng_throughputs/blob/master/plots/table2
Where the value in r for a dark sky for 30 seconds is: 796 which corresponds to a 15 second level of 398.

However, there are some discrepancies with the PhoSim calculation. For this previous history of this comparison please see issue #5.

This issue should be closed when we are confident that this discrepancy is understood.

For sersic2d objects in the phosim instance catalogs, the 8th, 9th, and 10th columns (the three following the redshift) are the weak lensing distortion parameters, gamma1, gamma2, and kappa. Currently, we aren't doing anything with them (and catsim doesn't set them, so they are zero in the Twinkles instance catalogs, for example). So the sims_GalSimInterface code needs to be updated to make use of them, and we need to pass those parameters from the instance catalog. Also, we need a means of filling those values in the instance catalogs given some lensing map.

It would be useful to embed configuration values into the output files themselves.

Currently (with a parameterized PSF) the PSF does not vary over the focal plane.

With the addition of #37 (passing the focal plane position to GalSim) this could be modified and a parameterized function could be used in GalSim.

On the other hand with a ray-traced atmosphere and an extra phase screen representing the perturbed optics (#9) I believe this would be naturally emergent. So hopefully we get this in DC2 by implementing the other two options.

Some valid phosim instance file commands (like rotskytel) are not in the list of valid commands.

imSim should either use these, or more gracefully ignore them. The current behavior results in a exception.

Also allowing commenting out lines with a '#' should be supported.

So, I don't forget:

We need to generate a realistic aperture with spider etc. As mentioned in #16 we may also want to make sure vignetting can be handled by correct pupil obscuration.

I recall @rmandelb was already working on a realistic LSST geometry in GalSim itself. So perhaps we can just call that? If not, and we need to implement it, we either need to do it there or define it here and then pass it to the proper routines if that is possible.

This requires us first building the stack in the KNL partition.

We might not need to do this for DC2 given imSim computational requirements but there is a good chance much of our allocated resources will be on KNL so we should understand stack and imSim performance there.

Add the ability to read a command file which will set non-visit specific configuration options.

We need to think both about what options go in it and the format/configuration language.

This issue comes from a conversation @craiglagegit and I had today at PACCD. In the the new GalSim code for the sensors, the angle for the photons are drawn from a distribution for the f1.4 beam before propagation into the silicon.

Currently, the code assumes you are and the center of the focal plane. When we call GalSim from imSim we actually know the global position on the focal plane. So, if that information gets passed along it should be possible to use an appropriate distribution.