PhoSim indexing fails with very large numbers of input objects about ssim_dc1 HOT 12 CLOSED

Thanks for moving/filing this!

from ssim_dc1.

@rbiswas4 and @drphilmarshall Is this an issue for us in DC1? DC2? According to PHOSM-27, John feels this shouldn't be an issue since really they are stored in double precision.

from ssim_dc1.

It is a DC1 issue. I think this was addressed by @danielsf and @SimonKrughoff in the context of Twinkles. The solution depended on the fact that we had a really small area. This may or may not be ok for DC2, depending on the area.

from ssim_dc1.

So, just to make sure I understand from reading the Twinkles PR: This an issue solely in making the catalogs correct?

If this is going to be an issue for DC2, some one should follow up on PhoSim-27 and let John know you don't think his comment really solves the issue.

from ssim_dc1.

So, just to make sure I understand from reading the Twinkles PR: This an issue solely in making the catalogs correct?

Not sure what you mean: The catalogs and phosim images are actually generated correctly (at least in the cases we have seen). However, if you want to do comparisons of the input catalogs and the phosim centroid files (which I think is a very useful step), then the indices are often different (because of the representation as doubles).

If this is going to be an issue for DC2, some one should follow up on PhoSim-27 and let John know you don't think his comment really solves the issue.

Whether it is an issue might depend on the size of DC2 ... I am traveling this week, so I don't have the details. I guess @danielsf would be able to help ...

from ssim_dc1.

I actually do not think this will be a problem for DC2 (nor was it a problem for DC1). It was only an issue in Twinkles. Someone should check my reasoning though:

PhoSim stores the indices as double-precision floats. This means they can store an integer of 2^52 ~ 4.5e15 exactly. Therefore, the PhoSim indexing would only break down if we ended up needing more than 4.5e15 independent sources. Twinkles ran afoul of this limit because, when we injected lensed AGN and SNe, we were forced to bit-pack their indexes to prevent them from colliding with the uniqueIDs of the ~ 3.5e10 galaxy uniqueIDs already allotted by CatSim.

So: as long as we aren't injecting new transients into DC2 (are we?), we should be able to live with the way PhoSim currently handles object indexes.

from ssim_dc1.

Now the thinking is that DC2 will include transients at normal density with an embedded Twinkles field.

from ssim_dc1.

In that case, we will have a problem. As Rahul said, we were able to fix this in Twinkles because it was a single chip on a single point in the sky, so we only had to keep a few million unique galaxy indices, rather than a few tens of billions. Once we break that assumption, we need PhoSim to store indices as ints or strings.

from ssim_dc1.

I thought @SimonKrughoff hacked together a quick solution on his own branch of PhoSim and issued a pull request. Whether or not that pull request gets merged to PhoSim master, could we just use Simon's branch with his quick fix?

from ssim_dc1.

transients at normal density with an embedded Twinkles field
@cwwalter Twinkles had transients/variables at higher than normal density and it was a 10 year survey.

So, to me the questions are:

• How many years is DC2 ?
• What is the density of transient/variables used? From your statement above does it mean that it is the normal density except for a particular chip location where it will be overdense as in Twinkles?

These two questions will be important in quantifying the number of transient/variables (as a hack) Scott is talking about.

from ssim_dc1.

Yes. Your understanding is correct. Right now we are planning 10 years, 6 bands, 300 sq degrees. We will have normal density for transients over the WFD field and an embedded Twinkles DDF. I'm not sure what the size of that is. That is for conversation at SUNYSB.

from ssim_dc1.

I think that this is not relevant anymore so I'm closing this (please, feel free to reopen @cwwalter)

from ssim_dc1.