Best practice for detectors other than CSPAD about dials_refinement_brewster2018 HOT 4 OPEN

With the hierachical strategy, I got:

Detector 1 RMSDs by panel:
+---------+--------+----------+----------+-----------------+
|   Panel |   Nref |   RMSD_X |   RMSD_Y |   RMSD_DeltaPsi |
|      id |        |     (px) |     (px) |           (deg) |
|---------+--------+----------+----------+-----------------|
|       0 |  10604 |  0.63188 |  0.41962 |        0.042608 |
|       1 |  18130 |  0.34461 |  0.47687 |        0.07479  |
|       2 |  19307 |  0.38917 |  0.46545 |        0.075072 |
|       3 |  10393 |  0.65391 |  0.38995 |        0.041234 |
|       4 |  10175 |  0.65817 |  0.41343 |        0.040321 |
|       5 |  19232 |  0.38799 |  0.48158 |        0.073116 |
|       6 |  18340 |  0.34653 |  0.48039 |        0.07747  |
|       7 |  10588 |  0.61312 |  0.41518 |        0.042859 |
+---------+--------+----------+----------+-----------------+

Interestingly, the outer panels (0, 3, 4, 7) have worse RMSD_X but better RMSD_Y and RMSD_DeltaPsi than inner panels. Worse RMSD_X is understandable, as the spots are more streaky due to bandwidth and parallax, but better RMSD_DeltaPsi is puzzling. Does it simply reflect the fact that you need smaller angles to rotate higher resolution spots (i.e. further from the origin in the reciprocal space) to move them the same distance as low resolution spots?

Do you observe similar effects on MPCCD, CSPAD, etc?

from dials_refinement_brewster2018.

Thanks for the questions! The program cspad.cbf_metrology wraps the following three programs:

1. dials.combine_experiments
2. cctbx.xfel.filter_experiments_by_rmsd
3. dials.refine

We recently described how these programs can be used outside of cspad.cbf_metrology, specifically using the SACLA Octal MPCCD detector, which is non-hierarchical but has 8 panels. The protocol is here:

Brewster AS, Young ID, Lyubimov A, Bhowmick A, Sauter NK (2019): Processing serial crystallographic data from XFELs or synchrotrons using the cctbx.xfel GUI. Computational Crystallography Newsletter 10, 22-39. (reprint)

Regarding the AGIPD and Jungfrau 16M, these detectors have 256 panels in a 4 level hierarchical arrangement. It's likely cspad.cbf_metrology could be re-worked to handle these detectors. The idea would be to re-write the expanding protocol to allow the user to specify a refinement 'plan'. Perhaps using a json file? Alternatively, the program could be made smarter to try and discover a good plan, but that's trickier.

More details:

1. Your paper shows the expanding strategy is better than the hierarchical strategy. Does this apply to other detectors as well? In this case, how do you perform it? I can hack cspad.cbf_metrology for other detectors, but have you done something similar already?

Expanding is likely better, but I've only used the 'hierarchical' method with other detectors (including the AGIPD and the Jungfrau16M). Here, I run dials.refine for hierarchy levels 0 (detector), 1 (quads), and 2 (modules), on the command line, and it works ok. I'd like to use the expanding method, and I'm making a note to look into automating it soon.

2. Do you have any comments if spherical_relp_model = True is useful?

This prediction model changes where the spots are predicted by not rotating them onto the Ewald sphere but instead projecting them onto the Ewald sphere. It's a small change that we haven't studied enough to say something conclusive. I think we need to properly model mosaicity and bandpass to get the predictions right, but that's a longer term project. That said, I'd welcome any feedback as to whether this parameter helps!

3. Do you routinely use real_space_grid_search? I noticed that the indexing rate of fft1d have improved over years (I don't know exactly which change was important).

We do use real_space_grid_search, but usually only when we have many multiple lattices per image. Generally we use fft1d for typical cases. If you have lots of computing time, you can use indexing.stills.method_list=fft1d,real_space_grid_search to try both on every frame, first ff1d and then if that fails, real_space_grid_search.

4. How about index_assignment.method = local and optimise_initial_basis_vectors = True? In my earlier experience, they sometimes dramatically increased the indexing rate of real_space_grid_search, but I am not sure if it is still the case.

I haven't tried these very much either. I'm intrigued to hear this report though!

Does it simply reflect the fact that you need smaller angles to rotate higher resolution spots (i.e. further from the origin in the reciprocal space) to move them the same distance as low resolution spots? Do you observe similar effects on MPCCD, CSPAD, etc?

This is what I would guess, and I do see it on the CSPAD also. Lower resolution panels have a higher delta psi RMSD.

from dials_refinement_brewster2018.

Hi Aaron, thank you very much for detailed explanation!

Expanding is likely better, but I've only used the 'hierarchical' method with other detectors

Since the central four panels of MPCCD extend to the edge of the detector, I don't know how effective it is, but I will try the expanding strategy and report here.

In your paper and CCN article, you used flat_refinement=True flat_refinement_with_distance=True. For MPCCD, it might be worth refining Tau2 and Tau3 at level 0. At SACLA, I had several datasets where RMSD remained relatively high (~ 1.4 px instead of 0.6 px) after refinement. Most of the time this was caused by wide bandpass; depending on the accelerator configuration (i.e. what BL3 is doing simultaneously), you might end up having ~1.5 % FWHM instead of 0.5 %. However, there were cases where the bandwidth was normal and beam line scientists suggested that the detector and the beam might not have been completely perpendicular. At that time, I was mostly using CrystFEL and could not verify this as it cannot refine tilt and twist. I will revisit this when I have time. Have you seen any problems refining tilt and twist?

from dials_refinement_brewster2018.

Good info. Thanks. I look forward to seeing your results.

from dials_refinement_brewster2018.