rs-station / laue-dials Goto Github PK

View Code? Open in Web Editor NEW

4.0 4.0 3.0 14.84 MB

A package for analyzing Laue x-ray crystallography data using the DIALS framework.

Home Page: https://rs-station.github.io/laue-dials/

License: MIT License

Python 100.00%

laue-dials's People

Contributors

Stargazers

Watchers

Forkers

princewalnut kmdalton hkwang

laue-dials's Issues

issue with multiprocessing during spot finding

In /n/hekstra_lab/projects/laue-dials-tests/chess_hewl/tutorial_CHESS.ipynb, this succeeds without the spotfinder.mp.nproc=4 but fails with it (or nproc)

laue.find_spots imported.expt \
    spotfinder.threshold.dispersion.gain=0.7 
#    spotfinder.mp.nproc=4 \

with the following error message (before the traceback):

easy_mp crash detected; subprocess trace: ----
Stacktrace:
exit code = -9

and traceback:

Traceback (most recent call last):
  File "/n/hekstra_lab/people/dhekstra/conda_envs/laue-dials/bin/laue.find_spots", line 8, in <module>
    sys.exit(run())
             ^^^^^
  File "/n/hekstra_lab/people/dhekstra/conda_envs/laue-dials/lib/python3.11/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/n/hekstra_lab/people/dhekstra/conda_envs/laue-dials/lib/python3.11/site-packages/laue_dials/command_line/find_spots.py", line 133, in run
    strong_refls = find_spots(params, imported_expts)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/hekstra_lab/people/dhekstra/conda_envs/laue-dials/lib/python3.11/site-packages/laue_dials/algorithms/monochromatic.py", line 22, in find_spots
    refls = do_spotfinding(expts, params)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/hekstra_lab/people/dhekstra/conda_envs/laue-dials/lib/python3.11/site-packages/dials/command_line/find_spots.py", line 127, in do_spotfinding
    reflections = flex.reflection_table.from_observations(experiments, params)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/hekstra_lab/people/dhekstra/conda_envs/laue-dials/lib/python3.11/site-packages/dials/array_family/flex_ext.py", line 183, in from_observations
    return spotfinder.find_spots(experiments)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/hekstra_lab/people/dhekstra/conda_envs/laue-dials/lib/python3.11/site-packages/dials/algorithms/spot_finding/finder.py", line 612, in find_spots
    table, hot_mask = self._find_spots_in_imageset(imageset)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/hekstra_lab/people/dhekstra/conda_envs/laue-dials/lib/python3.11/site-packages/dials/algorithms/spot_finding/finder.py", line 724, in _find_spots_in_imageset
    r, h = extract_spots(imageset[j0:j1])
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/hekstra_lab/people/dhekstra/conda_envs/laue-dials/lib/python3.11/site-packages/dials/algorithms/spot_finding/finder.py", line 414, in __call__
    return self._find_spots(imageset)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/hekstra_lab/people/dhekstra/conda_envs/laue-dials/lib/python3.11/site-packages/dials/algorithms/spot_finding/finder.py", line 503, in _find_spots
    batch_multi_node_parallel_map(
  File "/n/hekstra_lab/people/dhekstra/conda_envs/laue-dials/lib/python3.11/site-packages/dials/util/mp.py", line 194, in batch_multi_node_parallel_map
    return multi_node_parallel_map(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/hekstra_lab/people/dhekstra/conda_envs/laue-dials/lib/python3.11/site-packages/dials/util/mp.py", line 165, in multi_node_parallel_map
    result = libtbx.easy_mp.parallel_map(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/hekstra_lab/people/dhekstra/conda_envs/laue-dials/lib/python3.11/site-packages/libtbx/easy_mp.py", line 637, in parallel_map
    result = res()
             ^^^^^
  File "/n/hekstra_lab/people/dhekstra/conda_envs/laue-dials/lib/python3.11/site-packages/libtbx/scheduling/result.py", line 119, in __call__
    self.traceback( exception = self.exception() )
  File "/n/hekstra_lab/people/dhekstra/conda_envs/laue-dials/lib/python3.11/site-packages/libtbx/scheduling/stacktrace.py", line 117, in __call__
    self.raise_handler( exception = exception )
  File "/n/hekstra_lab/people/dhekstra/conda_envs/laue-dials/lib/python3.11/site-packages/libtbx/scheduling/stacktrace.py", line 134, in raise_with_traceback
    raise_(type(exception), exception, self.traceback)
  File "/n/hekstra_lab/people/dhekstra/conda_envs/laue-dials/lib/python3.11/site-packages/six.py", line 719, in reraise
    raise value
  File "/n/hekstra_lab/people/dhekstra/conda_envs/laue-dials/lib/python3.11/site-packages/libtbx/scheduling/mainthread.py", line 100, in poll
    value = target( *args, **kwargs )
            ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/hekstra_lab/people/dhekstra/conda_envs/laue-dials/lib/python3.11/site-packages/dials/util/mp.py", line 88, in __call__
    return libtbx.easy_mp.parallel_map(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/hekstra_lab/people/dhekstra/conda_envs/laue-dials/lib/python3.11/site-packages/libtbx/easy_mp.py", line 637, in parallel_map
    result = res()
             ^^^^^
  File "/n/hekstra_lab/people/dhekstra/conda_envs/laue-dials/lib/python3.11/site-packages/libtbx/scheduling/result.py", line 119, in __call__
    self.traceback( exception = self.exception() )
  File "/n/hekstra_lab/people/dhekstra/conda_envs/laue-dials/lib/python3.11/site-packages/libtbx/scheduling/stacktrace.py", line 88, in __call__
    raise exception
RuntimeError: Please report this error to [email protected]: exit code = -9

---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
File <timed eval>:1

File /n/hekstra_lab/people/dhekstra/conda_envs/laue-dials/lib/python3.11/site-packages/IPython/core/interactiveshell.py:2493, in InteractiveShell.run_cell_magic(self, magic_name, line, cell)
   2491 with self.builtin_trap:
   2492     args = (magic_arg_s, cell)
-> 2493     result = fn(*args, **kwargs)
   2495 # The code below prevents the output from being displayed
   2496 # when using magics with decorator @output_can_be_silenced
   2497 # when the last Python token in the expression is a ';'.
   2498 if getattr(fn, magic.MAGIC_OUTPUT_CAN_BE_SILENCED, False):

File /n/hekstra_lab/people/dhekstra/conda_envs/laue-dials/lib/python3.11/site-packages/IPython/core/magics/script.py:154, in ScriptMagics._make_script_magic.<locals>.named_script_magic(line, cell)
    152 else:
    153     line = script
--> 154 return self.shebang(line, cell)

File /n/hekstra_lab/people/dhekstra/conda_envs/laue-dials/lib/python3.11/site-packages/IPython/core/magics/script.py:314, in ScriptMagics.shebang(self, line, cell)
    309 if args.raise_error and p.returncode != 0:
    310     # If we get here and p.returncode is still None, we must have
    311     # killed it but not yet seen its return code. We don't wait for it,
    312     # in case it's stuck in uninterruptible sleep. -9 = SIGKILL
    313     rc = p.returncode or -9
--> 314     raise CalledProcessError(rc, cell)

CalledProcessError: Command 'b'\nlaue.find_spots imported.expt \\\n   spotfinder.threshold.dispersion.gain=0.7 \\\n   spotfinder.mp.nproc=4 \n\n# spotfinder.filter.max_separation=10 \n# spotfinder.threshold.dispersion.sigma_strong=3 \\\n'' returned non-zero exit status 1.

laue.sequence_to_stills crashes when using slice of data that does not start at image 0

dials=3.17.0
laue-dials=0.4

Using the hewl dataset from laue_tutorials, if I take the first 100 images and run through the reduction pipeline everything works as expected. However if I take a different slice of the images (e.g. 600-700),

laue.sequence_to_stills monochromatic_HEWL_anom_3049.expt monochromatic_HEWL_anom_3049.refl
fails with

Traceback (most recent call last):
  File "/n/home10/dmcdonagh/.conda/envs/manuscript_env/bin/laue.sequence_to_stills", line 8, in <module>
    sys.exit(run())
             ^^^^^
  File "/n/home10/dmcdonagh/.conda/envs/manuscript_env/lib/python3.11/contextlib.py", line 81, in inner
    return func(*args, **kwds)
           ^^^^^^^^^^^^^^^^^^^
  File "/n/home10/dmcdonagh/.conda/envs/manuscript_env/lib/python3.11/site-packages/laue_dials/command_line/sequence_to_stills.py", line 238, in run
    (new_experiments, new_reflections) = sequence_to_stills(
                                         ^^^^^^^^^^^^^^^^^^^
  File "/n/home10/dmcdonagh/.conda/envs/manuscript_env/lib/python3.11/site-packages/laue_dials/command_line/sequence_to_stills.py", line 140, in sequence_to_stills
    imageset=experiment.imageset.as_imageset()[
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/n/home10/dmcdonagh/.conda/envs/manuscript_env/lib/python3.11/site-packages/dxtbx/imageset.py", line 124, in __getitem__
    return self.partial_set(reader, start, stop)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OverflowError: Please report this error to [email protected]: can't convert negative value to unsigned int

Looking at the reflection tables in the two different cases, in the case where it fails there are some with bounding boxes where the z coordinate starts at -1, which might explain the error.

Steps to reproduce:

dials.import geometry.scan.oscillation=0,1 \
    geometry.goniometer.axes=-1,0,0 \
    geometry.beam.wavelength=1.05 \
    geometry.detector.panel.pixel=0.08854,0.08854 \
    input.template='PATH_TO_HEWL_DATA/HEWL_NaI_3_2_06##.mccd' \
    output.experiments=imported_HEWL_anom_3049.expt \
    output.log=dials.import_HEWL_anom_3049.log

laue.find_spots imported_HEWL_anom_3049.expt \
    spotfinder.mp.nproc=8 \
    spotfinder.threshold.dispersion.gain=0.15 \
    spotfinder.filter.max_separation=10 \
    output.reflections=strong_HEWL_anom_3049.refl \
    output.log=laue.find_spots_HEWL_anom_3049.log

laue.index imported_HEWL_anom_3049.expt strong_HEWL_anom_3049.refl \
    indexer.indexing.nproc=2 \
    indexer.indexing.known_symmetry.space_group=96 \
    indexer.indexing.refinement_protocol.mode=refine_shells \
    indexer.refinement.parameterisation.auto_reduction.action=fix \
    laue_output.index_only=False \
    laue_output.indexed.experiments=indexed_HEWL_anom_3049.expt \
    laue_output.indexed.reflections=indexed_HEWL_anom_3049.refl \
    laue_output.refined.experiments=refined_HEWL_anom_3049.expt \
    laue_output.refined.reflections=refined_HEWL_anom_3049.refl \
    laue_output.final_output.experiments=monochromatic_HEWL_anom_3049.expt \
    laue_output.final_output.reflections=monochromatic_HEWL_anom_3049.refl \
    laue_output.log=laue.index_HEWL_anom_3049.log

laue.sequence_to_stills monochromatic_HEWL_anom_3049.expt monochromatic_HEWL_anom_3049.refl \
    output.experiments=stills_HEWL_anom_3049.expt \
    output.reflections=stills_HEWL_anom_3049.refl \
    output.log=laue.sequence_to_stills_HEWL_anom_3049.log

the documentation badge does not link to the documentation

I think this one is self explanatory, team.

Workflow for testing against non-main DIALS branches

There are sometimes DIALS branches we might want to test against before they are merged into main and released. A Github Actions workflow that can be triggered manually and linked to a branch URL would be useful. See https://stackoverflow.com/questions/62736315/invoke-github-actions-workflow-manually-and-pass-parameters

Laue refiner

A proper Laue refinement object/target function would eliminate the need for beam and experiment generation prior to calls to laue.refine. This would save on computation speed and primarily on memory, and allow for refinement globally across a dataset which is not possible now because of the memory overhead

disable format check wherever possible

Only the initial solution and integrator subprograms need access to the images. For all the other subprograms, the format check on images during experiment list construction should be disabled.

Profile fitting does not include 0-intensity pixels

laue-dials/src/laue_dials/algorithms/integration.py

Line 266 in 51f0f07

mask = (distance <= radius) & (pixels > 0)

This should be pixels >= 0 to include 0-intensities in profile fitting.

Requested functionality for unit-cell/wavelength conversion

The laue.index command is monochromatic and frequently causes a deviation of 5-10% in unit cell axis lengths from the published cell, presumably with a corresponding shift in wavelength in the opposite direction. Indexing at the correct cell is frequently impossible (at least with the BioCARS spectrum), but incorrect unit cells can impact downstream analyses.

A function/command for rescaling the unit cell, wavelength, and any other coupled parameters would be helpful for users to correct the unit cell after laue.optimize_indexing assigns wavelengths.

@kmdalton @DHekstra, any ideas for algorithms regarding this? Would a simple arithmetic solution based on % difference between indexed/correct cell and multiplying the unit cell axes/wavelengths be sufficient?

Unit test coverage

There are a variety of classes+functions that are not tested -- this should be remedied

laue.refine should have aggregate statistics printed

The current log files print statistics per image from dials.refine which can be hard for the user to interpret. Assembling statistics at the end that are more representative of the refined data would be helpful for readers.

show the user how to set the unit cell dims in the final MTZ

since the unit cell dimensions coming out of laue data processing are only loosely tied to reality, perhaps we can add a cell at the end of the tutorial showing the user how to edit the unit cell of the final mtz by hand. something along these lines:

import reciprocalspaceship as rs

ds=rs.read_mtz(your.mtz)
ds.cell=(79.4,79.4,37.84,90,90,90)
ds.write_mtz(new_name.mtz)

Docs for laue.optimize_indexing unclear around d_min

The default value is None, but the allow_none=True and the example command in the documentation not showing required parameters implies incorrect functionality for the user. The documentation needs to be updated to more clearly reflect mandatory parameters and whether None values are truly allowed

laue.integrate takes longer with increasing nproc

dials=3.17.0
laue-dials=0.4

When running laue.integrate (using e.g the dataset in /n/hekstra_lab/projects/laue-dials-tests/laue_tutorials/simdhfr/data/bandwidth/shot1/shot_1_#####.cbf), running with nproc=1 produces the output:

Image 174 took 23.01715064048767 seconds.
Integrating image 175.
Image 175 took 24.365636825561523 seconds.
Integrating image 176.
Image 176 took 26.35458254814148 seconds.
Integrating image 177.
Image 177 took 25.817972421646118 seconds.
Integrating image 178.
Image 178 took 26.819148778915405 seconds.
Integrating image 179.
Image 179 took 27.399763584136963 seconds.
Integration finished.
Constructing reflection table
Converting to MTZ format.
Saving integrated reflections to laue/integrated_reference.mtz

Time Taken for Total Processing = 3841.383531 seconds

Repeating with nproc=2 produces:

Image 172 took 49.02370619773865 seconds.
Integrating image 174.
Image 173 took 46.737916707992554 seconds.
Integrating image 175.
Image 174 took 46.61181282997131 seconds.
Integrating image 176.
Image 175 took 42.28249502182007 seconds.
Integrating image 177.
Image 176 took 42.314101696014404 seconds.
Integrating image 178.
Image 177 took 41.87906455993652 seconds.
Integrating image 179.
Image 178 took 42.62828612327576 seconds.
Image 179 took 39.019068241119385 seconds.
Integration finished.
Constructing reflection table
Converting to MTZ format.
Saving integrated reflections to laue/integrated_reference.mtz

Time Taken for Total Processing = 3975.674198 seconds

Repeating with nproc=4 causes it to go even slower

Integrating image 177.
Image 174 took 71.01042747497559 seconds.
Integrating image 178.
Integrating image 179.
Image 175 took 75.81021571159363 seconds.
Image 177 took 61.80263161659241 seconds.
Image 178 took 54.61494040489197 seconds.
Image 179 took 45.53638052940369 seconds.

I've not managed to get this to finish with nproc=4, despite running for over 8 hours.

Looking at Top while this is running, it looks like more processes are correctly run with increasing nproc, but the %CPU is divided between them. E.g. nproc=4 has 4 processes each using 25% CPU.

laue.sequence_to_stills from laue-dials 0.3 leads to spot predictions with high RMSDs

With laue-dials 0.3 and DIALS 3.17.0, laue.sequence_to_stills causes one of my passes to be misindexed, leading to spot predictions with RMSDs on the order of 15 pixels, from ~0.25 pixels, with a previous version of laue.sequence_to_stills (see DMs with Rick on the Hekstra lab Slack). I have attached a .yml file for the conda environment in which this spot prediction works.

Decoy parameters for laue_dials.find_spots, potentially other functions.

As per the laue_dials.find_spots documentation, there are two different sets of output files: laue_output.* (where * is filename or log) and output.* (where * is reflections, shoeboxes, experiments, or log).

It looks like the laue_output files are the ones that are actually getting read, and so the others are "valid" parameters that do nothing. For example, I ran the code:

laue.find_spots imported_NaI_3_01.expt \
    spotfinder.mp.nproc=8 \
    spotfinder.threshold.dispersion.gain=0.35 \
    spotfinder.filter.max_separation=10 \
    output.reflections=strong_NaI_3_01.refl \
    output.log=dials.find_spots_NaI_3_01.log

which generated the files strong.refl and laue.find_spots.log, implying to me that the parameters I used to rename the reflection and log files are not actually relevant to this function. I'd originally thought that there were two redundant log (etc) files possible, but now it seems like there's actually the true logfile and the decoy logfile. This is made more confusing by the decoys being the set of parameters used for spot finding in non-laue dials...

laue.optimize_indexing should guess d_min

d_min should default to the resolution of the supplied reflection table not, the current hardcoded value.

Log files should include a version number

Log files for all commands should include a laue-dials version number, like DIALS does. A laue.version command may also be helpful.

laue.compute_rmsds y-axis should start at 0

It currently fits both the maximum and minimum RMSD values to the edges of the plot. A minimum of 0 (so that the y-axis is 0-max(RMSD)) would make for easier comparisons.

Issues with reading in multiple passes

With reference to /n/hekstra_lab/projects/laue-dials-tests/hemoglobin_TR/playground/tutorial-hemoglobin.ipynb, the following,

laue.initial_solution imported.expt \
                        indexer.refinement.parameterisation.auto_reduction.action=fix \
                        spotfinder.threshold.dispersion.gain=2 \
                        spotfinder.filter.d_min=1.6 \
                        indexer.indexing.known_symmetry.space_group=5 \
                        indexer.indexing.known_symmetry.unit_cell=93.25,43.98,83.5,90.0,122.03,90.0

fails with a DialsIndexError("No suitable lattice could be found."). The cause of this appears to be related to the angles of the four passes. Specifically, I did not specify geometry.scan.oscillation during dials.import, According to dials.show imported.expt in that case, the code advances the angle of each image by one degree in each pass (instead of the true step size of 6 degrees), and subsequent laue.initial_solution fails as described above. Presumably the cause of the error is therefore that the indexing routine is being fed garbage frame geometry.

When we do specify geometry.scan.oscillation as 0,6 during dials.import, dials.show imported.expt produces four scans with

Scan:
    number of images:   31
    image range:   {1,31}
    oscillation:   {0,6}
    exposure time: 0

and laue.initial_solution seems to succeed (very confusing given my comment below) but leads to trouble when invoking sequence_to_stills.

The difficulty is that the first ON/OFF pass uses angles (0:6:180) and the second pass uses (183:6:363).

Part of the problem seems to be that there are no angles/no angles are read from the MarCCD header, so we need to specify them externally. Naively it seems that there are three possible solutions:

somehow putting the angles in the headers of the MarCCD files (currently our MarCCD package doesn't support that but it could be extended)
some other way of getting the DIALS import done, e.g. by doing two separate imports and combining them. I describe this in a comment below.
use some DIALS function I do not know off to manipulate the MarCCD images.

doc error?

https://rs-station.github.io/laue-dials/cli/index.html lists fft3d as the default indexing algorithm. As I understand it, this is not actually the case. Is there a way to indicate that the DIALS default is overridden?

laue.[command] should output help message

DIALS commands like dials.index output help message when called with no arguments. laue-dials commands should do the same thing

Need to split refined_varying.* into stills

Need to implement capability to split the final reflection tables and experiment lists in laue.initial_solution into stills -- should include both one composite file or splitting into an image per file

Enforcing Unit Cell during laue.optimize_indexing can have bad orientation

The crystal U matrix is kept from the initial estimate, but in cases where the user supplies a unit cell that is not a simple rescale of the initial cell, this matrix needs to be recalculated accordingly

laue.predict KDE probabilities slow

Running laue.predict can be slow on large datasets because of the probability-assignment step for each reflection. Optimizing this could be beneficial.

laue.optimize indexing should have sequential mode

A mode for using crystal rotations from adjacent images as initial estimates (instead of hardcoded orientation matrices being used). This is similar to Precognition's progressive mode. Would imply a restriction on multiprocessing, but could account for errors like crystal slippage by propagating offsets throughout a dataset.

example data?

Could you include (a link to) example data needed to run the example script?

laue.version command

DIALS has a dials.version command for easily finding the DIALS version with running several imports from a DIALS script. It would be beneficial for a similar laue.version command to exist, possibly with the ability to list versions for all dependencies for reproducible purposes.

Python-level parallelism

Current pipeline is primarily serial computing in nature -- shifting to parallelism at the Python level would improve performance for users

Need to write PHIL parser for laue.initial_solution

It isn't clear if libtbx.phil has capability to split a phil_scope into multiple scopes based on indents or {} scopes. Current PHIL string has scopes for each subprocess in laue.initial_solution, and a parser to generate new PHIL scopes to pass to the respective processes is needed

laue.refine should not output a beam with a random wavelength

laue.refine outputs an experiment that can have a beam with a wavelength that is essentially based on which reflection's experiment wasn't deleted. The final output beam should be either the nominal wavelength or an average wavelength. Use nominal for now

PinkIndexer Option

We should use the crystals package to interface with PinkIndexer and allow for its use as an initial indexing solution in the pipeline

HKLs of (000) with nonzero wavelengths discovered after refinement

The poly_refined.* files have contained a few reflections given a Miller index of (000) and a nonzero wavelength. This suggests unindexed reflections have made it through filtration this far, or a bug in laue.optimize_indexing has occurred.

n_proc and nproc for different commands should all be nproc

DIALS commands use nproc by default. Some of our commands use n_proc as a parameter instead. For consistency purposes all parameters should be changed to nproc.

Coset prediction would be useful

Adding coset prediction for laue.predict would be helpful to certain users. A flag for turning on HKL.5 sets is my suggested solution

laue.integrate does nothing

laue.integrate currently does nothing, and functionality needs to be added here for the user

nproc issue?

For

the following fails in the CHESS HEWL tutorial

%%time
%%bash

laue.optimize_indexing stills* \
output.experiments="optimized.expt" \
output.reflections="optimized.refl" \
output.log="laue.optimize_indexing.log" \
wavelengths.lam_min=0.76 \
wavelengths.lam_max=0.81 \
reciprocal_grid.d_min=1.8 \
nproc=8

with error

Sorry: Unknown command line parameter definition: nproc = 8

The same cell completes just fine when omitting the last line.

laue.predict needs I/O script written

The LauePredictor class does the bulk of prediction and is currently implemented, but a script for parsing user input into acceptable input for the class and converting returned values in DIALS objects is needed in laue.predict

sequence_to_stills output parameters unsuccessful in renaming experiment & reflection files

I ran the code:

laue.sequence_to_stills monochromatic_NaI_3_02.expt monochromatic_NaI_3_02.refl \
    output.experiments=stills_NaI_3_02.expt \
    output.reflections=stills_NaI_3_02.refl \
    output.log=laue.sequence_to_stills_NaI_3_02.log

The initial output I got said the parameters had been changed:

The following parameters have been modified:

output {
  experiments = "stills_NaI_3_02.expt"
  reflections = "stills_NaI_3_02.refl"
  log = "laue.sequence_to_stills_NaI_3_02.log"
}
input {
  experiments = monochromatic_NaI_3_02.expt
  reflections = monochromatic_NaI_3_02.refl
}

However, the files that were actually written out were stills.expt, stills.refl, and laue.sequence_to_stills_NaI_3_02.log. For some reason, the log gets renamed but the other two files do not.

laue.refine should know to use cctbx.python when DIALS is sourced instead of conda installed

dials.refine is able to recognize and use a cctbx.python when DIALS is sourced instead of conda installed. We should enable a similar functionality in laue.refine and elsewhere in case the user opts to source an older DIALS build instead of using the conda version.

DIALS>=v3.15.0 broken

The parameter force_2d in dials.find_spots was removed in dials/dials#2422. This is a critical component of our pipeline and makes it incompatible with laue-dials. Follow up is needed with DIALS team -- pin installation to use DIALS v3.14.2 at most recent until resolution

Leftover embeds

Repo currently has leftover embed() statements/imports scattered throughout that need to be removed

account for multiplicity in spot prediction/res-dep BW

since we have miller indices and spacegroup by the time we predict spots, we know the multiplicities of each spot. perhaps there is merit in adjusting the threshold accordingly. (granting that will require some thought to determine what "accordingly" means)

IOCTL error in jupyter notebook

I just switched to using a local install of my cloned repo (fresh from git pull and pip install -e .).
Running the PYP example, I am encountering cells that run in the terminal but not from a notebook, with the following error:

Traceback (most recent call last):
  File "/n/hekstra_lab/people/dhekstra/conda_envs/laue-dials/bin/laue.find_spots", line 5, in <module>
    from laue_dials.command_line.find_spots import run
  File "/n/home12/dhekstra/ipython_notebooks/laue-dials/src/laue_dials/command_line/find_spots.py", line 17, in <module>
    laue_version()
  File "/n/home12/dhekstra/ipython_notebooks/laue-dials/src/laue_dials/utils/version.py", line 7, in laue_version
    width = os.get_terminal_size().columns
            ^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 25] Inappropriate ioctl for device

Use /n/hekstra_lab/projects/laue-dials-tests/hemoglobin_TR/playground/tutorial-hemoglobin-Copy1.ipynb to replicate.

SVG failed to converge in laue.integrate

The warning thrown when pinv() fails is not very clear to the user and suggests failure that is not really a problem -- rework the warning to be more informational

laue_dials.find_spots log does not contain meaningful information

The log file written by laue_dials.find_spots (which seems to be laue_output.log rather than output.log - see other issue) contains a lot of text that is formatted like a spot finding log but doesn't actually contain useable information:

Finding spots in image 1 to 999...
Setting chunksize=31
Extracting strong spots from images
 Using multiprocessing with 8 parallel job(s)

Found %d strong pixels on image %d

Extracted %d spots
Removed %d spots with size < %d pixels
Removed %d spots with size > %d pixels
Calculated %d spot centroids
Calculated %d spot intensities
Filtered %d of %d spots by peak-centroid distance
Found %d strong pixels on image %d

Extracted %d spots
Removed %d spots with size < %d pixels
Removed %d spots with size > %d pixels
Calculated %d spot centroids
Calculated %d spot intensities
Filtered %d of %d spots by peak-centroid distance
Found %d strong pixels on image %d

Extracted %d spots
Removed %d spots with size < %d pixels
Removed %d spots with size > %d pixels
Calculated %d spot centroids
Calculated %d spot intensities
Filtered %d of %d spots by peak-centroid distance
Found %d strong pixels on image %d

This continues for many more iterations. It seems like the spot finding does proceed successfully, but it's hard to evaluate much from the log!

laue.refine auto_reduction should be None

We need an option to toggle off auto_reduction in laue.refine. This will require a change to DIALS, and editing the master PHIL file in laue.refine to implement that change