Training fails upon empty spectrum about casanovo HOT 4 CLOSED

Hi Guhan,

I haven't encountered an early stopping problem during training before and it's actually weird that model calls `` 11929th at iteration despite correctly recognizing there would be a total of 19028 iterations. Did you try training the model with a smaller data set, maybe a subsample of the one you're using, and if so did you observe early stopping behavior?

from casanovo.

Hi @melihyilmaz,

I found the issue after some digging in the source code for pytorch lightning.

In the pytorch_lightning/loops/base.py" file, I noticed that the run() function excepts a StopIteration which effectively tells the training loop to end:

def run(self, *args: Any, **kwargs: Any) -> T:
        """The main entry point to the loop.

        Will frequently check the :attr:`done` condition and calls :attr:`advance`
        until :attr:`done` evaluates to ``True``.

        Override this if you wish to change the default behavior. The default implementation is:

        Example::

            def run(self, *args, **kwargs):
                if self.skip:
                    return self.on_skip()

                self.reset()
                self.on_run_start(*args, **kwargs)

                while not self.done:
                    self.advance(*args, **kwargs)

                output = self.on_run_end()
                return output

        Returns:
            The output of :attr:`on_run_end` (often outputs collected from each step of the loop)
        """
        if self.skip:
            return self.on_skip()

        self.reset()

        self.on_run_start(*args, **kwargs)

        while not self.done:
            try:
                self.on_advance_start(*args, **kwargs)
                self.advance(*args, **kwargs)
                self.on_advance_end()
                self.restarting = False
            except StopIteration:
                break

        output = self.on_run_end()
        return output

I removed the try/except and got a StopIteration on a particular spectrum in the .mgf that looked like this:

BEGIN IONS
TITLE=782.5711
PEPMASS=1088.5186
CHARGE=3+
SCANS=5711
RTINSECONDS=134.84
SEQ=QASTSEESDEMPVPDSESVFVIPGSALLWR
2731.2097 7026.7
END IONS

Since I had turned spectrum preprocessing on in the config.py file, and the default for max_mz in the preprocess_peaks function is 2500, the input into the _get_filter_intensity_mask function was an empty intensity array. This then threw an IndexError - min_intensity *= intensity[intensity_idx[-1]]: IndexError: index -1 is out of bounds for axis 0 with size 0.

TLDR, I think that it would be nice to include some error catching for empty spectra post-processing if possible, especially since the pytorch-lightning module has such bad error-messaging! I will remove this spectrum from the file for now.

from casanovo.

I see, thanks for raising this issue and letting us know. I'll level the issue open to add error handling in the future.

from casanovo.

Fixed in #55.

from casanovo.