redundant unnecessary repetitive calling of _validate_source_file() in nested loops about scaper HOT 2 OPEN

The reason every distribution tuple is validated when an event is added is that it can be different each time.

In your case you always add the same list of source files, but in general each time you add an event you could specify a different list of source files (or a different distribution tuple altogether) meaning we must validate the distribution tuple every time we call add_event.

I we don't validate the source files when an event is added it could cause Scaper to crash downstream. So from a design standpoint I believe what we are doing is the correct thing to do.

We could, potentially, add a flag to add_event to disable source file validation (e.g. disable_source_file_validation=False by default), since this is the only validation that involves file I/O - I believe all the other validation steps are insignificant computationally.

@moabaom To help us better understand how serious an issue this is, can you share a Minimal Working Example (MWE) and report the execution time with and without source file validation? Thanks!

from scaper.

p.s - actually another option would be to create a "cache" for validated files, such that if a scaper object has already validated that a file exists, it doesn't try to validate it again. This is probably safer than disabling validation and achieves the same speedup. If you create a new scaper object in a loop (like in the example), we could support providing a list of validated files as input, and the scaper object will not try to validate these source files.

Still, let's start with the MWE and timings to better scope the problem. thanks.

from scaper.