kolayne / find_most_similar_image Goto Github PK

It seems to be impossible to both keep using the multiprocessing module and fix the issue of tqdm bars displayed incorrectly when using forking (at least, at the moment). However, there is a PR fixing the issue for threading. I'd like to see the measurements of performance when using one thread, multiple threads, and multiple processes. Probably it's a good idea to switch to threads to fix the problem, but I'm not ready to sacrifice performance on a notable scale

Document functions in find_most_similar_image.py

At the moment the tool only consists of one file, which handles everything: analyzing images, saving storages, etc, and communicating with a user via a CLI. I find the latter function completely separate from the rest, and I want the tool to be split into (at least) two files, one of which would handle the search, why the other one would be dealing with cmd-line arguments (and most likely also managing files, while the core will only work with the data it is supposed to process, converted to the format it is going to be processed in).

This will also make it possible to create a good API: even though at the moment users can import the tool as a python module and use it in their programs, we currently force them to use files, wait for json.loads/np.asarray/etc conventions, which can be a tangible performance drawback when trying to process a large amount of data. Separating the core from the rest code would solve this problem

Can we delegate the following piece of code to numpy and thus speed it up?