[RFC] Hosting and Serving Models with Flash about lightning-flash HOT 5 CLOSED

Well there are various ways of creating RESTful service, say FastAPI, Flask, Django, Node.js, what not. But If we use Torchserve we don't have to worry about multi-processing over CPU, GPU, supporting all OSes, scaling to say Kubernetes, etc.

When I looked around in the context of ch 15 of our book, it seemed that the key questions with using Python (I used sanic over flask because it seemed like simple to use but async) were 1) is it safe (when it calls into extensions, e.g. for image processing)? 2) can you avoid copying as much as possible. Note that TorchScript pretty much solves the "GIL topic" for the model parts.
For the " multi-processing over CPU, GPU, supporting all OSes, scaling" bits, it might be even more natural to look at lightning over TorchServe for us here, so it might be good to have a comparison what is missing that lightning cannot provide here. Personally, I am not a huge fan of installing Java in more places than I have to.

from lightning-flash.

True that it would be possible with Python and there are ways to overcome with GIL.
But there are to of issues that could occur while we implement. Working with scripted models, metrics management, logging. Supporting more than 1 model serving parallel, distributing models over CPU Cores.

Installing Java is not needed for users !

All they would need to do is

curl server_ip/endpoint -T some_image.jpg

or send a POST Request.

Torchserve is very very well maintained and engineered. There was really a long discussion in PyTorch about this. Refer here

Let me know more thoughts. Maybe I should elaborate on idea ?

from lightning-flash.

Reposting my comment from slack:

Thanks for posting this @oke-aditya
As you mentioned Torchserve is a great tool for people who would like to productionize their model or even for a simple demo.
I am not sure about Flash Serving but a general PL Serving may help a wider audience I guess.

PS. It is a much-needed tool and I would love to contribute to this.

from lightning-flash.

Torchserve is very very well maintained and engineered. There was really a long discussion in PyTorch about this. Refer here

Torchserve allows serving by versioning the models and also provides both gRPC and REST API support.
It is well maintained and expected to be maintained in future too. We can follow the main core value of Lightning One less thing to remember and provide users with an API which they can use without any effort.

from lightning-flash.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

from lightning-flash.