Feature Request: Expose the models about ludwig HOT 10 CLOSED

Hey @w4nderlust
Thanks for the excellent work on ludwig.
I didn't open a new issue for this because it's heavily related to the above.
Would it be possible to export the model without the pre/post processing?
My scenario is I've got a text classifier with only 4-5 classes, so having the prediction returning an integer is already enough.
And the reason why I'm asking for this is because I can host it on Google cloud ML which supports tensorflow
Thanks!

from ludwig.

@jadjoubran there are other discussions about it. The tensorflow model weights are saved in the model directory, so you can actually just load it with tensorflow and do whatever you want with it. The problem is mostly that you will have to map the inputs yourself into tensors. You can do it using the train_set_metadata.json file that is also saved in the model directory. Ludwig preprocessing at prediction time is basically taking that file, taking the raw inputs and mapping them into tensors using those mappings.
There is a discussion on Gitq of someone trying to serve models trained with Ludwig in Java that has the same questions that you have, try to look here.

from ludwig.

I think the main problem there is data preprocessing and postprocessing. If you serve a model that returns integer numbers for category but you don't have the dictionary of how that maps into a category label, serving the model maybe a bit useless. (Ludwig does that for you, mapping inputs into tensors during preprocessing and tensors into strings and values during postprocessing.
Can you suggest how who you use such a model without the pre/postprocessing? That may inform me about the direction to pursue.
Also, the trained model is anyway saved with a TensorFlow saver in the model directory so you can load it in other ways if you want. Does this answer your request?

from ludwig.

Let's see if I am able to express my idea in the right way.

I would start by assuming that a user had previously created a model. I would like also to assume that the user that created the model (or the team) is able to understand what are the labels that will be predicted (using the same case you mentioned above) as well as how the data will look like when predict is called. In other words, the user(s) is responsible for the input and the output of the model.

Given that, the cli could expose something like this ludwig expose --model_path=/path/to/model --mapper=/path/to/JSON_mapper. Internally, what ludwig can do is to wrap the model into a POST REST endpoint - something like this

from ludwig import LudwigModel
from flask import Flask, request, jsonify
app = Flask(__name__)

model_path = ... # you get this from the cli
mapper_path = ... # you get this from the cli

@app.route('/model/predict', ['POST'])
def predict():
    body = request.get_json(silent=True)
    # map input based on JSON file containing mappings from strings to tensors
    mapper = read_mapper(mapper_path)
    input = mapper.transform_input(body)
   
    # load model
    model = LudwigModel.load(model_path)

    # obtain predictions
    predictions = model.predict(input)
   
    # map output based on JSON file containing mappings from strings to tensors
    result = mapper.transform_output(predictions)
    return jsonify(result)

if __name__ == "__main__":
    # read cli here
    model_path = ...
    mapper_path = ...

    app.run(host='0.0.0.0', debug=True)

The body of the POST request needs to respect some rules in order to map with the JSON file. You can argue with the fact that the JSON file may not be available but perhaps for the time being, the JSON file could be carried with the model and passed via the CLI. I am sure that there are smarter ways for doing so but maybe this can work.
It would be possible to extend the cli with other arguments such as: port, debug (true/false), host, etc. I would be happy to contribute on this if you think that could work :)

This is the first thing that popped into my mind on how to do it. I didn't test the code nor I don't know if it could work. The whole point is that the user should take care of the data in input and the expected output can be taken care by the webservice (or client in general).

The whole point of this expose is to help teams in integrating the data scientist work in the engineering development. This can also help to Dockerize the models and using it in a production environment (if suitable).

Anyway, I hope all this makes sense :)

from ludwig.

I think I get your point, but there's probably something you are missing about the API: Ludwig saves what you call mapper inside the model directory, and calls it train_set_metadata. moreover, when you call model.predict() Ludwig already performs the mapping of raw data into tensor using the content of that metadata file and already maps the outputs back into the data space again using the contents of that filw.
So it's already done for you :)
Moreover, model.predict() accepts a dictionary as input that looks like {'feature1_name': [...], 'feature2_name': [...]} (lists are there because you can provide more than one datapoint at a time) so you can basically directly parse the JSON coming from the request and give it to model.predict(data_dict=body). Take a look at the API docs for more details.

Additionally, a note on your code: you are loading the model every time you receive a request, that will be extremely slow. you want to load the model one time at the startup of the Flask app and then only call predict when you get a new request.

Finally, doing this sort of serving for the sake of building little demos is something we were considering anyway, so if you make it work, please consider contributing it, it would be much appreciated.

from ludwig.

Thus the trained model is saved with a TensorFlow saver in the model directory and we can use it for other projects. I am working in the computer vision area. I notice that ludwig already supports image claissification task, how about image detection task or segmentation, etc. On more thing, I am still not sure the underlying technology used in the ludwig to get the model. Is it similar to some kind of NAS(neural architecture search)?

from ludwig.

@w4nderlust Thanks for your clarification. I will take a deeper look to the APIs and understand better the underlying system.

Additionally, a note on your code: you are loading the model every time you receive a request, that will be extremely slow. you want to load the model one time at the startup of the Flask app and then only call predict when you get a new request.

Yeah you are absolutely right. In my defence, I wrote it quickly without much thinking 😛

Finally, doing this sort of serving for the sake of building little demos is something we were considering anyway, so if you make it work, please consider contributing it, it would be much appreciated.

All right, I will find the time to create a PR about it and we can eventually take it from there.
Grazie mille per il supporto :)

from ludwig.

@spaghettifunk you're welcome! You can also reach out to me directly if you need help for the PR.
@AllenDun that's a different discussion, please post it on the gitq page and let's discuss there, as I'm closing this issue now.

from ludwig.

Thanks so much for this wonderful set of tools!

Not sure if this is the right place to ask about extending the routes in serve.py. But it was the only existing, relevant issue I could find with the search tool.

I am running into problems when I try to add another route to ludwig.serve. I am trying to add a simple response for the root url, which I've added just above the /predict route. For some reason, this approach has not worked for me:

def server(model):
    app = FastAPI()

    input_features = {
        f['name'] for f in model.model_definition['input_features']
    }

    @app.get('/')
    def root():
        return {"Hello": "World!"}

    @app.post('/predict')
    async def predict(request: Request):
        form = await request.form()
        files, entry = convert_input(form)

        try:
            if (entry.keys() & input_features) != input_features:
                return JSONResponse(ALL_FEATURES_PRESENT_ERROR,
                                    status_code=400)
            try:
                resp = model.predict(data_dict=[entry]).to_dict('records')[0]
                return JSONResponse(resp)
            except Exception as e:
                logger.error("Error: {}".format(str(e)))
                return JSONResponse(COULD_NOT_RUN_INFERENCE_ERROR,
                                    status_code=500)
        finally:
            for f in files:
                os.remove(f.name)


    return app

from ludwig.

Answered in the other issue.

from ludwig.