aldelar / ray-serve-on-aml Goto Github PK

Need to code a simple load testing set of scripts based on load testers or document this process to validate and measure/document performance scenarios of a few classic use cases.

Note: test that this works + easily generate data for blog posts, more elaborated documents to outline the benefits of this solution

Need to abstract this code out of the main core base.

Exception handling when a model is not available in Redis

Need to be able to restrict one tenant from scoring for another tenant.

Mostly we need to document how you would implement multi-tenant RBAC on the scoring method.

Need to govern via config the # of Deployments in the service so this gets dynamically created instead of hardcoded.

Q: Do we need to enable a live reconfiguration of the number of deployments? Say the customer demand increases over time and this solution drives an HA system which can't go down.

We want to avoid having to create/update a zip of the code

Assuming AML trained many models deployed in the registry, we would probably rely on name conventions + tags to lazy load the proper models on inference, potential parameters for naming/tags:

• tenant name (or something more generic as some many models scenarios aren't going to be about tenant)
• versions
• stage tags (for instance: prod vs stage vs test, or blue, green, etc.
• tiers: do we have performance tiers, there may be a Tier1 that always needs to be in cache no matter what (nice to have feature potentially)
  Etc.

Hide secret using .Env

Find forecast model, etc.