The local_cache's discuss from triton-inference-server

local_cache's Issues

Local cache on sagemaker

Experiencing this error when starting a sagemaker endpoint with local-cache:
error: creating server: Invalid argument - unable to find 'libtritoncache_local.so' for cache. Searched: /opt/tritonserver/caches

endpoint image: 746233611703.dkr.ecr.eu-central-1.amazonaws.com/sagemaker-tritonserver:23.12-py3
environment variables:

"Environment": {
                "SAGEMAKER_TRITON_DEFAULT_MODEL_NAME": "ensemble",
                "SAGEMAKER_TRITON_THREAD_COUNT": "10",
                "SAGEMAKER_TRITON_LOG_INFO": "true",
                "SAGEMAKER_TRITON_LOG_VERBOSE": "true",
                
                "SAGEMAKER_TRITON_ALLOW_METRICS": "true",
                "SAGEMAKER_TRITON_PUBLISH_METRICS_TO_CLOUDWATCH": "true",
                "SAGEMAKER_TRITON_CLOUDWATCH_LOG_GROUP": f"/aws/SageMaker/Endpoints/TritonMetrics/{EXPERIMENT}",
                "SAGEMAKER_TRITON_CLOUDWATCH_METRIC_NAMESPACE": f"/aws/SageMaker/Endpoints/TritonMetrics/{EXPERIMENT}",
                "SAGEMAKER_TRITON_METRICS_PORT": "22000",
                "SAGEMAKER_TRITON_ADDITIONAL_ARGS": "--cache-config local,size=1000000",
            },

Any idea ho to add libtritoncache_local to sagemaker image ?

Recommend Projects