caikit-nlp-client

A client library for caikit-nlp

Installation

Install from PyPi

pip install caikit-nlp-client

Usage

A few examples follow, see example.py

http

To use the http protocol

from caikit_nlp_client import HttpClient

host = "localhost"
port = 8080
model_name = "flan-t5-small-caikit"
http_client = HttpClient(f"http://{host}:{port}")

text = http_client.generate_text(model_name, "What is the boiling point of Nitrogen?")

gRPC

To use the gRPC protocol

from caikit_nlp_client import GrpcClient

host = "localhost"
port = 8085
model_name = "flan-t5-small-caikit"
grpc_client = GrpcClient(host, port)

text = grpc_client.generate_text(model_name, "What is the boiling point of Nitrogen?")

Text generation methods may accept text generation parameters, which can be provided as kwargs to generate_text and generate_text_stream.

Available values and types be retrieved as a dict for both the grpc and http clients:

for param, default_value in client.get_text_generation_parameters():
    print(f"{param=}, {default_value=}")

Self-signed certificates

To use a self signed certificate, assuming we have a certificate authority cert ca.pem

http_client = HttpClient(f"https://{host}:{http_port}", ca_cert_path="ca.pem")

with open("ca.pem", "rb") as fh:
    ca_cert = fh.read()
grpc_client = GrpcClient(host, grpc_port, ca_cert=ca_cert)

To skip validation altogether:

# http
http_client = HttpClient(f"https://{host}:{http_port}", insecure=True)
# grpc
grpc_client = GrpcClient(host, port, insecure=True)

mTLS

Assuming we have a client.pem and client-key.pem certificate files, and we require ca.pem to validate the server certificate:

# http
http_client = HttpClient(
    f"https://{host}:{http_port}",
    ca_cert_path="ca.pem",
    client_cert_path="client.pem",
    client_key_path="client-key.pem"
)

# grpc
with open("ca.pem", "rb") as fh:
    ca_cert = fh.read()
with open("client.pem", "rb") as fh:
    client_cert = fh.read()
with open("client-key.pem", "rb") as fh:
    client_key = fh.read()

grpc_client = GrpcClient(
    host,
    port,
    ca_cert=ca_cert,
    client_key=client_key,
    client_cert
)

Contributing

Set up pre-commit for linting/style/misc fixes:

pip install pre-commit
pre-commit install

This project uses nox to manage test automation:

pip install nox
nox --list  # list available sessions
nox --python 3.10 -s tests # run tests session for a specific python version

Testing against a real caikit+tgis stack

Tests are run against a mocked instance of a TGIS backend by default. To test against a real caikit+tgis stack, it is sufficient to run using the --real-caikit flag when running pytest:

nox -s tests -- --real-caikit tests

Notes:

The required images (caikit-tgis-serving, text-generation-inference), so it could take a while for tests to start while compose is pulling the required images, it may seem like the tests are hanging.
This uses a real model (google/flan-t5-small), which will be downloaded to tests/fixtures/resources/flan-t5-small-caikit and is around 300MB in size.

vajain-kserve / caikit-nlp-client Goto Github PK