Dedicated Kafka Connector to track changes in
MLflow Model Registry
Kafka Connector to track model stage changes for configured MLflow Model Registry instance.
The purpose is to fetch ModelRegistry.ModelVersion which recently change status to Production and generate
Model Export Request as event on Kafka topic.
Event schema and sample payload of the event
{
"schema": {
"type": "struct",
"fields": [
{
"type": "string",
"optional": false,
"field": "Name"
},
{
"type": "string",
"optional": false,
"field": "Version"
},
{
"type": "int64",
"optional": false,
"field": "CreationTimestamp"
},
{
"type": "int64",
"optional": false,
"field": "lastUpdatedTimestamp"
},
{
"type": "string",
"optional": true,
"field": "userId"
},
{
"type": "string",
"optional": false,
"field": "currentStage"
},
{
"type": "string",
"optional": false,
"field": "description"
},
{
"type": "string",
"optional": false,
"field": "source"
},
{
"type": "string",
"optional": false,
"field": "runId"
},
{
"type": "string",
"optional": false,
"field": "status"
},
{
"type": "string",
"optional": true,
"field": "statusMessage"
},
{
"type": "array",
"items": {
"type": "struct",
"fields": [
{
"type": "string",
"optional": false,
"field": "Key"
},
{
"type": "string",
"optional": false,
"field": "Value"
}
],
"optional": false,
"name": "ModelExportRequest"
},
"optional": true,
"field": "tags"
},
{
"type": "string",
"optional": false,
"field": "runLink"
}
],
"optional": false,
"name": "ModelExportRequest"
},
"payload": {
"Name": "aaa",
"Version": "1",
"CreationTimestamp": 1609331596360,
"lastUpdatedTimestamp": 1609331610766,
"userId": "",
"currentStage": "Production",
"description": "",
"source": "file:///tmp/test/1/6ebcc72f3ad24c65b1821ff5283caa0d/artifacts/model",
"runId": "6ebcc72f3ad24c65b1821ff5283caa0d",
"status": "READY",
"statusMessage": "",
"tags": [],
"runLink": ""
}
}Connector is distributed as a jar file. In order to build the assembly use mvn clean install -f ./kafka-connect-mlflow.
Assembly will be available in directory ./kafka-connect-mlflow/target/kafka-connect-mlflow-${project_version}-assembly.jar.
Kafka Connect requires Apache Kafka and Apache Zookeeper servers.
There is docker-compose.yaml available to run all required components as containers.
Generated jar is mounted as the volume in kafka-connect container taken directly from ./target directory.
Additionally, docker-compose/yaml contains kafka-connect-ui service to setup kafka-connect-mlflow instance through browser.
It is exposed on port 8000 by default.
Connector should now be available through Kafka Connect UI:
Creating new instance of connector:
Event can be trigger by registering model version from MLflow Tracking UI:
And changing the stage of the model version to Production:
Maven support releases with maven-release-plugin.
Release can be generated using two-step procedure:
-
Prepare release (use flag
-DdryRun=trueif you want to verify it before creating release)mvn release:prepare -DignoreSnapshots=true -DskipTests=true -f ./kafka-connect-mlflow/two commits will be added on top of your commit and pushed to your branch:
- prepare release kafka-connect-mlflow-${version} with proper tag
- prepare for next development iteration
-
Tag pushed in the previous step triggers gitlab ci/cd
deploystage which publish your assembly toartifactory -
Clean release files after all
mvn release:clean -f ./kafka-connect-mlflow/
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent