Module of the Viam vision service to automatically build, deploy, and run inferences on custom object detectors and image classifiers via DirectAI's cloud offering. Since this is a beta test of such a service, the latency may be high and availability of the cloud service is not guaranteed. See here for a demo!

Internally, this module calls DirectAI's deploy_detector and deploy_classifier endpoints with the provided configuration, saves the ID of the deployed models, and then calls the models via their IDs on incoming images. See docs for an overview of DirectAI's publicly available API.

Note that this README closely follows Viam's in-house motion-detector documentation, as described here.

Start by configuring a camera on your robot. Remember the name you give to the camera, it will be important later.

Before calling DirectAI's API via Viam, you have to grab your client credentials. Generate them by clicking Get API Access on the DirectAI website. Save them in an Access JSON on the same machine that's running your Viam Server.

{
  "DIRECTAI_CLIENT_ID": "UE9S0AG9KS4F3",
  "DIRECTAI_CLIENT_SECRET": "L23LKkl0d5<M0R3S4F3"
}

Navigate to the Config tab of your robot’s page in the Viam app. Click on the Services subtab and click Create service. Select the vision type, then select the directai-beta model. Enter a name for your service and click Create.

On the new component panel, copy and paste the Example Detector or Classifier Attributes. Note that you can deploy classifier & detector attributes simultaneously if you'd like. Ensure that the Access JSON path that you provide in your Config is absolute, not relative. (e.g. /Users/janesmith/Downloads/directai_credential.json, not ~/Downloads/directai_credential.json)

{
  "access_json": "ABSOLUTE_PATH_TO_ACCESS_JSON_FILE",
  "deployed_detector": {
    "detector_configs": [
      {
        "name": "nose",
        "examples_to_include": [
          "nose"
        ],
        "detection_threshold": 0.1
      },
      {
        "examples_to_include": [
          "mouth"
        ],
        "examples_to_exclude": [
          "mustache"
        ],
        "detection_threshold": 0.1,
        "name": "mouth"
      },
      {
        "examples_to_include": [
          "eye"
        ],
        "detection_threshold": 0.1,
        "name": "eye"
      },
    ],
    "nms_threshold": 0.1
  }
}

{
  "access_json": "ABSOLUTE_PATH_TO_ACCESS_JSON_FILE",
  "deployed_classifier": {
    "classifier_configs": [
      {
        "name": "happy",
        "examples_to_include": [
          "happy person"
        ],
      },
      {
        "name": "sad",
        "examples_to_include": [
          "sad person"
        ]
      }
    ]
  }
}

The following attributes are available for directai:viamintegration:directai-beta vision services:

This module implements the following methods of the vision service API:

• GetDetections()
• GetDetectionsFromCamera()
• GetClassifications()
• GetClassificationsFromCamera()

The module behavior differs slightly for classifications and detections.

When returning classifications, the module will return a list of dictionaries. The list length will be equal to length of the classifier_configs list provided in the deployed classifier attributes. Each dictionary will include class_name and confidence key-value pair in decreaising confidence order.

When returning detections, the module will return a list of detections with bounding boxes that encapsulate the movement. Each detection will be of the following form:

{
  "confidence": float,
  "class_name": string,
  "x_min": float,
  "y_min": float,
  "x_max": float,
  "y_max": float
}

Once the directai:viamintegration:directai-beta modular service is in use, configure a transform camera to see classifications or detections appear in your robot's field of vision. View the transform camera from the Control tab.

To write code to use the motion detector output, use one of the available SDKs.

Please join DirectAI's discord, contact us at [email protected], or schedule time on our calendly if you have any questions or feedback!