Chaining models via the API

Since release v0.9.3.1, DeepDetect supports chaining multiple models and actions. This allows for instance to detect objects then run other models on object crops. A typical application is detecting faces, then estimating every face’s age.

Let’s define the key elements:

  • /chain is the REST resource defined by DeepDetect to execute a tree-like succession of models and actions
  • a model is represented by a service, as in regular API /predict calls
  • an actions is a stateless piece of code that takes a model’s output, transforms it as needed, and pass it on to the next set of models. As an example, the crop action crops objects after a model has detected them. Available actions are listed at the bottom of this page.

In a chain, the results from all executed model are then merged into the final JSON response. It is important to note that a chain is stateless, i.e. it is executed once and it’s structure does not remain into memory, only the existing services remain.

In order to use the PUT /chain REST API, you’ll first need to create the services used in the chain.

In the following example, faces and age_real services are already running on DeepDetect server:


curl -X PUT 'http://localhost:8080/chain/faces-crop-age_real'
  -d '{
 "chain": {
  "calls": [
   {
    "parameters": {
     "input": {
      "connector": "image",
      "keep_orig": true
     },
     "mllib": {
      "gpu": true
     },
     "output": {
      "bbox": true,
      "confidence_threshold": 0.3
     }
    },
    "service": "faces",
    "data": [
     "https://deepdetect.com/img/examples/multiple_faces.jpg"
    ]
   },
   {
    "id": "crop",
    "action": {
     "type": "crop"
    }
   },
   {
    "parent_id": "crop",
    "parameters": {
     "input": {
      "connector": "image",
      "keep_orig": true
     },
     "output": {
      "confidence_threshold": 0.01,
      "best": 1
     }
    },
    "service": "age_real"
   }
  ]
 }
}'

calls is an array of the services and actions to be executed. The call above executes a first service that detect faces in an image, then executes a cropping action, then runs the age estimation model.

Results are available in json as usual, with the age_real service results nested inside each faces result:


{
  "status": {
    "code": 200,
    "msg": "OK"
  },
  "head": {
    "method": "/chain",
    "time": 3467
  },
  "body": {
    "predictions": [
      {
        "uri": "https://deepdetect.com/img/examples/multiple_faces.jpg",
        "classes": [
          {
            "age_real": {
              "classes": [
                {
                  "prob": 0.04538746178150177,
                  "last": true,
                  "cat": "17"
                }
              ]
            },
            "cat": "1",
            "prob": 0.9733325242996216,
            "bbox": {
              "xmax": 1224.7215576171875,
              "ymax": 123.81903076171875,
              "ymin": 308.4881896972656,
              "xmin": 1007.9154663085938
            }
          },
                    ...
        ]
      }
    ]
  }
}

Chain tree structure

Chains are designed to support tree-like structures. In practice, this means one model’s output can serve as input to many models, thus branching the execution.

Tree-structured chains require the id attribute on every chain service and actions, so that they can be refered as parent_id by lower level services and actions. See the example below:


curl -X PUT 'http://localhost:8080/chain/test'
  -d '{
 "chain": {
  "calls": [
    {
            "id": "root",
            ...
        },
        {
            "parent_id": "root",
            "id": "leaf_child_1",
            ...
        },
        {
            "parent_id": "root",
            "id": "leaf_child_2",
            ...
        },
        {
            "parent_id": "leaf_child_2",
            "id": "leaf_great_child_2",
            ...
        },
        ...
    ]
 }
}

Action chain items

Between each service in a chain, action can by call with the following types:

Cropping

  • crop: Crops the bounding boxes from the parent call and send each box to the children services.
    • padding_ratio: extend the bounding boxes before cropping by a value that is passed as a ratio to the unpadded bounding box size
    • save_crop: saves cropped images on disk.

Example:


curl -X PUT 'http://localhost:8080/chain/test'
  -d '{
 "chain": {
  "calls": [
        ...
        {
            "action": {
                "type": "crop",
                "padding_ratio": 0.05,
                "save_crop": true
            }
        },
        ...
    ]
 }
}

Filtering

  • filter: Keeps parent results based on a list of class names
    • classes: classes name to keep

Example:

Keep results for the “car”, “dog” and “cat” classes only.


curl -X PUT 'http://localhost:8080/chain/test'
  -d '{
 "chain": {
  "calls": [
        ...
        {
            "action": {
                "type": "filter",
                "classnames": ["car", "dog", "cat"]
            }
        },
        ...
    ]
 }
}

Related