Deploy Models
JSON Responses
This article applies to these versions of LandingLens:
When you run inference on images, the prediction results display in a JSON format. The JSON schema varies based on model type. This article describes the JSON responses for each model type, and provides example code snippets and images.
The coordinates correspond to the points described in the following table and image.
The following table describes the objects in the JSON response for Segmentation and Visual Prompting models. See these Segmentation and Visual Prompting examples.
The following code snippet and image show the predictions for a Visual Prompting model trained to detect shipping containers in ports. The model has two classes: Shipping Containers and Other. Each class is a separate object nested in the
| LandingLens | LandingLens on Snowflake |
|---|---|
| (see exceptions below) |
Differences Between Cloud Deployment, LandingEdge, and Docker
The JSON schema is slightly different between inferences run via Cloud Deployment, and inferences run via LandingEdge and Docker. For example, some elements display in a different order, and the predictions are included in different elements. Differences between the schemas are noted in the tables. Example responses are from Cloud Deployments.Post-Processing and “Backbone” Elements
When an image is sent for inference via Cloud Deployment, it goes through the model, and then through a post-processing Classification step. This step classifies the image as"OK" or "NG" (short for “Not Good”). Because the process has two steps, the JSON schema for Cloud Deployment includes two sets of predictions:
backbonetypeandbackbonepredictions: These elements contain the data for the actual model that ran inference on the image.typeandpredictions: These elements contain the data for the Classification step.
type and predictions elements.
However, you can run a script that adds a post-processing model or Classification step when running inference with LandingEdge or Docker. If a post-processing step is added, the JSON will have the same prediction elements that are used for Cloud Deployments. In other words, the model predictions will move to the backbonetype and backbonepredictions elements, and the predictions for the post-processing step will be in the type and predictions elements.
JSON Responses for Object Detection Models
The following table describes the objects in the JSON response for Object Detection models. An example response is here.| Element | Description |
|---|---|
backbonetype (Cloud Deployment)type (LandingEdge, Docker) | The name of the prediction type. For Object Detection, this will always be ObjectDetectionPrediction. |
backbonepredictions (Cloud Deployment)predictions (LandingEdge, Docker) | This object contains the information for each region that the model predicted as an object of interest. Each prediction is a separate object nested in this element. |
xxxxxxxxxxxxx-xxxx-xxxx-xxxxxxxxxxxx | This alphanumeric number is an internal identifier for the region that the model predicted as an object of interest. |
score | The confidence score for the prediction. |
labelName | The predicted class. |
labelIndex | The internal identifier of the predicted class. Each class in a project is assigned a number, starting with 0. If you delete a class, that class’s number is not re-assigned to any current or future classes in that project. |
coordinates | The x and y coordinates (in pixels) of the edges of the bounding box around the predicted object of interest. |
defect_id | The unique identifier for the predicted class. |
predictions (Cloud Deployment) | When an image is sent for inference via Cloud Deployment, it goes through the model, and then through a post-processing Classification step. The predictions object contains the results from the Classification step. This step classifies the image as "OK" or "NG" (short for “Not Good”).This step simplifies defect detection. For example, if a model checks sheet metal for scratches, the metal should be marked as defective regardless if there are one or many scratches. score: Confidence score for the Classification prediction.labelName: If the number of predictions is > 0 then "NG", else "OK".This element is only applicable to Cloud Deployment. Inference run via LandingEdge and Docker does not include the post-processing Classification step. |
"type": "ClassificationPrediction" (Cloud Deployment) | When an image is sent for inference via Cloud Deployment, it goes through the model, and then through a post-processing Classification step. This object is the prediction type of the post-processing Classification step. This element is only applicable to Cloud Deployment. Inference run via LandingEdge and Docker does not include the post-processing Classification step. |
metadata (LandingEdge, Docker) | This element contains nested metadata from the image. If the Image Source is Folder Watcher, some data is populated by default. You can use scripts and web APIs to set or override the values. This element only displays in the JSON response for LandingEdge and Docker. |
image_id (LandingEdge, Docker) | If the Image Source is Folder Watcher, this is the file name of the image. Otherwise, this object is blank. You can use scripts and web APIs to set or override the values. This element only displays in the JSON response for LandingEdge and Docker. |
inspection_station_id (LandingEdge, Docker) | This element is blank by default. You can use scripts and web APIs to set or override the values. This element only displays in the JSON response for LandingEdge and Docker. |
location_id (LandingEdge, Docker) | If the Image Source is Folder Watcher, this is the directory that the image is in. Otherwise, this object is blank. You can use scripts and web APIs to set or override the values. This element only displays in the JSON response for LandingEdge and Docker. |
capture_timestamp (LandingEdge, Docker) | The time and date that OCR was run on the image. You can use scripts and web APIs to set or override the values. This element only displays in the JSON response for LandingEdge and Docker. |
latency | The latency object includes the detailed timing statistics of the inference call.Each key-value pair in the latency object represents a step in the inference process, and the duration of that step. All values are measured in seconds. |
model_id | The unique identifier for the model. For more information, go here. |
Coordinates in Object Detection Responses
The JSON response for Object Detection predictions includes acoordinates object, which communicates the size and location of the bounding box around the prediction region of interest.
Each section includes the coordinates of the pixels for each edge of the bounding box. The origin for the coordinates is the top left corner of the image.
For example, let’s say that the following snippet is the coordinate object in the JSON output for a prediction.
| Name | Description | Coordinate (in pixels) |
|---|---|---|
| xmin | The distance from the left edge of the image to the left side of the bounding box. | 3627 |
| ymin | The distance from the top of the image to the top side of the bounding box. | 979 |
| xmax | The distance from the left edge of the image to the right side of the bounding box. | 3965 |
| ymax | The distance from the top of the image to the bottom side of the bounding box. | 1299 |
Example: Object Detection JSON Response
The following code snippet and image show the predictions for an Object Detection model trained to detect hard hats. The model has one class: Hard Hat. The model correctly predicted two hard hats. Each prediction is nested in thebackbonepredictions object.
This JSON response is from Cloud Deployment.
JSON Responses for Segmentation and Visual Prompting Models
is not available in .
| Element | Description |
|---|---|
backbonetype (Cloud Deployment)type (LandingEdge, Docker) | The name of the prediction type. For Segmentation, this will always be SegmentationPrediction. For Visual Prompting, this will always be null. |
backbonepredictions (Cloud Deployment)predictions (LandingEdge, Docker) | For Segmentation, this object contains information about the image and the model’s predictions. For Visual Prompting, this will always be null. |
imageHeight | The height of the image in pixels. |
imageWidth | The width of the image in pixels. |
encoding | The information about how LandingLens encodes Segmentation predictions. Predictions are encoded with run-length encoding (RLE). 0 is mapped to Z, and 1 is mapped to N. This section will always be: “encoding”: { "algorithm": "rle", "options": { "map": { "Z": 0, "N": 1 } } }, |
bitmaps | This object contains the information for each class that the model predicted in the image. Each predicted class is a separate object nested in the bitmaps object.LandingEdge includes data for all classes in the project, and not just the ones predicted in the image. |
xxxxxxxxxxxxx-xxxx-xxxx-xxxxxxxxxxxx | This alphanumeric number is an internal identifier for the region that the model predicted as an object of interest. The prediction information is nested in this object. |
score | The confidence score for the prediction. |
bitmap | The prediction mask (the areas that the model predicted for a specific class) is formatted as a bitmap that is compressed with a run-length encoding (RLE) format. The Python library parses the JSON responses, which includes parsing and extracting bitmaps. We recommend using the library if you want to avoid manually parsing the bitmaps. Check out the Satellite Images and Post-Processing tutorial for an end-to-end example of running inference and visualizing the results with our Python library. You can use the decode_bitmap_rle API to decode the bitmap string into a NumPy array, which you can then convert into a PNG file. |
defect_id | The unique identifier for the predicted class. |
labelIndex | The internal identifier of the predicted class. Each class in a project is assigned a number, starting with 0. If you delete a class, that class’s number is not re-assigned to any current or future classes in that project. |
labelName | The predicted class. |
data (Cloud Deployment) | This will always be null.This element only displays in the JSON response for Cloud Deployment. |
labels (Cloud Deployment) | This will always be null.This element only displays in the JSON response for Cloud Deployment. |
num_classes (Cloud Deployment) | This is the total number of classes in the project. Segmentation and Visual Prompting projects have an internal class called “ok”. This class is included in the num_class object. So if you created 3 classes in the project, the num_class value is 4.This element only displays in the JSON response for Cloud Deployment. |
predictions (Cloud Deployment) | When an image is sent for inference via Segmentation or Visual Prompting, it goes through the model, and then through a post-processing Classification step. The predictions object contains the results from the Classification step. This step classifies the image as "OK" or "NG" (short for “Not Good”).This step simplifies defect detection. For example, if a model checks sheet metal for scratches, the metal should be marked as defective regardless if there are one or many scratches. score: Confidence score for the Classification prediction.labelName: If the number of predictions is > 0 then "NG", else "OK".This element is only applicable to Cloud Deployment. Inference run via LandingEdge and Docker does not include the post-processing Classification step. |
"type": "ClassificationPrediction" (Cloud Deployment) | When an image is sent for inference via Cloud Deployment, it goes through the model, and then through a post-processing Classification step. This object is the prediction type of the post-processing Classification step. This element is only applicable to Cloud Deployment. Inference run via LandingEdge and Docker does not include the post-processing Classification step. |
metadata (LandingEdge, Docker) | This element contains nested metadata from the image. If the Image Source is Folder Watcher, some data is populated by default. You can use scripts and web APIs to set or override the values. This element only displays in the JSON response for LandingEdge and Docker. |
image_id (LandingEdge, Docker) | If the Image Source is Folder Watcher, this is the file name of the image. Otherwise, this object is blank. You can use scripts and web APIs to set or override the values. This element only displays in the JSON response for LandingEdge and Docker. |
inspection_station_id (LandingEdge, Docker) | This element is blank by default. You can use scripts and web APIs to set or override the values. This element only displays in the JSON response for LandingEdge and Docker. |
location_id (LandingEdge, Docker) | If the Image Source is Folder Watcher, this is the directory that the image is in. Otherwise, this object is blank. You can use scripts and web APIs to set or override the values. This element only displays in the JSON response for LandingEdge and Docker. |
capture_timestamp (LandingEdge, Docker) | The time and date that OCR was run on the image. You can use scripts and web APIs to set or override the values. This element only displays in the JSON response for LandingEdge and Docker. |
latency | The latency object includes the detailed timing statistics of the inference call.Each key-value pair in the latency object represents a step in the inference process, and the duration of that step. All values are measured in seconds. |
model_id | The unique identifier for the model. For more information, go here. |
Example: Segmentation JSON Response
The following code snippet and image show the predictions for a Segmentation model trained to detect chips on pills. The model has one class: Chipped. Even though there is more than one chipped region, all of the data for Chipped predictions are included in onebitmap object, because these regions all correspond to the Chipped class.
This JSON response is from Cloud Deployment.
In the image, the purple overlay is the prediction for Chipped.
Example: Visual Prompting JSON Response
is not available in .
bitmaps object.
This JSON response is from Cloud Deployment.
In the image, the purple overlay is the prediction for Shipping Containers, and the yellow overlay is the prediction for Other.
Script for Parsing Segmentation and Visual Prompting Results
Use this Colab notebook to parse the JSON results from Segmentation and Visual Prompting models.JSON Responses for Classification Models
The following table describes the objects in the JSON response for Classification models. An example response is here.| Element | Description |
|---|---|
backbonetype (Cloud Deployment) | This object isn’t applicable to Classification models, because the prediction type is captured in the type object. For Classification, this will always be null.This element only displays in the JSON response for Cloud Deployment. |
backbonepredictions (Cloud Deployment) | This object isn’t applicable to Classification models, because the prediction type is captured in predictions. For Classification, this will always be null.This element only displays in the JSON response for Cloud Deployment. |
predictions | This object contains the information for the model’s prediction. |
score | The confidence score for the prediction. |
labelName | The predicted class. |
labelIndex | The internal identifier of the predicted class. Each class in a project is assigned a number, starting with 0. If you delete a class, that class’s number is not re-assigned to any current or future classes in that project. |
defectId (LandingEdge, Docker) | The unique identifier for the predicted class. This element only displays in the JSON response for LandingEdge and Docker. |
rawScores (LandingEdge, Docker) | This element only displays in the JSON response for LandingEdge and Docker. |
"type": "ClassificationPrediction" | The name of the prediction type. For Classification, this will always be ClassificationPrediction. |
metadata (LandingEdge, Docker) | This element contains nested metadata from the image. If the Image Source is Folder Watcher, some data is populated by default. You can use scripts and web APIs to set or override the values. This element only displays in the JSON response for LandingEdge and Docker. |
image_id (LandingEdge, Docker) | If the Image Source is Folder Watcher, this is the file name of the image. Otherwise, this object is blank. You can use scripts and web APIs to set or override the values. This element only displays in the JSON response for LandingEdge and Docker. |
inspection_station_id (LandingEdge, Docker) | This element is blank by default. You can use scripts and web APIs to set or override the values. This element only displays in the JSON response for LandingEdge and Docker. |
location_id (LandingEdge, Docker) | If the Image Source is Folder Watcher, this is the directory that the image is in. Otherwise, this object is blank. You can use scripts and web APIs to set or override the values. This element only displays in the JSON response for LandingEdge and Docker. |
capture_timestamp (LandingEdge, Docker) | The time and date that OCR was run on the image. You can use scripts and web APIs to set or override the values. This element only displays in the JSON response for LandingEdge and Docker. |
latency | The latency object includes the detailed timing statistics of the inference call.Each key-value pair in the latency object represents a step in the inference process, and the duration of that step. All values are measured in seconds. |
model_id | The unique identifier for the model. For more information, go here. |
Example: Classification JSON Response
The following code snippet and image show the predictions for a Classification model trained to detect defects on metal sheets. The model has three classes: Pitted, Scratched, and No Defect. The model correctly predicted the Scratched class. The prediction data is in thepredictions object.
This JSON response is from Cloud Deployment.
JSON Responses for Anomaly Detection Models
The following table describes the objects in the JSON response for Anomaly Detection models. An example response is here.| Element | Description |
|---|---|
backbonetype (Cloud Deployment)type (LandingEdge, Docker) | The name of the prediction type. For Anomaly Detection, this will always be AnomalyDetectionPrediction. |
backbonepredictions (Cloud Deployment)predictions (LandingEdge, Docker) | This object contains the prediction information. |
score | The confidence score for the prediction. |
defect_id | The unique identifier for the predicted class. |
labelIndex | The internal identifier of the predicted class. Normal is 0. Abnormal is 1. |
labelName | The predicted class. |
predictions (Cloud Deployment) | When an image is sent for inference via Cloud Deployment, it goes through the model, and then through a post-processing Classification step. The predictions object contains the results from the Classification step. This step classifies the image as "OK" or "NG" (short for “Not Good”).This step simplifies defect detection. For example, if a model checks sheet metal for scratches, the metal should be marked as defective regardless if there are one or many scratches. score: Confidence score for the Classification prediction.labelName: If the number of predictions is > 0 then "NG", else "OK".This element is only applicable to Cloud Deployment. Inference run via LandingEdge and Docker does not include the post-processing Classification step. |
"type": "ClassificationPrediction" (Cloud Deployment) | When an image is sent for inference via Cloud Deployment, it goes through the model, and then through a post-processing Classification step. This object is the prediction type of the post-processing Classification step. This element is only applicable to Cloud Deployment. Inference run via LandingEdge and Docker does not include the post-processing Classification step. |
metadata (LandingEdge, Docker) | This element contains nested metadata from the image. If the Image Source is Folder Watcher, some data is populated by default. You can use scripts and web APIs to set or override the values. This element only displays in the JSON response for LandingEdge and Docker. |
image_id (LandingEdge, Docker) | If the Image Source is Folder Watcher, this is the file name of the image. Otherwise, this object is blank. You can use scripts and web APIs to set or override the values. This element only displays in the JSON response for LandingEdge and Docker. |
inspection_station_id (LandingEdge, Docker) | This element is blank by default. You can use scripts and web APIs to set or override the values. This element only displays in the JSON response for LandingEdge and Docker. |
location_id (LandingEdge, Docker) | If the Image Source is Folder Watcher, this is the directory that the image is in. Otherwise, this object is blank. You can use scripts and web APIs to set or override the values. This element only displays in the JSON response for LandingEdge and Docker. |
capture_timestamp (LandingEdge, Docker) | The time and date that OCR was run on the image. You can use scripts and web APIs to set or override the values. This element only displays in the JSON response for LandingEdge and Docker. |
latency | The latency object includes the detailed timing statistics of the inference call.Each key-value pair in the latency object represents a step in the inference process, and the duration of that step. All values are measured in seconds. |
model_id | The unique identifier for the model. For more information, go here. |
Example: Anomaly Detection JSON Response
The following code snippet and image show the predictions for an Anomaly Detection model for detecting if PCB boards are assembled correctly. The model correctly predicted the image as Normal. The prediction data is in thebackbonepredictions object.
This JSON response is from Cloud Deployment.