Agreement-Based Labeling

This article applies to these versions of LandingLens:

LandingLens offers the ability to create and manage a Label Book. A Label Book is a living document to help guide users on how to properly label various categories of Classes. Now, you might be asking:

• How do we measure the consistency of labels across multiple users?
• How do we ensure that only clean, accurate labels will be added to images?

The answer is Agreement-Based Labeling. Agreement-Based Labeling is the concept of allowing multiple users to label the same set of images to analyze the consistency of their labels. This method allows you to identify the areas where users may need more direction with labeling.

For example, let’s say you have these Classes for Labelers to choose from:

• Scratch
• Rub Mark
• Crack

Now take a look at the image below. Some Labelers may choose to label the identified area as a Scratch, while others may select Rub Mark or Crack. To help clarify this ambiguity, you can use Agreement-Based Labeling, then update your Label Book accordingly.

​

Agreement-Based Labeling Workflow

This is how Agreement-Based Labeling works:

1. Send a Labeling Task to two or three users. This allows multiple users to label the same sets of images.
2. After the images have been labeled, LandingLens:
   • Calculates the similarities between the labels. This calculation is called the Agreement Score.
   • Ranks the images by Agreement Score, from lowest (least agreed and most confusion) to highest (most agreed and least confusion).
3. You can then review the labels and accept or reject them. When you reject a label, you can provide feedback to the Labeler. To prevent issues in the future, add clarifying information to the Label Book.

​

Labeling Tasks

Let’s say you labeled some images, and you want to compare your findings with another user. To do this, you can send a Labeling Task to that user.

LandingLens sends images in batches of 20 until all images have been reviewed by the required number of Labelers. However, if you send a Labeling Task with fewer than 20 images, LandingLens automatically divides those images into smaller batches. Let’s explore how LandingLens sends images in an example.

Let’s say you create a Labeling Task with these settings:

• The Labeling Task has 30 images.
• Two Labelers are required to label each image.

You then send the Labeling Task to these three Labelers: Anna, Cameron, and Eliza. The following steps will occur:

1. LandingLens divides the 30 images into two batches:
   • Batch One: Includes the first 20 images
   • Batch Two: Includes the remaining 10 images
2. LandingLens sends Batch One to two of the Labelers. (Let’s say, Anna and Cameron.)
3. Since two Labelers are required to label each image, the first person to complete their labels will be sent** Batch Two** . For example, if Cameron finishes labeling Batch One before Anna, then he will be sent Batch Two.
4. LandingLens sends Batch Two to the third Labeler. (In this case, Eliza.)

​

Send Labeling Tasks

1. Open the project.
2. Select the images you labeled.
3. Click Options in the action bar near the bottom of the screen and select Create Labeling Task. (You may need to scroll to see this option.)
4. Enter a brief, descriptive name for the Labeling Task in the Task Name field.
5. Select how many Labelers you want to label the images from the drop-down menu. For example, if you want two Labelers to label images, select 2.
6. If you’ve already labeled some images, you can choose whether to display (carry over) those existing labels. Select No to remove the labels, or select Yes to display the labels.
7. Select the Labelers you want to send a Labeling Task to from the Assign Labelers drop-down menu. To add multiple users, click the + icon.
8. Click Create Task. An email is sent to the Labelers, notifying them of the task.

​

Events That Occur After Labeling Tasks Are Sent

After a Labeling Task is sent, these events occur:

• The New Labeling Task email is sent to the Labeler. This email includes a link to the requested images.
• Images in a Labeling Task have the status In Task.
• Images in a Labeling Task can only be labeled by the requested Labelers. (The requested Labelers can only label images using the link from the New Labeling Task email.)

​

Labeling Tasks for Segmentation Projects

Labeling Tasks in Segmentation projects are assigned to users in batches of five images. For example, if you create a Labeling Task with 10 images and assign it to a user, that user will get two batches of five images each.

When the user opens a Labeling Task, they will see five images. When they submit that batch, the second batch of five images becomes available.

Limiting each batch to five images allows LandingLens to process each batch quickly.

​

Assign Additional Users to Labeling Tasks

1. Open the project.
2. Click Tasks in the left side panel.
3. On the Reviewing tab, click the vertical ellipses button and select Assign To.
4. Click the + icon.
5. Enter the name of the user you want to assign the Labeling Task to. (Remember, you can only invite users who have been invited to your project.)
6. Click Assign.

​

View and Submit Labeling Tasks

If a user has sent you a Labeling Task, you will receive the New Labeling Task email.

To view an assigned Labeling Task:

1. Click the link in the New Labeling Task email.
2. The task opens. Label the images.
3. After you’ve labeled all images, click Submit to share your labels.

​

Review Submitted Labeling Tasks

After all the Labelers have submitted their Labeling Task, you will receive the Task [#] Has Labels Ready for Your Review email. This email includes a link to the labeled images. Click this link to compare the labels. You can also click the Review button on the Tasks page to compare the labels.

The Review Task page with the labeled images will display. This page includes several settings to help you review the labels. The table below describes these settings.