Bounding Box Labeling

Bounding Box Project

The following guide will show you how to create a bounding box project and how to draw and label bounding boxes on your PDF/image/document. As a complimentary guide, watch this 2:00 minute Youtube video on how to create Bounding Box project.

Interface Legend

If you have already your project, use this legend for understand the basics of the Bounding Box UI. Also, see this :43 second Youtube video on the basics of drawing and labeling a bounding box!

Project Creation

You can create a Bounding Box Labeling project in Datasaur. It is pretty simple, you need to prepare the image or document file for this project. From your project home page, open "Create a Custom Project." In order to begin, you should upload the supported image or document file.

The type of Image files we accept: .bmp, .gif, .jpeg, .jpg, .png, .svg, .tiff, .tif, .webp

The type of document files we accept: .pdf

Here is an example:

527KB

Datasaur sample - Bbox.zip

archive

So now our window should look like this,

For the Step 2. Preview please make sure you select None for the Apply OCR Method under the preview settings. Afterward, we can move on to the next steps, under Step 3. Labeler’s tasks, you can add a label set by ticking the Bounding Box labeling checkbox. Here’s what the settings will do for your project:

Label Sets

You can utilize .csv, .tsv, or .json formats for the bounding box label set.

• For .csv/.tsv, we support color names (e.g., red), hex values (e.g., #00FF00), and RGB (e.g., rgb(0,0,255)). You can also utilize the label set with just names, as shown in the Datasaur sample - Bbox only name.csv below. Other values such as captionAllowed and captionRequired will use default settings.

• For .json, we support hex and RGB only.

24B

Datasaur sample - Bbox (only name).csv

201B

Datasaur sample - Bbox (with header).csv

158B

Datasaur sample - Bbox (without header).csv

564B

Datasaur sample - Bbox.json

2KB

Datasaur sample - Bbox (with custom attributes).json

Text Transcription

The Text Transcription setting allows the labeler to add corresponding text to a Bounding box. Disabling this setting means the labeler could not add the text.

By turning on the Text Transcription setting, the labeler can add text to a bounding box. You can choose whether a specific label must have a text by disabling or enabling the Require caption checkbox.

Require caption

For the rest of the Project Creation Process, you will then be able to take all the steps one takes in other project types: Preview, Labeler's Tasks, Assignment, and Project Settings.

Upload Pre-labeled Files

In Bounding Box Labeling Project, you also have the option to upload pre-labeled files. To do this, you can simply upload a JSON file along with the document that you want to label. Ensure that the JSON file follows the Datasaur Schema format. For further details about the Datasaur Schema, refer to this link.

Label Sets Capabilities

Label Set Modification

If you've created a project but forgot to add a label set, no worries. You can either drag and drop or click the three-dot button to include a new label set.

If you've initially set up a label set while creating a bounding box project but need to make edits, simply click the three-dot button and select 'Edit label set'. This will open a pop-up section where you can make your edits.

If you need to replace the current label set, you can do so by clicking the three dots and choosing 'Replace existing label set'. This will provide a drag-and-drop area where you can upload your desired label set.

By default, both Reviewers and Labelers have access to these capabilities. However, if you prefer to restrict this capability for your Labelers, you can adjust it in File > Settings > Administrator and toggle off the option.

Custom Attributes for Label Classes

Each label class in the label set can have multiple custom attributes (questions). The question types are limited to Text and Dropdown only. Each question can also have a default answer.

To modify the custom attributes within the label set, follow these steps:

1. Open the label set editor via Add new label set, Edit label set, or Replace existing label set. Please check the Label Set Modification section above.

   
2. Switch to the JSON tab.

   
3. Add or modify the questions field as needed for each label class.

   💡 We do not support the modification of the type and internalId value of existing questions, as it will affect labels that are already applied in unexpected ways.

   To add questions, include a questions field in the desired label class within the JSON structure. The questions field can contains multiple data with the following structures:

   1. id: ID of the question.

   2. label: The question name.

   3. type: The type of question. It can be TEXT or DROPDOWN.

   4. config: Configuration settings for the question

      1. required: Determines whether the question must be answered. It can be true or false.

      2. defaultValue: The default answer for the question

         1. For TEXT questions, it can be any text.

         2. For DROPDOWN questions, it must be an options.id value.

      3. multiline: Specifies if the TEXT question allows multiple lines. It can be true or false.

      4. options: An array of options for the DROPDOWN question, each containing:

         1. id: ID of the dropdown option. It can be used in the defaultValue.

         2. label: The text label for the dropdown option.

   5. internalId: readonly, unique id Datasaur uses to keep track of each individual question

   For a detailed example, refer to the Label Sets section for a sample of the BBox label set with custom attributes.

Creating Bounding Box(es)

The first thing you will see inside the Bounding Box labeling project will be the file viewer for the documents or images from the previous steps. You can start creating a Bounding Box by clicking and dragging it inside the viewer area. The Bounding Box will consist of four coordinate points.

You can also create a Multiple Bounding Box that will be considered as one by holding the CTRL key when you try to draw the next Bounding Box. When you are finished drawing the number of Bounding Boxes you desire, you can release the CTRL key and starts doing the next step.

Applying Label, Caption and Custom Attributes

Every time we finished drawing Bounding Box(es) on a certain area, a modal will be shown. This modal will have 2 fields by default consisting of Label and Caption, plus any attributes you have previously configured. You can select the label from the list, fill the caption and answer the attributes as needed. To submit the label, you can simply click anywhere inside the document viewer.

You can try to quickly review the value by hovering on the created Bounding Box, and it will show you a tooltip containing the values. Here’s what it will be looks like for a complete Bounding Box and the tooltip.

Automatically fill the caption using OCR

In addition to manual typing, we are now supporting an option to automatically fill captions using Tesseract! When you draw a bounding box around the desired text in the document, Tesseract will return it as the caption.

Please keep in mind that the processing time may be longer for longer text selections within the bounding box.

Enabling OCR for caption can be done in the Step 3. To do so, click on the Auto-caption provider dropdown and select Tesseract. After that, you can continue setting up your project configuration and launch it.\

Auto-caption provider dropdown is available when “Allow text caption” is enabled.

After your project has been successfully created, you are able to modify the auto-caption setting by clicking the triple-dots in the Bounding Box Labels extension. Disabling or enabling the setting will affect all documents within the project.

Auto-caption setting will not appear if no auto-caption provider has been selected in the Step 3.

There are a couple of things that you need to know about auto-caption.

1. Auto-captioning process only happens on drawing bounding box process, not resizing or repositioning the created bounding box

2. Auto-captioning process will be rerun when you resize or reposition the bounding box in a drawing state

3. Auto-captioning process can be canceled by clicking the (x) inside the Caption text field

   
4. Auto-captioning process will be canceled if you click outside the label box, and the bounding box will be created

   1. It will return an empty value if caption for label is optional

   2. It will throw an error if caption for label is required

5. Auto-caption can be used for multiple bounding boxes

   1. It will read all the bounding boxes

   2. It will be separated by a line break between the bounding box content

Single Vertex Editing

Initially, a bounding box is drawn as a rectangle. But to enhance accuracy for images and documents that are not perfectly straight, users can edit individual points of the bounding box by holding the Alt key (or the Option key on Mac), and then clicking and dragging a corner of the bounding box.

Managing Bounding Boxes

Copy and Paste

Copying and pasting a bounding box can only be done within the same document.

Copying and pasting a bounding box can be done in two ways:

1. The context menu: Right-click the bounding box, then select the option to copy or paste from the menu.

2. Using the keyboard shortcut: Select the bounding box, then press Ctrl-C and Ctrl-V (or Command-C and Command-V on Mac).

Delete

Deleting a bounding box can be done in two ways:

1. The context menu: Right-click the bounding box, then select the option to delete from the menu.

2. Using the keyboard shortcut: Select the bounding box, then press Del.

Multiple Selection

The multiple selection capability will be helpful when bulk copying, pasting, or deleting is needed.

Selecting multiple bounding boxes can be done by holding the Ctrl key while selecting each bounding box.

Reviewer Capabilities Overview

Capabilities

• View all labelers' work.

  • Note: Bounding box labels applied in similar positions may overlap.

• Apply labels.

• Update labels applied by labelers.

Limitations

• Unable to accept or reject labels.

• Unable to view all labelers' work with information on who applied each label.

• When updating a label applied by a labeler, there is no information or indicator showing that the label has been changed

Upload Pre-labeled Files

In Bounding Box Labeling Project, you also have the option to upload pre-labeled files.

Bounding Box Labeling supports several types of pre-labeled files. You can simply upload the appropriate file along with the document you want to label. Ensure that your pre-labeled file follows the correct format and that both files have the same name.

The supported formats for the pre-labeled files are the following.

1. Datasaur Schema (.json)

   Further details about the file format can be found here.

2. YOLO (.txt)

   Further details about the file format can be found here.

3. LabelMe (.xml)

   Further details about the file format can be found here.

4. Pascal VOC (.xml)

   Further details about the file format can be found here.

There are things to keep in mind when using YOLO, LabelMe, and Pascal VOC pre-labeled files in Datasaur:

1. When creating a pre-labeled project, make sure to also provide your label set.

2. During project creation, label classes from the file will be automatically mapped to the label classes in your provided label set.

3. If the file contains more label classes than your label set, the project creation process will create additional label classes based on the number found in the file.