Bounding Box Labeling
Last updated
Last updated
The following guide will show you how to create a bounding box project and how to draw and label bounding boxes on your PDF/image/document. As a complimentary guide, watch this 2:00 minute Youtube video on how to create Bounding Box project.
If you have already your project, use this legend for understand the basics of the Bounding Box UI. Also, see this :43 second Youtube video on the basics of drawing and labeling a bounding box!
You can create a Bounding Box Labeling project in Datasaur. It is pretty simple, you need to prepare the image or document file for this project. From your project home page, open "Create a Custom Project." In order to begin, you should upload the supported image or document file.
The type of Image files we accept: .bmp, .gif, .jpeg, .jpg, .png, .svg, .tiff, .tif, .webp
The type of document files we accept: .pdf
Here is an example:
So now our window should look like this,
For the Step 2. Preview please make sure you select None for the Apply OCR Method under the preview settings. Afterward, we can move on to the next steps, under Step 3. Labeler’s tasks, you can add a label set by ticking the Bounding Box labeling checkbox. Here’s what the settings will do for your project:
You can utilize .csv, .tsv, or .json formats for the bounding box label set.
For .csv/.tsv, we support color names (e.g., red), hex values (e.g., #00FF00), and RGB (e.g., rgb(0,0,255)). You can also utilize the label set with just names, as shown in the Datasaur sample - Bbox only name.csv below. Other values such as captionAllowed and captionRequired will use default settings.
For .json, we support hex and RGB only.
The Text Transcription setting allows the labeler to add corresponding text to a Bounding box. Disabling this setting means the labeler could not add the text.
By turning on the Text Transcription setting, the labeler can add text to a bounding box. You can choose whether a specific label must have a text by disabling or enabling the Require caption checkbox.
For the rest of the Project Creation Process, you will then be able to take all the steps one takes in other project types: Preview, Labeler's Tasks, Assignment, and Project Settings.
In Bounding Box Labeling Project, you also have the option to upload pre-labeled files. To do this, you can simply upload a JSON file along with the document that you want to label. Ensure that the JSON file follows the Datasaur Schema format. For further details about the Datasaur Schema, refer to this link.
If you've created a project but forgot to add a label set, no worries. You can either drag and drop or click the three-dot button to include a new label set.
If you've initially set up a label set while creating a bounding box project but need to make edits, simply click the three-dot button and select 'Edit label set'. This will open a pop-up section where you can make your edits.
If you need to replace the current label set, you can do so by clicking the three dots and choosing 'Replace existing label set'. This will provide a drag-and-drop area where you can upload your desired label set.
By default, both Reviewers and Labelers have access to these capabilities. However, if you prefer to restrict this capability for your Labelers, you can adjust it in File > Settings > Administrator and toggle off the option.
Each label class in the label set can have multiple custom attributes (questions). The question types are limited to Text and Dropdown only. Each question can also have a default answer.
To modify the custom attributes within the label set, follow these steps:
Open the label set editor via Add new label set, Edit label set, or Replace existing label set. Please check the Label Set Modification section above.
Switch to the JSON tab.
Add or modify the questions field as needed for each label class.
💡 We do not support the modification of the type and internalId value of existing questions, as it will affect labels that are already applied in unexpected ways.
To add questions, include a questions field in the desired label class within the JSON structure. The questions field can contains multiple data with the following structures:
id: ID of the question.
label: The question name.
type: The type of question. It can be TEXT or DROPDOWN.
config: Configuration settings for the question
required: Determines whether the question must be answered. It can be true or false.
defaultValue: The default answer for the question
For TEXT questions, it can be any text.
For DROPDOWN questions, it must be an options.id value.
multiline: Specifies if the TEXT question allows multiple lines. It can be true or false.
options: An array of options for the DROPDOWN question, each containing:
id: ID of the dropdown option. It can be used in the defaultValue.
label: The text label for the dropdown option.
internalId: readonly, unique id Datasaur uses to keep track of each individual question
For a detailed example, refer to the Label Sets section for a sample of the BBox label set with custom attributes.
The first thing you will see in the Bounding Box labeling project is the Document Viewer. This viewer displays the documents or images uploaded during project creation and allows you to create bounding boxes for labeling purposes.
There are two cursor types available for creating bounding boxes: Bounding box and Text selection.
The main method for creating a bounding box is by drawing it directly on the document. To do this, click and drag your cursor to define the desired area. This method allows for precise placement of bounding boxes over images or documents.
The Text Selection mode is another method to create bounding boxes by selecting text directly within documents. This approach eliminates the need to manually draw rectangles, making the labeling process quicker — especially when working with text-based documents.
This cursor type will only be shown in the Document Viewer toolbar for PDF documents.
This selection works only with native PDFs that contain selectable text. Some PDFs have text layers, while others do not.
Most PDFs generated from word processors (such as Microsoft Word) typically have text layers, while PDFs created from scanning documents or converting images do not.
To use this text selection mode, follow these steps:
Enable Text selection mode: Select Text selection icon to enable the Text selection mode.
Select text: Click and drag across text or double-click individual words to select them.
Automatic Box Creation: When you release the left mouse button, a bounding box will be automatically created around the selected text. A label box dialog will then appear, allowing you to assign the appropriate label class and the selected text will automatically populate the caption input field.
Every time we finished drawing Bounding Box(es) on a certain area, a modal will be shown. This modal will have 2 fields by default consisting of Label and Caption, plus any attributes you have previously configured. You can select the label from the list, fill the caption and answer the attributes as needed. To submit the label, you can simply click anywhere inside the document viewer.
You can try to quickly review the value by hovering on the created Bounding Box, and it will show you a tooltip containing the values. Here’s what it will be looks like for a complete Bounding Box and the tooltip.
In addition to manual typing, we are now supporting an option to automatically fill captions using Tesseract! When you draw a bounding box around the desired text in the document, Tesseract will return it as the caption.
Please keep in mind that the processing time may be longer for longer text selections within the bounding box.
Enabling OCR for caption can be done in the Step 3. To do so, click on the Auto-caption provider dropdown and select Tesseract. After that, you can continue setting up your project configuration and launch it.\
Auto-caption provider dropdown is available when “Allow text caption” is enabled.
After your project has been successfully created, you are able to modify the auto-caption setting by clicking the triple-dots in the Bounding Box Labels extension. Disabling or enabling the setting will affect all documents within the project.
Auto-caption setting will not appear if no auto-caption provider has been selected in the Step 3.
There are a couple of things that you need to know about auto-caption.
Auto-captioning process only happens on drawing bounding box process, not resizing or repositioning the created bounding box
Auto-captioning process will be rerun when you resize or reposition the bounding box in a drawing state
Auto-captioning process can be canceled by clicking the (x) inside the Caption text field
Auto-captioning process will be canceled if you click outside the label box, and the bounding box will be created
It will return an empty value if caption for label is optional
It will throw an error if caption for label is required
Auto-caption can be used for multiple bounding boxes
It will read all the bounding boxes
It will be separated by a line break between the bounding box content
Initially, a bounding box is drawn as a rectangle. But to enhance accuracy for images and documents that are not perfectly straight, users can edit individual points of the bounding box by holding the Alt key (or the Option key on Mac), and then clicking and dragging a corner of the bounding box.
Previously, users had to click on a bounding box before moving or resizing it. The new 'Show resize handles on hover' setting allows users to edit bounding boxes simply by hovering, providing a setting that can be customized to suit your preferences.
Only Admins can configure this setting.
During project creation, go to Step 5 and enable the "Show resize handles on hover" setting.
To enable it for an existing project, open the project settings from the Projects page by clicking the three-dot icon, then select "View project settings." Navigate to the Administrator tab and enable the "Show resize handles on hover" setting.
Alternatively, within the project, click the File menu, select "Settings," navigate to the Administrator tab, and enable the "Show resize handles on hover" setting.
Copying and pasting a bounding box can only be done within the same document.
Copying and pasting a bounding box can be done in two ways:
The context menu: Right-click the bounding box, then select the option to copy or paste from the menu.
Using the keyboard shortcut: Select the bounding box, then press Ctrl-C and Ctrl-V (or Command-C and Command-V on Mac).
Deleting a bounding box can be done in two ways:
The context menu: Right-click the bounding box, then select the option to delete from the menu.
Using the keyboard shortcut: Select the bounding box, then press Del.
The multiple selection capability will be helpful when bulk copying, pasting, or deleting is needed.
Selecting multiple bounding boxes can be done by holding the Ctrl key while selecting each bounding box.
Multiple bounding boxes can be merged into a single label containing multiple bounding boxes, which will be connected to each other. The merging process depends on the label classes of the selected bounding boxes:
If all selected bounding boxes share the same label class, they are automatically merged under that label class.
If the selected bounding boxes have different label classes, the label box will open, allowing the user to choose the label class to be applied. They will also be able to update the caption before saving the merged label.
The values of the merged label include the label class and caption. The caption for the merged label is determined by the selection order of the bounding boxes. For example, if the first selected bounding box has the caption "Merchant" and the second selected bounding box has the caption "name," the resulting caption will be "Merchant name”.
Merging bounding boxes can be accomplished by following these steps:
Select multiple bounding box (refer to the previous section for instructions).
Right-click on one of the selected labels to open the context menu.
In the context menu, select "Merge" to initiate the merging process.
When working with datasets containing numerous overlapping bounding boxes, the view can become cluttered, making it challenging to focus on specific areas. To address this, you can manage bounding box visibility by hiding or showing them based on their label class or individually, providing a cleaner and more organized view.
You can hide bounding boxes in three ways: by label class, individual bounding box from the context menu, and individual bounding box from the extension.
Hiding by label class
Navigate to the Bounding Box Labels extension.
For each label class, you will see an eye icon located next to the label instance counter.
Click the eye icon to hide all bounding boxes associated with that label class.
The interface will also display the count of bounding boxes currently visible and hidden.
Hiding individual bounding box from the context menu
Right-click on the bounding box you want to hide in the viewer.
Select Hide from the context menu.
Hiding individual bounding box from the extension
Navigate to the Bounding Box Labels extension.
Click on the instance counter of the bounding box label class you wish to hide.
Click the eye icon to hide the bounding box.
Bounding boxes added or updated after hiding will not be automatically hidden, ensuring newly created data is immediately visible for review.
You can show bounding boxes in two ways: by label class or by individual bounding box.
Showing by label class
Navigate to the Bounding Box Labels extension.
Simply click the eye icon. This will restore visibility for all the bounding boxes under that label class.
Showing individual bounding box
Navigate to the Bounding Box Labels extension.
Click on the instance counter of the bounding box label class you wish to show.
Click the eye icon to display the bounding box.
Tips & Tricks: Refreshing the page will reset the filter, displaying all bounding boxes. This will help when you need to quickly show all bounding boxes without manually unhiding each label class.
View all labelers' work.
Note: Bounding box labels applied in similar positions may overlap.
Apply labels.
Update labels applied by labelers.
Unable to accept or reject labels.
Unable to view all labelers' work with information on who applied each label.
When updating a label applied by a labeler, there is no information or indicator showing that the label has been changed
In Bounding Box Labeling Project, you also have the option to upload pre-labeled files.
Bounding Box Labeling supports several types of pre-labeled files. You can simply upload the appropriate file along with the document you want to label. Ensure that your pre-labeled file follows the correct format and that both files have the same name.
The supported formats for the pre-labeled files are the following.
Datasaur Schema (.json)
Further details about the file format can be found here.
YOLO (.txt)
Further details about the file format can be found here.
LabelMe (.xml)
Further details about the file format can be found here.
Pascal VOC (.xml)
Further details about the file format can be found here.
There are things to keep in mind when using YOLO, LabelMe, and Pascal VOC pre-labeled files in Datasaur:
When creating a pre-labeled project, make sure to also provide your label set.
During project creation, label classes from the file will be automatically mapped to the label classes in your provided label set.
If the file contains more label classes than your label set, the project creation process will create additional label classes based on the number found in the file.
