LogoLogo
API ReferenceGitHubSlackService StatusLogin
v3.8.16
v3.8.16
  • Deep Lake Docs
  • Vector Store Quickstart
  • Deep Learning Quickstart
  • Storage & Credentials
    • Storage Options
    • User Authentication
    • Storing Deep Lake Data in Your Own Cloud
      • Microsoft Azure
        • Provisioning Federated Credentials
        • Enabling CORS
      • Amazon Web Services
        • Provisioning Role-Based Access
        • Enabling CORS
  • List of ML Datasets
  • 🏢High-Performance Features
    • Introduction
    • Performant Dataloader
    • Tensor Query Language (TQL)
      • TQL Syntax
      • Sampling Datasets
    • Deep Memory
      • How it Works
    • Index for ANN Search
      • Caching and Optimization
    • Managed Tensor Database
      • REST API
      • Migrating Datasets to the Tensor Database
  • 📚EXAMPLE CODE
    • Getting Started
      • Vector Store
        • Step 1: Hello World
        • Step 2: Creating Deep Lake Vector Stores
        • Step 3: Performing Search in Vector Stores
        • Step 4: Customizing Vector Stores
      • Deep Learning
        • Step 1: Hello World
        • Step 2: Creating Deep Lake Datasets
        • Step 3: Understanding Compression
        • Step 4: Accessing and Updating Data
        • Step 5: Visualizing Datasets
        • Step 6: Using Activeloop Storage
        • Step 7: Connecting Deep Lake Datasets to ML Frameworks
        • Step 8: Parallel Computing
        • Step 9: Dataset Version Control
        • Step 10: Dataset Filtering
    • Tutorials (w Colab)
      • Vector Store Tutorials
        • Vector Search Options
          • Deep Lake Vector Store API
          • REST API
          • LangChain API
        • Image Similarity Search
        • Deep Lake Vector Store in LangChain
        • Deep Lake Vector Store in LlamaIndex
        • Improving Search Accuracy using Deep Memory
      • Deep Learning Tutorials
        • Creating Datasets
          • Creating Complex Datasets
          • Creating Object Detection Datasets
          • Creating Time-Series Datasets
          • Creating Datasets with Sequences
          • Creating Video Datasets
        • Training Models
          • Splitting Datasets for Training
          • Training an Image Classification Model in PyTorch
          • Training Models Using MMDetection
          • Training Models Using PyTorch Lightning
          • Training on AWS SageMaker
          • Training an Object Detection and Segmentation Model in PyTorch
        • Updating Datasets
        • Data Processing Using Parallel Computing
      • Concurrent Writes
        • Concurrency Using Zookeeper Locks
    • Playbooks
      • Querying, Training and Editing Datasets with Data Lineage
      • Evaluating Model Performance
      • Training Reproducibility Using Deep Lake and Weights & Biases
      • Working with Videos
    • Low-Level API Summary
  • 🔬Technical Details
    • Best Practices
      • Creating Datasets at Scale
      • Training Models at Scale
      • Storage Synchronization and "with" Context
      • Restoring Corrupted Datasets
      • Concurrent Writes
    • Data Layout
    • Version Control and Querying
    • Dataset Visualization
    • Tensor Relationships
    • Visualizer Integration
    • Shuffling in dataloaders
    • How to Contribute
Powered by GitBook
On this page
  • How to convert a YOLO object detection dataset to Deep Lake format
  • Create the Deep Lake Dataset
  • Inspect the Deep Lake Dataset

Was this helpful?

  1. EXAMPLE CODE
  2. Tutorials (w Colab)
  3. Deep Learning Tutorials
  4. Creating Datasets

Creating Object Detection Datasets

Converting an object detection dataset to Deep Lake format is a great way to get started with datasets of increasing complexity.

PreviousCreating Complex DatasetsNextCreating Time-Series Datasets

Was this helpful?

How to convert a YOLO object detection dataset to Deep Lake format

This tutorial is also available as a

Object detection using bounding boxes is one of the most common annotation types for Computer Vision datasets. This tutorial demonstrates how to convert an object detection dataset from YOLO format to Deep Lake, and a similar method can be used to convert object detection datasets from other formats such as COCO and PASCAL VOC.

Create the Deep Lake Dataset

The first step is to download the small dataset below called animals object detection.

The dataset has the following folder structure:

data_dir
|_images
    |_image_1.jpg
    |_image_2.jpg
    |_image_3.jpg
    |_image_4.jpg
|_boxes
    |_image_1.txt
    |_image_2.txt
    |_image_3.txt
    |_image_4.txt
    |_classes.txt

Now that you have the data, let's create a Deep Lake Dataset in the ./animals_od_deeplakefolder by running:

import deeplake
from PIL import Image, ImageDraw
import numpy as np
import os

ds = deeplake.empty('./animals_od_deeplake') # Create the dataset locally

Next, let's specify the folder paths containing the images and annotations in the dataset. In YOLO format, images and annotations are typically matched using a common filename such as image -> filename.jpeg and annotation -> filename.txt . It's also helpful to create a list of all of the image files and the class names contained in the dataset.

img_folder = './animals_od/images'
lbl_folder = './animals_od/boxes'

# List of all images
fn_imgs = os.listdir(img_folder)

# List of all class names
with open(os.path.join(lbl_folder, 'classes.txt'), 'r') as f:
    class_names = f.read().splitlines()

Since annotations in YOLO are typically stored in text files, it's useful to write a helper function that parses the annotation file and returns numpy arrays with the bounding box coordinates and bounding box classes.

def read_yolo_boxes(fn:str):
    """
    Function reads a label.txt YOLO file and returns a numpy array of yolo_boxes 
    for the box geometry and yolo_labels for the corresponding box labels.
    """
    
    box_f = open(fn)
    lines = box_f.read()
    box_f.close()
    
    # Split each box into a separate lines
    lines_split = lines.splitlines()
    
    yolo_boxes = np.zeros((len(lines_split),4))
    yolo_labels = np.zeros(len(lines_split))
    
    # Go through each line and parse data
    for l, line in enumerate(lines_split):
        line_split = line.split()
        yolo_boxes[l,:]=np.array((float(line_split[1]), float(line_split[2]), float(line_split[3]), float(line_split[4])))
        yolo_labels[l]=int(line_split[0]) 
         
    return yolo_boxes, yolo_labels

Finally, let's create the tensors and iterate through all the images in the dataset in order to upload the data in Deep Lake. Boxes and their labels will be stored in separate tensors, and for a given sample, the first axis of the boxes array corresponds to the first-and-only axis of the labels array (i.e. if there are 3 boxes in an image, the labels array is 3x1 and the boxes array is 3x4).

with ds:
    ds.create_tensor('images', htype='image', sample_compression = 'jpeg')
    ds.create_tensor('labels', htype='class_label', class_names = class_names)
    ds.create_tensor('boxes', htype='bbox')
    
    # Define the format of the bounding boxes
    ds.boxes.info.update(coords = {'type': 'fractional', 'mode': 'LTWH'})

    for fn_img in fn_imgs:

        img_name = os.path.splitext(fn_img)[0]
        fn_box = img_name+'.txt'
        
        # Get the arrays for the bounding boxes and their classes
        yolo_boxes, yolo_labels = read_yolo_boxes(os.path.join(lbl_folder,fn_box))
        
        # Append data to tensors
        ds.append({'images': deeplake.read(os.path.join(img_folder, fn_img)),
                   'labels': yolo_labels.astype(np.uint32),
                   'boxes': yolo_boxes.astype(np.float32)
                   })

In order for Activeloop Platform to correctly visualize the labels, class_names must be a list of strings, where the numerical labels correspond to the index of the label in the list.

Inspect the Deep Lake Dataset

Let's check out the third sample from this dataset, which contains two bounding boxes.

# Draw bounding boxes for the fourth image

ind = 3
img = Image.fromarray(ds.images[ind ].numpy())
draw = ImageDraw.Draw(img)
(w,h) = img.size
boxes = ds.boxes[ind].numpy()

for b in range(boxes.shape[0]):
    (xc,yc) = (int(boxes[b][0]*w), int(boxes[b][1]*h))
    (x1,y1) = (int(xc-boxes[b][2]*w/2), int(yc-boxes[b][3]*h/2))
    (x2,y2) = (int(xc+boxes[b][2]*w/2), int(yc+boxes[b][3]*h/2))
    draw.rectangle([x1,y1,x2,y2], width=2)
    draw.text((x1,y1), ds.labels.info.class_names[ds.labels[ind].numpy()[b]])
# Display the image and its bounding boxes
img

Congrats! You just created a beautiful object detection dataset! 🎉

Note: For optimal object detection model performance, it is often important for datasets to contain images with no annotations (See the 4th sample in the dataset above). Empty samples can be appended using:

ds.boxes.append(None)

or by specifying an empty array whose len(shape) is equal to that of the other samples in the tensor:

ds.boxes.append(np.zeros(0,4)) #len(sample.shape) == 2

📚
Colab Notebook
278KB
animals_od.zip
archive
animals object detection dataset