Compute Engine

C++ part of the Deep Lake codebase optimized for faster data fetching and computations

Overview of Deep Lake's Optimized Compute Engine built in C++

Compute Engine maximize performance of compute-heavy Deep Lake's features, such as distributed dataloading or large queries, by running certain operations in C++.

The interface for the Compute Engine is still in Python, and it is installed using pip install "deeplake[enterprise]", which delivers the compiled C++ code from binaries.

In order to use Compute Engine, Deep Lake data must be stored in Deep Lake Storage, or in the user's cloud while being connected to Deep Lake using Managed Credentials.

Features in the Optimized Compute Engine:

Tensor Query Language (TQL)Performant Dataloader