LogoLogo
API ReferenceGitHubSlackService StatusLogin
v3.9.16
v3.9.16
  • 🏠Deep Lake Docs
  • List of ML Datasets
  • 🏗️SETUP
    • Installation
    • User Authentication
      • Workload Identities (Azure Only)
    • Storage and Credentials
      • Storage Options
      • Setting up Deep Lake in Your Cloud
        • Microsoft Azure
          • Configure Azure SSO on Activeloop
          • Provisioning Federated Credentials
          • Enabling CORS
        • Google Cloud
          • Provisioning Federated Credentials
          • Enabling CORS
        • Amazon Web Services
          • Provisioning Role-Based Access
          • Enabling CORS
  • 📚Examples
    • Deep Learning
      • Deep Learning Quickstart
      • Deep Learning Guide
        • Step 1: Hello World
        • Step 2: Creating Deep Lake Datasets
        • Step 3: Understanding Compression
        • Step 4: Accessing and Updating Data
        • Step 5: Visualizing Datasets
        • Step 6: Using Activeloop Storage
        • Step 7: Connecting Deep Lake Datasets to ML Frameworks
        • Step 8: Parallel Computing
        • Step 9: Dataset Version Control
        • Step 10: Dataset Filtering
      • Deep Learning Tutorials
        • Creating Datasets
          • Creating Complex Datasets
          • Creating Object Detection Datasets
          • Creating Time-Series Datasets
          • Creating Datasets with Sequences
          • Creating Video Datasets
        • Training Models
          • Splitting Datasets for Training
          • Training an Image Classification Model in PyTorch
          • Training Models Using MMDetection
          • Training Models Using PyTorch Lightning
          • Training on AWS SageMaker
          • Training an Object Detection and Segmentation Model in PyTorch
        • Updating Datasets
        • Data Processing Using Parallel Computing
      • Deep Learning Playbooks
        • Querying, Training and Editing Datasets with Data Lineage
        • Evaluating Model Performance
        • Training Reproducibility Using Deep Lake and Weights & Biases
        • Working with Videos
      • Deep Lake Dataloaders
      • API Summary
    • RAG
      • RAG Quickstart
      • RAG Tutorials
        • Vector Store Basics
        • Vector Search Options
          • LangChain API
          • Deep Lake Vector Store API
          • Managed Database REST API
        • Customizing Your Vector Store
        • Image Similarity Search
        • Improving Search Accuracy using Deep Memory
      • LangChain Integration
      • LlamaIndex Integration
      • Managed Tensor Database
        • REST API
        • Migrating Datasets to the Tensor Database
      • Deep Memory
        • How it Works
    • Tensor Query Language (TQL)
      • TQL Syntax
      • Index for ANN Search
        • Caching and Optimization
      • Sampling Datasets
  • 🔬Technical Details
    • Best Practices
      • Creating Datasets at Scale
      • Training Models at Scale
      • Storage Synchronization and "with" Context
      • Restoring Corrupted Datasets
      • Concurrent Writes
        • Concurrency Using Zookeeper Locks
    • Deep Lake Data Format
      • Tensor Relationships
      • Version Control and Querying
    • Dataset Visualization
      • Visualizer Integration
    • Shuffling in Dataloaders
    • How to Contribute
Powered by GitBook
On this page
  • Setting up Role-Based Access for AWS S3
  • Step 1: Create the AWS IAM Policy
  • Step 2: Create the AWS IAM Role
  • Step 3: Grant Access to AWS KMS Key (only for buckets that are encrypted with customer managed KMS keys)
  • Step 4: Enter the created AWS Role ARN (Step 2) into the Activeloop App

Was this helpful?

Edit on GitHub
  1. SETUP
  2. Storage and Credentials
  3. Setting up Deep Lake in Your Cloud
  4. Amazon Web Services

Provisioning Role-Based Access

How to provision Role-Based Access in S3

PreviousAmazon Web ServicesNextEnabling CORS

Last updated 9 months ago

Was this helpful?

Setting up Role-Based Access for AWS S3

The most secure method for connecting data from your AWS account to Deep Lake is using Federated Credentials and Role-Based Access, which are set up using the steps below:

Step 1: Create the AWS IAM Policy

1. Login to the AWS account where the IAM Role will be created and where the data is stored.

2. Go to the IAM page in the AWS UI, which can be done by searching "IAM" in the console and locating the IAM page under Services.

3. In the left nav, open the Policies under Access management and on Create policy on the right.

5. Select the JSON tab instead of Visual editor.

6. Replace the code in the editor with the code below. Replace BUCKET_NAME with the bucket names for which you want to grant role-based access:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [ 
                "s3:ListBucket",
                "s3:GetBucketLocation",
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::BUCKET_NAME",
                "arn:aws:s3:::BUCKET_NAME/*"
            ]
        }
    ]
}

7. On the bottom right, click Next: Tags (create tags if needed) and Next: Preview, enter the policy name and description, and click Create policy

Step 2: Create the AWS IAM Role

1. On the IAM page, in the left nav, open the Roles under Access management, and click Create role on the right.

3. Select Custom trust policy from the list of options.

4. Replace the policy definition with the code below and click Next

{
    "Version": "2012-10-17",
    "Statement": 
    [
        {
            "Sid": "AllowAssumeRoleFromActiveloopSaaS",
            "Effect": "Allow",
            "Principal": {
                 "AWS": "arn:aws:iam::597713067985:role/activeloop_backend"
        },
        "Action": "sts:AssumeRole"
      }
   ]
}

5. From the provided policy list, select the previously created policy from Step 1 and click Next

6. Set the name and description for the role and click Create role at the bottom.

Step 3: Grant Access to AWS KMS Key (only for buckets that are encrypted with customer managed KMS keys)

1. Navigate to the bucket in the AWS S3 UI

2. Open the bucket Properties

3. Scroll down to Default encryption and copy the AWS KMS key ARN

4. In the Policy creation step (Step 1, Sub-step 6), use the JSON below in the policy statement, and replace YOUR_KMS_KEY_ARN with the copied Key ARN for the encrypted bucket.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
		 "s3:GetBucketLocation",
                "s3:*Object*"
            ],
            "Resource": [
                "arn:aws:s3:::BUCKET_NAME",
                "arn:aws:s3:::BUCKET_NAME/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "kms:Encrypt",
                "kms:Decrypt",
                "kms:ReEncrypt*",
                "kms:GenerateDataKey*",
                "kms:DescribeKey"
            ],
            "Resource": [
                "YOUR_KMS_KEY_ARN”
            ]
        }
    ]
}

Step 4: Enter the created AWS Role ARN (Step 2) into the Activeloop App

See the first video in the link below:

🏗️
Setting up Deep Lake in Your Cloud