AWS S3

Connect your own S3 buckets with Datasaur

File key

A file key is used during project creation to tell Datasaur which file should be used. You can get the file key from the path after the bucket name in the S3 URI, as shown in the example below.

• Bucket name: datasaur-test

• S3 URI: s3://datasaur-test/some-folder/image.png

• File key: /some-folder/image.png

Setup

By integrating your bucket with Datasaur, you can create projects using files directly from your S3.

1. Set up integration in the Settings page

Datasaur uses its own storage by default. Adding an external object storage integration allows you to use your own storage provider when creating projects.

1. Go to Settings > External object storage.

2. Click Add external object storage. A dialog will pop up. Do not close the dialog because we will use the External ID and it will be generated each time you close the dialog.

3. Enter the Bucket name. It is used to identify and differentiate the external object storage integration.

Leave this dialog open for the next steps.

2. Set up CORS for your S3 bucket

This step allows Datasaur to access files in your bucket.

1. In a new tab, log into your AWS account.

2. Open the S3 Management Console.

3. Select your preferred bucket.

   1. It's highly recommended to enable the lifecycle policy for both temp/ and export/ prefix to be automatically removed in 7 days.

4. Go to Permissions.

5. Edit the Cross-origin resource sharing (CORS) section, and paste the following configuration.

[
  { 
    "AllowedHeaders": ["*"], 
    "AllowedMethods": [
      "GET",
      "PUT",
      "POST",
      "HEAD",
      "DELETE"
    ],
    "AllowedOrigins": ["https://app.datasaur.ai"],
    "ExposeHeaders": []
  }
]

• Bucket name: Enter the name of the bucket you configured for CORS.

• Bucket prefix: This will be added at the start of the bucket path so you can organize files according to your needs. For example, entering test will create the path /{bucket-name}/test.

3. Create a policy for Datasaur role in AWS

You need to create an IAM policy that allows Datasaur to access your S3 bucket. If you already have a policy configured for the bucket, you can skip this step.

1. In your AWS IAM management console, go to Policies.

2. Click Create policy.

3. Open the JSON tab.

4. Paste the following configuration. Make sure to replace <your-bucket-name> with your actual bucket name. The write permissions are used to upload selected files to your bucket, while the GetBucketLocation permission is used to configure requests based on your bucket’s region.

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Action": [
           "s3:ListBucket",
           "s3:ListBucketVersions",
           "s3:PutObjectAcl",
           "s3:PutObject",
           "s3:GetObjectAcl",
           "s3:GetObject",
           "s3:DeleteObjectVersion",
           "s3:DeleteObject",
           "s3:GetBucketLocation"
         ],
         "Effect": "Allow",
         "Resource": [
           "arn:aws:s3:::<your-bucket-name>/*",
           "arn:aws:s3:::<your-bucket-name>"
         ]
       }
     ]
   }

5. Click Next: Tags. Adding tags is optional.

6. Click Next: Review.

7. Enter a policy name and optional description.

8. Click Create policy.

4. Create a role for Datasaur

After creating the policy for your S3 bucket, you need to attach it to an IAM role that Datasaur will assume to access your bucket.

1. In the AWS IAM Management Console, go to Roles.

2. Click Create role.

3. Under Trusted entity type, select AWS account.

4. Select Another AWS account and enter the Datasaur AWS Account ID: 682361690817.

5. Enable Require external ID and paste the external ID from the first step.

6. Click Next.

7. In the Add permissions step, select the policy created in the previous step.

8. Click Next.

9. Input a name and an optional description, then click Create role.

10. Return to the Roles page and open your newly created role.

11. Copy the Role ARN from the page.

12. Open the Datasaur tab and paste it into the Role ARN field in the Integrate your Amazon S3 bucket dialog.

5. Check connection

Click Check connection to verify that the configuration is correct. If the connection succeeds, you can continue creating the external object storage integration by clicking Add storage.

6. Setup complete

You can now create projects using files stored directly in your S3 bucket. You can also change the default object storage provider at any time from the Settings page.