# External Object Storage

## Overview

External object storage integration improves knowledge base accessibility by allowing users to connect their repositories directly. This enables straightforward data import, supporting a smoother and more efficient RAG process within the platform.

### Available providers

Right now, we support five external object storage services:

1. AWS S3
2. Google Cloud Storage
3. Azure Blob Storage
4. Dropbox
5. Google Drive

### Connect external object storage

1. Go to **Workspace settings.**
2. Navigate to **External object storage** section, then click **Add external object storage**.

<figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-35cb812e37d4f75dae90f7815be947a7a9a65c51%2FWorkspace%20settings%20-%20EOS%20-%20EOS%20added.png?alt=media" alt=""><figcaption><p>Add external object storage</p></figcaption></figure>

2. Choose a service and fill in the credentials.

<figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-624991e072aa8a2a673cbdad4e9d7075c879b372%2FWorkspace%20settings%20-%20EOS%20-%20Choose%20object%20storage%20providers.png?alt=media" alt=""><figcaption><p>List of object storage</p></figcaption></figure>

For more detailed guides on connecting to each external object storage, you can refer to this documentation:

1. [AWS S3](https://docs.datasaur.ai/integrations/external-object-storage/aws-s3)
2. [Google Cloud Storage](https://docs.datasaur.ai/integrations/external-object-storage/google-cloud-storage)
3. [Azure Blob Storage](https://docs.datasaur.ai/integrations/external-object-storage/azure-blob-storage)
4. [Dropbox](https://docs.datasaur.ai/integrations/external-object-storage/dropbox)
5. [Google Drive](https://docs.datasaur.ai/integrations/external-object-storage/google-drive)

### Connecting external object storage to knowledge base

1. Make sure you already connect the external object storage from the workspace settings.
2. Open your knowledge base, click the more menu next to **Upload files** button and select **Connect object storage.**

   <figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-1254bdc10bb8bb51fb89f95492c2cd23a12944f8%2FKnowledge%20base%20-%20empty%20-%20more%20add%20files%20menu.png?alt=media" alt=""><figcaption></figcaption></figure>
3. Select the object storage.

   <figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-db3973e951fc1f86dbd60876d4df7d3cc3d83ac1%2FKnowledge%20base%20-%20EOS%20-%20Select%20EOS.png?alt=media" alt=""><figcaption></figcaption></figure>
4. After you click your desired external object storage, a dialog will show. By default, all supported files will be added to the knowledge base.

   <figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-53fea2a591b582eb19b287dce7c7540d6e2055c6%2FKnowledge%20base%20-%20EOS%20-%20Connect%20object%20storage%20dialog%20-%20no%20rules.png?alt=media" alt=""><figcaption></figcaption></figure>
5. You can filter the files by writing [Rules](https://docs.datasaur.ai/llm-projects/vector-store/external-object-storage#rules-in-external-object-storage), then click **Test rules.** When the document list already matches your preference, click **Connect object storage.**

   <figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-af43ab2d1c78c8709f3bf9cdd3a6b2251ad8ba33%2FKnowledge%20base%20-%20EOS%20-%20Connect%20object%20storage%20dialog%20-%20rules%20applied.png?alt=media" alt=""><figcaption></figcaption></figure>
6. Once the process is finsihed, you can view your files within your knowledge base.

   <figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-b9c6e43a6a9e7f5259f7b7da0be37e70432afb2f%2FKnowledge%20base%20-%20EOS%20-%20files%20added.png?alt=media" alt=""><figcaption></figcaption></figure>

***

### Rules in external object storage

Rules help import specific files from external object storage into the knowledge base. Using [glob patterns](https://en.wikipedia.org/wiki/Glob_\(programming\)), you can include or exclude specific files.

<figure><img src="https://448889121-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-MbjY0HseEqu7LtYAt4d%2Fuploads%2Fgit-blob-fa606c0fe467dea73de54d4b3af2dc2509e1fd81%2FKnowledge%20base%20-%20EOS%20-%20Rules%20accordion%20highlight.png?alt=media" alt=""><figcaption></figcaption></figure>

#### How to write rules

#### Writing rules for file import

When setting up rules for importing files from your external object storage into the knowledge base, understanding how to effectively use Glob patterns is crucial. Here are steps and tips to guide you through the process:

**Step 1: Identify file patterns**

First, determine the common patterns in the names of files you wish to import or exclude. For example, if you want to import all `.pdf` files, your pattern would be `*.pdf`.

**Step 2: Utilize wildcards**

* `*` (asterisk) matches zero or more characters. For instance, `*.pdf` matches all files ending in `.pdf`.
* `?` (question mark) matches exactly one character. For example, `?.pdf` matches `a.pdf` but not `ab.pdf`.

**Step 3: Specifying directories**

If you want to specify files in a particular directory, include the directory name in your pattern. For example, `myfolder/*.pdf` matches all `.pdf` files in the `myfolder` directory.

**Step 4: Excluding files**

To exclude files, you can use the negation pattern `!`. For example, if you want to import all `.pdf` files except those starting with `temp`, your rules would include `*.pdf` and `!temp*.pdf`.

**Step 5: Combining patterns**

You can combine multiple patterns to fine-tune your selection. For example, to import `.pdf` and `.docx` files but exclude those in the `drafts` folder, use `*.pdf`, `*.docs`, and `!drafts/*`.

**Example rules**

1. Import all `.pdf` files: `*.pdf`
2. Import all files except those in the `temp` directory: `*`, `!temp/*`
3. Import all `.docx` and `.txt` files in the `data` directory: `data/*.docx`, `data/*.txt`

Remember, the order of rules matters. Patterns defined later can override those defined earlier, so plan your rules accordingly to ensure the correct files are imported.