External Object Storage
Last updated
Last updated
The External Object Storage Integration feature enhances the accessibility of Knowledge base by allowing users to directly connect their object storage repositories. This enables the easy import of data, facilitating a smooth and efficient RAG process within the platform.
Right now, we support three External Object Storage:
AWS S3
Google Cloud Storage
Azure Blob Storage
Connect your External Object Storage via the Workspace Settings by clicking the Add External Object Storage button.
Choose the Object Storage providers. And fill the form with your credentials.
For more detailed guides on connecting to each External Object Storage, you can refer to this link:
Make sure you already connect the External Object Storage in the Workspace Settings.
Open your Knowledge base, click the arrow button next and choose your desired object storage.
After you click your desired external object storage, a dialog will show. You can select which files you want to import to the knowledge base by using the rules. More information about How to write Rules can be found here.
Click the Connect and update knowledge base button, and your files will be imported into the knowledge base.
Once the import process is done, you can view your file within your knowledge base.
The Rules feature streamlines the process of importing data into Knowledge base from external object storage repositories. By incorporating Rules with Glob patterns, users can define specific criteria for importing files, providing a more tailored and automated approach to data integration.
Users can also filer the rules to include or exclude which files by using the Glob Pattern.
When setting up rules for importing files from your External Object Storage into the Knowledge base, understanding how to effectively use Glob patterns is crucial. Here are steps and tips to guide you through the process:
Step 1: Identify File Patterns
First, determine the common patterns in the names of files you wish to import or exclude. For example, if you want to import all .pdf files, your pattern would be *.pdf.
Step 2: Utilize Wildcards
* (asterisk) matches zero or more characters. For instance, *.pdf matches all files ending in .pdf.
? (question mark) matches exactly one character. For example, ?.pdf matches a.pdf but not ab.pdf.
Step 3: Specifying Directories
If you want to specify files in a particular directory, include the directory name in your pattern. For example, myfolder/*.pdf matches all .pdf files in the myfolder directory.
Step 4: Excluding Files
To exclude files, you can use the negation pattern !. For example, if you want to import all .pdf files except those starting with temp, your rules would include *.pdf and !temp*.pdf.
Step 5: Combining Patterns
You can combine multiple patterns to fine-tune your selection. For example, to import .pdf and .docx files but exclude those in the drafts folder, use *.pdf, *.docs, and !drafts/*.
Example Rules
Import all .pdf files: *.pdf
Import all files except those in the temp directory: *, !temp/*
Import all .docx and .txt files in the data directory: data/*.docx, data/*.txt
Remember, the order of rules matters. Patterns defined later can override those defined earlier, so plan your rules accordingly to ensure the correct files are imported.
