AWS S3 Destination
Learn how to configure automated data replication to Amazon S3
Overview
In addition to our unified API, Terminal supports automatically replicating normalized data to an Amazon S3 bucket in your AWS account. This enables:
- Ongoing data replication to power downstream data pipelines
- Complete data archival to maintain ownership of your data
- Simplified implementation without the need to implement APIs or orchestrate data ingestion
To learn more about Terminal’s S3 replication capabilities and determine if it fits your use case, reach out to our team and we can help you get set up.
Configuration Steps
Step 1: Create an Amazon S3 Bucket
-
Log in to the AWS Management Console
-
Navigate to S3 under Storage services
-
Create a new bucket:
- Click “Create bucket”
- Enter a unique bucket name (e.g.,
your-company-terminal-data
) - Choose your region
- Keep default security settings to block public access
- Enable default encryption
- Click “Create bucket”
-
(Optional) Enable S3 versioning to protect against accidental deletions or overwrites:
- Go to bucket Properties
- Find and enable Versioning to protect against accidental changes
Step 2: Configure the Bucket Policy
-
Go to your bucket’s Permissions tab
-
Add the bucket policy:
- Click “Edit” under Bucket Policy
- Paste the relevant policy provided below depending on your environment
- Save changes
Step 3: Configure your Destination
After configuring your bucket and bucket policy, you’ll need to set up your S3 destination with Terminal. Contact our team with the following information:
- Bucket name
- Environment (production or sandbox)
- Configuration preferences (see options below)
Self-service destination management through the Terminal dashboard is coming soon.
Review the configuration options below to optimize delivery for your use case. Our team can help recommend the best settings for your needs.
Configuration Options
Option | Description |
---|---|
Update Mode | Controls how partition updates are handled. Can choose to either overwrite or append . Both modes include all partition records. Defaults to overwrite .Overwrite: Replaces existing files Append: Adds timestamped versions of files |
Compression | Enable GZIP compression for objects. Defaults to true . |
Include Raw | Include raw data in output, similar to the API raw parameter. It’s important to note raw data can significantly increase file sizes. Defaults to false . |
Key Prefix | Optional prefix for object keys (e.g., /terminal-data/ ). Defaults to no prefix. |