Overview

In addition to our unified API, Terminal supports automatically replicating normalized data to an Amazon S3 bucket in your AWS account. This enables:

  • Ongoing data replication to power downstream data pipelines
  • Complete data archival to maintain ownership of your data
  • Simplified implementation without the need to implement APIs or orchestrate data ingestion

To learn more about Terminal’s S3 replication capabilities and determine if it fits your use case, reach out to our team and we can help you get set up.


Configuration Steps

Step 1: Create an Amazon S3 Bucket

  1. Log in to the AWS Management Console

  2. Navigate to S3 under Storage services

  3. Create a new bucket:

    • Click “Create bucket”
    • Enter a unique bucket name (e.g., your-company-terminal-data)
    • Choose your region
    • Keep default security settings to block public access
    • Enable default encryption
    • Click “Create bucket”
  4. (Optional) Enable S3 versioning to protect against accidental deletions or overwrites:

    • Go to bucket Properties
    • Find and enable Versioning to protect against accidental changes

Step 2: Configure the Bucket Policy

  1. Go to your bucket’s Permissions tab

  2. Add the bucket policy:

    • Click “Edit” under Bucket Policy
    • Paste the relevant policy provided below depending on your environment
    • Save changes

Step 3: Configure your Destination

After configuring your bucket and bucket policy, you’ll need to set up your S3 destination with Terminal. Contact our team with the following information:

  • Bucket name
  • Environment (production or sandbox)
  • Configuration preferences (see options below)

Self-service destination management through the Terminal dashboard is coming soon.

Review the configuration options below to optimize delivery for your use case. Our team can help recommend the best settings for your needs.

Configuration Options

OptionDescription
Update ModeControls how partition updates are handled. Can choose to either overwrite or append. Both modes include all partition records. Defaults to overwrite.

Overwrite: Replaces existing files
Append: Adds timestamped versions of files
CompressionEnable GZIP compression for objects. Defaults to true.
Include RawInclude raw data in output, similar to the API raw parameter. It’s important to note raw data can significantly increase file sizes. Defaults to false.
Key PrefixOptional prefix for object keys (e.g., /terminal-data/). Defaults to no prefix.