​
Syncing Data from Terminal

This guide explains how to efficiently sync data from Terminal connections using different access patterns and sync modes.
For real-time data queries (like current vehicle locations), see our Real-Time Data Guide. This guide focuses on syncing historical and record data.

​
Core Concepts

​
Sync Modes

Automatic Sync Mode (Default): Terminal automatically keeps your connection’s data up-to-date by running syncs in the background. Most connections use this mode and it’s typically configured during connection setup. Manual Sync Mode: You control when syncs happen by requesting them via the API. Useful when you need precise control over sync timing or costs. 
# Request a manual sync
curl --request POST \
  --url 'https://api.withterminal.com/tsp/v1/syncs' \
  --header 'Authorization: Bearer your_secret_key' \
  --header 'Connection-Token: con_tkn_your_token' \
  --header 'Content-Type: application/json' \
  --data '{"days": 90}'

​
Late-Arriving Data & Lookback Windows

Data doesn’t always become available in Terminal immediately when events occur. There can be hours or even days between when an event happens and when it’s available and ingested in our systems, creating a gap between record time (when the event occurred) and ingestion time (when Terminal processed it). When using record time for incremental access, apply a 24-48 hour lookback window to catch late-arriving data. Some endpoints such as HOS logs may require a longer lookback window due to the nature of when records can be updated.
Ingestion time access eliminates lookback windows by using Terminal’s processing time.

​
Access Patterns

Ingestion Time Access: Query data based on when Terminal processed it from the provider
  • Parameters: modifiedAfter, modifiedBefore
  • Best for: Incremental syncing by finding what data has changed since a point in time
  • Data is ordered by when it was ingested into Terminal (not record time)
  • Recommended to use when available since it eliminates the need for lookback windows
Do not pass record time parameters when using ingestion time access
Record Time Access: Query data based on when events actually occurred
  • Example parameters: startedAfter, startedBefore, startAt, endAt
  • Best for: Historical analysis and backfill scenarios
  • Data is ordered by record time (not ingestion time)
  • Required for: /vehicles/{id}/locations, /vehicles/{id}/stats/historical
Make sure to apply lookback windows when using for incremental replication (see above)

​
Data Types

Most data in Terminal is time-related and follows these patterns. Here are the three main categories: 1. Entities - Vehicles, drivers, groups, trailers, devices
  • Endpoints: /vehicles, /drivers, /groups, /trailers, /devices
  • How to sync: Use modifiedAfter for incremental updates
  • Notes: No lookback windows needed
2. Records - Safety events, trips, logs, and most other time-based data
  • Endpoints: /safety/events, /trips, /hos/logs, /hos/daily-logs, and most other time-based data
  • How to sync: Use modifiedAfter OR record time parameters
  • Notes: Can use lookback windows with record time
3. Historical Telematics - Vehicle sensor data over time
  • Endpoints: /vehicles/{id}/locations, /vehicles/{id}/stats/historical
  • How to sync: Use record time parameters (startAt/endAt)
  • Notes: Requires lookback windows if using for incremental syncing
Real-Time Data: For current state data like latest vehicle locations, see our Real-Time Data Guide. You can also simply poll these endpoints when you need the latest data - this is often the simplest integration approach.

​
Use Cases

​
Continuous Data Replication

Goal: Keep your data warehouse continuously updated
Best for: Real-time dashboards, operational systems, ongoing analytics

​
Step 1: Sync Entities & Records

Start with the simple data types using modifiedAfter: 
# Get vehicles modified since last sync
curl "https://api.withterminal.com/tsp/v1/vehicles?modifiedAfter=2024-01-15T10:00:00.000Z" \
  -H "Authorization: Bearer your_secret_key" \
  -H "Connection-Token: con_tkn_your_token"

# Get safety events since last sync
curl "https://api.withterminal.com/tsp/v1/safety/events?modifiedAfter=2024-01-15T10:00:00.000Z" \
  -H "Authorization: Bearer your_secret_key" \
  -H "Connection-Token: con_tkn_your_token"

​
Step 2: Sync Historical Telematics

Vehicle locations and stats require special handling with record time parameters: 
# Get recent vehicle locations (24-48 hour lookback for late-arriving data)
curl "https://api.withterminal.com/tsp/v1/vehicles/vcl_123/locations?startAt=2024-01-14T10:00:00.000Z" \
  -H "Authorization: Bearer your_secret_key" \
  -H "Connection-Token: con_tkn_your_token"

# Get recent vehicle stats
curl "https://api.withterminal.com/tsp/v1/vehicles/vcl_123/stats/historical?startAt=2024-01-14T10:00:00.000Z" \
  -H "Authorization: Bearer your_secret_key" \
  -H "Connection-Token: con_tkn_your_token"

​
Key Points for Steps 1 & 2

  • Process all pages using the cursor parameter (see Pagination Guide)
  • Store sync start time before beginning, update only after successful completion
  • Maintain separate “last sync time” checkpoints for each endpoint
  • Run endpoint syncs in parallel for optimal performance
  • Use default page sizes by omitting the limit parameter for optimal performance
  • Apply 24-48 hour lookback windows for vehicle locations and stats
  • Expect data to be mutable - upsert records to your data store to handle updates
  • Consider syncing data 1-2 days old to reduce update frequency (especially for locations/stats)

​
Step 3: Webhook Integration (Optional)

Webhooks are optional but helpful for triggering ingestion when new data is available. Key events for sync triggers:
  • sync.completed - Provider sync finished, trigger your incremental sync
  • vehicle.added, vehicle.modified - Sync vehicles endpoint or use event details directly
  • safety.added, safety.modified - Sync safety events endpoint or use event details directly
Since there are no ordering guarantees, you can either use the webhook data directly (if order doesn’t matter for your use case) or trigger a targeted sync to ensure consistency. See our Webhooks Guide for complete event details and setup instructions.

​
Historical Data Backfills

Goal: Load historical data for analysis or reporting
Best for: Data science, compliance reporting, one-time analysis projects

​
Step 1: Request Historical Sync

For new connections: Configure historical days during the connection setup process. For existing connections: Request a sync with historical data: 
# Sync historical data for replication
curl --request POST \
  --url 'https://api.withterminal.com/tsp/v1/syncs' \
  --header 'Authorization: Bearer your_secret_key' \
  --header 'Connection-Token: con_tkn_your_token' \
  --header 'Content-Type: application/json' \
  --data '{"startFrom": "2024-01-01T00:00:00.000Z"}'
Wait for the sync to complete using one of these methods:
  • Webhook: connection.first_sync_completed (for new connections only)
  • Webhook: sync.completed (for any sync completion)
  • Polling: GET /syncs/{id} or GET /connections/current endpoints

​
Step 2: Replicate Historical Data

Once the sync completes, query and replicate historical data to your data store: 
# Safety events for date range
curl "https://api.withterminal.com/tsp/v1/safety/events?startedAfter=2024-01-01T00:00:00.000Z&startedBefore=2024-01-31T23:59:59.999Z" \
  -H "Authorization: Bearer your_secret_key" \
  -H "Connection-Token: con_tkn_your_token"

# Trips for date range
curl "https://api.withterminal.com/tsp/v1/trips?startedAfter=2024-01-01T00:00:00.000Z&startedBefore=2024-01-31T23:59:59.999Z" \
  -H "Authorization: Bearer your_secret_key" \
  -H "Connection-Token: con_tkn_your_token"

# Vehicle locations (use month boundaries for efficiency)
curl "https://api.withterminal.com/tsp/v1/vehicles/vcl_123/locations?startAt=2024-01-01T00:00:00.000Z&endAt=2024-01-31T23:59:59.999Z" \
  -H "Authorization: Bearer your_secret_key" \
  -H "Connection-Token: con_tkn_your_token"
Key Points:
  • Process all pages using the cursor parameter (see Pagination Guide)
  • Split large time ranges into smaller chunks (weeks or months) for parallel processing
  • Use default page sizes by omitting the limit parameter for optimal performance
  • If you plan to use this for ongoing updates, apply lookback windows for late-arriving data
  • Stream data directly to your data store for efficiency
We recommend using an orchestration tool like Airflow, Step Functions, or Dagster for advanced parallel processing of large historical datasets.

​
Complete Data Pipeline (Historical + Continuous)

Goal: Build a complete data pipeline with historical data plus ongoing updates
Best for: Production data warehouses, comprehensive analytics platforms Approach: Start with historical backfill, then switch to continuous replication.

​
Step 1: Initial Historical Sync

For new connections: Configure historical days during the connection setup process. For existing connections: Request a sync with historical data: 
# Request 90 days of historical data
curl --request POST \
  --url 'https://api.withterminal.com/tsp/v1/syncs' \
  --header 'Authorization: Bearer your_secret_key' \
  --header 'Connection-Token: con_tkn_your_token' \
  --header 'Content-Type: application/json' \
  --data '{"days": 90}'

# Or specify exact start date
curl --request POST \
  --url 'https://api.withterminal.com/tsp/v1/syncs' \
  --header 'Authorization: Bearer your_secret_key' \
  --header 'Connection-Token: con_tkn_your_token' \
  --header 'Content-Type: application/json' \
  --data '{"startFrom": "2024-01-01T00:00:00.000Z"}'

​
Step 2: Replicate Historical Data

Use record time access patterns to replicate your historical data to your data store (same approach as Historical Data Backfills).

​
Step 3: Switch to Continuous Replication

Once historical data is processed, implement the ongoing sync pattern from Continuous Data Replication:
  • Use modifiedAfter for most endpoints
  • Use record time with lookback for vehicle locations/stats
  • Listen for webhook triggers to enable real-time updates
Key Points:
  • Replicate historical data first using record time parameters
  • Set your “last sync start time” to mark the transition point between historical and ongoing data
  • Switch to ingestion time patterns for ongoing updates
  • Consider orchestration tools for complex data pipelines

​
Understanding the Parameters

​
Record Time Parameters

These query by when events actually occurred:
  • startAt/endAt: Time range for vehicle locations and stats (ISO 8601)
  • startedAfter/startedBefore: Time range for events with duration (ISO 8601)
  • startDay/endDay: Events from a specific day (YYYY-MM-DD)
  • startMonth/endMonth: Events from a specific month (YYYY-MM)

​
Ingest Time Parameters

These query by when Terminal processed the data:
  • modifiedAfter: Records modified after this time
  • modifiedBefore: Records modified before this time

​
Best Practices

​
Error Handling & Rate Limits

  • Implement exponential backoff for retries (start with 1s, double each attempt)
  • Handle HTTP 429 responses by respecting the Retry-After header
  • Follow our Rate Limits Guide for proper throttling
  • Log failed requests with context for debugging

​
Data Replication Efficiency

  • Process data in batches using pagination cursors and time range chunking
  • Use default page sizes by omitting the limit parameter for optimal performance
  • Stream data directly to your data store to avoid memory accumulation
  • Use bulk insert operations for optimal database write performance
  • Use parallel processing for large historical datasets

​
Data Consistency

  • Store sync timestamps before starting, update only after successful completion
  • Use database transactions when processing batches
  • Track progress and resume from failures using pagination cursors or sub-jobs