Syncing Data

​

Overview

After a connection is created, Terminal syncs and normalizes data incrementally to ensure it is always up-to-date. Use the /connections/current or /syncs endpoints to track when data is synced from the source system to Terminal before querying for data.

​

Sync Modes

Terminal connections can be configured to sync data automatically or manually:

• Automatic syncing is configured by default and will sync data every 2 hours to Terminal.
  • Automatic syncing is configured by setting the connection’s syncMode to automatic.
• Manual syncing is useful to save costs if you only need to sync data periodically.
  • Manual syncing is configured by setting the connection’s syncMode to manual.

A sync can be requested at any time for a connection by calling the POST /syncs endpoint. This endpoint will return an id that can be used to check the status of the sync

​

Best Practices for Syncing

​

1. Create functions in your backend that are responsible for syncing data

Terminal provides 2 types of data: dynamic and event.

Syncing Dynamic Data

Dynamic data is data that may change. Examples include vehicles, drivers and HOS logs.

Use the modifiedAfter timestamp filter to only ingest data that has changed since the start time of when you last synced. This filter is important to ensure you only pull data that has been modified (or created) since the last time you synced.

Continue to pull data until the response does not include a next cursor.

Example

If you last started syncing at 2022-01-01T00:00:00.000Z you should make the following request

GET /hos/logs?modifiedAfter=2022-01-01T00:00:00.000Z

Syncing Event Data

Event data refers to records with a timestamp that are typically immutable, such as vehicle locations and stats.

For most event data endpoints, a ⁠modifiedAfter⁠ filter is not available.

Instead, these endpoints provide startAt⁠ and ⁠endAt⁠ timestamp filters that allow you to request data for a specific time window. Records are returned in ascending order by timestamp.

​

Recommended Sync Pattern:

• Set your sync window using the startAt and endAt parameters to define the desired time range.
• Store the cursor or the timestamp of the last successfully synced record to resume syncing from where you left off.
• To account for delayed inserts or retroactive changes, rather than syncing from the current checkpoint, you may choose to add a buffer of 1-2 days to your previous checkpoint. This gives the data time to “settle,” reducing the risk of missing late-arriving records.
• For example: If your sync window is one week long, and you performed a sync from June 4th to June 10th, your next sync should be from June 8/9th to June 16th. This ensures you have an ‘overlap’ of a few days, to account for any records on June 8th or 9th that may have arrived late or not yet settled.
• Apply de-duplication logic during ingestion to handle any overlapping data that may result from the lookback window.

​

Notes:

• In practice, some event data sources may occasionally update or insert records retroactively (for example, when vehicles are offline or when providers reclassify events).
  • We recommend using the modifiedAfter filter instead of startAt and endAt for endpoints where supported (such as safety events), as this will capture both new and updated records.

​

2. Orchestrate the sync

Setup a scheduled job on your backend to sync data periodically for each connection.

Some of our endpoints (ex: historical vehicle locations) require that you query them by vehicle. We recommend syncing vehicles first and then scheduling jobs to pull data for each vehicle.

Currently our system schedules syncs every 2 hours so please note that data will only be updated upon completion of syncing in our system.

For maintaing the most up to date information, we recommend initiating a sync after receiving a sync.complete webhook (learn more). Alternativley, you can use the /syncs or /connections/current endpoints to know when data has been synced from the source system to Terminal.