Syncing Data
Overview
After a connection is created, Terminal syncs and normalizes data incrementally to ensure it is always up-to-date. Use the /connections/current
or /syncs
endpoints to track when data is synced from the source system to Terminal before querying for data.
Sync Modes
Terminal connections can be configured to sync data automatically or manually:
- Automatic syncing is configured by default and will sync data every 2 hours to Terminal.
- Automatic syncing is configured by setting the connection’s
syncMode
toautomatic
.
- Automatic syncing is configured by setting the connection’s
- Manual syncing is useful to save costs if you only need to sync data periodically.
- Manual syncing is configured by setting the connection’s
syncMode
tomanual
.
- Manual syncing is configured by setting the connection’s
A sync can be requested at any time for a connection by calling the POST /syncs
endpoint. This endpoint will return an id
that can be used to check the status of the sync
Best Practices for Syncing
1. Create functions in your backend that are responsible for syncing data
Terminal provides 2 types of data: dynamic and event.
Syncing Dynamic Data
Dynamic data is data that may change. Examples include vehicles, drivers and HOS logs.
Use the modifiedAfter
timestamp filter to only ingest data that has changed since the start time of when you last synced. This filter is important to ensure you only pull data that has been modified (or created) since the last time you synced.
Continue to pull data until the response does not include a next
cursor.
Example
If you last started syncing at 2022-01-01T00:00:00.000Z
you should make the following request
Note: Ensure you store the start time of the last sync as data can change while you are syncing. This will ensure you never miss a change.
Syncing Event Data
Event data refers to records with a timestamp that are typically immutable, such as vehicle locations and stats.
For most event data endpoints, a modifiedAfter
filter is not available.
Instead, these endpoints provide startAt
and endAt
timestamp filters that allow you to request data for a specific time window. Records are returned in ascending order by timestamp.
Recommended Sync Pattern:
- Set your sync window using the
startAt
andendAt
parameters to define the desired time range. - Store the cursor or the timestamp of the last successfully synced record to resume syncing from where you left off.
- To account for delayed inserts or retroactive changes, rather than syncing from the current checkpoint, you may choose to add a buffer of 1-2 days to your previous checkpoint. This gives the data time to “settle,” reducing the risk of missing late-arriving records.
- For example: If your sync window is one week long, and you performed a sync from June 4th to June 10th, your next sync should be from June 8/9th to June 16th. This ensures you have an ‘overlap’ of a few days, to account for any records on June 8th or 9th that may have arrived late or not yet settled.
- Apply de-duplication logic during ingestion to handle any overlapping data that may result from the lookback window.
Note: Ensure you store the next cursor of the last request in order resume syncing from where you left off.
Notes:
- In practice, some event data sources may occasionally update or insert records retroactively (for example, when vehicles are offline or when providers reclassify events).
- We recommend using the
modifiedAfter
filter instead ofstartAt
andendAt
for endpoints where supported (such as safety events), as this will capture both new and updated records.
- We recommend using the
2. Orchestrate the sync
Setup a scheduled job on your backend to sync data periodically for each connection.
Some of our endpoints (ex: historical vehicle locations) require that you query them by vehicle. We recommend syncing vehicles first and then scheduling jobs to pull data for each vehicle.
Currently our system schedules syncs every 2 hours so please note that data will only be updated upon completion of syncing in our system.
For maintaing the most up to date information, we recommend initiating a sync after receiving a sync.complete
webhook (learn more). Alternativley, you can use the /syncs
or /connections/current
endpoints to know when data has been synced from the source system to Terminal.
If you need data more frequently - please reach out to our team and we’d be happy to help support your use case.