Choosing how to get data out of Terminal? See Data
Replication for sync vs replication and every
option. Kafka is one of them, alongside the API
and S3.
Overview
In addition to our unified API, Terminal can stream normalized data directly to a Kafka cluster you operate. This enables:- Low-latency ingestion into your event-driven systems and stream processors
- Routing the same data to your existing Kafka consumers, sinks, and downstream warehouses
How it Works
Terminal ingests data from telematics providers on an ongoing basis and normalizes it into our Common Models. As records are ingested or updated, Terminal produces them as events to topics on your Kafka cluster. Terminal authenticates to your cluster using credentials you provide (see Authentication) and produces records as soon as they are ingested or updated.Delivery model
Kafka delivery is a continuous, push-based stream:- Real-time: records are produced to your topics shortly after Terminal ingests or updates them, without waiting for a schedule.
- Per-model topics: you map each Common Model you want delivered to a topic name on your cluster. Terminal produces records for that model only to the configured topic.
- At-least-once: records can occasionally be redelivered (for example, after retries or recovery). Use the keys described in Message format to deduplicate downstream if your use case requires exactly-once semantics.
- Backfills: when you onboard a new connection or request historical data, Terminal will produce existing records to the same topics so your consumers see a consistent stream. Records older than an event age limit you set are skipped.
Message format
Each Kafka record contains:- Value: an Avro encoded
MessageWrapperevent. The schema mirrors Terminal’s webhook event structure so the same downstream code can consume both. - Key: a UTF-8 encoded string derived from the configured partition keys for the topic.
- Topic: the topic name you configured for that model.
- Headers:
terminal-*headers that let you identify the record without decoding the value. See Headers.
MessageWrapper record has the following shape:
eventType:insertfor a newly ingested record,modifyfor an update to an existing record.model: the Common Model name (for example,Vehicle,VehicleLocation,Trip).detail: a record where exactly one field is populated, matchingmodel. Each field follows the canonical Common Model schema.
connectionId field that identifies the source connection.
View supported models
View supported models
Headers
Terminal adds these headers to each record. Use them to route, filter, or count records without decoding the Avro value. Each value is a UTF-8 string.terminal-event-time looks like 2026-10-27T13:30:05Z, with fractional seconds when the source time has them. It comes from a different field on each model:
VehicleLocation:locatedAtVehicleStatLog:timestampTrip,SafetyEvent, andHOSLog:startedAtFaultCodeEvent:observedAtHOSDailyLogandVehicleUtilizationDay:date, at midnight UTC
Vehicle, Driver, Device, Group, Trailer, and IFTAVehicleMonth have no event time, so their records don’t carry this header.
Partition keys
Each topic has a configurable list of partition key components. Terminal builds the record key by joining the selected components with a colon (:), and Kafka uses that key to assign records to partitions consistently.
Available components:
CONNECTION: theconnectionIdof the source connection.PRIMARY_KEY: the model’s own primary identifier (for example,vehicleIdforVehicle,tripIdforTrip).FOREIGN_KEY: the related entity’s identifier (for example,vehicleIdfor aVehicleLocationorVehicleStatLog). Useful for keeping all activity for a single vehicle on the same partition.
conn_01ARZ.../veh_01ARZ.... If a record has no value for any of the configured components, the key falls back to a sentinel value so the record is still produced.
Choose components based on your downstream ordering and parallelism needs. A common choice for time-series models is [CONNECTION, FOREIGN_KEY] so that all events for a given vehicle on a given connection land on the same partition in order.
Ordering guarantees
Terminal does not provide global ordering guarantees on Kafka delivery. There are two reasons:- Provider ingestion is not strictly ordered. Telematics providers expose data through APIs and webhooks that can return events out of order, especially during catch-up after a sync delay.
- The replication pipeline is change-data-capture based and parallel, so the order in which records reach the producer is not the order in which they were originally generated by the device.
- Records are grouped by the configured partition key and sorted within a tumbling time window (currently 30 seconds, capped at 50,000 records per window).
- Within that window, records for the same partition key are produced in event-time order. Across windows, or across different partition keys, no ordering is implied.
ordering block on the destination. Today it is a single on/off toggle (enabled) and defaults to false, the 30-second window and 50,000-record cap are fixed. Enable it if you want Terminal to apply the per-partition sort before produce, or leave it off to have records flow through with the lowest possible latency and handle ordering entirely on your consumer side.
This is best-effort smoothing, not a contract. Consumers that require strict ordering, exactly-once semantics, or de-duplication should still:
- Pick a partition key that aligns with the entity you order around (for example,
[CONNECTION, FOREIGN_KEY]for time-series models keeps all activity for a vehicle on one partition). - Apply their own windowing or buffering by event timestamp on the consumer side. The
terminal-event-timeheader has that timestamp, so there’s no need to decode the value first.
Excluding old events
You can have Terminal skip records whose event time is older than a limit you set for each model. Use this when your consumers only act on recent activity and you don’t want late or historical records in your topics. Each limit is an ISO 8601 duration, such asP30D (30 days) or PT12H (12 hours):
- Terminal compares a record’s event time with the current time when it processes the record. If the record is older than its model’s limit, Terminal doesn’t produce it to your topic, and it isn’t delivered later.
- Event time is the value in the
terminal-event-timeheader, so you can check what arrives against the limit. - The limit applies to every record for that model, including updates and backfills. An update to a record whose event time is past the limit is also skipped.
- Event time is when the activity started. For
Trip,SafetyEvent, andHOSLogthat’sstartedAt, so a trip that lasts longer than the limit is skipped. HOSDailyLogandVehicleUtilizationDayuse their date at midnight UTC, so give them limits longer than a day.- Models without a limit are delivered as usual. Models with no event time (
Vehicle,Driver,Device,Group,Trailer, andIFTAVehicleMonth) are never skipped, even if you set a limit for them. - Changes take effect within a few minutes. Raising or removing a limit only affects records processed after the change.
Because of pipeline lag, a record can reach your topic a few seconds older
than the limit.
Topic options
You manage topics on your own cluster. For each Common Model you want delivered, you provide:- The topic name Terminal should produce to.
- The partition key components for that topic.
- Whether to include raw provider data alongside the normalized payload.
Reference implementation
These are the conventions we recommend when standing up a new Kafka destination. They are not enforced and you can deviate where it makes sense for your stack, but starting from these defaults makes most consumer patterns straightforward. Topics- One topic per Common Model, named with a stable prefix you control (for example,
terminal.vehicle,terminal.vehicle-location,terminal.trip). - Replication factor
3for production clusters. - Partition count sized for your largest model. For most fleets,
12to24partitions onVehicleLocationandVehicleStatLogis a good starting point. Entity topics (Vehicle,Driver,Trip) can be smaller (3to6). - Retention sized for your replay window.
7days of retention on time-series topics covers most operational replays. Entity topics work well as standard topics with longer retention rather than compacted, since Terminal emits bothinsertandmodifyevents and consumers usually want the history.
- Time-series models (
VehicleLocation,VehicleStatLog,SafetyEvent,FaultCodeEvent,Trip,HOSLog):[CONNECTION, FOREIGN_KEY]. This pins all activity for a given vehicle (or driver) on a given connection to one partition, which preserves the ordering window’s benefit and gives you natural per-vehicle parallelism on the consumer side. - Entity models (
Vehicle,Driver,Device,Group,Trailer):[CONNECTION, PRIMARY_KEY]. Updates to the same entity always land on the same partition. - Aggregate models (
HOSDailyLog,IFTAVehicleMonth,VehicleUtilizationDay):[CONNECTION, FOREIGN_KEY]if you join them by vehicle, otherwise[CONNECTION, PRIMARY_KEY].
Sandbox environment
Kafka delivery is not currently available in the sandbox environment. To validate your consumer end-to-end, work with our team to set up a production destination against a non-production Kafka cluster you control.Setup and configuration
Setup checklist
The fastest path to a working Kafka destination, in order:- Create the topics on your cluster. Use the conventions in Reference implementation if you don’t already have a standard.
- Provision a SASL user dedicated to Terminal (for example,
terminal-producer) and capture the username and password. - Grant ACLs to that user:
WriteandDescribeon each topic. See Required ACLs. - Decide on network connectivity: public internet, AWS PrivateLink, or AWS MSK VPC. Gather the corresponding identifiers (egress IP allow-list, VPC Endpoint Service name, or MSK cluster ARN). See Network connectivity.
- Send the destination config (template in Share your destination details) to your Terminal contact.
Prepare your Kafka cluster
- Identify the bootstrap servers Terminal will connect to (for example,
b-1.your-cluster.kafka.us-east-1.amazonaws.com:9096,b-2.your-cluster.kafka.us-east-1.amazonaws.com:9096). - Decide which Common Models you want delivered, and create one topic per model on your cluster with the partition count, replication factor, and retention that fit your use case.
- Choose how Terminal will reach your cluster: public internet, AWS PrivateLink, or AWS MSK VPC peering. See Network connectivity for details.
Authentication
Terminal authenticates with your cluster over SASL.- Security protocol:
SASL_SSL(default). All traffic is encrypted in transit using TLS to your brokers’ certificates. - SASL mechanism:
SCRAM-SHA-512(default).SCRAM-SHA-256is also supported on request. - Credentials: a username and password that Terminal uses to authenticate. Provision a dedicated user for Terminal so you can rotate or revoke access independently of other consumers.
Required ACLs
Grant the Terminal user the following ACLs on your cluster:Writeon each topic Terminal will produce to.Describeon each topic Terminal will produce to (used to verify topic existence and metadata).IdempotentWriteon the cluster (used so the producer can dedupe in-flight retries).
kafka-acls.sh, this looks like:
Network connectivity
Terminal supports three ways to reach your cluster.- Public Internet
- AWS PrivateLink
- AWS MSK VPC peering
The simplest setup. Terminal connects to your brokers over the public internet using the bootstrap servers you provide. All traffic is encrypted with
SASL_SSL.Use this option if your brokers are publicly reachable and you are comfortable allowing inbound connections from Terminal’s egress IPs. Contact our team for the current list of egress IPs to allow-list at your network boundary.Share your destination details
The fastest way to send us your configuration is to fill out the JSON template below and share it with your Terminal contact (over your shared Slack channel or by email). Send the password through a secure channel separately. We’ll confirm everything before provisioning.- Replace
networkwith one of the variants below if you are not using public internet. - Add or remove entries under
topicsfor the Common Models you want delivered. The key (Vehicle,VehicleLocation, etc.) must match the Common Model name exactly. excludeEventsOlderThanis optional. Omit it to deliver every record. The keys follow the same Common Model names. See Excluding old events.
network for AWS PrivateLink
network for AWS PrivateLink
serviceRegion if it matches the region your VPC Endpoint Service runs in. Omit availabilityZones to let Terminal auto-discover them.network for AWS MSK VPC
network for AWS MSK VPC
Self-service destination management through the Terminal dashboard is coming
soon.