Kafka Destination

Overview

In addition to our unified API, Terminal can stream normalized data directly to a Kafka cluster you operate. This enables:

Low-latency ingestion into your event-driven systems and stream processors
Routing the same data to your existing Kafka consumers, sinks, and downstream warehouses

How it Works

Terminal ingests data from telematics providers on an ongoing basis and normalizes it into our Common Models. As records are ingested or updated, Terminal produces them as events to topics on your Kafka cluster. Terminal authenticates to your cluster using credentials you provide (see Authentication) and produces records as soon as they are ingested or updated.

Delivery model

Kafka delivery is a continuous, push-based stream:

Real-time: records are produced to your topics shortly after Terminal ingests or updates them, without waiting for a schedule.
Per-model topics: you map each Common Model you want delivered to a topic name on your cluster. Terminal produces records for that model only to the configured topic.
At-least-once: records can occasionally be redelivered (for example, after retries or recovery). Use the keys described in Message format to deduplicate downstream if your use case requires exactly-once semantics.
Backfills: when you onboard a new connection or request historical data, Terminal will produce existing records to the same topics so your consumers see a consistent stream.

Message format

Each Kafka record contains:

Value: an Avro encoded MessageWrapper event. The schema mirrors Terminal’s webhook event structure so the same downstream code can consume both.
Key: a UTF-8 encoded string derived from the configured partition keys for the topic.
Topic: the topic name you configured for that model.

The MessageWrapper record has the following shape:

{
  "eventType": "insert",
  "model": "VehicleLocation",
  "detail": {
    "VehicleLocation": {
      "id": "vcl_loc_01ARZ3NDEKTSV4RRFFQ69G5FAV",
      "connectionId": "conn_01ARZ3NDEKTSV4RRFFQ69G5FBV",
      "vehicleId": "vcl_01ARZ3NDEKTSV4RRFFQ69G5FCV",
      "...": "..."
    }
  }
}

Top-level fields:

eventType: insert for a newly ingested record, modify for an update to an existing record.
model: the Common Model name (for example, Vehicle, VehicleLocation, Trip).
detail: a record where exactly one field is populated, matching model. Each field follows the canonical Common Model schema.

In addition to the fields documented in the Common Models, every record includes a connectionId field that identifies the source connection.

View supported models

Model	Reference
Vehicle	View model
Driver	View model
Device	View model
Group	View model
Trailer	View model
VehicleLocation	View model
VehicleStatLog	View model
SafetyEvent	View model
FaultCodeEvent	View model
Trip	View model
HOSDailyLog	View model
HOSLog	View model
IFTAVehicleMonth	View model
VehicleUtilizationDay	View model

Getting the Avro schemas. Terminal will share the .avsc schema files for the MessageWrapper and every Common Model you subscribe to. Reach out to your Terminal contact and we will send the latest schema bundle, which you can register in a schema registry, use for code generation, or load directly with Avro deserializers in any language. A self-serve download and schema registry compatibility are on the roadmap.

Partition keys

Each topic has a configurable list of partition key components. Terminal builds the record key by joining the selected components with a colon (:), and Kafka uses that key to assign records to partitions consistently. Available components:

CONNECTION: the connectionId of the source connection.
PRIMARY_KEY: the model’s own primary identifier (for example, vehicleId for Vehicle, tripId for Trip).
FOREIGN_KEY: the related entity’s identifier (for example, vehicleId for a VehicleLocation or VehicleStatLog). Useful for keeping all activity for a single vehicle on the same partition.

If multiple components are configured, the resulting key looks like conn_01ARZ.../veh_01ARZ.... If a record has no value for any of the configured components, the key falls back to a sentinel value so the record is still produced. Choose components based on your downstream ordering and parallelism needs. A common choice for time-series models is [CONNECTION, FOREIGN_KEY] so that all events for a given vehicle on a given connection land on the same partition in order.

Ordering guarantees

Terminal does not provide global ordering guarantees on Kafka delivery. There are two reasons:

Provider ingestion is not strictly ordered. Telematics providers expose data through APIs and webhooks that can return events out of order, especially during catch-up after a sync delay.
The replication pipeline is change-data-capture based and parallel, so the order in which records reach the producer is not the order in which they were originally generated by the device.

To improve practical ordering, Terminal applies a short windowed sort before producing to your topics:

Records are grouped by the configured partition key and sorted within a tumbling time window (currently 30 seconds, capped at 50,000 records per window).
Within that window, records for the same partition key are produced in event-time order. Across windows, or across different partition keys, no ordering is implied.

The windowed sort is configurable as an ordering block on the destination. Today it is a single on/off toggle (enabled) and defaults to false, the 30-second window and 50,000-record cap are fixed. Enable it if you want Terminal to apply the per-partition sort before produce, or leave it off to have records flow through with the lowest possible latency and handle ordering entirely on your consumer side. This is best-effort smoothing, not a contract. Consumers that require strict ordering, exactly-once semantics, or de-duplication should still:

Pick a partition key that aligns with the entity you order around (for example, [CONNECTION, FOREIGN_KEY] for time-series models keeps all activity for a vehicle on one partition).
Apply their own windowing or buffering by event timestamp on the consumer side.

Topic options

You manage topics on your own cluster. For each Common Model you want delivered, you provide:

The topic name Terminal should produce to.
The partition key components for that topic.
Whether to include raw provider data alongside the normalized payload.

Topic-level controls (partition count, replication factor, retention, compaction, ACLs) remain on your side. Make sure each topic exists before enabling the destination, and that the credentials you give Terminal have produce permissions on it.

Reference implementation

These are the conventions we recommend when standing up a new Kafka destination. They are not enforced and you can deviate where it makes sense for your stack, but starting from these defaults makes most consumer patterns straightforward. Topics

One topic per Common Model, named with a stable prefix you control (for example, terminal.vehicle, terminal.vehicle-location, terminal.trip).
Replication factor 3 for production clusters.
Partition count sized for your largest model. For most fleets, 12 to 24 partitions on VehicleLocation and VehicleStatLog is a good starting point. Entity topics (Vehicle, Driver, Trip) can be smaller (3 to 6).
Retention sized for your replay window. 7 days of retention on time-series topics covers most operational replays. Entity topics work well as standard topics with longer retention rather than compacted, since Terminal emits both insert and modify events and consumers usually want the history.

Partition keys

Time-series models (VehicleLocation, VehicleStatLog, SafetyEvent, FaultCodeEvent, Trip, HOSLog): [CONNECTION, FOREIGN_KEY]. This pins all activity for a given vehicle (or driver) on a given connection to one partition, which preserves the ordering window’s benefit and gives you natural per-vehicle parallelism on the consumer side.
Entity models (Vehicle, Driver, Device, Group, Trailer): [CONNECTION, PRIMARY_KEY]. Updates to the same entity always land on the same partition.
Aggregate models (HOSDailyLog, IFTAVehicleMonth, VehicleUtilizationDay): [CONNECTION, FOREIGN_KEY] if you join them by vehicle, otherwise [CONNECTION, PRIMARY_KEY].

Sandbox environment

Kafka delivery is not currently available in the sandbox environment. To validate your consumer end-to-end, work with our team to set up a production destination against a non-production Kafka cluster you control.

Setup and configuration

Setup checklist

The fastest path to a working Kafka destination, in order:

Create the topics on your cluster. Use the conventions in Reference implementation if you don’t already have a standard.
Provision a SASL user dedicated to Terminal (for example, terminal-producer) and capture the username and password.
Grant ACLs to that user: Write and Describe on each topic. See Required ACLs.
Decide on network connectivity: public internet, AWS PrivateLink, or AWS MSK VPC. Gather the corresponding identifiers (egress IP allow-list, VPC Endpoint Service name, or MSK cluster ARN). See Network connectivity.
Send the destination config (template in Share your destination details) to your Terminal contact.

Prepare your Kafka cluster

Identify the bootstrap servers Terminal will connect to (for example, b-1.your-cluster.kafka.us-east-1.amazonaws.com:9096,b-2.your-cluster.kafka.us-east-1.amazonaws.com:9096).
Decide which Common Models you want delivered, and create one topic per model on your cluster with the partition count, replication factor, and retention that fit your use case.
Choose how Terminal will reach your cluster: public internet, AWS PrivateLink, or AWS MSK VPC peering. See Network connectivity for details.

Authentication

Terminal authenticates with your cluster over SASL.

Security protocol: SASL_SSL (default). All traffic is encrypted in transit using TLS to your brokers’ certificates.
SASL mechanism: SCRAM-SHA-512 (default). SCRAM-SHA-256 is also supported on request.
Credentials: a username and password that Terminal uses to authenticate. Provision a dedicated user for Terminal so you can rotate or revoke access independently of other consumers.

The credentials you provide are stored encrypted at rest in Terminal and are only used to authenticate the replication producer to your cluster.

Required ACLs

Grant the Terminal user the following ACLs on your cluster:

Write on each topic Terminal will produce to.
Describe on each topic Terminal will produce to (used to verify topic existence and metadata).
IdempotentWrite on the cluster (used so the producer can dedupe in-flight retries).

The exact command depends on your broker tooling. On an Apache Kafka or MSK cluster using kafka-acls.sh, this looks like:

kafka-acls.sh --bootstrap-server <broker> --command-config admin.properties \
  --add --allow-principal "User:terminal" \
  --producer --topic terminal.vehicle

kafka-acls.sh --bootstrap-server <broker> --command-config admin.properties \
  --add --allow-principal "User:terminal" \
  --operation IdempotentWrite --cluster

Repeat the topic ACL command for each topic Terminal will produce to.

Network connectivity

Terminal supports three ways to reach your cluster.

Public Internet
AWS PrivateLink
AWS MSK VPC peering

The simplest setup. Terminal connects to your brokers over the public internet using the bootstrap servers you provide. All traffic is encrypted with SASL_SSL.Use this option if your brokers are publicly reachable and you are comfortable allowing inbound connections from Terminal’s egress IPs. Contact our team for the current list of egress IPs to allow-list at your network boundary.

Expose your cluster as a VPC Endpoint Service and Terminal will connect through a private endpoint, avoiding the public internet.You’ll need to provide:

Service name: the VPC Endpoint Service name (for example, com.amazonaws.vpce.us-east-1.vpce-svc-0123456789abcdef0).
Service region: the AWS region the service runs in. If different from Terminal’s region, the connection will be cross-region.
Allowed principal: add Terminal’s AWS account as an allowed principal on the endpoint service so we can create the endpoint. Our team will share the account ID for your environment.
Availability zones (optional): AZ IDs (for example, use1-az1) to constrain which subnets Terminal uses. Required only for same-region setups when you want to pin specific AZs.

Make sure your bootstrap servers resolve to addresses reachable through the endpoint (for example, MSK brokers’ private DNS).

For Amazon MSK clusters, Terminal can connect directly to your MSK brokers through a managed VPC connection.You’ll need to provide:

Cluster ARN: the ARN of your MSK cluster (for example, arn:aws:kafka:us-east-1:123456789012:cluster/your-cluster/abc123).
Availability zones: the AZ IDs of your MSK cluster’s broker subnets (at least two). Terminal aligns its subnets to match.
Authentication: SASL credentials are required for MSK VPC connectivity (IAM auth is not supported today).

Authorize Terminal’s AWS account on the cluster so the VPC connection can be established. Our team will share the account ID for your environment.

The fastest way to send us your configuration is to fill out the JSON template below and share it with your Terminal contact (over your shared Slack channel or by email). Send the password through a secure channel separately. We’ll confirm everything before provisioning.

{
  "bootstrapServers": "b-1.your-cluster.kafka.us-east-1.amazonaws.com:9096,b-2.your-cluster.kafka.us-east-1.amazonaws.com:9096",
  "auth": {
    "type": "sasl",
    "username": "terminal-producer",
    "password": "<sensitive: send through a secure channel>",
    "mechanism": "SCRAM-SHA-512"
  },
  "network": {
    "type": "public"
  },
  "ordering": {
    "enabled": false
  },
  "topics": {
    "Vehicle": {
      "topic": { "name": "terminal.vehicle" },
      "partitionKeys": ["CONNECTION", "PRIMARY_KEY"],
      "includeRaw": false
    },
    "VehicleLocation": {
      "topic": { "name": "terminal.vehicle-location" },
      "partitionKeys": ["CONNECTION", "FOREIGN_KEY"],
      "includeRaw": false
    },
    "Trip": {
      "topic": { "name": "terminal.trip" },
      "partitionKeys": ["CONNECTION", "PRIMARY_KEY"],
      "includeRaw": false
    }
  }
}

Notes on the template:

Replace network with one of the variants below if you are not using public internet.
Add or remove entries under topics for the Common Models you want delivered. The key (Vehicle, VehicleLocation, etc.) must match the Common Model name exactly.

network for AWS PrivateLink

{
  "type": "private_link",
  "serviceName": "com.amazonaws.vpce.us-east-1.vpce-svc-0123456789abcdef0",
  "serviceRegion": "us-east-1",
  "availabilityZones": ["use1-az1", "use1-az2"]
}

Omit serviceRegion if it matches the region your VPC Endpoint Service runs in. Omit availabilityZones to let Terminal auto-discover them.

network for AWS MSK VPC

{
  "type": "msk_vpc",
  "clusterArn": "arn:aws:kafka:us-east-1:123456789012:cluster/your-cluster/abc123",
  "availabilityZones": ["use1-az1", "use1-az2"]
}

For the full list of configuration options, see Configuration options. Our team is here to help recommend the best settings for your needs.

Self-service destination management through the Terminal dashboard is coming soon.

Configuration options

Option	Description
Bootstrap servers	Comma-separated list of broker host:port pairs Terminal will use to connect (for example, `broker1:9096,broker2:9096`).
Network mode	How Terminal reaches your cluster: `Public`, `PrivateLink`, or `MSK VPC`. See Network connectivity.
PrivateLink service	When network mode is `PrivateLink`: the VPC Endpoint Service name, region, and (optionally) availability zones.
MSK cluster	When network mode is `MSK VPC`: the MSK cluster ARN and the broker availability zones.
Authentication	SASL credentials Terminal uses to authenticate to your brokers.
Username	SASL username for the dedicated Terminal user on your cluster.
Password	SASL password for the user. Stored encrypted at rest and never returned in API responses.
Security protocol	Transport-layer protocol. Defaults to `SASL_SSL`.
SASL mechanism	Defaults to `SCRAM-SHA-512`. `SCRAM-SHA-256` is also supported on request.
Ordering	Optional. Enable or disable Terminal’s per-partition windowed sort before produce. Defaults to `enabled: false`. See Ordering guarantees.
Topics	Mapping of Common Model to topic configuration (see options below). At least one model is required.
Topic name	The topic on your cluster that Terminal produces records for this model to.
Partition keys	Ordered list of components used to build the record key: `CONNECTION`, `PRIMARY_KEY`, `FOREIGN_KEY`. See Partition keys.
Include raw	Optional. Include raw provider data with the normalized payload. Note: this increases data volume by 2-3x.
Disabled	Optional. Pause delivery for a specific topic without removing the configuration.

About Us

Implementation Guides

Use Case Guides

Data Destinations

Terminal Platform

Advanced

Kafka Destination

Overview

How it Works

Delivery model

Message format

Partition keys

Ordering guarantees

Topic options

Reference implementation

Sandbox environment

Setup and configuration

Setup checklist

Prepare your Kafka cluster

Authentication

Required ACLs

Network connectivity

Configuration options

​Overview

​How it Works

​Delivery model

​Message format

​Partition keys

​Ordering guarantees

​Topic options

​Reference implementation

​Sandbox environment

​Setup and configuration

​Setup checklist

​Prepare your Kafka cluster

​Authentication

​Required ACLs

​Network connectivity

​Share your destination details

​Configuration options

Overview

How it Works

Delivery model

Message format

Partition keys

Ordering guarantees

Topic options

Reference implementation

Sandbox environment

Setup and configuration

Setup checklist

Prepare your Kafka cluster

Authentication

Required ACLs

Network connectivity

Share your destination details

Configuration options