> ## Documentation Index > Fetch the complete documentation index at: https://docs.withterminal.com/llms.txt > Use this file to discover all available pages before exploring further. # Kafka Destination > Real-time data replication to your Kafka cluster ## Overview In addition to our unified API, Terminal can stream normalized data directly to a Kafka cluster you operate. This enables: * Low-latency ingestion into your event-driven systems and stream processors * Routing the same data to your existing Kafka consumers, sinks, and downstream warehouses ## How it Works Terminal ingests data from telematics providers on an ongoing basis and normalizes it into our [Common Models](/models). As records are ingested or updated, Terminal produces them as events to topics on your Kafka cluster. ```mermaid theme={null} %%{init: {'theme':'base', 'themeVariables': {'primaryColor':'#e5e7eb', 'primaryTextColor':'#111827', 'primaryBorderColor':'#9ca3af', 'lineColor':'#6b7280', 'secondaryColor':'#f3f4f6', 'tertiaryColor':'#f9fafb'}}}%% flowchart TB subgraph Terminal["Terminal AWS"] TI["Ingestion Service
(Normalizes TSP data)"] TWD["Data Warehouse"] KD["Kafka Replication Service"] end subgraph Consumer["Consumer Kafka"] Brokers["Kafka Cluster
(your brokers)"] Stream["Stream Processors
(Flink, Kafka Streams, etc.)"] DW["Data Warehouse
(Snowflake, Redshift, etc.)"] Apps["Application Services"] end TI -->|writes| TWD TI -->|stream| KD TWD -->|cdc| KD KD -->|produces records over SASL| Brokers Brokers -->|optional| Stream Brokers -->|optional| DW Brokers -->|optional| Apps ``` Terminal authenticates to your cluster using credentials you provide (see [Authentication](#authentication)) and produces records as soon as they are ingested or updated. ### Delivery model Kafka delivery is a continuous, push-based stream: * **Real-time**: records are produced to your topics shortly after Terminal ingests or updates them, without waiting for a schedule. * **Per-model topics**: you map each Common Model you want delivered to a topic name on your cluster. Terminal produces records for that model only to the configured topic. * **At-least-once**: records can occasionally be redelivered (for example, after retries or recovery). Use the keys described in [Message format](#message-format) to deduplicate downstream if your use case requires exactly-once semantics. * **Backfills**: when you onboard a new connection or request historical data, Terminal will produce existing records to the same topics so your consumers see a consistent stream. ### Message format Each Kafka record contains: * **Value**: an [Avro](https://avro.apache.org/) encoded `MessageWrapper` event. The schema mirrors Terminal's [webhook event](/terminal-platform/webhooks) structure so the same downstream code can consume both. * **Key**: a UTF-8 encoded string derived from the configured [partition keys](#partition-keys) for the topic. * **Topic**: the topic name you configured for that model. The `MessageWrapper` record has the following shape: ```json theme={null} { "eventType": "insert", "model": "VehicleLocation", "detail": { "VehicleLocation": { "id": "vcl_loc_01ARZ3NDEKTSV4RRFFQ69G5FAV", "connectionId": "conn_01ARZ3NDEKTSV4RRFFQ69G5FBV", "vehicleId": "vcl_01ARZ3NDEKTSV4RRFFQ69G5FCV", "...": "..." } } } ``` Top-level fields: * `eventType`: `insert` for a newly ingested record, `modify` for an update to an existing record. * `model`: the [Common Model](/models) name (for example, `Vehicle`, `VehicleLocation`, `Trip`). * `detail`: a record where exactly one field is populated, matching `model`. Each field follows the canonical Common Model schema. In addition to the fields documented in the Common Models, every record includes a `connectionId` field that identifies the source connection. | Model | Reference | | --------------------- | --------------------------------------------- | | Vehicle | [View model](/models/vehicle) | | Driver | [View model](/models/driver) | | Device | [View model](/models/device) | | Group | [View model](/models/group) | | Trailer | [View model](/models/trailer) | | VehicleLocation | [View model](/models/vehicle-location) | | VehicleStatLog | [View model](/models/vehicle-stat-log) | | SafetyEvent | [View model](/models/safety-event) | | FaultCodeEvent | [View model](/models/fault-code-event) | | Trip | [View model](/models/trip) | | HOSDailyLog | [View model](/models/hosdaily-log) | | HOSLog | [View model](/models/hoslog) | | IFTAVehicleMonth | [View model](/models/iftavehicle-month) | | VehicleUtilizationDay | [View model](/models/vehicle-utilization-day) | **Getting the Avro schemas.** Terminal will share the `.avsc` schema files for the `MessageWrapper` and every Common Model you subscribe to. Reach out to your Terminal contact and we will send the latest schema bundle, which you can register in a schema registry, use for code generation, or load directly with Avro deserializers in any language. A self-serve download and schema registry compatibility are on the roadmap. ### Partition keys Each topic has a configurable list of partition key components. Terminal builds the record key by joining the selected components with a colon (`:`), and Kafka uses that key to assign records to partitions consistently. Available components: * `CONNECTION`: the `connectionId` of the source connection. * `PRIMARY_KEY`: the model's own primary identifier (for example, `vehicleId` for `Vehicle`, `tripId` for `Trip`). * `FOREIGN_KEY`: the related entity's identifier (for example, `vehicleId` for a `VehicleLocation` or `VehicleStatLog`). Useful for keeping all activity for a single vehicle on the same partition. If multiple components are configured, the resulting key looks like `conn_01ARZ.../veh_01ARZ...`. If a record has no value for any of the configured components, the key falls back to a sentinel value so the record is still produced. Choose components based on your downstream ordering and parallelism needs. A common choice for time-series models is `[CONNECTION, FOREIGN_KEY]` so that all events for a given vehicle on a given connection land on the same partition in order. ### Ordering guarantees Terminal does not provide global ordering guarantees on Kafka delivery. There are two reasons: * Provider ingestion is not strictly ordered. Telematics providers expose data through APIs and webhooks that can return events out of order, especially during catch-up after a sync delay. * The replication pipeline is change-data-capture based and parallel, so the order in which records reach the producer is not the order in which they were originally generated by the device. To improve practical ordering, Terminal applies a short windowed sort before producing to your topics: * Records are grouped by the configured [partition key](#partition-keys) and sorted within a tumbling time window (currently 30 seconds, capped at 50,000 records per window). * Within that window, records for the same partition key are produced in event-time order. Across windows, or across different partition keys, no ordering is implied. The windowed sort is configurable as an `ordering` block on the destination. Today it is a single on/off toggle (`enabled`) and defaults to `false`, the 30-second window and 50,000-record cap are fixed. Enable it if you want Terminal to apply the per-partition sort before produce, or leave it off to have records flow through with the lowest possible latency and handle ordering entirely on your consumer side. This is best-effort smoothing, not a contract. Consumers that require strict ordering, exactly-once semantics, or de-duplication should still: * Pick a partition key that aligns with the entity you order around (for example, `[CONNECTION, FOREIGN_KEY]` for time-series models keeps all activity for a vehicle on one partition). * Apply their own windowing or buffering by event timestamp on the consumer side. ### Topic options You manage topics on your own cluster. For each Common Model you want delivered, you provide: * The topic name Terminal should produce to. * The partition key components for that topic. * Whether to include [raw provider data](/terminal-platform/raw-data) alongside the normalized payload. Topic-level controls (partition count, replication factor, retention, compaction, ACLs) remain on your side. Make sure each topic exists before enabling the destination, and that the credentials you give Terminal have produce permissions on it. ### Reference implementation These are the conventions we recommend when standing up a new Kafka destination. They are not enforced and you can deviate where it makes sense for your stack, but starting from these defaults makes most consumer patterns straightforward. **Topics** * One topic per Common Model, named with a stable prefix you control (for example, `terminal.vehicle`, `terminal.vehicle-location`, `terminal.trip`). * Replication factor `3` for production clusters. * Partition count sized for your largest model. For most fleets, `12` to `24` partitions on `VehicleLocation` and `VehicleStatLog` is a good starting point. Entity topics (`Vehicle`, `Driver`, `Trip`) can be smaller (`3` to `6`). * Retention sized for your replay window. `7` days of retention on time-series topics covers most operational replays. Entity topics work well as standard topics with longer retention rather than compacted, since Terminal emits both `insert` and `modify` events and consumers usually want the history. **Partition keys** * Time-series models (`VehicleLocation`, `VehicleStatLog`, `SafetyEvent`, `FaultCodeEvent`, `Trip`, `HOSLog`): `[CONNECTION, FOREIGN_KEY]`. This pins all activity for a given vehicle (or driver) on a given connection to one partition, which preserves the ordering window's benefit and gives you natural per-vehicle parallelism on the consumer side. * Entity models (`Vehicle`, `Driver`, `Device`, `Group`, `Trailer`): `[CONNECTION, PRIMARY_KEY]`. Updates to the same entity always land on the same partition. * Aggregate models (`HOSDailyLog`, `IFTAVehicleMonth`, `VehicleUtilizationDay`): `[CONNECTION, FOREIGN_KEY]` if you join them by vehicle, otherwise `[CONNECTION, PRIMARY_KEY]`. ### Sandbox environment Kafka delivery is not currently available in the [sandbox environment](/terminal-platform/sandbox). To validate your consumer end-to-end, work with our team to set up a production destination against a non-production Kafka cluster you control. *** ## Setup and configuration ### Setup checklist The fastest path to a working Kafka destination, in order: 1. **Create the topics** on your cluster. Use the conventions in [Reference implementation](#reference-implementation) if you don't already have a standard. 2. **Provision a SASL user** dedicated to Terminal (for example, `terminal-producer`) and capture the username and password. 3. **Grant ACLs** to that user: `Write` and `Describe` on each topic. See [Required ACLs](#required-acls). 4. **Decide on network connectivity**: public internet, AWS PrivateLink, or AWS MSK VPC. Gather the corresponding identifiers (egress IP allow-list, VPC Endpoint Service name, or MSK cluster ARN). See [Network connectivity](#network-connectivity). 5. **Send the destination config** (template in [Share your destination details](#share-your-destination-details)) to your Terminal contact. ### Prepare your Kafka cluster 1. Identify the bootstrap servers Terminal will connect to (for example, `b-1.your-cluster.kafka.us-east-1.amazonaws.com:9096,b-2.your-cluster.kafka.us-east-1.amazonaws.com:9096`). 2. Decide which Common Models you want delivered, and create one topic per model on your cluster with the partition count, replication factor, and retention that fit your use case. 3. Choose how Terminal will reach your cluster: public internet, [AWS PrivateLink](#aws-privatelink), or [AWS MSK VPC peering](#aws-msk-vpc-peering). See [Network connectivity](#network-connectivity) for details. ### Authentication Terminal authenticates with your cluster over SASL. * **Security protocol**: `SASL_SSL` (default). All traffic is encrypted in transit using TLS to your brokers' certificates. * **SASL mechanism**: `SCRAM-SHA-512` (default). `SCRAM-SHA-256` is also supported on request. * **Credentials**: a username and password that Terminal uses to authenticate. Provision a dedicated user for Terminal so you can rotate or revoke access independently of other consumers. The credentials you provide are stored encrypted at rest in Terminal and are only used to authenticate the replication producer to your cluster. #### Required ACLs Grant the Terminal user the following ACLs on your cluster: * `Write` on each topic Terminal will produce to. * `Describe` on each topic Terminal will produce to (used to verify topic existence and metadata). * `IdempotentWrite` on the cluster (used so the producer can dedupe in-flight retries). The exact command depends on your broker tooling. On an Apache Kafka or MSK cluster using `kafka-acls.sh`, this looks like: ```bash theme={null} kafka-acls.sh --bootstrap-server --command-config admin.properties \ --add --allow-principal "User:terminal" \ --producer --topic terminal.vehicle kafka-acls.sh --bootstrap-server --command-config admin.properties \ --add --allow-principal "User:terminal" \ --operation IdempotentWrite --cluster ``` Repeat the topic ACL command for each topic Terminal will produce to. ### Network connectivity Terminal supports three ways to reach your cluster. The simplest setup. Terminal connects to your brokers over the public internet using the bootstrap servers you provide. All traffic is encrypted with `SASL_SSL`. Use this option if your brokers are publicly reachable and you are comfortable allowing inbound connections from Terminal's egress IPs. Contact our team for the current list of egress IPs to allow-list at your network boundary. Expose your cluster as a [VPC Endpoint Service](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-share-your-services.html) and Terminal will connect through a private endpoint, avoiding the public internet. You'll need to provide: * **Service name**: the VPC Endpoint Service name (for example, `com.amazonaws.vpce.us-east-1.vpce-svc-0123456789abcdef0`). * **Service region**: the AWS region the service runs in. If different from Terminal's region, the connection will be cross-region. * **Allowed principal**: add Terminal's AWS account as an allowed principal on the endpoint service so we can create the endpoint. Our team will share the account ID for your environment. * **Availability zones** (optional): AZ IDs (for example, `use1-az1`) to constrain which subnets Terminal uses. Required only for same-region setups when you want to pin specific AZs. Make sure your bootstrap servers resolve to addresses reachable through the endpoint (for example, MSK brokers' private DNS). For Amazon MSK clusters, Terminal can connect directly to your MSK brokers through a managed VPC connection. You'll need to provide: * **Cluster ARN**: the ARN of your MSK cluster (for example, `arn:aws:kafka:us-east-1:123456789012:cluster/your-cluster/abc123`). * **Availability zones**: the AZ IDs of your MSK cluster's broker subnets (at least two). Terminal aligns its subnets to match. * **Authentication**: SASL credentials are required for MSK VPC connectivity (IAM auth is not supported today). Authorize Terminal's AWS account on the cluster so the VPC connection can be established. Our team will share the account ID for your environment. ### Share your destination details The fastest way to send us your configuration is to fill out the JSON template below and share it with your Terminal contact (over your shared Slack channel or by email). Send the password through a secure channel separately. We'll confirm everything before provisioning. ```json theme={null} { "bootstrapServers": "b-1.your-cluster.kafka.us-east-1.amazonaws.com:9096,b-2.your-cluster.kafka.us-east-1.amazonaws.com:9096", "auth": { "type": "sasl", "username": "terminal-producer", "password": "", "mechanism": "SCRAM-SHA-512" }, "network": { "type": "public" }, "ordering": { "enabled": false }, "topics": { "Vehicle": { "topic": { "name": "terminal.vehicle" }, "partitionKeys": ["CONNECTION", "PRIMARY_KEY"], "includeRaw": false }, "VehicleLocation": { "topic": { "name": "terminal.vehicle-location" }, "partitionKeys": ["CONNECTION", "FOREIGN_KEY"], "includeRaw": false }, "Trip": { "topic": { "name": "terminal.trip" }, "partitionKeys": ["CONNECTION", "PRIMARY_KEY"], "includeRaw": false } } } ``` Notes on the template: * Replace `network` with one of the variants below if you are not using public internet. * Add or remove entries under `topics` for the Common Models you want delivered. The key (`Vehicle`, `VehicleLocation`, etc.) must match the [Common Model](/models) name exactly. ```json theme={null} { "type": "private_link", "serviceName": "com.amazonaws.vpce.us-east-1.vpce-svc-0123456789abcdef0", "serviceRegion": "us-east-1", "availabilityZones": ["use1-az1", "use1-az2"] } ``` Omit `serviceRegion` if it matches the region your VPC Endpoint Service runs in. Omit `availabilityZones` to let Terminal auto-discover them. ```json theme={null} { "type": "msk_vpc", "clusterArn": "arn:aws:kafka:us-east-1:123456789012:cluster/your-cluster/abc123", "availabilityZones": ["use1-az1", "use1-az2"] } ``` For the full list of configuration options, see [Configuration options](#configuration-options). Our team is here to help recommend the best settings for your needs. Self-service destination management through the Terminal dashboard is coming soon. *** ### Configuration options | Option | Description | | ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Bootstrap servers | Comma-separated list of broker host:port pairs Terminal will use to connect (for example, `broker1:9096,broker2:9096`). | | Network mode | How Terminal reaches your cluster: `Public`, `PrivateLink`, or `MSK VPC`. See [Network connectivity](#network-connectivity). | | PrivateLink service | When network mode is `PrivateLink`: the VPC Endpoint Service name, region, and (optionally) availability zones. | | MSK cluster | When network mode is `MSK VPC`: the MSK cluster ARN and the broker availability zones. | | Authentication | SASL credentials Terminal uses to authenticate to your brokers. | | Username | SASL username for the dedicated Terminal user on your cluster. | | Password | SASL password for the user. Stored encrypted at rest and never returned in API responses. | | Security protocol | Transport-layer protocol. Defaults to `SASL_SSL`. | | SASL mechanism | Defaults to `SCRAM-SHA-512`. `SCRAM-SHA-256` is also supported on request. | | Ordering | Optional. Enable or disable Terminal's per-partition windowed sort before produce. Defaults to `enabled: false`. See [Ordering guarantees](#ordering-guarantees). | | Topics | Mapping of Common Model to topic configuration (see options below). At least one model is required. | | Topic name | The topic on your cluster that Terminal produces records for this model to. | | Partition keys | Ordered list of components used to build the record key: `CONNECTION`, `PRIMARY_KEY`, `FOREIGN_KEY`. See [Partition keys](#partition-keys). | | Include raw | Optional. Include [raw provider data](/terminal-platform/raw-data) with the normalized payload. Note: this increases data volume by 2-3x. | | Disabled | Optional. Pause delivery for a specific topic without removing the configuration. |