Kafka
Jump to Section
The Kafka Connection allows the Intelligence Hub to connect to Apache Kafka brokers for reading (input) and writing (output) of messages. Below are details on the connection, input, and output settings and supported features.
Connection Settings
Broker URIs
Specify one or more Kafka broker URIs to connect to. Example: `broker1:9092,broker2:9092’. The connection uses the first broker in the list that successfully connects.
Security Type
Security type used for Kafka client connection.
- Plaintext: No security is used.
- SSL: Enables SSL for the client connection.
- Truststore CA Certificate: The CA certificate used for the client connection.
- Keystore Client Keypair: The client keypair used to encrypt/decrypt the client communication.
Input Settings
Kafka inputs subscribe to topics and consume messages. The following settings are available:
| Setting | Description |
|---|---|
| Topic | The topic to subscribe to. |
| Group | The consumer group ID to join. |
| Initial Offset | Where to start consuming if no offset exists (e.g., earliest, latest). |
| Key Format | The expected format for the message key (Any, JSON, String, Binary). See options below. |
| Value Format | The expected format for the message value (Any, JSON, String, Binary). See options below. |
| Max Messages | Maximum number of records per read. |
| Include Metadata | Include metadata such as topic, partition, offset, timestamp, headers, key. |
Key/Value Format Options
| Type | Description |
|---|---|
| Any | The payload can be JSON, String, or Binary |
| JSON | The payload is deserialized as JSON. Non-JSON payloads are ignored. |
| String (UTF-8) | The payload is treated as a string. |
| Binary | The payload is treated as a byte array. |
Output Settings
Kafka outputs publish messages to topics. The following settings are available:
| Setting | Description |
|---|---|
| Topic | The topic to publish messages to. |
| Key | The key to use for the message (optional). |
| Value | The value to publish (payload). |