Apache Kafka v2

Provides integration with Apache Kafka to publish records, perform bounded consumer-group reads, listen continuously, and inspect topic metadata.

Revision History

0.0.0.5 - Moves reusable format, timeout and consumer-policy settings to the Connection.

0.0.0.4 - Adds Listen.

0.0.0.3 - Corrected the display of Enums.

0.0.0.2 - Corrected the response shape to use a single Response envelope.

0.0.0.1 - Initial release.

Connection

Create an Apache Kafka Connection to define the broker endpoints, default topic and consumer group, and the security and operation settings shared by its Methods. Use separate Connections when operations need different formats or consumer policies.

Setup Notes

Apache Kafka is a distributed event-streaming platform used to publish, retain, and process records across applications and services. Use this Node when a Workflow needs to exchange JSON, text, or binary data with a Kafka cluster.

• Ensure the selected Cluster can resolve and reach at least one endpoint in Bootstrap Servers.
• Configure broker ACLs for the operations the Workflow performs. Producing requires topic write access, consuming requires topic and consumer-group read access, and metadata methods require permission to inspect the relevant topics.
• The Node does not automatically create topics. Create each topic and configure its partitions and replication before running the Workflow.
• Use Ssl for TLS without SASL authentication. Use SaslSsl when the cluster requires SASL authentication over TLS.
• SaslPlaintext authenticates but does not encrypt network traffic. Use it only on a trusted private network.
• SaslPlaintext and SaslSsl require SASL Username and SASL Password.
• TLS Connections verify the certificate chain and broker hostname. Supply CA Certificate PEM when the broker certificate is not trusted by the platform trust store.
• For mutual TLS, supply both Client Certificate PEM and Client Private Key PEM. Supply Client Key Password only when the private key is encrypted.
• Use the Connection test to verify broker discovery and authentication before adding the Node to a Workflow.

Methods

This Node exposes Methods for publishing records, performing bounded consumer-group reads, listening continuously, and inspecting topics and partitions.

Produce Messages

Use this method to publish one Kafka record for each row in Items. The first record is sent while the Node invocation starts, so initial validation, connection, and delivery errors surface before output streaming begins.

Produce Item Properties

Each Headers row contains a required string Key and an optional Value encoded using Header Value Format.

Produce Response Properties

After startup, a recoverable per-item error returns a failed Flowgear result and processing continues with the next row. Fatal producer errors stop the invocation.

Consume Messages

Use this method to read a bounded sequence of records through one Kafka consumer group. The Node confirms the topic and performs the first poll while the invocation starts, so initial provider errors surface before output streaming begins.

Consumed Record Properties

CommitAfterDelivery commits a record when downstream processing requests the next row or completes the stream. If processing abandons or fails on the current row, Kafka can deliver that record again. DoNotCommit leaves the consumer group offsets unchanged.

Listen

Starts a continuous listener for one exact topic and consumer group. Use this trigger when the Workflow should process arriving records until it is stopped.

Listen remains active across empty polls. It commits each record after that record's downstream Workflow path completes and requests another item. The Connection's bounded-consume Offset Commit Mode and Idle Timeout Seconds do not change that listener behavior.

List Topics

Use this method to list topics visible to the Connection. Kafka provider errors surface to the Workflow; a cluster with no visible topics returns no rows.

Topic Properties

Describe Topic

Use this method to inspect partition leadership and replica placement for one topic. A missing topic or Kafka provider error surfaces to the Workflow.

Partition Properties

Usage Notes

• Templates use the Connection formats for sample keys, values, and headers.
• Produce Messages and Consume Messages are bounded Workflow operations. Use Listen for a continuous Kafka event trigger.
• Kafka preserves ordering within a partition, not across all partitions in a topic.
• Producer idempotence reduces duplicate writes within one producer session. A Workflow retry can still publish a duplicate business event.
• Json requires valid JSON data, Utf8Text uses strict UTF-8, and Base64 preserves arbitrary bytes. A record that does not match the selected format causes an error instead of being silently changed.
• Tombstone records carry a Kafka null value and are commonly used by compacted topics to represent deletion.
• The Node does not provide Schema Registry serializers, Kafka transactions, or consume-transform-produce exactly-once processing.

Groups, offsets, and duplicate handling

Consumers using the same group share partitions and compete for records. Use different group IDs for Workflows or Environments that each need every event. Auto Offset Reset applies only when a valid committed offset is unavailable; selecting Earliest does not rewind an existing group's committed position.

Kafka ordering is partition-local. Failures, shutdown, rebalances, or uncertain commits can cause redelivery. Producer idempotence does not prevent duplicates caused by retrying a Workflow. Design downstream changes so that processing the same event again does not duplicate the business operation; see Eventual Consistency.

Set Maximum Processing Interval Seconds above the longest expected time between polls, including downstream work for a listener event. Exceeding it can cause a group rebalance and a rejected commit. It is not a timeout that cancels the downstream business operation for you.

Formats and upgrades

Use Json when values are JSON, Utf8Text for text, and Base64 to preserve arbitrary bytes. Decode failures stop the listener without committing the failing record. Choose compatible key and header formats as well as a value format.

When upgrading an older Kafka Step, move non-default format, timeout and consumer-policy settings into its Connection, then refresh the Step's inputs. Topic and Group ID remain Method inputs with Connection defaults. Check other Steps sharing that Connection before changing those defaults.

Examples

To listen for order events, configure a JSON value format and a consumer group dedicated to the intended Workflow and Environment. Add a Listen Step for the orders topic, then map Response.Value into the Steps that process the order. Store a stable event or business identifier after the target change succeeds so that a redelivered event can be recognized.

For a short diagnostic read, use Consume Messages with a small Maximum Messages value and DoNotCommit. This leaves the group's committed offsets unchanged; use a separate diagnostic group if you should not compete with an active production consumer.

See also

• Create a Listener Workflow.
• Eventual Consistency.
• Handle Partial Batch Failures.
• Correlate Records Across Systems.