@theotherwillembotha/node-red-kafka 0.0.55

Apache Kafka producer and consumer nodes for Node-RED

npm install @theotherwillembotha/node-red-kafka

node-red-kafka

Apache Kafka producer and consumer nodes for Node-RED, built on node-red-plugincore.

What is Kafka?

Apache Kafka is a distributed event streaming platform. At its core it works like a persistent, high-throughput message bus:

• Topics - named channels that producers write to and consumers read from. Think of a topic as a log file that multiple services can append to and tail simultaneously.
• Brokers - the Kafka server(s) that store and serve messages. A cluster has one or more brokers.
• Producers - clients that publish messages to a topic.
• Consumers - clients that subscribe to a topic and process messages as they arrive.
• Consumer Groups - a group of consumers that share the workload of a topic. Each partition is assigned to exactly one consumer in the group at a time.
• Partitions - each topic is split into partitions for parallelism and scalability. A key on a message controls which partition it lands on.
• Offsets - every message in a partition has a sequential offset number. Kafka remembers where each consumer group is up to, so consumers can resume after a restart without missing or re-reading messages.

Kafka is commonly used for real-time data pipelines, event-driven architectures, IoT telemetry ingestion, and microservice integration.

Nodes

[!IMPORTANT] This plugin requires @theotherwillembotha/node-red-plugincore to be installed.

node-red-plugincore is declared as a dependency and npm will install it automatically alongside this package. However, due to a known Node-RED limitation, packages that arrive as transitive npm dependencies are only discovered by the Node-RED runtime on the next startup.

You have two options:

• Install @theotherwillembotha/node-red-plugincore via the palette manager or npm install first, then install this plugin - both will be available immediately without a restart.
• Install this plugin directly - node-red-plugincore will be installed automatically alongside it. Restart Node-RED once and both packages will be fully loaded.

Getting Started

1. Start a Kafka broker

Save the following as kafka-compose.yaml and run docker compose -f kafka-compose.yaml up -d:

services:
  kafka:
    image: apache/kafka:4.3.0
    container_name: kafka
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_NUM_PARTITIONS: 3
    networks:
      - dev_nodered

networks:
  dev_nodered:
    name: dev_nodered
    external: true

Note: KAFKA_LISTENERS must bind to 0.0.0.0 (not localhost) so that other containers on the same Docker network can reach the broker. KAFKA_ADVERTISED_LISTENERS must use the container name (kafka) so that clients resolve it correctly via Docker DNS.

The Kafka container and your Node-RED container must be on the same Docker network (or otherwise reachable from each other). The compose file above attaches Kafka to dev_nodered - make sure your Node-RED container is on that same network. If you are using a different network name, update the networks section accordingly. You can verify connectivity by pinging the kafka container from within the Node-RED container before configuring the cluster node.

2. Configure the cluster in Node-RED

Add a Kafka Cluster config node and point it at kafka:9092. Use Test Connection to verify before saving.

3. Add a topic

Add a Kafka Topic config node, select your cluster, and enter a topic name.

4. Wire up consumer and/or producer nodes

Drop a Kafka Consumer or Kafka Producer node into your flow and select the topic config.

Node Reference

Kafka Cluster Config

Connection configuration for a Kafka broker or cluster. All other Kafka nodes reference this.

Use the Test Connection button to verify connectivity before saving. The test times out after 6 seconds.

Kafka Topic Config

Names a Kafka topic and links it to a cluster. Both consumer and producer nodes reference this, so the topic name is defined in one place.

Kafka Consumer

Subscribes to a Kafka topic and emits a Node-RED message for each record received.

Warning - Read from beginning: This flag only takes effect the first time a consumer group connects to a topic (i.e. when there is no committed offset for that group). On a busy or long-lived topic this can replay millions of historical messages, potentially flooding your flow. Only enable this on a fresh consumer group, or when you specifically intend to reprocess historical data. To reset, either change the Group ID or manually reset offsets via a Kafka admin tool.

Output message properties:

The node status shows Connected (green) when the consumer is running, or Error (red) if the connection fails.

Kafka Producer

Publishes an incoming Node-RED message to a Kafka topic.

Input: msg.payload is used as the message value. Objects are serialised to JSON automatically; all other values are converted to a string.

Output: The original input message is forwarded unchanged after the publish completes successfully.

The node status shows Connected (green) when ready, or Error (red) if the connection fails. Messages arriving before the producer is connected will trigger an error.

A note on message keys: Kafka uses the key to determine which partition a message is routed to. Messages with the same key always land on the same partition, which guarantees ordering for that key. If no key is set, messages are distributed round-robin across partitions.

Example

The flow below uses an Inject node to trigger a producer, then a consumer reads it back and logs it.

Requirements

• Node-RED >= 3.0.0
• Node.js >= 18.0.0
• A running Apache Kafka broker (see Getting Started above)

License

ISC