Kafka Engine

Streaming events from Kafka into ClickHouse is one of the most common production patterns, and it's built from two familiar ideas: a table engine that stores no data itself (the same shape as Distributed, covered later) and a materialized view that reacts to inserts. Kafka wiring just connects them to a topic instead of a client.

The three-part shape

-- 1. the consumer: stores nothing, just describes how to read the topic
CREATE TABLE events_queue
(
  user_id UInt64,
  event   String,
  ts      DateTime
)
ENGINE = Kafka
SETTINGS
  kafka_broker_list = 'kafka:9092',
  kafka_topic_list = 'events',
  kafka_group_name = 'clickhouse_events_consumer',
  kafka_format = 'JSONEachRow';

-- 2. the real table: this is what you actually query
CREATE TABLE events
(
  user_id UInt64,
  event   String,
  ts      DateTime
)
ENGINE = MergeTree
ORDER BY (user_id, ts);

-- 3. the glue: fires on every batch Kafka delivers
CREATE MATERIALIZED VIEW events_consumer_mv
TO events
AS SELECT user_id, event, ts FROM events_queue;

Why not just query the Kafka table directly?

You can — SELECT against a Kafka engine table actually consumes messages from the topic as a side effect, advancing the consumer offset. That makes it fundamentally unlike a normal table: reading it twice does not give you the same rows twice, and two people querying it concurrently split the messages between them rather than both seeing everything. In practice it should only ever be read by exactly one thing — the materialized view attached to it — which is why the standard advice is to never query it directly yourself.

What the materialized view is really doing here

ClickHouse's Kafka integration polls the topic in the background and, for each batch of messages it reads, inserts them into the Kafka engine table — which immediately fires every materialized view attached to it, exactly like any other insert. The MV's SELECT can reshape, filter, or aggregate the raw message on the way into the target table, not just copy it verbatim.