<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Michael Loh]]></title><description><![CDATA[Michael Loh]]></description><link>https://mloh.hashnode.dev</link><generator>RSS for Node</generator><lastBuildDate>Mon, 16 Nov 2026 20:17:07 GMT</lastBuildDate><atom:link href="https://mloh.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[When to skip Kafka and Event Hubs with Databricks Zerobus Ingest]]></title><description><![CDATA[When to write events straight into Delta with Databricks Zerobus Ingest, when Kafka or Event Hubs still wins, and what each costs on Azure.]]></description><link>https://mloh.hashnode.dev/databricks-zerobus-ingest-vs-kafka-event-hubs</link><guid isPermaLink="true">https://mloh.hashnode.dev/databricks-zerobus-ingest-vs-kafka-event-hubs</guid><category><![CDATA[Databricks]]></category><category><![CDATA[data-engineering]]></category><category><![CDATA[Azure]]></category><category><![CDATA[Apache Kafka]]></category><category><![CDATA[streaming]]></category><dc:creator><![CDATA[Ming Yao (Michael)]]></dc:creator><pubDate>Sun, 15 Nov 2026 00:23:59 GMT</pubDate><content:encoded><![CDATA[<p>Getting data into Databricks is easy when it already sits in files or a database. It gets harder when other systems push data at you, like webhook events or readings from devices.</p>
<p>The usual answer is a message queue or event stream such as Kafka or Azure Event Hubs, plus a streaming job that copies the events into a Databricks table. Both run all day. Often the lakehouse is the only thing reading those events.</p>
<p>Zerobus Ingest lets apps and devices skip both. They send events straight into a Databricks table, using Zerobus's own client library, plain HTTPS, or protocols they may already use, like Kafka, MQTT and OpenTelemetry. Databricks runs the service, and rows can be queried within seconds.</p>
<img src="https://cdn.hashnode.com/uploads/covers/6af828aac8022181e86dd689/1e402413-dd3f-4ec1-8095-97e67728a017.svg" alt="Usual setup with Kafka or Event Hubs and a streaming job, compared with producers writing through Zerobus Ingest" style="display:block;margin:0 auto" />

<h2>How they compare</h2>
<table>
<thead>
<tr>
<th></th>
<th>Kafka or Event Hubs + streaming job</th>
<th>Zerobus Ingest</th>
</tr>
</thead>
<tbody><tr>
<td>What you run</td>
<td>A Kafka cluster or Event Hubs, plus a job that copies events into the table</td>
<td>Nothing. Databricks runs it</td>
</tr>
<tr>
<td>Copies of the data</td>
<td>Two: the events wait in the queue first (up to 7 days on Event Hubs Standard), then land in the table</td>
<td>One: the table</td>
</tr>
<tr>
<td>Checking the fields</td>
<td>The queue itself accepts anything. A schema registry (Confluent for Kafka, Azure Schema Registry for Event Hubs) can check the fields, but you set it up and use it in your own code</td>
<td>Each record must match the table's columns, and Zerobus never changes the table</td>
</tr>
<tr>
<td>Other readers and replay</td>
<td>Several apps can read the same events as they arrive, and go back and re-read them from any point</td>
<td>No. Anything else reads the table</td>
</tr>
<tr>
<td>Delay before data is in the table</td>
<td>Depends on how often the job runs, from seconds to minutes</td>
<td>About 5 seconds</td>
</tr>
<tr>
<td>Delivery</td>
<td>Each event lands in the table exactly once, in order within each slice of the queue</td>
<td>At least once, so a retry can duplicate an event. Events stay in order within one connection</td>
</tr>
<tr>
<td>Code that sends events</td>
<td>A Kafka or Event Hubs client library</td>
<td>Zerobus's client library or a plain web request (REST), with records shaped like the table</td>
</tr>
<tr>
<td>Sources you don't control</td>
<td>Some can send in directly, like IoT Hub</td>
<td>Need something in between to receive and forward the data</td>
</tr>
<tr>
<td>Size limits</td>
<td>1 MB per send</td>
<td>10 MB per message, limits on each connection, and <a href="https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/zerobus-limits">not every region</a> is supported</td>
</tr>
<tr>
<td>Billing</td>
<td>By the hour, even when idle</td>
<td>Per GB sent</td>
</tr>
</tbody></table>
<h2>Cost</h2>
<p>I priced both at Azure list prices (East US, Premium tier) with 1 KB events: Event Hubs Standard plus a streaming job on two D4ds v5 nodes running all month, against Zerobus at its per-GB rate, compute included. Storage and downstream processing cost the same either way.</p>
<table>
<thead>
<tr>
<th>Events per second</th>
<th>Data per month</th>
<th>Event Hubs + streaming job</th>
<th>Zerobus</th>
</tr>
</thead>
<tbody><tr>
<td>100</td>
<td>263 GB</td>
<td>about $800</td>
<td>about $17</td>
</tr>
<tr>
<td>1,000</td>
<td>2.6 TB</td>
<td>about $860</td>
<td>about $170</td>
</tr>
<tr>
<td>10,000</td>
<td>26 TB</td>
<td>about $1,720</td>
<td>about $1,680</td>
</tr>
</tbody></table>
<p>At low to mid volume, Zerobus is clearly cheaper. The streaming job costs about $770 a month whether it is busy or not, and Zerobus has no fixed part, so you only pay for what you send.</p>
<p>As volume grows, there comes a point where running your own Event Hubs and job gets cheaper. In this model that is around 10,000 events a second. But the saving comes with a high engineering cost: someone has to size the cluster, keep the job healthy and fix it when it falls behind, and neither column counts those hours.</p>
<h2>Picking between them</h2>
<table>
<thead>
<tr>
<th>Situation</th>
<th>Better choice</th>
</tr>
</thead>
<tbody><tr>
<td>Your own apps, devices or webhook receivers push events, and only the lakehouse reads them</td>
<td>Zerobus</td>
</tr>
<tr>
<td>Producers already speak Kafka, MQTT or OpenTelemetry, and the topics only feed the lakehouse</td>
<td>Zerobus, through that protocol</td>
</tr>
<tr>
<td>Other systems read the same events, or you need replay</td>
<td>Kafka or Event Hubs</td>
</tr>
<tr>
<td>High, steady volume with a team that already runs Kafka or Event Hubs well</td>
<td>Kafka or Event Hubs</td>
</tr>
<tr>
<td>A source only ships an Event Hubs connector</td>
<td>Turn it on and read the stream</td>
</tr>
<tr>
<td>Files, database changes, or code already in Databricks</td>
<td>Auto Loader, Lakeflow Connect, or Spark</td>
</tr>
</tbody></table>
<h2>My take</h2>
<p>If the lakehouse is the only thing reading your events and the volume isn't huge, I'd reach for Zerobus first, mostly because it's simple and cheap. It's serverless, so the data team has no cluster or namespace to size, patch or keep awake, and it pays per GB sent instead of per hour.</p>
<h2>What I built</h2>
<p>Two Python producers stand in for an app and a device, and a Lakeflow pipeline cleans up what lands. Since Zerobus won't change the table's columns, I kept them to a few fields every event has and put the event body in a <code>VARIANT</code> column, so producers can add fields freely. The code, Terraform and Databricks Asset Bundle are <a href="https://github.com/mingyao1993/mloh/tree/main/databricks-zerobus">on GitHub</a>, and the <a href="https://docs.databricks.com/aws/en/ingestion/lakeflow-connect/zerobus-ingest">Zerobus Ingest docs</a> cover the rest.</p>
]]></content:encoded></item></channel></rss>