Dec 7, 2025
You Gotta Push If You Wanna Pull
Historically, data management systems have been built around the notion of pull queries: users query data which, for instance, is stored in tables in an RDBMS, Parquet files in a data lake, or a full-text index in Elasticsearch. When a user issues a query, the engine will produce the result set at that point in time by churning through the data set and finding all matching records (oftentimes sped up by utilizing indexes).
Read More...
Nov 25, 2025
On Idempotency Keys
In distributed systems, there’s a common understanding that it is not possible to guarantee exactly-once delivery of messages. What is possible though is exactly-once processing. By adding a unique idempotency key to each message, you can enable consumers to recognize and ignore duplicate messages, i.e. messages which they have received and successfully processed before.
Read More...
Nov 20, 2025
Building a Durable Execution Engine With SQLite
Lately, there has been a lot of excitement around Durable Execution (DE) engines. The basic idea of DE is to take (potentially long-running) multi-step workflows, such as processing a purchase order or a user sign-up, and make their individual steps persistent. If a flow gets interrupted while running, for instance due to a machine failure, the DE engine can resume it from the last successfully executed step and drive it to completion.
Read More...
Nov 3, 2025
"You Don't Need Kafka, Just Use Postgres" Considered Harmful
Looking to make it to the front page of HackerNews? Then writing a post arguing that "Postgres is enough", or why "you don’t need Kafka at your scale" is a pretty failsafe way of achieving exactly that. No matter how often it has been discussed before, this topic is always doing well. And sure, what’s not to love about that? I mean, it has it all: Postgres, everybody’s most favorite RDBMS—check! Keeping things lean and easy—sure, count me in! A somewhat spicy take—bring it on!
Read More...
Sep 17, 2025
Let's Take a Look at... Lower Java Tail Latencies With ZGC
Java 25 was released earlier this week, and it is the first Java release with long-term support (LTS) which ships with Generational ZGC as the one (and only) flavor of the ZGC garbage collector. ZGC itself is a relatively new concurrent collector, originally added in Java 11.
Read More...
Aug 5, 2025
Postgres Replication Slots: Confirmed Flush LSN vs. Restart LSN
Replication slots in Postgres keep track of how far consumers have read a replication stream. After a restart, consumers—either Postgres read replicas or external tools for change data capture (CDC), like Debezium—resume reading from the last confirmed log sequence number (LSN) of their replication slot. The slot prevents the database from disposing of required log segments, allowing safe resumption after downtime.
In this post, we are going to take a look at why Postgres replication slots don’t have one but two LSN-related attributes: restart_lsn and confirmed_flush_lsn. Understanding the difference between the two is crucial for troubleshooting replication issues, optimizing WAL retention, and avoiding common pitfalls in production environments.
Read More...
Jul 17, 2025
Converting Future to CompletableFuture With Java Virtual Threads
Since Java 8, the CompletableFuture API provides a convenient way for performing asynchronous operations in a functional, composable way. This makes it very simple to call some long-running methods—for instance involving external I/O—asynchronously and process each result as soon as it is available, without blocking on any threads:
Read More...
Jul 8, 2025
Mastering Postgres Replication Slots: Preventing WAL Bloat and Other Production Issues
Over the last couple of years, I’ve helped dozens of users and organizations to build Change Data Capture (CDC) pipelines for their Postgres databases. A key concern in that process is setting up and managing replication slots, which are Postgres' mechanism for making sure that any segments of the write-ahead log (WAL) of the database are kept around until they have been processed by registered replication consumers.
When not being careful, a replication slot may cause unduly large amounts of WAL segments to be retained by the database. This post describes best practices helping to prevent this and other issues, discussing aspects like heartbeats, replication slot failover, monitoring, the management of Postgres publications, and more. While this is primarily based on my experience of using replication slots via Debezium’s Postgres connector, the principles are generally applicable and are worth considering also when using other CDC tools for Postgres based on logical replication.
Read More...
Jun 18, 2025
This AI Agent Should Have Been a SQL Query
Table of Contents Agents Need to Interact With LLMs Agents Should Be Event-Driven Agents Need Context Agents Require Memory When SQL Is Not Enough Parting Thoughts AI Agents have improved in leaps and bounds in recent times, moving beyond simple chatbots to sophisticated, autonomous systems. This post explores a novel approach to building agentic systems: using the power of streaming SQL queries. Discover how platforms like Apache Flink can transform the development of AI Agents, offering benefits in consistency, scalability, and developer experience.
Read More...
May 26, 2025
Backfilling Postgres TOAST Columns in Debezium Data Change Events
Table of Contents Debezium Reselect Postprocessor Flink DataStream API Flink SQL With OVER Aggregation Flink Process Table Functions Summary and Discussion Postgres logical replication, while powerful for capturing real-time data changes, presents challenges with TOAST columns, whose values can be absent from data change events in specific situations. This post discusses how Debezium addresses this through its built-in reselect post processor, then explores more robust solutions leveraging Apache Flink’s capabilities for stateful stream processing, including Flink SQL and the brand-new process table functions (PTFs) in Flink 2.1.
Read More...
