Apache Flink: what it solves and what it costs to run

State and checkpoints

Almost every useful job keeps something between one record and the next: a count per key, an open time window, the read position in the source. In Flink that is state, and the framework takes care of it. You don’t have to keep that position in a separate Redis and hope it gets saved at the right moment.

Every so often Flink takes a checkpoint. It injects barriers into the stream starting at the sources, and each operator saves its state when the barrier passes through it. The result is a consistent snapshot of the whole job, with the source positions and the state of each operator, written to durable storage (S3, in our case). If a task fails, the job goes back to the last checkpoint and rereads the source from the position saved with it.

State lives in a state backend. hashmap keeps everything on the JVM heap and is fast; RocksDB stores it on local disk and handles state larger than memory. Savepoints are checkpoints you trigger by hand, used to update the job’s code without losing its position.

Backpressure

Between operators, Flink uses credit-based flow control: an operator only sends data when the next one has free buffer space. If the sink gets slow, the slowdown travels up the graph to the source, which starts reading more slowly. The system slows down, and pending work stays at the source (in the oplog, in the Kafka topic) instead of piling up in the process’s memory.

Event time

For windowed aggregations, Flink separates the moment an event happened from the moment it arrived, and uses watermarks to decide when a window can close. This barely shows up in CDC, but it is one of the main reasons to use Flink for streaming analytics.

How it scales

A Flink job is split into subtasks. If an operator has parallelism 8, there are 8 copies of it spread across the TaskManagers’ slots, each one processing a slice of the data. To handle more load, you raise the parallelism and add TaskManagers. The job’s code stays the same.

When there is keyed state, Flink splits the key space into key groups, and each subtask owns a set of them. When you change the parallelism, the job stops with a savepoint and comes back with the new number of subtasks, and Flink redistributes the key groups (and each one’s state) among them. Each key keeps going to the same place, so per-entity ordering holds. The Kubernetes operator also has an autoscaler that adjusts parallelism to the load.

How far reading scales depends on the source. A Kafka source spreads the topic’s partitions across the subtasks, and more partitions allow more parallelism. A MongoDB change stream can’t be split, so scale comes from somewhere else there: each collection becomes an independent pipeline, and whatever comes after the read (building the event, lookups, writing) can run with higher parallelism.

Scaling isn’t only about Flink. What it publishes to Kafka scales on the consumer side: with the topic partitioned by the entity key, you can run one consumer replica per partition in the same consumer group. That was impossible with the Node process reading straight from the change stream, which has no consumer group and would process everything twice with a second replica.

How it stays available

Flink’s availability comes from recovery. It doesn’t run two active copies of the same job. When something fails, it goes back to the last consistent state and continues from there.

If a TaskManager goes down, the JobManager notices, reschedules the affected tasks on other slots and restores state from the last checkpoint. With regional failover, only the tasks connected to the failed part restart; in a job with several independent sources, the others keep running. The restart strategy decides how many times it retries and how long it waits between attempts.

The JobManager itself can also go down. With high availability turned on (on Kubernetes, using ConfigMaps for leader election and metadata), a new JobManager takes over, finds the last registered checkpoint and resumes the job. Since checkpoints live in durable storage like S3, losing a pod or an entire node doesn’t take the state with it.

Upgrades follow the same logic: the job stops with a savepoint, the new version starts from it and continues from the same position in the source.

This model has a cost. During recovery the job sits still for a few seconds or longer, and everything processed after the last checkpoint gets processed again. Whoever consumes the output has to tolerate that delay and those duplicates, and the checkpoint interval becomes a trade-off between snapshot cost and how much work gets redone on each failure. In our case, Kafka between Flink and the consumers absorbs the pauses, and the event’s deterministic id takes care of the duplicates.

An explicit delivery guarantee

The checkpoint ties the source position to the state and to the output. The KafkaSink in AT_LEAST_ONCE mode only lets the checkpoint complete after flushing what it sent. So no record that was read gets lost: in the worst case, after a failure, the records since the last checkpoint are sent again. In EXACTLY_ONCE mode, the sink writes inside a Kafka transaction and only commits when the checkpoint closes.

In a hand-written consumer, that guarantee depends on someone remembering to save the position after the side effect, and it’s easy to get wrong. In the Node process we replaced, the position was saved before the handler ran, and any crash in between lost the event without leaving a log.

End-to-end backpressure

In Node, an EventEmitter doesn’t wait for an async listener’s Promise. Each event fires a new handler, and under load that turns into unbounded concurrency, then out-of-memory, then data loss. In Flink, a slow sink makes the source read more slowly. The extra load shows up as lag, which you can measure and fix.

Scale and isolation

Parallelism is configuration, and with keyBy on the right key you scale without losing per-entity ordering. For jobs with several independent sources, Flink also has isolation tools that make a difference:

• slot sharing groups, which put the heavy pipelines in their own slots;
• regional failover, where a failure in one pipeline restarts only the tasks connected to it;
• restart strategies like exponential-delay, which retries with a growing wait instead of giving up after N failures.

In our events job, each MongoDB collection is a separate pipeline. If one collection’s source has a problem, only that one restarts.

Efficiency

In our measurements, the Flink events job uses about 10 times less CPU on average than the Node process it replaced. The comparison isn’t exact: part of the work moved out of the old process into consumers outside Flink, and the old process did an extra read of the document on every update. Even so, the order-of-magnitude difference was clear.

Ecosystem

Connectors that already take part in checkpoints, per-operator metrics, a web UI with the backpressure graph, and a Kubernetes operator. In a home-grown consumer, each of those would be code to write and maintain.

Running Flink takes work

A Flink job is a stateful distributed system. You end up operating the JobManager, TaskManagers, checkpoints in S3, the Kubernetes operator, and savepoints on every upgrade.

The most dangerous failure mode is silent. A job in FAILED doesn’t take down any API, it just stops producing. We’ve had the indexing job stopped for almost two days after database maintenance, and a staging environment stopped for a week without anyone noticing. Another time, deploys for two jobs stopped reaching the operator because a resource got stuck in Terminating, while the CI pipeline stayed green.

If Flink is on the critical path, it needs alerts on job state, the age of the last checkpoint and source lag, and on-call coverage at the same level as any production service.

Reserved memory

The JVM reserves memory up front. Our events job takes a little over 2 GiB even under light load. For small jobs, that weighs more than the CPU you save.

A different language

If the rest of the company writes TypeScript, any domain logic that has to run inside the job must be ported to Java or fit in SQL. In our case, the charge event had to carry the Pix BR Code. That became about 1,400 lines of Java, plus parity tests generated from the TypeScript code, and now every change on one side has to be mirrored on the other.

Exactly-once costs latency

Flink’s exactly-once applies to the job’s internal state. For Kafka, it relies on transactions, and a read_committed consumer only sees the records when the transaction closes, at the next checkpoint. With a checkpoint every 60 seconds, that can mean almost a minute of delay, which doesn’t work for a screen that shows “payment confirmed”.

That’s why we use at-least-once. The price is that duplicates become part of the contract: the event needs a deterministic id, and every consumer with an external side effect has to deduplicate.

Parallelism stops at the source

Flink scales as far as the source allows. A MongoDB change stream can’t be split, so each collection is read by a single task. And if reading from the source gets slow, the whole job inherits the lag. Our CDC jobs read from a single node of the replica set and fall tens of minutes behind while the database’s nightly backup runs. The problem isn’t in Flink, but Flink is where the lag shows up.

Learning curve

Keyed state, type serialization, watermarks, state compatibility across job versions, JVM memory tuning and checkpoint tuning. For people coming from stateless Node.js services, that’s a lot of new things at once.