In the past few posts, we have discussed approaches to implement at-least-once processing on the consumer side, i.e. mechanisms that make sure that every record in the partition is only processed once. Today, we will look at a similar problem on the producer side - how can we make sure that every record is written … Continue reading Learning Kafka with Python – retries and idempotent writes
In the previous posts, we have already used the Python client to implement Kafka consumers. Today, we will take a closer look at the components that make up a consumer and discuss their inner workings and how they communicate with the Kafka cluster. High level overview of the consumer Our discussion will be based on … Continue reading Learning Kafka with Python – a deep dive into consumers and rebalancing
Very often, either the source or the target of a Kafka based message queue is a classical relational database. Consuming data and using it to update a database table sounds straightforward, but poses a few challenges around reliability and delivery semantics. In this post, we look into two options to realize such an architecture. The … Continue reading Learning Kafka with Python – implementing a database sink
We now understand how Kafka producers add data to partitions. So let us move on and take a look at consumers - how they operate, how they are configured and how different levels of reliability and delivery guarantees can be achieved. Consumer groups In the previous post on producers, we have seen that the interaction … Continue reading Learning Kafka with Python – consuming data
As proud owners of a brand new Kafka installation, we are now ready to explore how applications interact with Kafka. Today, we will look at producers and understand how they write data to Kafka. Basic design considerations At first glance, writing data to Kafka sounds easy - connect to a Kafka broker and submit a … Continue reading Learning Kafka with Python – producing data
In this post, we will install a Kafka cluster with three nodes on our lab PC, using KVM, Vagrant and Ansible. The setup Of course it is possible (and actually easy, see the instructions in the quickstart section of the excellent Apache Kafka documentation) to run Kafka as a single-node cluster on your PC. The … Continue reading Learning Kafka with Python – installation
More or less by accident, I recently took a closer look at Kafka, trying to understand how it is installed, how it works and how the Python API can be used to write Kafka applications with Python. So I decided its time to create a series of blog posts on Kafka to document my learnings … Continue reading Learning Kafka with Python – the basics
