Asynchronous I/O with Python part III – native coroutines and the event loop

In the previous post, we have seen how iterators and generators can be used in Python to implement coroutines. With this approach, a coroutine is simply a function that contains a yield statement somewhere. This is nice, but makes the code hard to read, as the function signature does not immediately give you a hint … Continue reading Asynchronous I/O with Python part III – native coroutines and the event loop →

Asynchronous I/O with Python part II – iterators and generators

As explained in my previous post, historically coroutines in Python have evolved from iterators and generators, and understanding generators is still vital to understanding native coroutines. In this post, we take a short tour through iterators in Python and how generators have traditionally been implemented. Iterables and iterators In Python (and in other programming languages), … Continue reading Asynchronous I/O with Python part II – iterators and generators →

Asynchronous I/O with Python part I – the basics

Though not really new, a programming model commonly known as asynchronous I/O has been attracting a lot of attention over the last couple of years and even influenced the development of languages like Java, Go or Kotlin. In this and the next few posts, we will take a closer look at this model and how … Continue reading Asynchronous I/O with Python part I – the basics →

Learning Kafka with Python – retries and idempotent writes

In the past few posts, we have discussed approaches to implement at-least-once processing on the consumer side, i.e. mechanisms that make sure that every record in the partition is only processed once. Today, we will look at a similar problem on the producer side - how can we make sure that every record is written … Continue reading Learning Kafka with Python – retries and idempotent writes →

Learning Kafka with Python – a deep dive into consumers and rebalancing

In the previous posts, we have already used the Python client to implement Kafka consumers. Today, we will take a closer look at the components that make up a consumer and discuss their inner workings and how they communicate with the Kafka cluster. High level overview of the consumer Our discussion will be based on … Continue reading Learning Kafka with Python – a deep dive into consumers and rebalancing →

Learning Kafka with Python – implementing a database sink

Very often, either the source or the target of a Kafka based message queue is a classical relational database. Consuming data and using it to update a database table sounds straightforward, but poses a few challenges around reliability and delivery semantics. In this post, we look into two options to realize such an architecture. The … Continue reading Learning Kafka with Python – implementing a database sink →

Learning Kafka with Python – consuming data

We now understand how Kafka producers add data to partitions. So let us move on and take a look at consumers - how they operate, how they are configured and how different levels of reliability and delivery guarantees can be achieved. Consumer groups In the previous post on producers, we have seen that the interaction … Continue reading Learning Kafka with Python – consuming data →

Learning Kafka with Python – producing data

As proud owners of a brand new Kafka installation, we are now ready to explore how applications interact with Kafka. Today, we will look at producers and understand how they write data to Kafka. Basic design considerations At first glance, writing data to Kafka sounds easy - connect to a Kafka broker and submit a … Continue reading Learning Kafka with Python – producing data →