In the previous posts, we have already used the Python client to implement Kafka consumers. Today, we will take a closer look at the components that make up a consumer and discuss their inner workings and how they communicate with the Kafka cluster. High level overview of the consumer Our discussion will be based on … Continue reading Learning Kafka with Python – a deep dive into consumers and rebalancing
Very often, either the source or the target of a Kafka based message queue is a classical relational database. Consuming data and using it to update a database table sounds straightforward, but poses a few challenges around reliability and delivery semantics. In this post, we look into two options to realize such an architecture. The … Continue reading Learning Kafka with Python – implementing a database sink
We now understand how Kafka producers add data to partitions. So let us move on and take a look at consumers - how they operate, how they are configured and how different levels of reliability and delivery guarantees can be achieved. Consumer groups In the previous post on producers, we have seen that the interaction … Continue reading Learning Kafka with Python – consuming data
As proud owners of a brand new Kafka installation, we are now ready to explore how applications interact with Kafka. Today, we will look at producers and understand how they write data to Kafka. Basic design considerations At first glance, writing data to Kafka sounds easy - connect to a Kafka broker and submit a … Continue reading Learning Kafka with Python – producing data
In this post, we will install a Kafka cluster with three nodes on our lab PC, using KVM, Vagrant and Ansible. The setup Of course it is possible (and actually easy, see the instructions in the quickstart section of the excellent Apache Kafka documentation) to run Kafka as a single-node cluster on your PC. The … Continue reading Learning Kafka with Python – installation
More or less by accident, I recently took a closer look at Kafka, trying to understand how it is installed, how it works and how the Python API can be used to write Kafka applications with Python. So I decided its time to create a series of blog posts on Kafka to document my learnings … Continue reading Learning Kafka with Python – the basics
In my previous post, we have seen how the libvirt toolset can be used to directly create virtual volumes, virtual networks and KVM virtual machines. If this is not the first time you visit my post, you will know that I am a big fan of automation, so let us investigate today how we can … Continue reading Managing KVM virtual machines part III – using libvirt with Ansible
In the previous post, we have seen how Vagrant can be used to define, create and destroy KVM virtual machines. Today, we will dig a bit deeper into the objects managed by the libvirt library and learn how to create virtual machines using the libvirt toolkit directly Creating a volume When creating a virtual machine, … Continue reading Managing KVM virtual machines part II – the libvirt toolkit
When you first install and play with Vagrant, chances are that you will be using the VirtualBox VM provider initially, which is supported out-of-the-box by Vagrant and open source. However, in some situations, VirtualBox might not be your preferred hypervisor. Luckily, with the help of a plugin, Vagrant can also be used with KVM. In … Continue reading Managing KVM virtual machines part I – Vagrant and libvirt
When you are playing with virtualization and cloud technology, you will sooner or later realize that the resources of an average lab PC are limited. Especially memory can easily become a bottleneck if you need to spin up more than just a few virtual machines on an average desktop computer. Public cloud platforms, however, offer … Continue reading A cloud in the cloud – running OpenStack on a public cloud platform
