Learning Kafka with Python – a deep dive into consumers and rebalancing

In the previous posts, we have already used the Python client to implement Kafka consumers. Today, we will take a closer look at the components that make up a consumer and discuss their inner workings and how they communicate with the Kafka cluster. High level overview of the consumer Our discussion will be based on … Continue reading Learning Kafka with Python – a deep dive into consumers and rebalancing

Learning Kafka with Python – implementing a database sink

Very often, either the source or the target of a Kafka based message queue is a classical relational database. Consuming data and using it to update a database table sounds straightforward, but poses a few challenges around reliability and delivery semantics. In this post, we look into two options to realize such an architecture. The … Continue reading Learning Kafka with Python – implementing a database sink

Learning Kafka with Python – consuming data

We now understand how Kafka producers add data to partitions. So let us move on and take a look at consumers - how they operate, how they are configured and how different levels of reliability and delivery guarantees can be achieved. Consumer groups In the previous post on producers, we have seen that the interaction … Continue reading Learning Kafka with Python – consuming data

Learning Kafka with Python – producing data

As proud owners of a brand new Kafka installation, we are now ready to explore how applications interact with Kafka. Today, we will look at producers and understand how they write data to Kafka. Basic design considerations At first glance, writing data to Kafka sounds easy - connect to a Kafka broker and submit a … Continue reading Learning Kafka with Python – producing data

Managing KVM virtual machines part III – using libvirt with Ansible

In my previous post, we have seen how the libvirt toolset can be used to directly create virtual volumes, virtual networks and KVM virtual machines. If this is not the first time you visit my post, you will know that I am a big fan of automation, so let us investigate today how we can … Continue reading Managing KVM virtual machines part III – using libvirt with Ansible

Managing KVM virtual machines part II – the libvirt toolkit

In the previous post, we have seen how Vagrant can be used to define, create and destroy KVM virtual machines. Today, we will dig a bit deeper into the objects managed by the libvirt library and learn how to create virtual machines using the libvirt toolkit directly Creating a volume When creating a virtual machine, … Continue reading Managing KVM virtual machines part II – the libvirt toolkit

Managing KVM virtual machines part I – Vagrant and libvirt

When you first install and play with Vagrant, chances are that you will be using the VirtualBox VM provider initially, which is supported out-of-the-box by Vagrant and open source. However, in some situations, VirtualBox might not be your preferred hypervisor. Luckily, with the help of a plugin, Vagrant can also be used with KVM. In … Continue reading Managing KVM virtual machines part I – Vagrant and libvirt

A cloud in the cloud – running OpenStack on a public cloud platform

When you are playing with virtualization and cloud technology, you will sooner or later realize that the resources of an average lab PC are limited. Especially memory can easily become a bottleneck if you need to spin up more than just a few virtual machines on an average desktop computer. Public cloud platforms, however, offer … Continue reading A cloud in the cloud – running OpenStack on a public cloud platform