This presentation discusses the process of building highly scalable, mission-critical microservices with Apache Kafka, Kafka Streams, and Apache Mesos/DCOS.
I had a talk at MesosCon 2017 Europe in Prague about building highly scalable, mission-critical microservices with Apache Kafka, Kafka Streams and Apache Mesos / DCOS. I would like to share the slides and a video recording of the live demo.
Microservices establish many benefits like agile, flexible development and deployment of business logic. However, a Microservice architecture also creates many new challenges. This includes increased communication between distributed instances, the need for orchestration, new fail-over requirements, and resiliency design patterns.
A live demo shows how to develop real-time applications for your core business with Kafka messaging brokers and Kafka Streams API. You see how to deploy/manage/scale them on a DC/OS cluster using different deployment options.
The following picture shows the architecture. You can either run Kafka Brokers and Kafka Streams microservices natively on DC/OS via Marathon or leverage Kubernetes as a Docker container orchestration tool (which is also supported by Mesosphere in the meantime).
Here are the slides from my talk .
The following video shows the live demo. It is built on AWS using Mesosphere’s Cloud Formation script to set up a DC/OS cluster in ten minutes.
Here, I deployed both – Kafka brokers and Kafka Streams microservices – directly to DC/OS without leveraging Kubernetes. I expect to see many people continue to deploy Kafka brokers directly on DC/OS. For microservices many teams might move to the following stack: Microservice –> Docker –> Kubernetes –> DC/OS.
Do you also use Apache Mesos respectively DC/OS to run Kafka? Only the brokers or also Kafka clients (producers, consumers, Streams, Connect, KSQL, etc)? Or do you prefer another tool like Kubernetes (maybe on DC/OS)?
