Apache Kafka is used at Pinterest for transporting data for real time streaming applications, logging and visibility metrics for monitoring. Hosted on AWS, Pinterest’s Kafka installation uses the MirrorMaker and DoctorKafka tools for replication and high availability.
Apache Kafka is used at Pinterest for transporting data for real time streaming applications, logging and visibility metrics for monitoring. Hosted on AWS, Pinterest’s Kafka installation uses the MirrorMaker and DoctorKafka tools for replication and high availability.
Pinterest’s Kafka installation runs on over 2000 brokers, wrote Yu Yang, Tech Lead at Pinterest, and handles more than 800 million messages and 1.2 Petabytes per day, and is spread over 3 AWS regions. Their primary Kafka toolset includes Kafka’s MirrorMaker and Pinterest’s own DoctorKafka. MirrorMaker consumes data from a source cluster and publishes them to a target cluster, effectively creating a replica of the source cluster. Pinterest’s team uses it to spread data among the 3 AWS regions.
