You want to partition messages according to your business scenario. But default partitioner cannot do so. Use Kafka Custom Partitioning strategy.
Join the DZone community and get the full member experience.
Imagine that you are running an e-commerce store for electronic devices. Going into the holiday season, your business forecast predicts a significant increase in the sales of other brands when compared to Apple devices.
Every sale transaction goes through a Kafka broker, and you want to ensure there are no resource issues with the data flow. Out of the three Kafka partitions for handling sales data, you want to dedicate two for non-Apple devices and only one for Apple devices.
Check out the below illustration that describes the requirements.
The reason behind custom partitioning is often a business requirement. Even though Kafka has a default partitioning mechanism, the business requirement creates a need for a custom partitioning strategy.
Of course, the example requirement is a little contrived. But it does not matter. All that matters is that you need to perform custom partitioning or the business might suffer due to excessive load.
Thankfully, Kafka provides a ready-to-use mechanism to implement custom partitioning.
We need a place to keep our custom partitioning logic. For this purpose, Kafka provides a Partitioner interface. We need to implement this interface and override the necessary methods with our custom logic.
Check out the below code for the BrandPartitioner class:
To implement the Partitioner interface successfully, we need to implement three specific methods:
partition() – This is where we keep the actual partitioning logic
configure() – This is where we receive any custom properties that might be needed to determine the correct partition.
