Susheel Aroskar from Netflix’s Engineering team spoke at the recent QCon New York 2018 Conference about Zuul Push, a scalable push notification service that asynchronously pushes data like personalized movie recommendations from cloud to devices.
Susheel Aroskar from Netflix ’s Engineering team spoke at the recent QCon New York 2018 Conference about Zuul Push, a scalable push notification service that asynchronously pushes data like personalized movie recommendations from cloud to devices.
The Netflix team uses Zuul Push to improve the user experience for their customers. A typical user on Netflix spends a significant amount of time picking a movie compared to watching a movie. This framework helps in providing better engagement, by generating customized recommendations in the cloud that you can show the user. Also, if there is a better recommendation generated, it lets the client know in real time.
Zuul Push is a high performance async service based on Apache Netty, a NIO based network application framework. It can be used to scale to large number of persistent connections using Netty and async I/O. Zuul Push supports WebSocket and Server-Sent Events ( SSE) protocols for push notifications and handles more than 5.5 million connected clients at peak.
Aroskar discussed the architecture of Zuul Push solution, which includes the following components:
The replicated client registry makes it possible to scale the concurrent persistent connections and deliver push notifications. Push registry, which is based on Redis and Cassandra, features a checklist which includes low read latency, record expiry, sharding and replication. They use Dynomite which is basically Redis key value data store with additional capabilities like auto-sharding, read/write quorum, and cross-region replication.
He also reviewed the design of the backend message routing infrastructure which uses Kafka server to decouple message senders from receivers. Message processing is responsible for queuing, routing, and delivering the messages. The solution uses the „Fire and Forget“ approach to message delivery and there are different queues for different priorities of the messages. They also run multiple message processor instances in parallel to scale the message processing throughput.
Zuul Push supports custom authentication policies so you can authenticate using cookies, JSON Web Tokens ( JWT) or some custom scheme. Aroskar discussed the operational aspects of Zuul Push and how they managed long lived stable connections. They use Amazon’s Application Load Balancer ( ALB) to manage the push cluster.
Zuul Push framework is based on Zuul which is Netflix’s API gateway and all of the Netflix HTTP API traffic passes through Zuul. If you are interested in learning more about Zuul Push notificaitons framework, checkout their website and the sample Java application which shows how to use WebSockets as well as SSE to enable push messaging for Zuul.
