Saurabh Pathak, engineering manager at Twitter, discussed the sites notification architecture at QCon London 2017. This included highlighting key challenges which are unique to Twitter, such as the bimodal nature of the social network, dealing with spikes, and the requirement to serve notifications in real time.
Saurabh Pathak, engineering manager at Twitter, discussed the sites notification architecture at QCon London 2017. This included highlighting key challenges which are unique to Twitter, such as the bimodal nature of the social network, dealing with spikes, and the requirement to serve notifications in real-time.
Pathak explained that unlike a typical social network, Twitter is asymmetric. Some users can have millions of followers, where others can have less than a hundred. This makes notifications bimodal in nature, and also creates challenges in being able to serve them in real-time. For example, a popular celebrity Tweeting is going to generate a far greater load than a typical user.
Pathak explains that it’s these different types of users, combined with stringent performance requirements, that present the following architectural challenges:
Before addressing these challenges, notifications are first divided into either a push or pull based model. The notification timeline that people see when visiting their Twitter feed adopts pull, whereas SMS and email are push.
For the pull based model, Pathak explained that notifications are usually be served from a cache, as timeline generation is an expensive operation. For this, Twitter makes use of Manhatten, Twitters real-time distributed backing store, and a fork of Redis. By using caching, latency is kept to a minimum, providing a better user experience. The notifications timeline is also asynchronously replicated across data centres, the goal being that the user will see the same timeline even in the event of failover.
With push notifications, in order to deal with latency and spikes, Pathak explains that whilst they leverage horizontal scaling, they also make use of short lived caches. This helps in situations where there are millions of events for the same user.
In order to deal with heterogeneous calls, Twitter makes use of priority queues to stop important calls being blocked; a celebrity tweeting would never block somebodies login call. Also, different types of calls are profiled and grouped together, meaning if an external dependency is down then failures tend to be isolated.
Finally, Pathak concludes some key takeaways of the architecture to be:
The full talk can be watched online, and is also followed by a talk on personalised notifications from Gary Lam, staff engineer at Twitter
