Automated alerting? Escalating robo-calling? All ok, but beware the inebriated meatbag in the system
On Call It is Friday the somethingth of Marchtober. No, we’re not sure anymore either. Still, even in these troubled times there remains a crumb of comfort to be gleaned from the oopsies of others. Welcome to On Call. Cast your mind back to The Before Times. The mid 2000s, to be precise, where our protagonist, “Bob”, was gainfully employed within the walls of a major financial institution (which must remain nameless for Reasons.) Bob was at the human end of a lengthy incident response chain. Well-designed systems monitored the applications in this well-known institution and, if something happened that didn’t match expectations, an alert was generated. This alert would normally fire off a SMS to whoever was unlucky enough to be carrying the on-call phone. SLAs dictated an acknowledgement within 15 minutes, and failure to do so would provoke robo-calls. Continued failure to respond would see those robo-calls redirected to the bosses, something which would likely prove somewhat “career-altering,” as Bob put it. On the night in question, Bob had the dreaded phone. A transaction being conducted in Japanese Yen managed to burst through the maximum allowable value in the system. This was not particularly uncommon “due to a combination of the exchange rate and the size of the transactions” with which the finance house was entrusted.
