Sounding the Alarm | Customers, Etc.
Does your organization have the muscles to know how to sound the alarm when it matters?
Let’s imagine. It’s late on Friday at the end of a long week. Everyone has signed off but you’re hanging on to catch up on some admin work before calling it a day. As you finish combing through unread Slack messages, you check the customer support inbox one last time and notice an email with the subject line “URGENT! DATA LOST!!!”. The message reads:
Help! I need someone to reply ASAP. I was working in the app and the next thing I knew, the data in my account was deleted.
Although your alert level is raised, your reaction is tempered as you remember there was a user interface bug a few weeks ago that made it look like data was missing when it wasn’t. But wasn’t that fixed?
Access to user-generated content (UGC) at your company is strictly controlled, so you would need an engineer to confirm if the underlying data is in fact deleted. You don’t really want to page the entire engineering team on a Friday night. Maybe the customer just needs to refresh so they can get the bug fix that shipped several weeks ago.
You’re typing your reply to the customer to get more information when you see another ticket show up in the inbox with the subject “Um… did I just lose all my data?”.
What do you do?
When to sound the alarm
When we think of emergency situations, we often think of them as if the emergency is immediately obvious. We see a building on fire and we quickly reason, “the building is obviously on fire.”
What we forget are the moments leading up to the clarifying point where we realize that the situation is actually an emergency. We think we smell smoke—“is that really smoke?”—but even after we’re pretty sure we smell smoke, we might not see the fire. Our alert is raised, but we don’t want to be the person that yelled “Fire!” when there wasn’t actually a fire. We usually wait until we know it’s an emergency before we cause a major disruption to everyone else’s day.
In related news, did you see the story about the roller coaster in North Carolina1 that developed a crack over the weekend while guests were riding on it? Check this out:
It’s crazy. You can actually see the track separate from the support beam as passengers go screaming past. This ride at Carowinds is now shut down (of course), but what I want to talk about are the moments leading up to the ride actually being shut down. The Washington Post article shares the story of Jeremy Wagner, who was at the park and noticed the crack while waiting in the parking lot to pick up his kids (emphasis mine):
Wagner scanned the support systems of other rides to support that notion but saw nothing else like it. He turned his attention back to the Fury 325 as another car hit the turn. This time, he saw light shooting through the expanded crack before the car passed and the crack contracted. He knew something was wrong.
Wagner got out of the truck, flagged down a Carowinds employee and pointed out the crack.
“I was like, ‘Y’all need to shut this ride down. That’s bad news,’” Wagner recalled telling the man.
Unsatisfied with what he described as the man’s “lack of urgency,” Wagner went back into the park, where he spoke to several more employees. He eventually talked to someone in guest services, who asked him to send her the video he’d shot of the crack. Once he had, Wagner said, she walked away.
“The biggest thing for me — there was no sense of urgency,” he said. Carowinds did not immediately respond late Sunday night to a request for comment on Wagner’s allegation about a lack of urgency.
Wagner left, unnerved. During the hour-long drive home, his dread ate at him. He knew Fury 325 was one of the amusement park’s marquee attractions and would be heavily used over the Fourth of July weekend during the park’s 50th anniversary. He feared the pillar would fail, sending a car shooting off the tracks and diving into a crowd below. If tragedy struck and he hadn’t done more to shut down the ride, Wagner knew he would blame himself and forever wonder whether he could have prevented it.
On the trip home, Wagner called Carowinds but got the park’s automated phone system, he said. After getting back, he called the fire department in Carowinds’s jurisdiction. Someone there told Wagner, who is a volunteer firefighter, that he had a direct line to the park’s security people and would contact them. Ten minutes later, the firefighter called back to tell Wagner that the amusement park had shut down the ride.
Relief replaced Wagner’s dread.
When you watch the video embedded in the news report, the emergency was obvious, right? But what about the multiple park employees that saw the exact same video and didn’t register that the situation was urgent, that there was a possible emergency situation on their hands?
What’s your “sound the alarm” procedure?
We like to think we’ll recognize a disaster when we see it, but that’s not always the case. I’m sure that Carowinds park employees are all on high alert right now. If someone shows them a video of a ride with a crack in it, they’re going to immediately escalate and shut the ride down. You can probably picture what the corporate email to shift managers looked like. In hindsight, it’s obvious.
What’s actually needed is training for when the situation isn’t obvious, but the potential for disaster is very high. “I don’t know if this is an emergency, but if it is, it’s a big deal.” In those situations, you need a very clear process for who to call and what to do.
It’s important to write down and clarify your escalation procedures because you don’t want team members to be surprised when someone raises the alarm. If the person receiving the escalation acts bothered and goes “Ugh, what?” before looking into it, people are going to be less likely to escalate things they think there could be an emergency.
Escalate quickly, deescalate quickly
If you define a procedure for escalation in case of emergency, your process should also clarify how to deescalate the situation. The point of escalation isn’t to have an emergency, it’s to determine as quickly as possible if there’s an emergency—and respond accordingly, if so—but if there’s not an emergency, you should quickly deescalate.
In the case of the roller coaster, imagine if their procedure had been to immediately shut down the ride to investigate, urgently have an engineer review the evidence, and then decide whether there was an actual emergency or if the situation could be deescalated.
Defining when a situation can be deescalated also forces clarity about what is an actual emergency. Emergencies in a business context aren’t always obvious. If one customer can’t access their data, is that an emergency? What if its ten customers? The threshold for what is and isn’t an emergency is going to differ from business to business. It’s important to define what an escalation looks like for your company.
Practice your escalation procedure
Do you remember when you had fire drills when you were in school? You don’t want to wait until there’s an actual fire to have everyone learn the process. Nobody wants to learn in a fire; they want to act! That’s why it’s important to teach the process and practice it before you have an emergency.
In a business context, you can do this through an interactive table-top exercise. Come up with a scenario, sit around a table (virtual or otherwise), and play through it. You’ll need to let internal teams know you’re having a drill, but then you can run through your process as if it’s real. Practice getting an urgent email. Practice having the support team send an email to Pager Duty (where was that email address written down?). Practice making sure the Pager Duty escalations actually work. And lastly, practice deescalation.
If you work for a startup and have never had an emergency that affected your business, I’m going to have a hard time convincing you that you need to prioritize coming up with an emergency escalation procedure. But after you have your first real emergency—which you most certainly will—circle back and define your process, train your team, and practice it to make sure it works.
I’m linking to the Washington Post version of the article because I intend to quote from it, but if you hit a paywall, this version from the Charlotte Observer will give you the gist and let you watch the video.