🔥Fire Protocol: What to do if POWR.io is down or there's another Fire

Step #1: Identify Priority

If powr.io is down (check in multiple browser to verify), it's definitely a Fire 🔥

What else qualifies as a fire?

The product is completely unusable (for example, settings don't load, app view does not load on users' websites)
A core feature of a product is non-functional (for examples, payments not going through, form submissions not submitting, email alerts not sending)
The issue affects all users or at least a large subset of paying users (for example, all Business users)

Not sure if something's a fire? Check the linked Priority Framework document.

Step #2: Create Master/Problem Ticket

Create a Master Ticket with at least 5 incident tickets linked to it in Zendesk, following the instructions in our internal KB.

A description will be required for the ticket, make it short and clear.

Step #3: Get back to the affected users

Use the macro "Incident / Fire - first contact" to respond, link with the related Problem/Master ticket that was just created and submit the response as On-hold.

Repeat the same process for each ticket. All affected tickets should be responded within 5mns after identifying the incident and creating the Master.

Step #4: Post in #incident channel

Post in #incident channel in Slack. Don't just post, "Forms aren't submitting!". Provide as much details as possible, and include link to the Master Ticket.

Also include details about troubleshooting steps you have already gone through, such as "I tested in free and paid forms and the issue is happening in every form I tested. The responses do not appear on the dashboard and I see an error in the Console when clicking Submit (screenshot of error included."

It's recommended to use AI help to structure it better for Engineers to have a clearer understanding of the incident.

Step #5: Alert Support

Post in #secretsupport channel to let the team know there is an issue. Link to the Master Ticket you created and ask everyone to apply the steps #3 and #4.

Do not get distracted by other tickets, focus on just the ones related to the issue at hand so we can be sure we have all related tickets responded and linked as fast as possible while the engineering team is working to push a fix.

Step #6: Check in with engineers

Check #incident to see if the engineering team is actively working on resolving the issue. Join the call so you know what the status of the issue is.

Step #7: Call and text engineers

If no one has responded to you in #incident within 5-10 minutes and all of POWR is down or the incident is highly urgent and no one is online, start CALLING engineers on their phones.

How to call: Their phone numbers are in Airtable in the "Employees" table. Most engineers also have their number in their Slack profile.

Use your own phone, Zendesk or Google Voice, to make an international call. POWR will reimburse you if you have to pay for the call.

Who to call: Call any engineer with the title "Senior Software Engineer" in the Employees Airtable.

If the issue is more specific, call a Senior Software Engineer in the team that is likely to be in charge of the feature that is broken, or even our CEO (Brent, Juan Varela, Alikhan, Anuarbek, Puru, Praneeta).

Don't obsess over calling the right person too much, as long as at least one senior engineer is aware and looking into it, that's great. Have the engineer guide you through what to do next.

IMPORTANT: Some of us might not answer the phone if we don't recognize the number, so feel free to send a Whatsapp text telling the engineer who you are and why you are reaching out.

Example: Text: "Hey Brent, this is Rich. POWR is down. Please come online."

CALL AT LEAST 3 TIMES. If the engineer is asleep and/or has notifications turned off, they might not hear the call until the 3rd attempt (that's when most phones go into emergency mode).

Step #8: Communicate, Communicate, Communicate!

Communicate updates to the Support team and engineering team. Make sure there is never a gap between information known to the engineering team and information known to the support team.

Please also don't just quote engineers when sending an update to support. What makes sense to an engineer does not automatically make sense to a support agent. Translate the information as best as you can so we are all clear on what the status is.

DON'T STOP COMMUNICATING UNTIL THE ISSUE IS FULLY RESOLVED!

Step #9: Documentation

At least one engineer dealing with the issue is in charge of creating a document with a timeline of events.

Please make sure Dupport is represented in that timeline. Some of the things we want to include:

When did the first support ticket come in about this issue?
How many tickets did we get in total?
Were users generally thankful that we jumped on the issue fast or are people generally angry and upset?

Step #10: Follow up, Verify & Resolve

Follow up:

If the issue is still not fixed within 2 hours after the first contact, send a follow-up on for each ticket separately with the macro "incident/fire: follow-up".

Verify:

Once the engineering team confirms that their fix is live, verify that things are working for you.

Pick 5 random tickets (out of the ones linked to the Master Ticket). Verify that things are working in the user's app.

NEVER TELL A USER SOMETHING IS WORKING WHEN YOU HAVEN'T CHECKED FIRST.

Nothing is more frustrating for a user than if support says everything's dandy now and the user is still seeing the bug. If we have tens of linked tickets, we of course can't check every single one of them. But at the very least, check a handful to make sure we're not sending out incorrect information.

If the issue is NOT resolved, let the engineering team know IMMEDIATELY.

Resolve:

Once you've verified that things are working, send a general update by going to the Master ticket, enter your response there with the macro "incident/fire: final update" and submit as solve.

Do not solve the Master Ticket yourself. Coordinate with the Support Manager on this if he's available.