Runbook recipes

The following are recommended runbook architectures for the categories outlined in Runbook Best Practices:

In the runbook recipes steps are formatted as Step Category > Step Description.  Unless otherwise noted, the recommended default step execution rules are: Always execute and Execute automatically.  You may wish to modify this according to your business rules.

Note: You will need to have the desired integration configured before you can add an integration-driven runbook step.  Runbook conditions are available to paid FireHydrant plans only. If you're on a free plan, you can specify to execute those steps manually or choose not to include them.

Default Runbook

Execution rule:  Always attach

Recommended Runbook Steps

SEV-1 (or equivalent) Incident Runbook

Execution rule:  Execute when Current Severity is SEV-1 (or equivalent) 

Recommended Runbook Steps

  • Remind responders to provide high frequency internal communication updates. For example:
  • Assign the appropriate incident roles.
  • Publish incident status updates to the internal status page.
    • Step: FireHydrant Status Page > Publish to status page 
      • Conditions & Scheduling: Automatically execute the step when the Current Milestone is [insert key milestones] (e.g. Acknowledged, Investigating, Mitigated, etc.)
  • Publish the incident and send updates to the external status page. (e.g. Statuspage.io)
  • Attach a task list for each critical assigned role.
  • Notify other relevant Slack channels with periodic incident status updates.
    • Prerequisite: The FireHydrant app must be added to the Slack channels for FireHydrant to post your message. Please make sure to /invite @FireHydrant to any channels you list in the configuration.
    • Step: Slack > Notify channel with a custom message
      • Add relevant team channels (e.g. #support)
      • Conditions & Scheduling
        • Current milestone is one of: Started, Detected, Acknowledged, Investigating, Identified, Mitigated
        • Execute automatically every 30 minutes
  •  

SEV-2 (or equivalent) Incident Runbook

Execution rule:  Execute when Current Severity is SEV-2 (or equivalent) 

Recommended Runbook Steps

  • Remind responders to provide medium frequency internal communication updates. For example:
  • Publish incident status updates to the internal status page.
    • Step: FireHydrant Status Page > Publish to status page
      • Conditions and Scheduling: Automatically execute the step when the Current Milestone is [insert key milestones] (e.g. Acknowledged, Investigating, Mitigated, etc.)
  • Attach a general task list for the incident.
  • Notify other relevant Slack channels with periodic incident status updates until the incident is resolved.
    • Prerequisite: The FireHydrant app must be added to the Slack channels for FireHydrant to post your message. Please make sure to/invite @FireHydrantto any channels you list in the configuration.
    • Step: Slack > Notify channel with a custom message
      • Add relevant team channels (e.g. #support)
      • Condition and scheduling:
        • Execute automatically
        • Repeat every 60 minutes
        • When the Current milestone is one of: Started, Detected, Acknowledged, Investigating, Identified, Mitigated
  •  

Advanced Runbooks

Service-based Runbook

  • Add a team
    • Step: FireHydrant > Assign A Team
  • Page out to an alerting provider
  • Notify incident channel of available documentation
  • Tasklist with specific steps to be taken to restore the service based on condition
  • Run a custom script
    • Step: FireHydrant > Script
      • Add your description and custom script to the runbook step configuration.

Team-based Runbook

  • Notify a team’s home channel of engagement
    • Step: Slack > Notify channel
      • Comma Separated Channels: [#team-channel]
  • Add a Role
    • Step: FireHydrant > Assign a Role
      • Add the role and user to the runbook step configuration.
  • Email Manager
    • Firehydrant > Send an email notification
      • Add the manager's email address, subject, and message to the runbook step configuration.
  • Reminder to team’s home channel for long-running incidents
    • Step: Slack > Notify incident channel with a custom message
      • Your Message: [Friendly reminder to post an update on the incident status page(s)]
      • Action button: Post an update
      • Comma Separated Channels: [#team-channel]
    • Conditions and Scheduling:
      • Execute automatically
      • Repeat every 60 minutes
      • When the Time since incident opened: greater than 5 hours
  • Tasklist

Customer Facing Issue Runbook

  • Let the customer support team know an incident has been opened.
  • Send reminders to update customer-facing information while the incident is open.
    • Slack > Notify incident channel with a custom message
      • Your Message: [Friendly reminder to post an update to the external status page.]
      • Action button: Post an update
      • Condition and scheduling:
        • Execute automatically
        • Repeat every 60 minutes
        • When the Current milestone is one of: Started, Detected, Acknowledged, Investigating, Identified, Mitigated
Was this article helpful?
0 out of 0 found this helpful

Articles in this section

See more
Contact Our Support Team
Got a Question or Suggestion? We're here to help!
Follow us on Twitter
Get the latest news and updates first