<p>There are a number of things that I think it's important to know before going on-call:</p>

<ol>
<li><strong>How to triage an issue.</strong>  There should be a defined list of priorities with examples so that you can identify how serious a problem is.</li>
<li><strong>What the priority levels mean.</strong>  Can you ack this and then go back to sleep?</li>
<li><strong>Who to contact post-triage.</strong> The on-caller's primary responsibility is <em>triage</em>, not fixing.  Fixing the problem is good, too, but once you've triaged the issue, then it's up to you to rope in other folks from other teams and manage communications between them.</li>
</ol>

<p>People jumping straight into fixing problems before triaging is the most common mistake I've seen (because, after all, we're used to fixing things and think of that as our job).  But communications are often vital, and doing things in the wrong order can mean customers are left in the dark because you got caught up in trying to fix a problem before letting your support people know there <em>is</em> a problem.</p>

CC BY-SA 4.0
2018-11-21T23:51:38.363Z
2018-11-21T23:51:38.363Z
Xiong Chiamiov
<p>Here's my two pieces of advice:</p>
<ol>
<li><p><strong>Work out schedules so that you're not on-call 24 hours a day</strong>.  There should be a hand-off between daytime operations and after-hours support. I encourage people to split half their regular time between support weeks and project weeks; after-hours support should come during a project week.</p>
</li>
<li><p><strong>Get in the habit of taking notes during an incident.</strong> That's the most important thing; even more important than &quot;fixing&quot; the issue.  If bandaging the service requires you to stop taking notes, then it's time to call in additional support.  <strong>There should always be someone taking notes.</strong></p>
</li>
</ol>

CC BY-SA 4.0
2021-05-05T01:07:47.090Z
2021-05-05T01:07:47.090Z
Stuart Ainsworth
<p>[[ Based on tags and the wording this sounds like a PagerDuty-specific question, but it could apply to any tech or <a href="/questions/tagged/monitoring" class="post-tag" title="show questions tagged &#39;monitoring&#39;" rel="tag">monitoring</a> stack in my mind.... PagerDuty is just so dominant in SF-based jobs I've seen that is like tech Kleenex.... ]]</p>

<p>I've been searching in my spare time for a few months for an answer to: <strong>what do you tell a new recruit to pager duty</strong> (or PagerDuty or <a href="/questions/tagged/pagerduty" class="post-tag" title="show questions tagged &#39;pagerduty&#39;" rel="tag">pagerduty</a>)???  And I don't mainly mere "install the app and prey" <a href="/questions/tagged/training" class="post-tag" title="show questions tagged &#39;training&#39;" rel="tag">training</a>, but those are key parts of it.  I'm more looking for generic, general, globalizable advice like:</p>

<ul>
<li>ask for help as early as possible</li>
<li>don't go sleep deprived "too" many nights in a row <code>~3</code></li>
<li>know who is likely to answer with low latency and who will notice work again at Monday 9am <code>DOH!</code></li>
<li>it is easier to find answer on StackExchange than random forums at 3am <code>free plug</code></li>
<li>triage: how big of a fire is it?  how many fire trucks do you call in?  Etc.</li>
<li>when to merge incidents</li>
<li>when to snooze incidents... but that is so scary!!  "<em>You don't get fired for snoozing here?</em>"  <code>hahaha.</code></li>
<li>good notification settings</li>
<li>good phone/volume settings for getting woken up, even if you sleep like a log.</li>
<li>scheduling and calendar layers</li>
<li>rules of thumb and recommended tools</li>
<li>know who to call for what subsystem (varies based on locality and org) but using a wiki or Confluence is often part of that</li>
</ul>

<p>Maybe I'm searching wrong.  I'm open to meta-suggestions for better ways to find this needle in the haystack.  Searching for "starting pagerduty" makes sense to me, but it gets results of how to setup these things programmatically or technically, not how to instruct the humans in dealing with sorts of things.  <a href="https://support.pagerduty.com/docs/quick-start-guide" rel="nofollow noreferrer">PagerDuty does try</a> to address this, but it doesn't include the breadth of responsibilities that the job entails to me.</p>

<p>I also admit that I'm asking for everything and the kitchen sink and the city water supply for a bonus round.  Getting little pieces of this as links and pointing folks at it would be a huge improvement over what I've found so far.  I doubt anybody has done all of this in one doc or would ever want to do so.</p>

<p>I've actually already turned this question into an internal class.  I'm happy to get permission from management to share my fun version of this after I pull out the proprietary bits.  I'm still holding out for it having been done at some point before and I'm just too lazy to go that far into the search results.  :)</p>

CC BY-SA 4.0
2018-10-21T09:10:30.857Z
I'm very sorry if this is too chatty for your taste.  **Feel free to edit it to make it more "professional".** I wrote this after getting woken up for a production incident so I'm not self-editing as well as I might otherwise.  :-)
chicks
CC BY-SA 4.0
2018-10-27T16:52:10.857Z
Ian in DevOps.Slack suggested https://response.pagerduty.com/ which is a great resource.  I knew that PagerDuty had done videos about their process, but having it in readable form is a huge asset.
chicks
CC BY-SA 4.0
2018-11-21T23:52:18.007Z
I would describe this question as being about "going on-call".
Xiong Chiamiov
CC BY-SA 4.0
2018-11-21T23:54:36.497Z
That's a good suggestion.  I tried to work it into the title.
chicks
CC BY-SA 4.0
2021-05-05T02:42:57.843Z
This seems super wordy, and not really asking a question.
Stuart Ainsworth
CC BY-SA 4.0
2021-05-05T02:53:39.977Z
It is super wordy, but slicing out all of the fluff makes it less fun to read.  Thanks for trying to edit it into shape, but throwing out the puns seemed overly harsh.  And thanks for your answer.  +1
chicks
CC BY-SA 4.0
2021-05-05T03:02:34.193Z
ok, kudos for positive attitude, but how can we make this more structured and clearer as to what you're asking?   IS it advice specific to PagerDuty the product, or just on call? Are you asking for things to search or looking for specific guidance?
Stuart Ainsworth
CC BY-SA 4.0
2021-05-05T11:33:44.033Z
I'm looking for general oncall advice.  I'm referencing PagerDuty because they are the most often used app for this and they have the most useful docs for this that I have found.
chicks
<monitoring><pagerduty><training>
What do you tell someone new to going oncall or PagerDutyl?
chicks
# What do you tell someone new to going oncall or PagerDutyl?

(Asked by: username_0 on 2018-10-21)

[[ Based on tags and the wording this sounds like a PagerDuty-specific question, but it could apply to any tech or [monitoring](/questions/tagged/monitoring "show questions tagged 'monitoring'") stack in my mind.... PagerDuty is just so dominant in SF-based jobs I've seen that is like tech Kleenex.... ]]

I've been searching in my spare time for a few months for an answer to: **what do you tell a new recruit to pager duty** (or PagerDuty or [pagerduty](/questions/tagged/pagerduty "show questions tagged 'pagerduty'"))??? And I don't mainly mere "install the app and prey" [training](/questions/tagged/training "show questions tagged 'training'"), but those are key parts of it. I'm more looking for generic, general, globalizable advice like:

* ask for help as early as possible
* don't go sleep deprived "too" many nights in a row `~3`
* know who is likely to answer with low latency and who will notice work again at Monday 9am `DOH!`
* it is easier to find answer on StackExchange than random forums at 3am `free plug`
* triage: how big of a fire is it? how many fire trucks do you call in? Etc.
* when to merge incidents
* when to snooze incidents... but that is so scary!! "*You don't get fired for snoozing here?*" `hahaha.`
* good notification settings
* good phone/volume settings for getting woken up, even if you sleep like a log.
* scheduling and calendar layers
* rules of thumb and recommended tools
* know who to call for what subsystem (varies based on locality and org) but using a wiki or Confluence is often part of that

Maybe I'm searching wrong. I'm open to meta-suggestions for better ways to find this needle in the haystack. Searching for "starting pagerduty" makes sense to me, but it gets results of how to setup these things programmatically or technically, not how to instruct the humans in dealing with sorts of things. [PagerDuty does try](https://support.pagerduty.com/docs/quick-start-guide) to address this, but it doesn't include the breadth of responsibilities that the job entails to me.

I also admit that I'm asking for everything and the kitchen sink and the city water supply for a bonus round. Getting little pieces of this as links and pointing folks at it would be a huge improvement over what I've found so far. I doubt anybody has done all of this in one doc or would ever want to do so.

I've actually already turned this question into an internal class. I'm happy to get permission from management to share my fun version of this after I pull out the proprietary bits. I'm still holding out for it having been done at some point before and I'm just too lazy to go that far into the search results. :)

**Tags:** monitoring, pagerduty, training

> **Comments:**
> **username_0**: I'm very sorry if this is too chatty for your taste.  **Feel free to edit it to make it more "professional".** I wrote this after getting woken up for a production incident so I'm not self-editing as well as I might otherwise.  :-)  
> **username_0**: Ian in DevOps.Slack suggested https://response.pagerduty.com/ which is a great resource.  I knew that PagerDuty had done videos about their process, but having it in readable form is a huge asset.  
> **username_1**: I would describe this question as being about "going on-call".  
> **username_0**: That's a good suggestion.  I tried to work it into the title.  
> **username_2**: This seems super wordy, and not really asking a question.  
> **username_0**: It is super wordy, but slicing out all of the fluff makes it less fun to read.  Thanks for trying to edit it into shape, but throwing out the puns seemed overly harsh.  And thanks for your answer.  +1  
> **username_2**: ok, kudos for positive attitude, but how can we make this more structured and clearer as to what you're asking?   IS it advice specific to PagerDuty the product, or just on call? Are you asking for things to search or looking for specific guidance?  
> **username_0**: I'm looking for general oncall advice.  I'm referencing PagerDuty because they are the most often used app for this and they have the most useful docs for this that I have found.  

---

## 2 answers

### Answer by username_1 on 2018-11-21. Score: 2 | Accepted answer

There are a number of things that I think it's important to know before going on-call:

1. **How to triage an issue.** There should be a defined list of priorities with examples so that you can identify how serious a problem is.
2. **What the priority levels mean.** Can you ack this and then go back to sleep?
3. **Who to contact post-triage.** The on-caller's primary responsibility is *triage*, not fixing. Fixing the problem is good, too, but once you've triaged the issue, then it's up to you to rope in other folks from other teams and manage communications between them.

People jumping straight into fixing problems before triaging is the most common mistake I've seen (because, after all, we're used to fixing things and think of that as our job). But communications are often vital, and doing things in the wrong order can mean customers are left in the dark because you got caught up in trying to fix a problem before letting your support people know there *is* a problem.


---

### Answer by username_2 on 2021-05-05. Score: 1

Here's my two pieces of advice:

1. **Work out schedules so that you're not on-call 24 hours a day**. There should be a hand-off between daytime operations and after-hours support. I encourage people to split half their regular time between support weeks and project weeks; after-hours support should come during a project week.
2. **Get in the habit of taking notes during an incident.** That's the most important thing; even more important than "fixing" the issue. If bandaging the service requires you to stop taking notes, then it's time to call in additional support. **There should always be someone taking notes.**


---

