<p>As with any outage communications, a non-technical reader will be primarily looking to understand:</p>

<ul>
<li>How long was it?</li>
<li>How bad was it?</li>
</ul>

<p>Amazon CloudWatch metrics provides the following <a href="http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/sqs-metricscollected.html" rel="noreferrer">metrics for SQS queues</a> that can help answer these questions:</p>

<ul>
<li><strong>NumberOfMessagesSent:</strong> The number of messages added to a queue.</li>
<li><strong>NumberOfMessagesReceived:</strong> The number of messages returned by calls to the ReceiveMessage API action.</li>
<li><strong>ApproximateNumberOfMessagesVisible:</strong> The number of messages available for retrieval from the queue.</li>
</ul>

<p>When graphed correctly, these metrics can be powerful visual aides in describing queue processing delays. Here are a couple of examples from an outage I experienced where the job's capacity to process queue messages was severely degraded:</p>

<h2>NumberOfMessagesSent &amp; NumberOfMessagesReceived</h2>

<ul>
<li><strong>Graph Type:</strong> Line Graph</li>
<li><strong>Statistic:</strong> Sum</li>
<li><strong>Period:</strong> 5-minutes</li>
</ul>

<p><a href="https://i.sstatic.net/3xFKQ.png" rel="noreferrer"><img src="https://i.sstatic.net/3xFKQ.png" alt="NumberOfMessagesSent &amp; NumberOfMessagesReceived"></a></p>

<p>This graphs the contrast between messages sent and received, which helps isolate which processing component is responsible for the delay. In this graph, the received metric drops dramatically while the sent metric continues on its normal trend, so we can infer that the issue lies in the queue reading component instead of the queue writing component.</p>

<p><strong>Does this answer how long/how bad event was?</strong> Yes; describes processes impacted over time.</p>

<h2>NumberOfMessagesReceived &amp; ApproximateNumberOfMessagesVisible</h2>

<ul>
<li><strong>Graph Type:</strong> Stacked Area Graph </li>
<li><strong>Statistic:</strong> Sum</li>
<li><strong>Period:</strong> 5-minutes</li>
</ul>

<p><a href="https://i.sstatic.net/An7Dw.png" rel="noreferrer"><img src="https://i.sstatic.net/An7Dw.png" alt="NumberOfMessagesReceived &amp; ApproximateNumberOfMessagesVisible"></a></p>

<p>This graphs the queue depth on top of messages received, which helps show how far the queue backed up and how it recovered. In this graph, we can see that the queue depth backed up dramatically while the queue reading component was having issues, and began recovering when the queue reading component began reading messages again.</p>

<p><strong>Does this answer how long/how bad event was?</strong> Yes; describes messages impacted over time.</p>

<hr>

<h2>Graphing Discussion</h2>

<p>In both graphs, queue processing can generally be considered healthy when the lines overlap, and unhealthy when the lines diverge.  This is an easy pattern to teach to a non-technical team member, and can help them to quickly dissemenate where and how there are problems when presented with these graphs.</p>

<p>To further communicate specific points in graphs, you can simply annotate them:</p>

<p><a href="https://i.sstatic.net/kumc6.png" rel="noreferrer"><img src="https://i.sstatic.net/kumc6.png" alt="Both previous graphs with annotations."></a></p>

<p><strong>Graphing Tips:</strong></p>

<ul>
<li>Label units and axes.</li>
<li>Use consistent colors for matching metrics across graphs.  Note that NumberOfMessagesReceived is orange in both graphs; this will help visualize the same metric across different graphs.</li>
<li>Vertically align graphs that describe similar metrics so that they are easier to compare across time.</li>
</ul>

<p><sub><sub><strong>Note:</strong> I've formatted these graphs for presentation on StackExchange, so these aren't necessarily how I would present them in an outage post-mortem. I have explicitly removed values from the left axis here to obscure them from StackExchange; you'll want to keep them in your post-mortems.</sub></sub>  </p>

<hr>

<h2>Additional tips</h2>

<ul>
<li><strong>Empower Your Team:</strong> After training your team members to read these graphs, there is no reason keep them hidden away. Consider setting up a <a href="http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Dashboards.html" rel="noreferrer">CloudWatch Dashboard</a> and giving your non-technical team members <a href="http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/iam-identity-based-access-control-cw.html#read-only-access-example-cw" rel="noreferrer">read-only IAM access to CloudWatch</a>, so that they can view these graphs any time.</li>
<li><strong>Set Up Notifications:</strong> Consider setting up <a href="http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html" rel="noreferrer">Cloudwatch Alarms</a> based on the ApproximateNumberOfMessagesVisible metric if it exceeds some agreed-upon high value, and subscribe team members to notify them of potential issues. Cloudwatch Alarms have description fields that are sent along with the notification emails -- make sure to include a human-readable description to help your non-technical members disseminate the alarm.</li>
<li><strong>Explore Other Data:</strong> Per <a href="https://devops.stackexchange.com/questions/370/how-can-i-communicate-queue-based-processing-delays-to-non-technical-team-member/371#comment488_371">Evgeny's comment</a>, explore other data beyond what CloudWatch provides and think about how you can convey that data to your team.  His example of using message lifetime in the queue to create a histogram is a great example of this creative thinking, and can be accomplished by logging both the message send and message receive times in your application.  You can get the message Sent Timestamp via the SentTimeStamp Attribute on each <a href="http://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_Message.html" rel="noreferrer">queue message</a> of the ReceiveMessage API response.  More details <a href="http://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html" rel="noreferrer">here</a>.</li>
</ul>

CC BY-SA 3.0
2017-03-06T05:37:18.097Z
It is also very useful to look at data from different view points, not just those provided by CloudWatch. For example if you can show a histogram of how *long* each message stays in the queue, showing that some messages stay for X time while others stay for X*2 time. And during outages the histogram moves its high points towards X*4 or something ... extremely powerful to see.
Evgeny Zislis
CC BY-SA 3.0
2017-03-06T05:38:00.913Z
Also, just want to say: this is one absolutely amazing answer.
Evgeny Zislis
CC BY-SA 3.0
2017-03-06T17:05:12.580Z
Thanks @Evgeny! That is a great idea and I've added another tip to the answer based on it, with credit to your comment.
Anthony Neace
CC BY-SA 3.0
2017-03-06T03:34:46.890Z
2017-03-06T17:04:53.433Z
2017-04-13T13:01:29.297Z
Anthony Neace
<p>I am responsible for a set of SQS queue-processing jobs with a scaling policy on the <code>ApproximateNumberOfMessagesVisible</code> CloudWatch metric.  These jobs can fail to keep up with the amount of messages sent for any number of reasons:</p>

<ul>
<li>Service degradation reduces capacity of messages able to be processed.</li>
<li><code>AutoScaling</code> max limit reached while queue depth continues to rise.</li>
<li>S3 Outage impacts other dependent AWS services (<code>AutoScaling</code> service) that the queue processing job uses to keep up with demand.</li>
</ul>

<p>When discussing outages with non-technical team members, I would like to communicate specific delays of queue processing that can translate into customer-visible degradations.  How can I do this with SQS queues?</p>

<amazon-web-services><amazon-sqs><amazon-cloudwatch><post-mortem>
How to communicate queue-based processing delays to non-technical team members?
Anthony Neace
# How to communicate queue-based processing delays to non-technical team members?

(Asked by: username_0 on 2017-03-06)

I am responsible for a set of SQS queue-processing jobs with a scaling policy on the `ApproximateNumberOfMessagesVisible` CloudWatch metric. These jobs can fail to keep up with the amount of messages sent for any number of reasons:

* Service degradation reduces capacity of messages able to be processed.
* `AutoScaling` max limit reached while queue depth continues to rise.
* S3 Outage impacts other dependent AWS services (`AutoScaling` service) that the queue processing job uses to keep up with demand.

When discussing outages with non-technical team members, I would like to communicate specific delays of queue processing that can translate into customer-visible degradations. How can I do this with SQS queues?

**Tags:** amazon-web-services, amazon-sqs, amazon-cloudwatch, post-mortem


---

## 1 answers

### Answer by username_0 on 2017-03-06. Score: 16 | Accepted answer

As with any outage communications, a non-technical reader will be primarily looking to understand:

* How long was it?
* How bad was it?

Amazon CloudWatch metrics provides the following [metrics for SQS queues](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/sqs-metricscollected.html) that can help answer these questions:

* **NumberOfMessagesSent:** The number of messages added to a queue.
* **NumberOfMessagesReceived:** The number of messages returned by calls to the ReceiveMessage API action.
* **ApproximateNumberOfMessagesVisible:** The number of messages available for retrieval from the queue.

When graphed correctly, these metrics can be powerful visual aides in describing queue processing delays. Here are a couple of examples from an outage I experienced where the job's capacity to process queue messages was severely degraded:

## NumberOfMessagesSent & NumberOfMessagesReceived

* **Graph Type:** Line Graph
* **Statistic:** Sum
* **Period:** 5-minutes

[![NumberOfMessagesSent & NumberOfMessagesReceived](https://i.sstatic.net/3xFKQ.png)](https://i.sstatic.net/3xFKQ.png)

This graphs the contrast between messages sent and received, which helps isolate which processing component is responsible for the delay. In this graph, the received metric drops dramatically while the sent metric continues on its normal trend, so we can infer that the issue lies in the queue reading component instead of the queue writing component.

**Does this answer how long/how bad event was?** Yes; describes processes impacted over time.

## NumberOfMessagesReceived & ApproximateNumberOfMessagesVisible

* **Graph Type:** Stacked Area Graph
* **Statistic:** Sum
* **Period:** 5-minutes

[![NumberOfMessagesReceived & ApproximateNumberOfMessagesVisible](https://i.sstatic.net/An7Dw.png)](https://i.sstatic.net/An7Dw.png)

This graphs the queue depth on top of messages received, which helps show how far the queue backed up and how it recovered. In this graph, we can see that the queue depth backed up dramatically while the queue reading component was having issues, and began recovering when the queue reading component began reading messages again.

**Does this answer how long/how bad event was?** Yes; describes messages impacted over time.

---

## Graphing Discussion

In both graphs, queue processing can generally be considered healthy when the lines overlap, and unhealthy when the lines diverge. This is an easy pattern to teach to a non-technical team member, and can help them to quickly dissemenate where and how there are problems when presented with these graphs.

To further communicate specific points in graphs, you can simply annotate them:

[![Both previous graphs with annotations.](https://i.sstatic.net/kumc6.png)](https://i.sstatic.net/kumc6.png)

**Graphing Tips:**

* Label units and axes.
* Use consistent colors for matching metrics across graphs. Note that NumberOfMessagesReceived is orange in both graphs; this will help visualize the same metric across different graphs.
* Vertically align graphs that describe similar metrics so that they are easier to compare across time.

**Note:** I've formatted these graphs for presentation on StackExchange, so these aren't necessarily how I would present them in an outage post-mortem. I have explicitly removed values from the left axis here to obscure them from StackExchange; you'll want to keep them in your post-mortems.

---

## Additional tips

* **Empower Your Team:** After training your team members to read these graphs, there is no reason keep them hidden away. Consider setting up a [CloudWatch Dashboard](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Dashboards.html) and giving your non-technical team members [read-only IAM access to CloudWatch](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/iam-identity-based-access-control-cw.html#read-only-access-example-cw), so that they can view these graphs any time.
* **Set Up Notifications:** Consider setting up [Cloudwatch Alarms](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/AlarmThatSendsEmail.html) based on the ApproximateNumberOfMessagesVisible metric if it exceeds some agreed-upon high value, and subscribe team members to notify them of potential issues. Cloudwatch Alarms have description fields that are sent along with the notification emails -- make sure to include a human-readable description to help your non-technical members disseminate the alarm.
* **Explore Other Data:** Per [Evgeny's comment](https://devops.stackexchange.com/questions/370/how-can-i-communicate-queue-based-processing-delays-to-non-technical-team-member/371#comment488_371), explore other data beyond what CloudWatch provides and think about how you can convey that data to your team. His example of using message lifetime in the queue to create a histogram is a great example of this creative thinking, and can be accomplished by logging both the message send and message receive times in your application. You can get the message Sent Timestamp via the SentTimeStamp Attribute on each [queue message](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_Message.html) of the ReceiveMessage API response. More details [here](http://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ReceiveMessage.html).

> **Comments:**
> **username_1**: It is also very useful to look at data from different view points, not just those provided by CloudWatch. For example if you can show a histogram of how *long* each message stays in the queue, showing that some messages stay for X time while others stay for X*2 time. And during outages the histogram moves its high points towards X*4 or something ... extremely powerful to see.  
> **username_1**: Also, just want to say: this is one absolutely amazing answer.  
> **username_0**: Thanks @Evgeny! That is a great idea and I've added another tip to the answer based on it, with credit to your comment.  

---

