<p>One way of preventing these retry storms is by using backoff mechanisms.</p>

<p>From the <a href="https://cloud.google.com/appengine/articles/scalability#backoff" rel="noreferrer">Implement backoff on retry</a> section of Google App Engine <a href="https://cloud.google.com/appengine/articles/scalability" rel="noreferrer">Designing for Scale</a> guide:</p>

<blockquote>
  <p>Your code can retry on failure, whether calling a service such as
  Cloud Datastore or an external service using URL Fetch or the Socket
  API. In these cases, you should always implement a randomized
  exponential backoff policy in order to avoid the <a href="https://wikipedia.org/wiki/Thundering_herd_problem" rel="noreferrer">thundering herd
  problem</a>. You should also limit the total number of retries and
  handle failures after the maximum retry limit is reached.</p>
</blockquote>

<p>Most of the GAE APIs already have such backoff mechanisms/policies enabled by default.</p>

CC BY-SA 4.0
2024-08-13T10:53:34.013Z
Exponential backoff won't avoid retry storm on it's own. In the case of constant 100% failing RPS, exponential backoff will just delay the storm a little bit. https://medium.com/yandex/good-retry-bad-retry-an-incident-story-648072d3cee6
Paku
CC BY-SA 3.0
2017-04-15T19:43:00.647Z
Thanks, implementing backoff mechanisms is great advice, I usually go for configurable exponential backoff using the [Transient Fault Handling Application Block](https://msdn.microsoft.com/en-us/library/hh680934(v=pandp.50).aspx).  However, through 5+ years of operational experience operating hyper-scale applications in Azure, even with exponential backoffs in place "retry storms" still happen fairly often - I've never been able to find a workable strategy for avoiding them.
Richard Slater
CC BY-SA 3.0
2017-04-15T16:44:57.980Z
2017-04-15T16:44:57.980Z
Dan Cornilescu
<p>The answer is a retry budget. E.g. <a href="https://github.com/grpc/proposal/blob/master/A6-client-retries.md#throttling-retry-attempts-and-hedged-rpcs" rel="nofollow noreferrer">grpc go client has a retry budget of 10%</a>. Meaning that if requests to a certain server are failing above a rate of 10%, then retries won't be done.</p>
<p>Excellent blog post on this topic here:
<a href="https://medium.com/yandex/good-retry-bad-retry-an-incident-story-648072d3cee6" rel="nofollow noreferrer">https://medium.com/yandex/good-retry-bad-retry-an-incident-story-648072d3cee6</a></p>

CC BY-SA 4.0
2024-08-13T10:59:19.257Z
2024-08-13T10:59:19.257Z
Paku
<p>It depends on what you're trying to avoid.</p>

<p>If you are trying to avoid any service interuption of something which is a genuinely critical service (I'm thinking in terms of "people will die if my API call is not appropriately served") the you need to just budget for the huge inefficiencies that come from vastly over provisioning dedicated resources. And yes they have to be dedicated, none of this allowing for traffic spikes stuff, multiple services spiking would thus cause an outage.</p>

<p>In the <em>far</em> more likely scenario that you're service going down would be inconvenient you can tackle the problem both from the client and server sides. Although it's worth noting that it's logically impossible to actually solve the problem of to much traffic because without processing the traffic (which consumes resources) you can't know if it's a retry, if it's a retry for a request that was successful but incorrectly handled by the client, if it's a DDOS, etc. But you <em>can</em> mitigate impact.</p>

<p>In the <strong>client code</strong> write sensible retry logic which has an upper limit and a mechanism for gracefully failing. That way you don't stick your users in an infinite loop of failing requests and you just give them an error telling them to try whatever they just did in little while.</p>

<p>For your <strong>server side</strong> infrastructure the simplest solution is to throttle. Hard limits on requests, especially if you can try and spread them logically based on your specific use case (ie. If you have a centralised service make some hard decisions, do you want to start blocking geographically distant requests which might be resulting in threads hanging server side? Or do you want to distribute your inevitable yet minor outage evenly? etc) It basically boils down to the fact that returning a 503 intentionally from a gateway is a hell of a lot cheaper than letting the request go through and sending a 504 anyway. Basically force clients to behave based on what you can currently provide and provide the correct responses so that clients can react appropriately.</p>

CC BY-SA 3.0
2017-04-18T21:28:44.177Z
2017-04-18T21:28:44.177Z
hvindin
<p>A "retry storm" is caused when clients are configured to retry a set number of times before giving up, a retry policy is necessary because of packets loss will occur in normal operation of a service.</p>

<p>Take this example:</p>

<p><a href="https://i.sstatic.net/gCgAH.png" rel="noreferrer"><img src="https://i.sstatic.net/gCgAH.png" alt="Sample Architecture"></a></p>

<p>If for example the services as a whole were scaled to support 80,000 requests per seconds and run at about 80% of capacity, a spike in traffic that caused the service to receive 101,000 requests per second would cause 1,000 of those requests to fail.</p>

<p>When the retry policies kick in, you end up with additional 1,000+ requests, depending on where the failure was detected which would push the service as a whole up to 102,000 requests per second - from there your service goes into a death spiral doubling the number of failed requests every second.</p>

<p>Other than massive over-provisioning of services beyond the projected peak transaction, which would be inefficient.  What strategies can you employ to avoid "retry storms"?</p>

CC BY-SA 3.0
2017-04-26T14:53:48.407Z
If 100kQPS is 80% of capacity, then 101kQPS should not result in 1k failures, it should result in zero failures - isn't that the point of overprovisioning?
Adrian
CC BY-SA 3.0
2017-04-26T16:12:50.883Z
@Adrian your right, it was a contrived example to explain the point - I was trying to be reductive enough to make my point clear without being overly abstract.  I have corrected the "scaled to support 100,000" to "scaled to support 80,000".
Richard Slater
<serverless><cloud><distributed-systems>
How to avoid "retry storms" in distributed services?
Richard Slater
# How to avoid "retry storms" in distributed services?

(Asked by: username_0 on 2017-04-15)

A "retry storm" is caused when clients are configured to retry a set number of times before giving up, a retry policy is necessary because of packets loss will occur in normal operation of a service.

Take this example:

[![Sample Architecture](https://i.sstatic.net/gCgAH.png)](https://i.sstatic.net/gCgAH.png)

If for example the services as a whole were scaled to support 80,000 requests per seconds and run at about 80% of capacity, a spike in traffic that caused the service to receive 101,000 requests per second would cause 1,000 of those requests to fail.

When the retry policies kick in, you end up with additional 1,000+ requests, depending on where the failure was detected which would push the service as a whole up to 102,000 requests per second - from there your service goes into a death spiral doubling the number of failed requests every second.

Other than massive over-provisioning of services beyond the projected peak transaction, which would be inefficient. What strategies can you employ to avoid "retry storms"?

**Tags:** serverless, cloud, distributed-systems

> **Comments:**
> **username_1**: If 100kQPS is 80% of capacity, then 101kQPS should not result in 1k failures, it should result in zero failures - isn't that the point of overprovisioning?  
> **username_0**: @username_1 your right, it was a contrived example to explain the point - I was trying to be reductive enough to make my point clear without being overly abstract.  I have corrected the "scaled to support 100,000" to "scaled to support 80,000".  

---

## 3 answers

### Answer by username_4 on 2017-04-18. Score: 12 | Accepted answer

It depends on what you're trying to avoid.

If you are trying to avoid any service interuption of something which is a genuinely critical service (I'm thinking in terms of "people will die if my API call is not appropriately served") the you need to just budget for the huge inefficiencies that come from vastly over provisioning dedicated resources. And yes they have to be dedicated, none of this allowing for traffic spikes stuff, multiple services spiking would thus cause an outage.

In the *far* more likely scenario that you're service going down would be inconvenient you can tackle the problem both from the client and server sides. Although it's worth noting that it's logically impossible to actually solve the problem of to much traffic because without processing the traffic (which consumes resources) you can't know if it's a retry, if it's a retry for a request that was successful but incorrectly handled by the client, if it's a DDOS, etc. But you *can* mitigate impact.

In the **client code** write sensible retry logic which has an upper limit and a mechanism for gracefully failing. That way you don't stick your users in an infinite loop of failing requests and you just give them an error telling them to try whatever they just did in little while.

For your **server side** infrastructure the simplest solution is to throttle. Hard limits on requests, especially if you can try and spread them logically based on your specific use case (ie. If you have a centralised service make some hard decisions, do you want to start blocking geographically distant requests which might be resulting in threads hanging server side? Or do you want to distribute your inevitable yet minor outage evenly? etc) It basically boils down to the fact that returning a 503 intentionally from a gateway is a hell of a lot cheaper than letting the request go through and sending a 504 anyway. Basically force clients to behave based on what you can currently provide and provide the correct responses so that clients can react appropriately.


---

### Answer by username_2 on 2017-04-15. Score: 8

One way of preventing these retry storms is by using backoff mechanisms.

From the [Implement backoff on retry](https://cloud.google.com/appengine/articles/scalability#backoff) section of Google App Engine [Designing for Scale](https://cloud.google.com/appengine/articles/scalability) guide:

> Your code can retry on failure, whether calling a service such as
> Cloud Datastore or an external service using URL Fetch or the Socket
> API. In these cases, you should always implement a randomized
> exponential backoff policy in order to avoid the [thundering herd
> problem](https://wikipedia.org/wiki/Thundering_herd_problem). You should also limit the total number of retries and
> handle failures after the maximum retry limit is reached.

Most of the GAE APIs already have such backoff mechanisms/policies enabled by default.

> **Comments:**
> **username_0**: Thanks, implementing backoff mechanisms is great advice, I usually go for configurable exponential backoff using the [Transient Fault Handling Application Block](https://msdn.microsoft.com/en-us/library/hh680934(v=pandp.50).aspx).  However, through 5+ years of operational experience operating hyper-scale applications in Azure, even with exponential backoffs in place "retry storms" still happen fairly often - I've never been able to find a workable strategy for avoiding them.  
> **username_3**: Exponential backoff won't avoid retry storm on it's own. In the case of constant 100% failing RPS, exponential backoff will just delay the storm a little bit. https://medium.com/yandex/good-retry-bad-retry-an-incident-story-648072d3cee6  

---

### Answer by username_3 on 2024-08-13. Score: 1

The answer is a retry budget. E.g. [grpc go client has a retry budget of 10%](https://github.com/grpc/proposal/blob/master/A6-client-retries.md#throttling-retry-attempts-and-hedged-rpcs). Meaning that if requests to a certain server are failing above a rate of 10%, then retries won't be done.

Excellent blog post on this topic here:
<https://medium.com/yandex/good-retry-bad-retry-an-incident-story-648072d3cee6>


---

