<p>Consider using the <a href="https://github.com/rabbitmq/rabbitmq-autocluster" rel="nofollow noreferrer">rabbitmq/rabbitmq-autocluster</a> plugin:</p>

<blockquote>
  <p>A RabbitMQ plugin that clusters nodes automatically using a number of peer discovery mechanisms:</p>
  
  <ul>
  <li><a href="https://www.consul.io/" rel="nofollow noreferrer">Consul</a>,</li>
  <li><a href="https://github.com/coreos/etcd" rel="nofollow noreferrer">etcd2</a></li>
  <li>DNS A records</li>
  <li>AWS EC2 tags</li>
  <li>AWS Autoscaling Groups</li>
  </ul>
</blockquote>

<p>There is a <a href="https://github.com/aweber/rabbitmq-autocluster/wiki/AWS%20Configuration" rel="nofollow noreferrer">fair bit of configuration</a> to plug in to get this setup including setting IAM policies and adding EC2 tags to the instances you want to be party to your cluster.</p>

<p>If you were to use AWS Autoscaling Groups then you would add the following to your <code>rabbitmq.config</code>:</p>

<pre><code>[
  {rabbit, [ ... ]},
  {autocluster, [
    {backend, aws},
    {aws_autoscaling, true},
    {aws_ec2_region, "us-west-2"}
  ]}
].
</code></pre>

<p>If you are not using AWS Autoscaling Groups you can still achieve the desired result using tags on your EC2 Instances:</p>

<pre><code>[
  {rabbit, [ ... ]},
  {autocluster, [
    {backend, aws},
    {aws_ec2_tags, [{"region", "us-west-2"}, {"service", "rabbitmq"}]},
    {aws_ec2_region, "us-east-1"},
    {aws_access_key, "..."},
    {aws_secret_key, "..."}
  ]}
].
</code></pre>

<p>With all of that said I strongly recommend using <a href="https://www.consul.io/" rel="nofollow noreferrer">Consul by HashiCorp</a> as your service discovery mechanism, in the long run, you get significantly more flexibility in terms of decoupling your parts of your system from each other.</p>

CC BY-SA 3.0
2017-03-25T22:06:11.887Z
thank you for the detailed explanation and the diagram, so AFAIK `rabbitmq/rabbitmq-autocluster plugin` will know also to remove node from the replication list once the node is down, one more thing if may I ask, I thought to start with `2-node` cluster, are you suggest to start with `3-node` cluster as you describe in your diagram with policy  ` rabbitmqctl set_policy ha-all "" '{"ha-mode":"all","ha-sync-mode":"automatic"}' ` ? or should I post it in a another question ?
Berlin
CC BY-SA 3.0
2017-03-26T08:47:05.147Z
Possibly another question as the formatting in comments isn't all that easy to use.  I have always advised clients when adopting cloud to use larger number of small instances rather than the inverse.  This is because failure is expected in the cloud, if you have a 2-node cluster and 1 fails you have lost 50% of your capacity, whereas if you have a 3-node cluster and 1 fails you only lose 33%.
Richard Slater
CC BY-SA 3.0
2017-04-01T15:13:36.257Z
I have configured my cluster using `rabbitmq/rabbitmq-autocluster plugin` and it is work pretty well however when node goes down RabbitMQ does not remove it from the replication list, any idea why ?
Berlin
CC BY-SA 3.0
2017-04-01T16:18:01.497Z
Have you enabled the Cluster Cleanup configuration option?
Richard Slater
CC BY-SA 3.0
2017-04-01T22:50:28.037Z
thanks, found it `https://github.com/aweber/rabbitmq-autocluster/wiki/General-Settings`, I will try that.
Berlin
CC BY-SA 4.0
2019-10-11T12:26:25.887Z
Links are great https://github.com/aweber/rabbitmq-autocluster/wiki/General-Settings
Oliver Salzburg
CC BY-SA 3.0
2017-03-25T18:47:05.343Z
2017-03-25T18:47:05.343Z
Richard Slater
<p>I am planning to create RabbitMQ cluster using Ansible on AWS VPC with Amazon internal load balancer as the frontend to point connections to it.</p>

<p>Any suggestion how to remove a dead node from RabbitMQ cluster based on autoscaling rule where nodes can go up and down, or if you are using spot instances?</p>

<p>When a node goes down, RabbitMQ does not remove it from the replication list automatically, I can see <code>Node not running</code> in the management UI.</p>

<p>I managed to join to the cluster a scaled instance automatically via Ansible and userdata.</p>

<p><a href="https://i.sstatic.net/2O2Tx.png" rel="noreferrer"><img src="https://i.sstatic.net/2O2Tx.png" alt="Diagram of Infrastructure"></a></p>

CC BY-SA 3.0
2017-03-25T15:33:23.517Z
@Pierre.Vriens, I changed to only 1 question, thanks.
Berlin
CC BY-SA 3.0
2017-03-25T15:35:09.747Z
Merci! You didn't destroy the other 4 (or so) questions I hope. Maybe keep them as folowup question, to the extend they are still relevant?
Pierre.Vriens
CC BY-SA 3.0
2017-03-25T15:39:20.067Z
Yep, the questions are still relevant, but this one is the most important. I will post another question later :) thanks!
Berlin
CC BY-SA 3.0
2017-03-25T18:59:45.530Z
@Berlin I drew a diagram that represents what I think you are describing, if you meant something else then please do let me know and I will adapt.
Richard Slater
CC BY-SA 3.0
2017-03-25T19:45:42.730Z
Hey @Pierre.Vriens - I don't mind I had a few minutes and wanted to clarify my assumption, theoretically, I could add it to my answer, and I may well do that.
Richard Slater
<amazon-web-services><ansible><aws-vpc><rabbitmq>
How to automatically remove a dead node from RabbitMQ cluster
Berlin
# How to automatically remove a dead node from RabbitMQ cluster

(Asked by: username_0 on 2017-03-25)

I am planning to create RabbitMQ cluster using Ansible on AWS VPC with Amazon internal load balancer as the frontend to point connections to it.

Any suggestion how to remove a dead node from RabbitMQ cluster based on autoscaling rule where nodes can go up and down, or if you are using spot instances?

When a node goes down, RabbitMQ does not remove it from the replication list automatically, I can see `Node not running` in the management UI.

I managed to join to the cluster a scaled instance automatically via Ansible and userdata.

[![Diagram of Infrastructure](https://i.sstatic.net/2O2Tx.png)](https://i.sstatic.net/2O2Tx.png)

**Tags:** amazon-web-services, ansible, aws-vpc, rabbitmq

> **Comments:**
> **username_0**: @username_1, I changed to only 1 question, thanks.  
> **username_1**: Merci! You didn't destroy the other 4 (or so) questions I hope. Maybe keep them as folowup question, to the extend they are still relevant?  
> **username_0**: Yep, the questions are still relevant, but this one is the most important. I will post another question later :) thanks!  
> **username_2**: @username_0 I drew a diagram that represents what I think you are describing, if you meant something else then please do let me know and I will adapt.  
> **username_2**: Hey @username_1 - I don't mind I had a few minutes and wanted to clarify my assumption, theoretically, I could add it to my answer, and I may well do that.  

---

## 1 answers

### Answer by username_2 on 2017-03-25. Score: 4 | Accepted answer

Consider using the rabbitmq/rabbitmq-autocluster plugin:

A RabbitMQ plugin that clusters nodes automatically using a number of peer discovery mechanisms:

Consul,
etcd2
DNS A records
AWS EC2 tags
AWS Autoscaling Groups


There is a fair bit of configuration to plug in to get this setup including setting IAM policies and adding EC2 tags to the instances you want to be party to your cluster.
If you were to use AWS Autoscaling Groups then you would add the following to your rabbitmq.config:
[
  {rabbit, [ ... ]},
  {autocluster, [
    {backend, aws},
    {aws_autoscaling, true},
    {aws_ec2_region, "us-west-2"}
  ]}
].

If you are not using AWS Autoscaling Groups you can still achieve the desired result using tags on your EC2 Instances:
[
  {rabbit, [ ... ]},
  {autocluster, [
    {backend, aws},
    {aws_ec2_tags, [{"region", "us-west-2"}, {"service", "rabbitmq"}]},
    {aws_ec2_region, "us-east-1"},
    {aws_access_key, "..."},
    {aws_secret_key, "..."}
  ]}
].

With all of that said I strongly recommend using Consul by HashiCorp as your service discovery mechanism, in the long run, you get significantly more flexibility in terms of decoupling your parts of your system from each other.

> **Comments:**
> **username_0**: thank you for the detailed explanation and the diagram, so AFAIK `rabbitmq/rabbitmq-autocluster plugin` will know also to remove node from the replication list once the node is down, one more thing if may I ask, I thought to start with `2-node` cluster, are you suggest to start with `3-node` cluster as you describe in your diagram with policy  ` rabbitmqctl set_policy ha-all "" '{"ha-mode":"all","ha-sync-mode":"automatic"}' ` ? or should I post it in a another question ?  
> **username_2**: Possibly another question as the formatting in comments isn't all that easy to use.  I have always advised clients when adopting cloud to use larger number of small instances rather than the inverse.  This is because failure is expected in the cloud, if you have a 2-node cluster and 1 fails you have lost 50% of your capacity, whereas if you have a 3-node cluster and 1 fails you only lose 33%.  
> **username_0**: I have configured my cluster using `rabbitmq/rabbitmq-autocluster plugin` and it is work pretty well however when node goes down RabbitMQ does not remove it from the replication list, any idea why ?  
> **username_2**: Have you enabled the Cluster Cleanup configuration option?  
> **username_0**: thanks, found it `https://github.com/aweber/rabbitmq-autocluster/wiki/General-Settings`, I will try that.  
> **username_3**: Links are great https://github.com/aweber/rabbitmq-autocluster/wiki/General-Settings  

---

