<p>This might not be the answer to your specific cross data-center-ip setup.</p>

<p>I occasionally run into one or more swarm-nodes being status: <code>Down</code> and Availability: <code>Active</code>. Having the  <code>Status.Message</code>: <code>"heartbeat failure"</code>. This can happen after a reboot.</p>

<p>What helped was to stop the docker daemon, remove <code>/var/lib/docker/swarm/worker/tasks.db</code> and start the docker daemon again.</p>

<p>from:
<a href="https://github.com/moby/moby/issues/34827#issuecomment-457678500" rel="nofollow noreferrer">https://github.com/moby/moby/issues/34827#issuecomment-457678500</a></p>

<p>sometimes it fixes itself: <a href="https://stackoverflow.com/a/54126180/2087704">https://stackoverflow.com/a/54126180/2087704</a></p>

CC BY-SA 4.0
2020-04-08T07:34:20.440Z
2020-04-08T07:34:20.440Z
invad0r
<p>My cluster is currently located in a single data center. I've been trying to change that by adding a single worker node from another data center, but so far it hasn't worked.</p>

<p>I'm able to make this node join the swarm and get listed by the managers, but it is always shown as "Down". Here is what "docker inspect" shows me about this node:</p>

<pre><code>    "Status": {
        "State": "down",
        "Message": "heartbeat failure",
        "Addr": "xxx.xxx.xxx.xxx"
    }
</code></pre>

<p>I've opened the following ports in both sides:</p>

<pre><code>2377 tcp
7946 tcp+udp
4789 udp
</code></pre>

<p>How do I troubleshoot and fix this?</p>

CC BY-SA 4.0
2018-10-09T02:25:05.723Z
Found it. My nodes are advertising their private IPs to the external node. Not sure how to fix that, but I guess it's another question
ivarec
CC BY-SA 4.0
2019-12-23T09:22:04.310Z
Could you post an answer?
030
<docker><docker-swarm><networking>
How to fix a "heartbeat failure" in Docker Swarm?
ivarec
# How to fix a "heartbeat failure" in Docker Swarm?

(Asked by: username_0 on 2018-10-09)

My cluster is currently located in a single data center. I've been trying to change that by adding a single worker node from another data center, but so far it hasn't worked.
I'm able to make this node join the swarm and get listed by the managers, but it is always shown as "Down". Here is what "docker inspect" shows me about this node:
    "Status": {
        "State": "down",
        "Message": "heartbeat failure",
        "Addr": "xxx.xxx.xxx.xxx"
    }

I've opened the following ports in both sides:
2377 tcp
7946 tcp+udp
4789 udp

How do I troubleshoot and fix this?

**Tags:** docker, docker-swarm, networking

> **Comments:**
> **username_0**: Found it. My nodes are advertising their private IPs to the external node. Not sure how to fix that, but I guess it's another question  
> **username_1**: Could you post an answer?  

---

## 1 answers

### Answer by username_2 on 2020-04-08. Score: 2

This might not be the answer to your specific cross data-center-ip setup.

I occasionally run into one or more swarm-nodes being status: `Down` and Availability: `Active`. Having the `Status.Message`: `"heartbeat failure"`. This can happen after a reboot.

What helped was to stop the docker daemon, remove `/var/lib/docker/swarm/worker/tasks.db` and start the docker daemon again.

from:
<https://github.com/moby/moby/issues/34827#issuecomment-457678500>

sometimes it fixes itself: <https://stackoverflow.com/a/54126180/2087704>


---

