The Watch method of the discovery backend calls cluster.UpdateNodes method to update the node list according the return value of the Fetch method of the discovery service.
But it only inserts the nodes that aren't exist in the node list and doesn't touch the nodes that are "removed" from the discovery service.
If a node is removed from the discovery service, the swarm list command returns the correct result because it calls the Fetch method of the discovery service directly. But in fact the removed node is still in the node list of the cluster.
That's correct, right now, the node isn't removed, but flagged as unhealthy and is skipped by the scheduler.
We are not sure about how we want to handle this, because we don't want to loose information in case of the node comes back up.
Did you encounter a specific issue ?
As far as I can see the removed node is not flagged as unhealthy unless the refreshContainers method returns an error.
My case is that after I add a node to the cluster and then need to remove it for debugging, I don't want new containers to be launched on the removed node.
What about add another command such as swarm leave --addr=1.2.3.4 or swarm remove --addr=1.2.3.4 that sends a "leave" message to the discovery service? The swarm daemon can remove the corresponding node when it receives the message.
Or perhaps we can send a specific signal to the swarm join process, when the process receives the signal it send the "leave" message to the discovery service and quit.
For tracking the information of the removed node when it comes back up, perhaps a persistent storage of the nodes information is required.
@mountkin I don't feel live a new command would be the solution. If you're using etcd or consul, there is a TTL so your node will disappear. This feature is coming to the hosted registry service.
@aluzzardi what do you think ?
Eventually, we will need to handle the node removal case correctly.
Right now we don't do any house keeping even if the node gets removed through etcd / consul.
We need to find a way to make the distinction between node removal and unhealthy node.
Regarding the debugging, I don't think nodes should be removed straight from discovery as every implementation will act differently. Perhaps this should be a super set of the API (e.g. DELETE /v1/nodes/foo) and the swarm binary would provide easy access to it.
Different schedulers also will have their own health check and tracking of different task status, such as Mesos.
I don't have a good picture how different schedulers are suppose to plug into swarm yet, but I think ideally is that there is both a way to notify swarm for actually node health, or let swarm's node detection and tracking to be pluggable so it can call out to existing services. In Mesos case it can simply contact the master to get all the information.
👍 for node removal being a priority. If you have long running jobs, you need some way of signaling that a node is going to be removed but is still present while it is finishing its outstanding jobs.
what is the priority and timeline on node removal - housekeeping? It seems that this is integral to using Swarm. Otherwise it presents only a half solution for Docker cluster management. Would be happy to help test.
The node list is synced from discovery services periodically.
If we simply remove it from swarm's cached node list, it'll be added again when the next UpdateNodes is triggered by discovery service.
Add a new member to the Cluster struct to keep all the nodes that had been "removed". And modify the UpdateNodes function to exclude the nodes that are in the removed list.
The list of removed nodes should be stored persistently.
DELETE /nodes/:node_id which is responsible for add an entry to the removed list and delete the node from cluster.nodes.
POST /nodes/:node_id which is responsible for remove the node from the removed list.
The user removed the node from discovery service first, then call swarm's API to delete it from the cache.
But this may not work for the discovery services that haven't exposed a DELETE API, such as nodes and token.
@vieux @aluzzardi What do you think?
@vieux saw from your comments in linked issues that this was going to be addressed soon - can you give us any updates on how this is coming together?
@vincentwoo We have a few PRs cooking for this - should be completed for 0.3.0.
@aluzzardi I see that this was marked resolved in #811, but I don't see the topic addressed in the official docs or changelog. Can you describe the procedure required to remove a node from the cluster? Does it depend on the discovery service?
Hi @vincentwoo, Yes it depends on the discovery service. If using the kv discovery or the token discovery, there is a TTL field attached to the node entry. If the node does not heartbeat to notify of its presence in the cluster (ie. stopping the swarm join or the node crashes), the TTL will expire and the Manager will be notified to remove the node entry.
So if I understand you correctly, the way to remove a node is to explicitly kill whatever background process swarm join is running in, and wait for the TTL to expire? Are there plans to encapsulate or document this?
Yes exactly. The documentation mentions the heartbeat flag but I agree that this should be explained better.. Especially about the whole process off tagging a node as dead to evict it from any scheduling decision and then removing it from the cluster, the process of heartbeating and watch notifications, etc.
You can maybe open an issue to track this. I'll look into enhancing the docs for this aspect.