<p>The key here is the behavior of AWS AMIs in the context of ECS. By default the AWS ECS Amazon Linux AMI optimized for ECS is in charge of VOLUME management but there's a problem if you copy that AMI and try to launch a new instance -- the SECONDARY volume that ECS uses to deploy docker on can go missing.</p>

<p>Troubleshooting steps:</p>

<ul>
<li><p>Check <code>/var/log/cloud-init-output.log</code> and see if you have an error message like:</p>

<blockquote>
  <p>ERROR: /dev//dev/xvdcz is not a valid block device.</p>
</blockquote></li>
<li><p>Check your instance mounted volumes and make sure that the Secondary Volume exists. Usually if you see the error above it means that only <code>/dev/xvda</code> is mounted.</p></li>
</ul>

<p><strong>FIX</strong>:</p>

<p>Enforce your Launch Configuration, Launch Template or script to always deploy the secondary volume <code>/dev/xvdcz</code> <strong>regardless</strong> of AMI configuration. Do not rely on the AMI as unfortunately the AMI flow has changed and might not preserve all the necessary configuration.</p>

CC BY-SA 4.0
2019-04-24T20:28:59.050Z
2019-09-05T17:51:20.233Z
2019-09-05T17:51:20.233Z
eco
<p>The ECS instance fails to join the ECS cluster because it relies on docker to launch its ecs agent. If in this case docker fails to start then the whole process halts.</p>

<p>Docker fails to start in AWS ECS instances with the following message in /var/log/docker :</p>

<pre><code>Error starting daemon: error initializing graphdriver: devicemapper: Non 
existing device docker-docker--pool
</code></pre>

<p>Notice the following.</p>

<p>Instance AMI is a COPY of the AWS ECS optimized AMI with the embedded ECS agent(and docker config for ECS)</p>

<docker><amazon-web-services><amazon-ec2><amazon-ecs><amazon-ami>
Why does Docker fail to start on AWS ECS instances. ECS stateful instance fails initialization. ECS stateful instance doesn't join the ECS cluster
eco
# Why does Docker fail to start on AWS ECS instances. ECS stateful instance fails initialization. ECS stateful instance doesn't join the ECS cluster

(Asked by: username_0 on 2019-04-24)

The ECS instance fails to join the ECS cluster because it relies on docker to launch its ecs agent. If in this case docker fails to start then the whole process halts.
Docker fails to start in AWS ECS instances with the following message in /var/log/docker :
Error starting daemon: error initializing graphdriver: devicemapper: Non 
existing device docker-docker--pool

Notice the following.
Instance AMI is a COPY of the AWS ECS optimized AMI with the embedded ECS agent(and docker config for ECS)

**Tags:** docker, amazon-web-services, amazon-ec2, amazon-ecs, amazon-ami


---

## 1 answers

### Answer by username_0 on 2019-04-24. Score: 1

The key here is the behavior of AWS AMIs in the context of ECS. By default the AWS ECS Amazon Linux AMI optimized for ECS is in charge of VOLUME management but there's a problem if you copy that AMI and try to launch a new instance -- the SECONDARY volume that ECS uses to deploy docker on can go missing.

Troubleshooting steps:

* Check `/var/log/cloud-init-output.log` and see if you have an error message like:

  > ERROR: /dev//dev/xvdcz is not a valid block device.
* Check your instance mounted volumes and make sure that the Susername_0ndary Volume exists. Usually if you see the error above it means that only `/dev/xvda` is mounted.

**FIX**:

Enforce your Launch Configuration, Launch Template or script to always deploy the susername_0ndary volume `/dev/xvdcz` **regardless** of AMI configuration. Do not rely on the AMI as unfortunately the AMI flow has changed and might not preserve all the necessary configuration.


---

