<p>So I found some answers over time, posting them here in case it helps anyone.</p>
<ul>
<li><p>The Userdata never ran, I can't find see any of the above echo statements in the logs /var/log/cloud-init.log. When I SSH into the instance I can't find any of these files. How do I debug this better?</p>
<p>ANS: There was an error in my Userdata, I SSHed into the instance and ran the script manually to debug and fix all issues.</p>
</li>
<li><p>On code update, Cloudformation stack is updated, this does not run Userdata(?) (I know it only runs when an instance is first created, but I would like some confirmation that Cloudformation update does not trigger this</p>
<p>ANS: Cloudformation terminates the old instance and spins up a new one on each deploy, this causes the Userdata to be run each time.</p>
</li>
<li><p>Is this an ideal flow? Are there any improvements I can make here, considering I'm not an expert, but willing to spend some time learning where required.</p>
<p>ANS: It was not. The primary reason I was doing this was to fire async requests from one lambda to another, without waiting for the results. This is not possible, as the lambda will not wait for any requests that you do not <code>await</code> meaning you still have to wait for those operations. I tried asynchronous invocation but it still did not work (For those interested I posted a question about it <a href="https://stackoverflow.com/questions/72493614/serverless-express-lambda-async-invoke-not-working">here</a>). So I switched to using SNS to publish messages from one lambda to trigger another, achieving my requirement for fire and forget requests.</p>
</li>
</ul>

CC BY-SA 4.0
2022-07-26T17:20:41.897Z
2022-07-26T17:20:41.897Z
kvnam
<p>I'm relatively new to Cloudformation, and had a few questions about my template and best practices. I'm facing a few hurdles, and there is a lot of information out there, it's a bit overwhelming, so any feedback would be highly appreciated. I'm not looking for detailed code etc. just some good insights on how I can improve my steps.</p>
<p>I'm trying to set up a basic Node/Express API:</p>
<ul>
<li>On push to Git repo
<ul>
<li>Build Docker image and push to private AWS ECR repo</li>
<li>After successful push, deploy Cloudformation template that provisions
<ul>
<li>An EC2 + security group with Elastic IP assigned</li>
<li>Run docker compose in Userdata of EC2 to get app up and running</li>
</ul>
</li>
</ul>
</li>
</ul>
<p>This is my UserData (I do need some specific help here!)</p>
<pre><code>UserData: !Base64 |
        #!/bin/bash -ex
        yum update -y
        yum install docker -y
        service docker start
        usermod -a -G docker ec2-user
        echo &quot;Start Docker service&quot;
        apt-get update
        apt-get install docker-compose-plugin
        apt install amazon-ecr-credential-helper
        echo &quot;APT-GET update complete&quot;
        echo &quot;{ \&quot;credHelpers\&quot;: { \&quot;&lt;my_acc_id&gt;.dkr.ecr.&lt;my_region&gt;.amazonaws.com\&quot;: \&quot;ecr-login\&quot; } }&quot; &gt; ~/.docker/config.json
        systemctl restart docker
        echo &quot;
        version: &quot;3.9&quot;
        services:
          my-app:
            image: &lt;acc_id&gt;.dkr.ecr.&lt;my_region&gt;.amazonaws.com/my-repo
            environment:
              STAGE: staging
              VIRTUAL_HOST: my-customdomain.com
              VIRTUAL_PORT: 3000
            ports:
              - &quot;3000:3000&quot;
            restart: always
            networks:
              - my-network

          https-portal:
            image: steveltn/https-portal:1
            ports:
              - '80:80'
              - '443:443'
            links:
              - my-app
            environment:
              STAGE: production
            volumes:
              - /var/run/docker.sock:/var/run/docker.sock:ro
            networks:
              - my-network

        volumes:
          https-portal-data:
        networks:
          my-network:
            driver: bridge
        &quot; &gt; docker-compose.yaml
        docker compose up -d
</code></pre>
<p>Status:
Cloudformation template deploys successfully, all resources set up. But the Userdata doesn't run, so my EC2 never sets up my app.</p>
<p>Issues / Questions:</p>
<ul>
<li>The Userdata never ran, I can't find see any of the above echo statements in the logs <code>/var/log/cloud-init.log</code>. When I SSH into the instance I can't find any of these files. How do I debug this better?</li>
<li>Is there a better way to get the docker-compose data in there? writing the whole file in the UserData script seems inefficient. Is there a better way to do this?</li>
<li>On code update, Cloudformation stack is updated, this does not run Userdata(?) (I know it only runs when an instance is first created, but I would like some confirmation that Cloudformation update does not trigger this.</li>
<li>What is the best practice here if I want to re-run docker compose in my Ec2 after every Cloudformation deploy? If it does trigger Userdata, what could be wrong here?</li>
<li>Is this an ideal flow? Are there any improvements I can make here, considering I'm not an expert, but willing to spend some time learning where required.</li>
</ul>
<p>I appreciate anyone taking the time to answer these questions. Thanks!</p>

<docker><docker-compose><amazon-ec2><aws-cloudformation><aws-ecr>
Cloudformation template with EC2 using docker compose
kvnam
# Cloudformation template with EC2 using docker compose

(Asked by: username_0 on 2022-07-21)

I'm relatively new to Cloudformation, and had a few questions about my template and best practices. I'm facing a few hurdles, and there is a lot of information out there, it's a bit overwhelming, so any feedback would be highly appreciated. I'm not looking for detailed code etc. just some good insights on how I can improve my steps.
I'm trying to set up a basic Node/Express API:

On push to Git repo

Build Docker image and push to private AWS ECR repo
After successful push, deploy Cloudformation template that provisions

An EC2 + security group with Elastic IP assigned
Run docker compose in Userdata of EC2 to get app up and running





This is my UserData (I do need some specific help here!)
UserData: !Base64 |
        #!/bin/bash -ex
        yum update -y
        yum install docker -y
        service docker start
        usermod -a -G docker ec2-user
        echo "Start Docker service"
        apt-get update
        apt-get install docker-compose-plugin
        apt install amazon-ecr-credential-helper
        echo "APT-GET update complete"
        echo "{ \"credHelpers\": { \".dkr.ecr..amazonaws.com\": \"ecr-login\" } }" > ~/.docker/config.json
        systemctl restart docker
        echo "
        version: "3.9"
        services:
          my-app:
            image: .dkr.ecr..amazonaws.com/my-repo
            environment:
              STAGE: staging
              VIRTUAL_HOST: my-customdomain.com
              VIRTUAL_PORT: 3000
            ports:
              - "3000:3000"
            restart: always
            networks:
              - my-network

          https-portal:
            image: steveltn/https-portal:1
            ports:
              - '80:80'
              - '443:443'
            links:
              - my-app
            environment:
              STAGE: production
            volumes:
              - /var/run/docker.sock:/var/run/docker.sock:ro
            networks:
              - my-network

        volumes:
          https-portal-data:
        networks:
          my-network:
            driver: bridge
        " > docker-compose.yaml
        docker compose up -d

Status:
Cloudformation template deploys successfully, all resources set up. But the Userdata doesn't run, so my EC2 never sets up my app.
Issues / Questions:

The Userdata never ran, I can't find see any of the above echo statements in the logs /var/log/cloud-init.log. When I SSH into the instance I can't find any of these files. How do I debug this better?
Is there a better way to get the docker-compose data in there? writing the whole file in the UserData script seems inefficient. Is there a better way to do this?
On code update, Cloudformation stack is updated, this does not run Userdata(?) (I know it only runs when an instance is first created, but I would like some confirmation that Cloudformation update does not trigger this.
What is the best practice here if I want to re-run docker compose in my Ec2 after every Cloudformation deploy? If it does trigger Userdata, what could be wrong here?
Is this an ideal flow? Are there any improvements I can make here, considering I'm not an expert, but willing to spend some time learning where required.

I appreciate anyone taking the time to answer these questions. Thanks!

**Tags:** docker, docker-compose, amazon-ec2, aws-cloudformation, aws-ecr


---

## 1 answers

### Answer by username_0 on 2022-07-26. Score: 1

So I found some answers over time, posting them here in case it helps anyone.

* The Userdata never ran, I can't find see any of the above echo statements in the logs /var/log/cloud-init.log. When I SSH into the instance I can't find any of these files. How do I debug this better?

  ANS: There was an error in my Userdata, I SSHed into the instance and ran the script manually to debug and fix all issues.
* On code update, Cloudformation stack is updated, this does not run Userdata(?) (I know it only runs when an instance is first created, but I would like some confirmation that Cloudformation update does not trigger this

  ANS: Cloudformation terminates the old instance and spins up a new one on each deploy, this causes the Userdata to be run each time.
* Is this an ideal flow? Are there any improvements I can make here, considering I'm not an expert, but willing to spend some time learning where required.

  ANS: It was not. The primary reason I was doing this was to fire async requests from one lambda to another, without waiting for the results. This is not possible, as the lambda will not wait for any requests that you do not `await` meaning you still have to wait for those operations. I tried asynchronous invocation but it still did not work (For those interested I posted a question about it [here](https://stackoverflow.com/questions/72493614/serverless-express-lambda-async-invoke-not-working)). So I switched to using SNS to publish messages from one lambda to trigger another, achieving my requirement for fire and forget requests.


---

