<h1>Failure to Merge - YAML whitespace</h1>
<p>This turned out to be because of whitespace. You can confirm this by sshing to the machine and running</p>
<pre class="lang-sh prettyprint-override"><code>sudo grep WARNING /var/log/cloud-init.log
</code></pre>
<p>Which should return this,</p>
<pre><code>util.py[WARNING]: Failed loading yaml blob. Invalid format at line 11 column 1: &quot;while scanning for the next token
util.py[WARNING]: Failed at merging in cloud config part from part-001
</code></pre>
<p>Basically what happens is this,</p>
<ol>
<li>You upload the <code>user_data</code> to the providers instance's metadata.</li>
<li>The client you're provisioning launches <code>cloud-init</code>.</li>
<li>That downloads the <code>user_data</code> from the instance's metadata at <code>http://169.254.169.254/</code></li>
<li>It sees the <code>#cloud-config</code> on line one of your <code>user_data</code></li>
<li>You merge your own cloud_config with the downloaded <code>user_data</code></li>
<li>The merge-algorithm doesn't normalize for whitespace, so the yaml is invalid.</li>
<li>The merge-algorithm returns a blank cloud-config.</li>
</ol>
<p>You can verify this by trying to re-run <code>cloud-init init</code> which will produce something like this,</p>
<pre><code>2022-05-16 04:50:04,351 - util.py[WARNING]: Failed loading yaml blob. Invalid format at line 11 column 1: &quot;while scanning for the next token
found character '\t' that cannot start any token
  in &quot;&lt;unicode string&gt;&quot;, line 11, column 1:
        - podman
    ^&quot;
2022-05-16 04:50:04,351 - util.py[WARNING]: Failed at merging in cloud config part from part-001
</code></pre>
<h2>Full Log</h2>
<pre><code>__init__.py[DEBUG]: {'MIME-Version': '1.0', 'Content-Type': 'text/cloud-config', 'Content-Disposition': 'attachment; filename=&quot;part-001&quot;'}
__init__.py[DEBUG]: Calling handler CloudConfigPartHandler: [['text/cloud-config', 'text/cloud-config-jsonp']] (text/cloud-config, part-001, 3) with frequency once-per-instance
util.py[DEBUG]: Attempting to load yaml from string of length 252 with allowed root types (&lt;class 'dict'&gt;,)
util.py[WARNING]: Failed loading yaml blob. Invalid format at line 11 column 1: &quot;while scanning for the next token
found character '\t' that cannot start any token
  in &quot;&lt;unicode string&gt;&quot;, line 11, column 1:
        - podman
    ^&quot;
util.py[WARNING]: Failed at merging in cloud config part from part-001
util.py[DEBUG]: Failed at merging in cloud config part from part-001
Traceback (most recent call last):
  File &quot;/usr/lib/python3/dist-packages/cloudinit/handlers/cloud_config.py&quot;, line 140, in handle_part
    self._merge_part(payload, headers)
  File &quot;/usr/lib/python3/dist-packages/cloudinit/handlers/cloud_config.py&quot;, line 116, in _merge_part
    (payload_yaml, my_mergers) = self._extract_mergers(payload, headers)
  File &quot;/usr/lib/python3/dist-packages/cloudinit/handlers/cloud_config.py&quot;, line 95, in _extract_mergers
    mergers_yaml = mergers.dict_extract_mergers(payload_yaml)
  File &quot;/usr/lib/python3/dist-packages/cloudinit/mergers/__init__.py&quot;, line 79, in dict_extract_mergers
    raw_mergers = config.pop('merge_how', None)
AttributeError: 'NoneType' object has no attribute 'pop'
__init__.py[DEBUG]: Calling handler CloudConfigPartHandler: [['text/cloud-config', 'text/cloud-config-jsonp']] (__end__, None, 3) with frequency once-per-instance
util.py[DEBUG]: Writing to /var/lib/cloud/instances/i-0804054440b55ca17/cloud-config.txt - wb: [600] 26 bytes
</code></pre>

CC BY-SA 4.0
2022-05-16T04:44:16.043Z
2022-05-16T06:04:17.403Z
2022-05-16T06:04:17.403Z
Evan Carroll
<p>I'm currently provisioning a machine with Cloud Config in <code>user_data</code> (I've also tried <code>user_data_base64</code>). When I run my command in Terraform I have,</p>
<pre><code>resource &quot;aws_instance&quot; &quot;web&quot; {
    user_data_base64 = filebase64(&quot;${path.module}/scripts/web-init-script.yml&quot;)
</code></pre>
<p>However, nothing happens. It silently fails. What's the problem? The contents of <code>web-init.script.yml</code> are,</p>
<pre class="lang-yml prettyprint-override"><code>❯ cat scripts/db-init-script.sh
#cloud-config
package_update: true
package_upgrade: true

fqdn: db.acme.com
prefer_fqdn_over_hostname: true
hostname: db

packages:
    - podman
</code></pre>

<terraform><yaml><cloud-init>
Cloud-config in Terraform does not work, nothing seems to happen and it fails silently
Evan Carroll
# Cloud-config in Terraform does not work, nothing seems to happen and it fails silently

(Asked by: username_0 on 2022-05-16)

I'm currently provisioning a machine with Cloud Config in user_data (I've also tried user_data_base64). When I run my command in Terraform I have,
resource "aws_instance" "web" {
    user_data_base64 = filebase64("${path.module}/scripts/web-init-script.yml")

However, nothing happens. It silently fails. What's the problem? The contents of web-init.script.yml are,
❯ cat scripts/db-init-script.sh
#cloud-config
package_update: true
package_upgrade: true

fqdn: db.acme.com
prefer_fqdn_over_hostname: true
hostname: db

packages:
    - podman

**Tags:** terraform, yaml, cloud-init


---

## 1 answers

### Answer by username_0 on 2022-05-16. Score: 0 | Accepted answer

Failure to Merge - YAML whitespace
This turned out to be because of whitespace. You can confirm this by sshing to the machine and running
sudo grep WARNING /var/log/cloud-init.log

Which should return this,
util.py[WARNING]: Failed loading yaml blob. Invalid format at line 11 column 1: "while scanning for the next token
util.py[WARNING]: Failed at merging in cloud config part from part-001

Basically what happens is this,

You upload the user_data to the providers instance's metadata.
The client you're provisioning launches cloud-init.
That downloads the user_data from the instance's metadata at http://169.254.169.254/
It sees the #cloud-config on line one of your user_data
You merge your own cloud_config with the downloaded user_data
The merge-algorithm doesn't normalize for whitespace, so the yaml is invalid.
The merge-algorithm returns a blank cloud-config.

You can verify this by trying to re-run cloud-init init which will produce something like this,
2022-05-16 04:50:04,351 - util.py[WARNING]: Failed loading yaml blob. Invalid format at line 11 column 1: "while scanning for the next token
found character '\t' that cannot start any token
  in "", line 11, column 1:
        - podman
    ^"
2022-05-16 04:50:04,351 - util.py[WARNING]: Failed at merging in cloud config part from part-001

Full Log
__init__.py[DEBUG]: {'MIME-Version': '1.0', 'Content-Type': 'text/cloud-config', 'Content-Disposition': 'attachment; filename="part-001"'}
__init__.py[DEBUG]: Calling handler CloudConfigPartHandler: [['text/cloud-config', 'text/cloud-config-jsonp']] (text/cloud-config, part-001, 3) with frequency once-per-instance
util.py[DEBUG]: Attempting to load yaml from string of length 252 with allowed root types (,)
util.py[WARNING]: Failed loading yaml blob. Invalid format at line 11 column 1: "while scanning for the next token
found character '\t' that cannot start any token
  in "", line 11, column 1:
        - podman
    ^"
util.py[WARNING]: Failed at merging in cloud config part from part-001
util.py[DEBUG]: Failed at merging in cloud config part from part-001
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/handlers/cloud_config.py", line 140, in handle_part
    self._merge_part(payload, headers)
  File "/usr/lib/python3/dist-packages/cloudinit/handlers/cloud_config.py", line 116, in _merge_part
    (payload_yaml, my_mergers) = self._extract_mergers(payload, headers)
  File "/usr/lib/python3/dist-packages/cloudinit/handlers/cloud_config.py", line 95, in _extract_mergers
    mergers_yaml = mergers.dict_extract_mergers(payload_yaml)
  File "/usr/lib/python3/dist-packages/cloudinit/mergers/__init__.py", line 79, in dict_extract_mergers
    raw_mergers = config.pop('merge_how', None)
AttributeError: 'NoneType' object has no attribute 'pop'
__init__.py[DEBUG]: Calling handler CloudConfigPartHandler: [['text/cloud-config', 'text/cloud-config-jsonp']] (__end__, None, 3) with frequency once-per-instance
util.py[DEBUG]: Writing to /var/lib/cloud/instances/i-0804054440b55ca17/cloud-config.txt - wb: [600] 26 bytes


---

