<p>I want to add eight public keys via instance metadata to avoid adding them manually (i.e.: ssh to VMs, pasting the keys to <code>.ssh/authorized_keys</code>, etc.).<br />
I added the keys in Terraform (four distinct keys for two users) using the metadata attribute of the <code>google_compute_instance</code>:</p>
<pre><code>resource &quot;google_compute_instance&quot; &quot;host&quot; {
  count         = var.number_of_hosts
  
  // vm details...

  metadata = {
    &quot;ssh-keys&quot; = &lt;&lt;EOF
    user1:${file(&quot;${path.root}/key1.pub&quot;)}
    user1:${file(&quot;${path.root}/key2.pub&quot;)}
    user1:${file(&quot;${path.root}/key3.pub&quot;)}
    user1:${file(&quot;${path.root}/key4.pub&quot;)}
    user2:${file(&quot;${path.root}/key1.pub&quot;)}
    user2:${file(&quot;${path.root}/key2.pub&quot;)}
    user2:${file(&quot;${path.root}/key3.pub&quot;)}
    user2:${file(&quot;${path.root}/key4.pub&quot;)}
    EOF
  }
</code></pre>
<p>I ran <code>terraform apply</code>. I opened the GCP console and clicked on one of the deployed machines. In the &quot;Details&quot; tab, I can see all eight keys in the SSH Keys tab.
Now, when I ssh from my local computer, i.e., <code>ssh user2@EXTERNAL_IP</code> (I deliberately started with user2, not user1 - not a typo) and then <code>cat ~/.ssh/authorized_keys</code>, I can only see the following:</p>
<pre><code>user1 : key1
user2 : key4
</code></pre>
<p>Thus, I can't ssh to VM2 because the public part of the key pair that USER 2 has access to is not <code>~/.ssh/authorized_keys</code> even though it is declared in the instance metadata.</p>
<p>On the other hand, when I do <code>user1@EXTERNAL_IP</code> and <code>cat ~/.ssh/authorized_keys</code>, I can see:</p>
<pre><code>user1 : key1
user2 : key4
user1 : key1 (duplicate)
</code></pre>
<p>Since the private key that corresponds to <code>user1 : key1</code> is there, I can ssh to VM2 successfully.</p>
<p>What baffles me:</p>
<ol>
<li>Why are not all keys declared in the instance metadata added to the <code>authorized_keys</code>?</li>
<li>Why is there a difference in the content of the <code>authorized_keys</code> depending on the user?</li>
<li>Where does the duplicate come from?</li>
</ol>
<hr />
<p>Edit - some additional information:</p>
<ol>
<li>the image used - <code>ubuntu-minimal-2004-focal-v20230427</code></li>
<li><code>ssh_config</code> (only uncommented lines):</li>
</ol>
<pre><code>Include /etc/ssh/ssh_config.d/*.conf
Host *
SendEnv LANG LC_*
HashKnownHosts yes
GSSAPIAuthentication yes
</code></pre>
<ol start="3">
<li><code>sshd_config</code> (only uncommented lines):</li>
</ol>
<pre><code>Include /etc/ssh/sshd_config.d/*.conf
PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM yes
X11Forwarding yes
PrintMotd no
AcceptEnv LANG LC_*
Subsystem       sftp    /usr/lib/openssh/sftp-server

</code></pre>

CC BY-SA 4.0
2023-05-25T15:51:46.787Z
What software are you using in your VM image to interpret the metadata and generate those `authorized_keys` files? I expect it's that software which is making the decision about which files to write and what to write into each of them. From Terraform's perspective and GCP's perspective this is just an opaque string, so how it gets interpreted is up to the software running inside your VM.
Martin Atkins
CC BY-SA 4.0
2023-05-25T17:21:02.057Z
I edited my answer with the `sshd_config` and `ssh_config`, but I'm not sure that is what you meant with "software are you using in your VM image to interpret the metadata and generate those `authorized_keys` files".
SkogensKonung
CC BY-SA 4.0
2023-05-25T23:25:44.133Z
The way this metadata concept works is that Google Compute Engine just exposes [an internal API that returns the metadata values](https://cloud.google.com/compute/docs/metadata/querying-metadata) and then some software running in your VM -- typically during the boot process -- will make requests to that API to fetch the data and then reconfigure your system based on what it reads. It's that software that would be responsible for generating these `authorized_keys` files. If you're using a generic Linux distribution image then perhaps you can name that distribution so we can find docs about it.
Martin Atkins
CC BY-SA 4.0
2023-05-26T06:19:59.717Z
Here's the info from the description tab in the GCP console: `Canonical, Ubuntu, 20.04 LTS Minimal, amd64 focal minimal image built on 2023-04-27, supports Shielded VM features`
SkogensKonung
CC BY-SA 4.0
2023-05-26T15:19:48.723Z
According to Ubuntu's documentation, [official Ubuntu cloud images since 18.04 use CloudInit for boot-time initialization](https://help.ubuntu.com/community/CloudInit). That suggests that this is a CloudInit question, and so I'm going to add a label for that in the hope that it will be more visible to those who have CloudInit expertise.
Martin Atkins
<terraform><google-cloud-platform><ssh><cloud-init>
SSH Keys not propagating correctly from instance metadata to authorized_keys: missing keys, user discrepancy, and duplicate key
SkogensKonung
# SSH Keys not propagating correctly from instance metadata to authorized_keys: missing keys, user discrepancy, and duplicate key

(Asked by: username_0 on 2023-05-25)

I want to add eight public keys via instance metadata to avoid adding them manually (i.e.: ssh to VMs, pasting the keys to .ssh/authorized_keys, etc.).
I added the keys in Terraform (four distinct keys for two users) using the metadata attribute of the google_compute_instance:
resource "google_compute_instance" "host" {
  count         = var.number_of_hosts
  
  // vm details...

  metadata = {
    "ssh-keys" = <
I ran terraform apply. I opened the GCP console and clicked on one of the deployed machines. In the "Details" tab, I can see all eight keys in the SSH Keys tab.
Now, when I ssh from my local computer, i.e., ssh user2@EXTERNAL_IP (I deliberately started with user2, not user1 - not a typo) and then cat ~/.ssh/authorized_keys, I can only see the following:
user1 : key1
user2 : key4

Thus, I can't ssh to VM2 because the public part of the key pair that USER 2 has access to is not ~/.ssh/authorized_keys even though it is declared in the instance metadata.
On the other hand, when I do user1@EXTERNAL_IP and cat ~/.ssh/authorized_keys, I can see:
user1 : key1
user2 : key4
user1 : key1 (duplicate)

Since the private key that corresponds to user1 : key1 is there, I can ssh to VM2 successfully.
What baffles me:

Why are not all keys declared in the instance metadata added to the authorized_keys?
Why is there a difference in the content of the authorized_keys depending on the user?
Where does the duplicate come from?


Edit - some additional information:

the image used - ubuntu-minimal-2004-focal-v20230427
ssh_config (only uncommented lines):

Include /etc/ssh/ssh_config.d/*.conf
Host *
SendEnv LANG LC_*
HashKnownHosts yes
GSSAPIAuthentication yes


sshd_config (only uncommented lines):

Include /etc/ssh/sshd_config.d/*.conf
PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM yes
X11Forwarding yes
PrintMotd no
AcceptEnv LANG LC_*
Subsystem       sftp    /usr/lib/openssh/sftp-server

**Tags:** terraform, google-cloud-platform, ssh, cloud-init

> **Comments:**
> **username_1**: What software are you using in your VM image to interpret the metadata and generate those `authorized_keys` files? I expect it's that software which is making the decision about which files to write and what to write into each of them. From Terraform's perspective and GCP's perspective this is just an opaque string, so how it gets interpreted is up to the software running inside your VM.  
> **username_0**: I edited my answer with the `sshd_config` and `ssh_config`, but I'm not sure that is what you meant with "software are you using in your VM image to interpret the metadata and generate those `authorized_keys` files".  
> **username_1**: The way this metadata concept works is that Google Compute Engine just exposes [an internal API that returns the metadata values](https://cloud.google.com/compute/docs/metadata/querying-metadata) and then some software running in your VM -- typically during the boot process -- will make requests to that API to fetch the data and then reconfigure your system based on what it reads. It's that software that would be responsible for generating these `authorized_keys` files. If you're using a generic Linux distribution image then perhaps you can name that distribution so we can find docs about it.  
> **username_0**: Here's the info from the description tab in the GCP console: `Canonical, Ubuntu, 20.04 LTS Minimal, amd64 focal minimal image built on 2023-04-27, supports Shielded VM features`  
> **username_1**: According to Ubuntu's documentation, [official Ubuntu cloud images since 18.04 use CloudInit for boot-time initialization](https://help.ubuntu.com/community/CloudInit). That suggests that this is a CloudInit question, and so I'm going to add a label for that in the hope that it will be more visible to those who have CloudInit expertise.  

---

## 0 answers

