<p>(Kublr CTO) Take a look at Kublr - <a href="https://kublr.com/" rel="nofollow noreferrer">https://kublr.com/</a></p>

<p>It is designed exactly for a use-case like this: centralized management and operations of multiple Kubernetes clusters in multiple and/or heterogeneous target environments. Currently it supports AWS, Azure, Google Cloud, vSphere, vCloud Director, and on-prem/BYOI (Bring Your Own Infrastructure mode.</p>

<p>In this response - <a href="https://devops.stackexchange.com/a/9281/17283">https://devops.stackexchange.com/a/9281/17283</a> - I described in more details which requirements we considered important when designing Kublr.</p>

CC BY-SA 4.0
2019-09-25T18:51:24.867Z
2019-09-25T18:51:24.867Z
Oleg Chunikhin
<p>At work we are currenly using a multi AZ Kubernetes cluster on AWS and we're using <code>kops</code> along with Terraform (<code>kops</code> generates the Terraform configuration files) to provision the cluster. What is not clear to me is if your intention is to run a single multi-cloud cluster or if you want to run multiple clusters in multiple clouds.</p>

<p>Anyway, our current setup is a multi-master highly-available multi-AZ Kubernetes cluster. I'll try to explain step by step.</p>

<p>The first thing in order to create the cluster is to generate the Terraform configuration with <code>kops</code> (you could directly apply the changes in AWS by directly using <code>kops</code>, but in our case we think it's best to keep the Terraform files versioned in git, to be able to change details about this configuration and inspect them, also). For instance, this could be the command we used:</p>

<p><code>kops create cluster \
--name=my_new_cluster \
--ssh-public-key=k8s.pub \
--dns-zone=&lt;your-name&gt;.&lt;your-domain&gt; \
--cloud=aws \
--master-size=t2.medium \
--node-size=t2.medium \
--vpc=&lt;vpc-id&gt; \
--zones=us-east-1a,us-east-1b,us-east-1c \
--master-zones=us-east-1a,us-east-1b,us-east-1c \
--out=my_new_cluster \
--target=terraform</code></p>

<p>This creates the Terraform configurations (in the directory specified by the <code>--out</code> flag) for a Kubernetes cluster which nodes and masters are distributed along 3 Availability Zones (us-east-1a, us-east-1b and us-east-1c). This implies creating:</p>

<ul>
<li>all necessary subnets, security groups and route tables</li>
<li>IAM roles</li>
<li>AWS Auto Scaling Groups to manage the cluster nodes and masters (it's the magic of all this, they're the ones that maintain the high availability of the cluster by launching extra nodes as traffic increases, and replaces faulty ones)</li>
<li>all EC2 instances and EBS volumes</li>
<li>Route53 DNS records to refer to the cluster API and internal DNS for k8s to communicate with pods.</li>
</ul>

<p><code>kops</code> uses <code>kubectl</code> internally to create the cluster and all necessary configurations in Kubernetes.</p>

<p>After that, you could <code>terraform apply</code> the configurations and watch it all go up in AWS. It's quite simple, really, but you have to learn how Terraform works. I've found it very intuitive.</p>

<hr>

<p>Later on, you may need to edit you cluster. There's a command: <code>kops edit cluster &lt;cluster-name&gt;</code> that let you do that. It's all configurations in  <code>yaml</code> format. I won't get into specifics but after some changes you may need to <code>kops rolling-update &lt;cluster-name&gt;</code> to apply them.</p>

<p>Hope I've been clear.</p>

<p>Regards.</p>

CC BY-SA 3.0
2017-09-10T05:33:53.447Z
Great, detailed answer. Long live Kubernetes :-) and thank you Terraform!
Eldad Assis
CC BY-SA 3.0
2017-09-10T18:26:08.517Z
@EldadAK I'm glad I've helped you.
Extra: You can also build your cluster using spot instances ;) as per https://github.com/kubernetes/kops/blob/master/docs/instance_groups.md#converting-an-instance-group-to-use-spot-instances
You need only set the `maxPrice` and that's it. If this matter interests you, I can get into specifics when implementing this.
klaus
CC BY-SA 3.0
2017-09-09T19:29:32.863Z
2017-09-09T19:57:21.333Z
2017-09-09T19:57:21.333Z
klaus
<p>I want to install a <strong>highly available kubernetes cluster</strong> (multiple master nodes) on a public cloud (AWS, Azure, GCP and maybe others).<br>
I'm looking for a non-vendor specific option to do it.</p>

<p>I'm aware of several options like</p>

<ul>
<li><a href="https://github.com/kubernetes/kops" rel="noreferrer">Kops</a></li>
<li><a href="https://jujucharms.com/" rel="noreferrer">Juju</a></li>
<li>Other Github projects I found...</li>
</ul>

<p>I was wondering if anyone has a good automated example of doing this with such tools like <a href="https://www.terraform.io/" rel="noreferrer">Terraform</a>, <a href="https://www.ansible.com/" rel="noreferrer">Ansible</a> or others.</p>

<p>The <strong>HA</strong> is a must as this if for production use!</p>

CC BY-SA 3.0
2017-09-08T22:41:37.363Z
Kops is already using Terraform and ansible under the hood IIRC
Tensibai
CC BY-SA 3.0
2017-09-10T18:22:48.173Z
@Tensibai Actually, no. `kops` interacts directly with AWS API if you run it as is. With the `--target=terraform` flag, it generates the configuration for you to apply, as I detailed in my answer.
klaus
CC BY-SA 3.0
2017-09-10T20:09:58.007Z
@klaus blame my faulty memory :)
Tensibai
<kubernetes>
Deploying a Kubernetes cluster for production on multiple clouds
Eldad Assis
# Deploying a Kubernetes cluster for production on multiple clouds

(Asked by: username_0 on 2017-07-30)

I want to install a **highly available kubernetes cluster** (multiple master nodes) on a public cloud (AWS, Azure, GCP and maybe others).  
I'm looking for a non-vendor specific option to do it.

I'm aware of several options like

* [Kops](https://github.com/kubernetes/kops)
* [Juju](https://jujucharms.com/)
* Other Github projects I found...

I was wondering if anyone has a good automated example of doing this with such tools like [Terraform](https://www.terraform.io/), [Ansible](https://www.ansible.com/) or others.

The **HA** is a must as this if for production use!

**Tags:** kubernetes

> **Comments:**
> **username_1**: Kops is already using Terraform and ansible under the hood IIRC  
> **username_2**: @username_1 Actually, no. `kops` interacts directly with AWS API if you run it as is. With the `--target=terraform` flag, it generates the configuration for you to apply, as I detailed in my answer.  
> **username_1**: @username_2 blame my faulty memory :)  

---

## 2 answers

### Answer by username_2 on 2017-09-09. Score: 6 | Accepted answer

At work we are currenly using a multi AZ Kubernetes cluster on AWS and we're using `kops` along with Terraform (`kops` generates the Terraform configuration files) to provision the cluster. What is not clear to me is if your intention is to run a single multi-cloud cluster or if you want to run multiple clusters in multiple clouds.

Anyway, our current setup is a multi-master highly-available multi-AZ Kubernetes cluster. I'll try to explain step by step.

The first thing in order to create the cluster is to generate the Terraform configuration with `kops` (you could directly apply the changes in AWS by directly using `kops`, but in our case we think it's best to keep the Terraform files versioned in git, to be able to change details about this configuration and inspect them, also). For instance, this could be the command we used:

`kops create cluster \
--name=my_new_cluster \
--ssh-public-key=k8s.pub \
--dns-zone=. \
--cloud=aws \
--master-size=t2.medium \
--node-size=t2.medium \
--vpc= \
--zones=us-east-1a,us-east-1b,us-east-1c \
--master-zones=us-east-1a,us-east-1b,us-east-1c \
--out=my_new_cluster \
--target=terraform`

This creates the Terraform configurations (in the directory specified by the `--out` flag) for a Kubernetes cluster which nodes and masters are distributed along 3 Availability Zones (us-east-1a, us-east-1b and us-east-1c). This implies creating:

* all necessary subnets, security groups and route tables
* IAM roles
* AWS Auto Scaling Groups to manage the cluster nodes and masters (it's the magic of all this, they're the ones that maintain the high availability of the cluster by launching extra nodes as traffic increases, and replaces faulty ones)
* all EC2 instances and EBS volumes
* Route53 DNS records to refer to the cluster API and internal DNS for k8s to communicate with pods.

`kops` uses `kubectl` internally to create the cluster and all necessary configurations in Kubernetes.

After that, you could `terraform apply` the configurations and watch it all go up in AWS. It's quite simple, really, but you have to learn how Terraform works. I've found it very intuitive.

---

Later on, you may need to edit you cluster. There's a command: `kops edit cluster`  that let you do that. It's all configurations in `yaml` format. I won't get into specifics but after some changes you may need to `kops rolling-update`  to apply them.

Hope I've been clear.

Regards.

> **Comments:**
> **username_0**: Great, detailed answer. Long live Kubernetes :-) and thank you Terraform!  
> **username_2**: @EldadAK I'm glad I've helped you. Extra: You can also build your cluster using spot instances ;) as per https://github.com/kubernetes/kops/blob/master/docs/instance_groups.md#converting-an-instance-group-to-use-spot-instances You need only set the `maxPrice` and that's it. If this matter interests you, I can get into specifics when implementing this.  

---

### Answer by username_3 on 2019-09-25. Score: 1

(Kublr CTO) Take a look at Kublr - <https://kublr.com/>

It is designed exactly for a use-case like this: centralized management and operations of multiple Kubernetes clusters in multiple and/or heterogeneous target environments. Currently it supports AWS, Azure, Google Cloud, vSphere, vCloud Director, and on-prem/BYOI (Bring Your Own Infrastructure mode.

In this response - <https://devops.stackexchange.com/a/9281/17283> - I described in more details which requirements we considered important when designing Kublr.


---

