<p>"best suited" is the question. How do you measure this? what are the metrics and why do you choose them? I think you are looking for some kind of "tools combination", earlier tested on production by someone else's if preferred. What you want to do is try some opensource/free private cloud on-premises and run IaC on this private environment. You can test tools forever. How do you compare those tools? I believe all of them works fine as long as there is not any inconsistency problem. I think there is a missing/main part in your search when it comes to IaC. Because of few options for that missing angle as well. I think it is better to create a chart for all of us as you are going this process in order to benefit people here. Do you have a working environment? If so could you please detail all the tools you used to create this environment?</p>

<p>Some vendors run open source projects. How about them?</p>

<p>The Cloud Native Computing Foundation seeks to drive adoption of this paradigm by fostering and sustaining an ecosystem of open source, vendor-neutral projects.<a href="https://github.com/cncf/toc/blob/master/DEFINITION.md" rel="nofollow noreferrer">CNCF Cloud Native Definition v1.0</a></p>

<p>I started to wonder if this is cloud-native then what is it that is not cloud-native?</p>

<p>I think this is what you are looking for. The distinction between provider and provisioner is blurred. A workflow automation tool should be doing what cloud orchestrator with GUI does. Terraform makes a distinction between provider and provisioner but Cloudify removes them as I see it. For an architect designing the whole workflow with GUI might be useful to reach the desired state. Writing desired state for which you do DRY tests already beforehand or showing with graphs. There is something you know already where it goes since you already tested, seeing it flowing in graphics workflow makes sense? Then go for it. To me it is painting the history I do not see the benefit.</p>

<p>There are many ways to break down the types of automation, whether it’s imperative versus declarative, or orchestration versus desired configuration state.</p>

<p>Most orchestrators are fully CLI based, or offer very limited GUI operations.
A distinct advantage here is that Cloudify enables you to one-click your way through architectural relationship diagrams. See: <a href="https://cloudify.co/2018/10/22/terraform-vs-cloudify/" rel="nofollow noreferrer">Terraform vs. Cloudify</a>.</p>

<h3>Providers</h3>

<p>A provider is responsible for understanding API interactions and exposing resources. Providers generally are an IaaS (e.g. AWS, GCP, Microsoft Azure, OpenStack), PaaS (e.g. Heroku), or SaaS services (e.g. Terraform Enterprise, DNSimple, CloudFlare).
See: <a href="https://www.terraform.io/docs/providers/index.html" rel="nofollow noreferrer">Providers</a>.</p>

<h3>Provisioners</h3>

<p>Provisioners are used to execute scripts on a local or remote machine as part of resource creation or destruction. Provisioners can be used to bootstrap a resource, cleanup before destroy, run configuration management, etc.
See: <a href="https://www.terraform.io/docs/provisioners/index.html" rel="nofollow noreferrer">Provisioners</a>.</p>

CC BY-SA 4.0
2018-10-20T02:06:46.867Z
I'm looking for "All-in-one" tool. So if this is possible with combination of tools the metric to measure is "number of tools used to setup k8s cluster from scratch". If this is not possible with one CLI command, the next metric is "number of commands needed". If there's no CLI, the metric is "number of clicks". The next metric is "number of clicks/commands to install the tool".
Vanuan
CC BY-SA 4.0
2018-10-20T02:16:04.007Z
Added metrics into question.
Vanuan
CC BY-SA 4.0
2018-10-22T11:34:20.870Z
What you are looking for is in your "clarification" words and also in the title of the link you gave. I think you focus too much on tools and lose the sight of your problem. I do not want to go further because I do not know what kind of problem you are facing. I have to create your environment and try. You want to deploy clustered-containers on some cloud. Inconsistency would be a problem in most cases.
hakkican
CC BY-SA 4.0
2018-10-18T10:08:11.730Z
2018-11-06T12:21:20.773Z
2018-11-06T12:21:20.773Z
hakkican
<p>Here's what I've tried:</p>

<h2>Terraform + libvirt</h2>

<p>There's unofficial libvirt provider for terraform. It works, but there are a few gotchas:</p>

<ul>
<li><p>Doesn't support block devices:
<a href="https://github.com/dmacvicar/terraform-provider-libvirt/issues/418" rel="nofollow noreferrer">https://github.com/dmacvicar/terraform-provider-libvirt/issues/418</a></p></li>
<li><p>Existing state not being removed:
<a href="https://github.com/dmacvicar/terraform-provider-libvirt/issues/423" rel="nofollow noreferrer">https://github.com/dmacvicar/terraform-provider-libvirt/issues/423</a></p></li>
</ul>

<h2>Docker machine</h2>

<p>Docker machine has unofficial kvm driver.</p>

<ul>
<li>Works quite well</li>
<li>Isn't supported anymore</li>
<li>kvm plugin lacks remote qemu+ssh:
<a href="https://github.com/dhiltgen/docker-machine-kvm/pull/63" rel="nofollow noreferrer">https://github.com/dhiltgen/docker-machine-kvm/pull/63</a></li>
</ul>

CC BY-SA 4.0
2018-10-09T18:12:30.347Z
2018-10-09T18:19:05.750Z
2018-10-09T18:19:05.750Z
Vanuan
<p>There is a new generation of tooling developed that tries to integrate different IaaS providers (Azure/AWS/GCE/DigitalOcean) into one "infrastructure codebase". It's dubbed "Cloud Native" and it's supposed to result in "one tool to rule them all". There's Terraform from HashiCorp, InfraKit from Docker, Juju from Canonical, independent Foreman, Pulumi and others.</p>

<p>It's easy to get lost. Which of these tools is best suited to IaaS on premises? Let's suppose you have a bunch of metal servers and you want to provision some VMs there. Let's suppose you don't want VirtualBox/VMWare/ESX/Xen, you want <a href="https://en.wikipedia.org/wiki/Kernel-based_Virtual_Machine" rel="nofollow noreferrer">KVM</a>. And on top of that an obvious choice is <a href="https://en.wikipedia.org/wiki/Libvirt" rel="nofollow noreferrer">libvirt</a>. To manage multiple hypervisors we need to connect to the libvirt daemon remotely using <a href="https://libvirt.org/guide/html/Application_Development_Guide-Connections-URI_Formats.html" rel="nofollow noreferrer">connection URI</a>. For KVM we can use <a href="https://wiki.libvirt.org/page/SSHSetup" rel="nofollow noreferrer">qemu+ssh</a>. Let's suppose we set those up using PXE boot. Now we have a bunch of <code>qemu+ssh://</code> links along with ssh access.</p>

<p>Now we need a central component on top of that to manage resources and some tool that will provision virtual hosts according to some declarative specification.</p>

<p><a href="https://i.sstatic.net/KL1uW.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/KL1uW.jpg" alt="IaaS on premises using libvirt qemu kvm"></a></p>

<p>On top of those VMs I want to run containers, so automatic setup of container orchestration cluster is desired. In addition, some way of attaching storage is desired. At least directly attached block devices or fs mounts.</p>

<p>Any suggestions?</p>

<p>Note: Found <a href="https://stackoverflow.com/questions/3754392/">a similar question on SO</a>, but for Xen.</p>

<p>Clarification: "best suited" needs metrics. So these are the metrics I'm looking to optimize:</p>

<ul>
<li>Zero closed source LOC</li>
<li>Number of tools if not "All-in-one"</li>
<li>UI actions to setup 3 master + 5 workers k8s/swarm cluster

<ul>
<li>number of commands (for CLI tool)</li>
<li>number of clicks (for Web based/GUI tool)</li>
<li>length/size of the config file</li>
</ul></li>
<li>UI actions to install the tool</li>
<li>UI actions to maintain configuration (preferably none, self-healing infrastructure)</li>
</ul>

<infrastructure-as-code><virtualization><provisioning><infrastructure>
Open source private cloud VM provisioning for container workload
Vanuan
# Open source private cloud VM provisioning for container workload

(Asked by: username_0 on 2018-10-09)

There is a new generation of tooling developed that tries to integrate different IaaS providers (Azure/AWS/GCE/DigitalOcean) into one "infrastructure codebase". It's dubbed "Cloud Native" and it's supposed to result in "one tool to rule them all". There's Terraform from HashiCorp, InfraKit from Docker, Juju from Canonical, independent Foreman, Pulumi and others.

It's easy to get lost. Which of these tools is best suited to IaaS on premises? Let's suppose you have a bunch of metal servers and you want to provision some VMs there. Let's suppose you don't want VirtualBox/VMWare/ESX/Xen, you want [KVM](https://en.wikipedia.org/wiki/Kernel-based_Virtual_Machine). And on top of that an obvious choice is [libvirt](https://en.wikipedia.org/wiki/Libvirt). To manage multiple hypervisors we need to connect to the libvirt daemon remotely using [connection URI](https://libvirt.org/guide/html/Application_Development_Guide-Connections-URI_Formats.html). For KVM we can use [qemu+ssh](https://wiki.libvirt.org/page/SSHSetup). Let's suppose we set those up using PXE boot. Now we have a bunch of `qemu+ssh://` links along with ssh access.

Now we need a central component on top of that to manage resources and some tool that will provision virtual hosts according to some declarative specification.

[![IaaS on premises using libvirt qemu kvm](https://i.sstatic.net/KL1uW.jpg)](https://i.sstatic.net/KL1uW.jpg)

On top of those VMs I want to run containers, so automatic setup of container orchestration cluster is desired. In addition, some way of attaching storage is desired. At least directly attached block devices or fs mounts.

Any suggestions?

Note: Found [a similar question on SO](https://stackoverflow.com/questions/3754392/), but for Xen.

Clarification: "best suited" needs metrics. So these are the metrics I'm looking to optimize:

* Zero closed source LOC
* Number of tools if not "All-in-one"
* UI actions to setup 3 master + 5 workers k8s/swarm cluster
  + number of commands (for CLI tool)
  + number of clicks (for Web based/GUI tool)
  + length/size of the config file
* UI actions to install the tool
* UI actions to maintain configuration (preferably none, self-healing infrastructure)

**Tags:** infrastructure-as-code, virtualization, provisioning, infrastructure


---

## 2 answers

### Answer by username_1 on 2018-10-18. Score: 3

"best suited" is the question. How do you measure this? what are the metrics and why do you choose them? I think you are looking for some kind of "tools combination", earlier tested on production by someone else's if preferred. What you want to do is try some opensource/free private cloud on-premises and run IaC on this private environment. You can test tools forever. How do you compare those tools? I believe all of them works fine as long as there is not any inconsistency problem. I think there is a missing/main part in your search when it comes to IaC. Because of few options for that missing angle as well. I think it is better to create a chart for all of us as you are going this process in order to benefit people here. Do you have a working environment? If so could you please detail all the tools you used to create this environment?

Some vendors run open source projects. How about them?

The Cloud Native Computing Foundation seeks to drive adoption of this paradigm by fostering and sustaining an ecosystem of open source, vendor-neutral projects.[CNCF Cloud Native Definition v1.0](https://github.com/cncf/toc/blob/master/DEFINITION.md)

I started to wonder if this is cloud-native then what is it that is not cloud-native?

I think this is what you are looking for. The distinction between provider and provisioner is blurred. A workflow automation tool should be doing what cloud orchestrator with GUI does. Terraform makes a distinction between provider and provisioner but Cloudify removes them as I see it. For an architect designing the whole workflow with GUI might be useful to reach the desired state. Writing desired state for which you do DRY tests already beforehand or showing with graphs. There is something you know already where it goes since you already tested, seeing it flowing in graphics workflow makes sense? Then go for it. To me it is painting the history I do not see the benefit.

There are many ways to break down the types of automation, whether it’s imperative versus declarative, or orchestration versus desired configuration state.

Most orchestrators are fully CLI based, or offer very limited GUI operations.
A distinct advantage here is that Cloudify enables you to one-click your way through architectural relationship diagrams. See: [Terraform vs. Cloudify](https://cloudify.co/2018/10/22/terraform-vs-cloudify/).

### Providers

A provider is responsible for understanding API interactions and exposing resources. Providers generally are an IaaS (e.g. AWS, GCP, Microsoft Azure, OpenStack), PaaS (e.g. Heroku), or SaaS services (e.g. Terraform Enterprise, DNSimple, CloudFlare).
See: [Providers](https://www.terraform.io/docs/providers/index.html).

### Provisioners

Provisioners are used to execute scripts on a local or remote machine as part of resource creation or destruction. Provisioners can be used to bootstrap a resource, cleanup before destroy, run configuration management, etc.
See: [Provisioners](https://www.terraform.io/docs/provisioners/index.html).

> **Comments:**
> **username_0**: I'm looking for "All-in-one" tool. So if this is possible with combination of tools the metric to measure is "number of tools used to setup k8s cluster from scratch". If this is not possible with one CLI command, the next metric is "number of commands needed". If there's no CLI, the metric is "number of clicks". The next metric is "number of clicks/commands to install the tool".  
> **username_0**: Added metrics into question.  
> **username_1**: What you are looking for is in your "clarification" words and also in the title of the link you gave. I think you focus too much on tools and lose the sight of your problem. I do not want to go further because I do not know what kind of problem you are facing. I have to create your environment and try. You want to deploy clustered-containers on some cloud. Inconsistency would be a problem in most cases.  

---

### Answer by username_0 on 2018-10-09. Score: 1

Here's what I've tried:

## Terraform + libvirt

There's unofficial libvirt provider for terraform. It works, but there are a few gotchas:

* Doesn't support block devices:
  <https://github.com/dmacvicar/terraform-provider-libvirt/issues/418>
* Existing state not being removed:
  <https://github.com/dmacvicar/terraform-provider-libvirt/issues/423>

## Docker machine

Docker machine has unofficial kvm driver.

* Works quite well
* Isn't supported anymore
* kvm plugin lacks remote qemu+ssh:
  <https://github.com/dhiltgen/docker-machine-kvm/pull/63>


---

