<p>A storage solution for kubernetes is a pain for those of us who chose to run kubernetes on premises.</p>

<p>What storage solution one picks would certainly depend on potential usage, a solution that will perform well for simple file operations would not pan out very well if you want to run a database in kubernetes.</p>

<p>Given your configuration, I think "vSphere Cloud Provider" is your best option</p>

<blockquote>
  <p>VMware offers a Cloud Provider known as the vSphere Cloud Provider (VCP) for Kubernetes which allows Pods to use enterprise grade persistent storage.</p>
</blockquote>

<p><a href="https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/overview.html" rel="nofollow noreferrer">architecture overview of vSphere Cloud Provider</a></p>

<p>If you don't want to run databases you can use NFS using HDFS setup you have or any other NFS server.</p>

<p><strong>Disclaimer:</strong> I have not used vSphere Cloud Provider or ceph myself. Personally, I have used NFS from our storage server for production and it works fine for moderate database loads. We also have a large gluster implementation supporting close to thousand volumes, however, based on issues we have seen with gluster I will not suggest that to anyone.</p>

<p>On a side note, you don't "have" to use all of your cluster nodes as "storage nodes" for Ceph &amp; Rook. Only storage nodes will have restrictions on them. If you are going with Ceph &amp; Rook, I will suggest to use dedicated storage nodes otherwise it may impact performance of your workloads.</p>

CC BY-SA 4.0
2020-02-05T13:40:31.743Z
Hello @haroon, thanks a lot for your answer. I'll take a look at VCP, it might indeed be a good option. i'm not going to run any classic RDBMS into kubernetes. But i may have some elasticsearch, mongo or these kind of thinks running in the cluster. I may end up using two storage NFS for regular volumes that does not requires any specific performances and Ceph or VCP for fast storage then...
Anthony Raymond
CC BY-SA 4.0
2020-02-05T15:38:41.530Z
Also about the NFS, did you used an external NFS server or an in-cluster NFS provider? I would like to use the HDFS over NFS so i'd just need the NFS-client-provisioner, but the only one i see seems outdated and no longer maintained. Which one did you used?
Anthony Raymond
CC BY-SA 4.0
2020-02-06T03:23:49.390Z
I am using external NFS server. For NFS provisioning I am using `quay.io/external_storage/nfs-client-provisioner`, you can deploy it using helm chart `nfs-client-provisioner`.  
If you are using HDFS over NFS, one thing to keep in mind that it does not support random writes so Mongo, Elastic may have to go to "fast storage".
haroon
CC BY-SA 4.0
2020-02-06T11:00:35.707Z
Ok thanks :) What does that means "it does not support random writes" and how does that affects mongo, elastic etc?
Anthony Raymond
CC BY-SA 4.0
2020-02-05T03:12:56.220Z
2020-02-05T03:12:56.220Z
haroon
<h3>Problem in a nutshell</h3>

<p>What kind of storage for dynamic provisioning in a bare-metal kubernetes should i use.</p>

<h3>Our configuration</h3>

<ul>
<li>Each of our kubernetes nodes (worker and master) are running on VMs. The VMs themselves run on a VCenter on top of an Hyperconverged architecture</li>
<li>So far the cluster have no persistent storage configured (other than the host storage)</li>
<li>We do have an HDFS (Hadoop Distributed File System) running in bare metal as well, to make it short, hdfs is a <em>distributed</em> and <em>Highly Available</em> storage.</li>
</ul>

<h3>What do i want</h3>

<p>I've seen Ceph&amp;Rook being a popular solution. But if i understand it properly it required each kubernetes nodes (if i want it to be fault resistant) to have an attached physical storage, which will all contains more or less the same datas.</p>

<h3>What's the problem with that</h3>

<p>If each nodes have 1Tb of storage, i will have 1Tb of storage as a ceph&amp;rook storage, and any time i want to add a node to the cluster i will be forced to add 1To of storage to the new node as well.</p>

<p>Ceph will duplicate data across the cluster to make it available in case of failure. But since the cluster is running on top of a hyperconverged architecture, the cepth duplication will be duplicated again by the hyperconverged architecture... you may start to see the problem.</p>

<p>The above example makes no sense to me, especially when i have an HDFS next to this cluster.</p>

<p>My question is what is my best option to use the HDFS as a dynamic storage provisioner?</p>

<kubernetes><kubernetes-volume>
Bare-Metal Kubernetes storage in enterprise
Anthony Raymond
# Bare-Metal Kubernetes storage in enterprise

(Asked by: username_0 on 2020-02-03)

### Problem in a nutshell

What kind of storage for dynamic provisioning in a bare-metal kubernetes should i use.

### Our configuration

* Each of our kubernetes nodes (worker and master) are running on VMs. The VMs themselves run on a VCenter on top of an Hyperconverged architecture
* So far the cluster have no persistent storage configured (other than the host storage)
* We do have an HDFS (Hadoop Distributed File System) running in bare metal as well, to make it short, hdfs is a *distributed* and *Highly Available* storage.

### What do i want

I've seen Ceph&Rook being a popular solution. But if i understand it properly it required each kubernetes nodes (if i want it to be fault resistant) to have an attached physical storage, which will all contains more or less the same datas.

### What's the problem with that

If each nodes have 1Tb of storage, i will have 1Tb of storage as a ceph&rook storage, and any time i want to add a node to the cluster i will be forced to add 1To of storage to the new node as well.

Ceph will duplicate data across the cluster to make it available in case of failure. But since the cluster is running on top of a hyperconverged architecture, the cepth duplication will be duplicated again by the hyperconverged architecture... you may start to see the problem.

The above example makes no sense to me, especially when i have an HDFS next to this cluster.

My question is what is my best option to use the HDFS as a dynamic storage provisioner?

**Tags:** kubernetes, kubernetes-volume


---

## 1 answers

### Answer by username_1 on 2020-02-05. Score: 1

A storage solution for kubernetes is a pain for those of us who chose to run kubernetes on premises.

What storage solution one picks would certainly depend on potential usage, a solution that will perform well for simple file operations would not pan out very well if you want to run a database in kubernetes.

Given your configuration, I think "vSphere Cloud Provider" is your best option

> VMware offers a Cloud Provider known as the vSphere Cloud Provider (VCP) for Kubernetes which allows Pods to use enterprise grade persistent storage.

[architecture overview of vSphere Cloud Provider](https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/overview.html)

If you don't want to run databases you can use NFS using HDFS setup you have or any other NFS server.

**Disclaimer:** I have not used vSphere Cloud Provider or ceph myself. Personally, I have used NFS from our storage server for production and it works fine for moderate database loads. We also have a large gluster implementation supporting close to thousand volumes, however, based on issues we have seen with gluster I will not suggest that to anyone.

On a side note, you don't "have" to use all of your cluster nodes as "storage nodes" for Ceph & Rook. Only storage nodes will have restrictions on them. If you are going with Ceph & Rook, I will suggest to use dedicated storage nodes otherwise it may impact performance of your workloads.

> **Comments:**
> **username_0**: Hello @username_1, thanks a lot for your answer. I'll take a look at VCP, it might indeed be a good option. i'm not going to run any classic RDBMS into kubernetes. But i may have some elasticsearch, mongo or these kind of thinks running in the cluster. I may end up using two storage NFS for regular volumes that does not requires any specific performances and Ceph or VCP for fast storage then...  
> **username_0**: Also about the NFS, did you used an external NFS server or an in-cluster NFS provider? I would like to use the HDFS over NFS so i'd just need the NFS-client-provisioner, but the only one i see seems outdated and no longer maintained. Which one did you used?  
> **username_1**: I am using external NFS server. For NFS provisioning I am using `quay.io/external_storage/nfs-client-provisioner`, you can deploy it using helm chart `nfs-client-provisioner`.   If you are using HDFS over NFS, one thing to keep in mind that it does not support random writes so Mongo, Elastic may have to go to "fast storage".  
> **username_0**: Ok thanks :) What does that means "it does not support random writes" and how does that affects mongo, elastic etc?  

---

