<blockquote>
  <p>we'd prefer not to bake them into the actual Docker image itself (plus we want the ability to change the assets without having to recompile the application images).</p>
</blockquote>

<p>This is indeed good properties to have. You are on the right track.</p>

<blockquote>
  <p>The problem, however, comes with our read-only data assets (similar to a game's asset files).</p>
</blockquote>

<p>This can be solved with multiple solutions, it's hard to tell what is the best for you. Here are some alternatives.</p>

<p><strong>Persistent Volume - ReadOnlyMany</strong></p>

<blockquote>
  <p>I know that we're essentially looking for a persistent volume for Kubernetes, but the problem for us is bundling all of the data into a single package that has the same delivery convenience as a Docker image.</p>
</blockquote>

<p>Yes, you could create a <a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/" rel="nofollow noreferrer">Persistent Volume</a> with <a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes" rel="nofollow noreferrer">access modes</a>:</p>

<pre><code>accessModes:
- ReadWriteOnce
- ReadOnlyMany
</code></pre>

<p>then (by an automatic process)</p>

<ol>
<li>Load your data onto the volume with a <a href="https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/" rel="nofollow noreferrer">Job</a> using access mode <code>ReadWriteOnce</code></li>
<li>Load the volumes to your pods/services using access mode <code>ReadOnlyMany</code></li>
</ol>

<p>and keep the volume immutable, you create a new when you need to change content. You can also use multiple volumes this way.</p>

<p><strong>CDN</strong></p>

<blockquote>
  <p>our read-only data assets (similar to a game's asset files).</p>
</blockquote>

<p>This inherently also sounds what typically is served by <a href="https://en.wikipedia.org/wiki/Content_delivery_network" rel="nofollow noreferrer">Content Deliver Networks</a>. But it depends on your use case if this is a good solution or not.</p>

<p><strong>Buckets</strong></p>

<p>Data for e.g. CDN is usually stored by "buckets" at cloud providers, that may also be an option, then your applications load data using HTTP typically. <a href="https://min.io/" rel="nofollow noreferrer">Minio</a> may be an alternative if you want to do this in your cluster.</p>

CC BY-SA 4.0
2020-05-04T16:29:29.287Z
But does this provide a volume "file" that I can then take and plug into another cluster, or do I have to repeat the "create volume, load volume" steps every time I want to deploy the data?
stix
CC BY-SA 4.0
2020-05-04T16:39:21.347Z
PVC is only within cluster. CDN is "global". And buckets is also widely accessible.
Jonas
CC BY-SA 4.0
2020-05-01T19:35:19.233Z
2020-05-01T19:35:19.233Z
Jonas
<p>This may not have been available when the question was asked, but a relatively new option is to use &quot;Image Volumes&quot;:</p>
<p><a href="https://kubernetes.io/docs/tasks/configure-pod-container/image-volumes/" rel="nofollow noreferrer">https://kubernetes.io/docs/tasks/configure-pod-container/image-volumes/</a></p>
<p>Basically, package your assets into an image that can be distributed using the same channels as your application images, and mount that image into your application containers at runtime.</p>
<p>I'm looking at using this technique to inject client-specific branding assets (logos, some CSS tweaks) into a standard image which would be shared by multiple clients.</p>

CC BY-SA 4.0
2024-12-18T21:30:00.370Z
2024-12-18T21:30:00.370Z
Simon Geard
<p>I'm migrating an HPC app to Kubernetes, and am trying to determine the best way to provide read only data assets as a configuration managed snapshot. </p>

<p>Previously, my team had delivered our application as a set of RPMs, but as we move to Kubernetes, we're delivering Docker images. This works fine for our application binary, as instead of a pile of RPMs that all have to agree, we can just deliver a known working image. </p>

<p>The problem, however, comes with our read-only data assets (similar to a game's asset files). Several different Docker images might rely on a single set of data assets, and so we'd prefer not to bake them into the actual Docker image itself (plus we want the ability to change the assets without having to recompile the application images). </p>

<p>We're unsure as to the best approach for this. The first idea would be to create a "data container" that simply runs NFS and serves out the data. This successfully isolates the data from the application and allows us to collapse a set of data RPMs into a single tagged docker image, but to me it seems like it might be overkill. </p>

<p>I know that we're essentially looking for a persistent volume for Kubernetes, but the problem for us is bundling all of the data into a single package that has the same delivery convenience as a Docker image. </p>

<p><strong>Is there a better way to provide this read-only data as a version controlled snapshot?</strong></p>

CC BY-SA 4.0
2020-05-01T17:30:55.053Z
Could you indicate what you mean with "data"? E.g. text files, blobs that are larger than 1 GB?
030
CC BY-SA 4.0
2020-05-01T17:36:30.883Z
A directory tree with files. Basically I'm looking for something like an ISO snapshot that we can version control and deliver as a mount point that the application pods pick up. 

Our biggest dataset is on the order of 15 GB, but it's broken into multiple files.
stix
CC BY-SA 4.0
2020-05-01T19:01:50.377Z
What cloud provider do you use? E.g. AWS, GCP, Azure, bare-metal?
030
CC BY-SA 4.0
2020-05-02T23:47:00.237Z
@030 Bare metal. These are our own clusters.
stix
<kubernetes><data>
How can I best deliver read only data assets to a Kubernetes application?
stix
# How can I best deliver read only data assets to a Kubernetes application?

(Asked by: username_0 on 2020-05-01)

I'm migrating an HPC app to Kubernetes, and am trying to determine the best way to provide read only data assets as a configuration managed snapshot.

Previously, my team had delivered our application as a set of RPMs, but as we move to Kubernetes, we're delivering Docker images. This works fine for our application binary, as instead of a pile of RPMs that all have to agree, we can just deliver a known working image.

The problem, however, comes with our read-only data assets (similar to a game's asset files). Several different Docker images might rely on a single set of data assets, and so we'd prefer not to bake them into the actual Docker image itself (plus we want the ability to change the assets without having to recompile the application images).

We're unsure as to the best approach for this. The first idea would be to create a "data container" that simply runs NFS and serves out the data. This successfully isolates the data from the application and allows us to collapse a set of data RPMs into a single tagged docker image, but to me it seems like it might be overkill.

I know that we're essentially looking for a persistent volume for Kubernetes, but the problem for us is bundling all of the data into a single package that has the same delivery convenience as a Docker image.

**Is there a better way to provide this read-only data as a version controlled snapshot?**

**Tags:** kubernetes, data

> **Comments:**
> **username_1**: Could you indicate what you mean with "data"? E.g. text files, blobs that are larger than 1 GB?  
> **username_0**: A directory tree with files. Basically I'm looking for something like an ISO snapshot that we can version control and deliver as a mount point that the application pods pick up.   Our biggest dataset is on the order of 15 GB, but it's broken into multiple files.  
> **username_1**: What cloud provider do you use? E.g. AWS, GCP, Azure, bare-metal?  
> **username_0**: @username_1 Bare metal. These are our own clusters.  

---

## 2 answers

### Answer by username_2 on 2020-05-01. Score: 1

we'd prefer not to bake them into the actual Docker image itself (plus we want the ability to change the assets without having to recompile the application images).

This is indeed good properties to have. You are on the right track.

The problem, however, comes with our read-only data assets (similar to a game's asset files).

This can be solved with multiple solutions, it's hard to tell what is the best for you. Here are some alternatives.
Persistent Volume - ReadOnlyMany

I know that we're essentially looking for a persistent volume for Kubernetes, but the problem for us is bundling all of the data into a single package that has the same delivery convenience as a Docker image.

Yes, you could create a Persistent Volume with access modes:
accessModes:
- ReadWriteOnce
- ReadOnlyMany

then (by an automatic process)

Load your data onto the volume with a Job using access mode ReadWriteOnce
Load the volumes to your pods/services using access mode ReadOnlyMany

and keep the volume immutable, you create a new when you need to change content. You can also use multiple volumes this way.
CDN

our read-only data assets (similar to a game's asset files).

This inherently also sounds what typically is served by Content Deliver Networks. But it depends on your use case if this is a good solution or not.
Buckets
Data for e.g. CDN is usually stored by "buckets" at cloud providers, that may also be an option, then your applications load data using HTTP typically. Minio may be an alternative if you want to do this in your cluster.

> **Comments:**
> **username_0**: But does this provide a volume "file" that I can then take and plug into another cluster, or do I have to repeat the "create volume, load volume" steps every time I want to deploy the data?  
> **username_2**: PVC is only within cluster. CDN is "global". And buckets is also widely accessible.  

---

### Answer by username_3 on 2024-12-18. Score: 0

This may not have been available when the question was asked, but a relatively new option is to use "Image Volumes":

<https://kubernetes.io/docs/tasks/configure-pod-container/image-volumes/>

Basically, package your assets into an image that can be distributed using the same channels as your application images, and mount that image into your application containers at runtime.

I'm looking at using this technique to inject client-specific branding assets (logos, some CSS tweaks) into a standard image which would be shared by multiple clients.


---

