<p>Docker-in-docker has gotten better over the years and the <a href="https://hub.docker.com/_/docker" rel="nofollow noreferrer">docker dind</a> image is getting more maturity. However there are still a number of caveats that make its use relatively difficult to get right.</p>
<h3>Storage driver</h3>
<p>The main challenge of getting docker-in-docker to work well is the storage driver of the docker installation. On your host you would have filesystems like <code>ext4</code>, which docker can run <code>overlayfs2</code> on top of. However, within a docker container, docker is unable to use <code>overlayfs2</code> as the backing filesystem of another <code>overlayfs2</code>. It doesn't work recursively. There are no easy setups to create a recursive docker-in-docker storage driver stack that works well.</p>
<p>The only storage driver that works on anything is <code>vfs</code>. However, <code>vfs</code> does not have copy-on-write capabilities, meaning that every docker build layer is actually a complete copy of the previous layers. In docker builds with many layers, high amounts of duplication cause performance issues as well as they quickly exhaust disk space. So it's usually not a real solution.</p>
<p>A possible workaround is to use a volume mount for the directory of the inner docker daemon's storage (<code>/var/lib/docker</code>). Then the inner docker daemon can use the host filesystem directly and can thus run <code>overlayfs2</code> on it.</p>
<p>Another workaround would be to create a loop device formatted as <code>ext4</code> which can then be used as the backing filesystem for the inner docker daemon. To avoid pre-allocating space on the disk, you can use a sparse file for the loop device. For example</p>
<pre><code>dd if=/dev/zero of=&quot;$image_file&quot; bs=1M count=0 seek=&quot;$sparse_loop_device_size_mb&quot;
mkfs.ext4 &quot;$image_file&quot;
mount -n -o loop,noatime,nodiratime,noexec,noauto &quot;$image_file&quot; /var/lib/docker
</code></pre>
<p>(The size of the loop device here is the maximum size, not the initial size)</p>
<p>Note however that docker recommends that write-heavy containers should not use the container's writeable layer due to performance reasons. So it is anyway advisable to use a volume mount for the docker daemon directory.</p>
<h3>Image cache</h3>
<p>Another typical issue is the cache of docker builds or of docker pulls within docker-in-docker. If you spin up docker-in-docker daemon for a build every time, it will have zero cache. Or if you spin up a docker-in-docker daemon for an integration test, you will have to pull down everything all over again. This is obvious when you think about it, but it can be a key performance issue if not taken into account. There is currently no easy way to share cache between the host docker daemon and the docker-in-docker.</p>
<p>However, you may be able to mount a volume for the docker dir (<code>/var/lib/docker</code>), to use it across multiple runs. Just make sure you never mount it to two docker daemons at the same time as that's an easy way to get unpredictable behavior due to corruption. The docker dir was not designed to be accessed by more than one daemon concurrently.</p>
<h3>Performance</h3>
<p>I have seen no evidence in the community of any significant performance impact of using docker-in-docker (aside from the storage driver and cache considerations mentioned above).</p>
<h3>Security</h3>
<p>Docker-in-docker requires privileged mode, which can be an additional attack vector. There is ongoing work to support docker-in-docker in rootless mode, including using a fuse variant of <code>overlayfs</code>. This is less mature however.</p>
<h3>Workaround: Mounting docker.sock</h3>
<p>There is, of course, the possibility of mounting <code>/var/run/docker.sock</code> within a container and you can use the docker CLI within that container. This is not a real docker-in-docker, as the container merely uses a client to connect to the host's docker daemon. It is however a popular, hassle-free option for certain use cases. This is far easier to achieve, if your circumstances permit.</p>

CC BY-SA 4.0
2020-07-31T22:43:25.040Z
2020-07-31T22:43:25.040Z
Vlad A. Ionescu
<h1>Continuous Integration Concerns</h1>

<p>In short: Docker in Docker (dind) doesn't handle concurrency well.</p>

<p>The reason why you shouldn't use dind for CI is because Docker was designed to have exclusive access to the directory it uses for storage (normally <code>/var/lib/docker</code>). Dind doesn't respect this as all child processes use this directory concurrently. Every time you rebuild (from CI for example), anything related to your app in this directory could get wiped out and forced to start from zero. How would your users like it if they entered their payment details, clicked "Purchase", and suddenly found themselves back on the login screen as if they'd never done anything? That's just not good UX. Two rebuilds occur at once? That's really going to end badly for everyone involved (including your data integrity).</p>

<h1>Other Concerns</h1>

<p>From the link the OP posted, security concerns arise as the system will try to apply security policies in a very "CSS-like" fashion where a lower container could have access to an outer container's resources unless explicitly prohibited. Remember when you could access web server resources by doing something like "mywebsite.com/../another_folder/private_resource.txt"? Also, sometimes filesystems just don't play well with each other when they're nested in this way.</p>

<h1>The Fix</h1>

<p>Thankfully, the blog post in the OP has a good solution for the issue. Unless your needs are not met by "build/run/push Docker containers from your CI system itself running on Docker", you can use <a href="https://docs.docker.com/engine/tutorials/dockervolumes/#adding-a-data-volume" rel="noreferrer"><code>-v</code> mode</a> (add a data volume to your container) on the Docker socket (usually <code>/var/run/docker.sock:/var/run/docker.sock</code>) to allow the kind of access you need to the "shared" data volume. These containers will be started alongside the parent, instead of underneath, forcing synchronous IO. Now you have the same thing (almost) as dind but without the downsides that come with Docker not being build for concurrency.</p>

<p>Reference (from OP): <a href="http://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/" rel="noreferrer">Using Docker-in-Docker for your CI or testing environment? Think twice.</a></p>

CC BY-SA 3.0
2017-03-27T23:26:20.430Z
Here's one example of described approach (dood) for Jenkins, yet several issues reported while using it https://hub.docker.com/r/psharkey/jenkins-dood/
rombob
CC BY-SA 3.0
2017-11-12T11:41:41.377Z
From this explanation I still can't really tell if dind should be avoided in my case... 
My build agent runs in a docker container, and does the following: 1. `Checkout repo.` 2. `Start container & mount repo.` 3. `Run some build-/test script inside container.` Per agent, there is ever only one 'dind'-container running. Are there still issues with dind in this use-case?
helmesjo
CC BY-SA 4.0
2021-11-15T14:54:07.483Z
-1 this answer answer has several problems. CI/CD systems are better if they are stateless. You're not storing session data or anything related to runtime operations in the filesystem of your container!

"Sometimes filesystems just don't play well with each other when they're nested in this way" <-- citation needed.

Using docker-in-docker for CICD is now a standard practice, in fact both GitLab and GitHub have this as an officially supported configuration for their CICD systems.
Segfault
CC BY-SA 4.0
2022-01-24T13:33:31.890Z
Is this answer still updated, or is there a real solution nowadays?
Gulzar
CC BY-SA 4.0
2023-07-14T14:42:15.233Z
Oh great, now you can't run docker directly in k8s since 1.24 without a custom AMI, so I don't think so @Gulzar in fact, trying to find the current best practice has been a nightmare for me
Shanteva
CC BY-SA 3.0
2017-03-27T02:59:43.800Z
2018-04-05T15:55:40.763Z
2018-04-05T15:55:40.763Z
Peter G
<p>In August 2013 <a href="https://github.com/jpetazzo" rel="noreferrer">Jérôme Petazzoni</a> created Docker in Docker, <code>dind</code> for short, this allowed Docker containers to be created inside of Docker Containers, this functionality proved very popular resulting in <a href="https://github.com/jpetazzo/dind" rel="noreferrer">Jérôme's GitHub Repository</a> receiving over a thousand stars and three hundred forks.</p>

<p>As of Docker 1.8, released two years later in August 2015, Docker in Docker is directly supported by Docker out of the box.  However, the use of Docker in Docker comes with a warning, seemingly related to <a href="http://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/" rel="noreferrer">Jérôme's Post: Using Docker-in-Docker for your CI or testing environment? Think twice.</a> which focuses on the reasons why Docker in Docker is not a great choice for Continuous Integration.</p>

<p>Why is it considered bad to use Docker in Docker?  Is it simply a case for avoiding <a href="https://en.wikipedia.org/wiki/Turtles_all_the_way_down" rel="noreferrer">Turtles all the way down?</a> or performance considerations? </p>

<p><a href="https://i.sstatic.net/tnNkH.jpg" rel="noreferrer"><img src="https://i.sstatic.net/tnNkH.jpg" alt="Turtles all the way down!"></a></p>

CC BY-SA 3.0
2017-03-26T21:03:27.813Z
I'm not familiar with docker other than having read about it.  But thinking about it, it feel like you have the host OS on the hardware, that host loads a container, then that container loads another one.  Seems like a lot of overhead given that the idea is to deploy an image.  A picture of a picture of a picture ... Also interested in actual answers to this q.
No Refunds No Returns
CC BY-SA 3.0
2018-04-05T19:05:38.967Z
You are linking the answer to your question... or am I missing something?
AnoE
<docker><docker-in-docker>
Why is Docker-in-Docker considered bad?
Richard Slater
# Why is Docker-in-Docker considered bad?

(Asked by: username_0 on 2017-03-26)

In August 2013 [Jérôme Petazzoni](https://github.com/jpetazzo) created Docker in Docker, `dind` for short, this allowed Docker containers to be created inside of Docker Containers, this functionality proved very popular resulting in [Jérôme's GitHub Repository](https://github.com/jpetazzo/dind) receiving over a thousand stars and three hundred forks.

As of Docker 1.8, released two years later in August 2015, Docker in Docker is directly supported by Docker out of the box. However, the use of Docker in Docker comes with a warning, seemingly related to [Jérôme's Post: Using Docker-in-Docker for your CI or testing environment? Think twice.](http://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/) which focuses on the reasons why Docker in Docker is not a great choice for Continuous Integration.

Why is it considered bad to use Docker in Docker? Is it simply a case for avoiding [Turtles all the way down?](https://en.wikipedia.org/wiki/Turtles_all_the_way_down) or performance considerations?

[![Turtles all the way down!](https://i.sstatic.net/tnNkH.jpg)](https://i.sstatic.net/tnNkH.jpg)

**Tags:** docker, docker-in-docker

> **Comments:**
> **username_1**: I'm not familiar with docker other than having read about it.  But thinking about it, it feel like you have the host OS on the hardware, that host loads a container, then that container loads another one.  Seems like a lot of overhead given that the idea is to deploy an image.  A picture of a picture of a picture ... Also interested in actual answers to this q.  
> **username_2**: You are linking the answer to your question... or am I missing something?  

---

## 2 answers

### Answer by username_4 on 2017-03-27. Score: 20 | Accepted answer

# Continuous Integration Concerns

In short: Docker in Docker (dind) doesn't handle concurrency well.

The reason why you shouldn't use dind for CI is because Docker was designed to have exclusive access to the directory it uses for storage (normally `/var/lib/docker`). Dind doesn't respect this as all child processes use this directory concurrently. Every time you rebuild (from CI for example), anything related to your app in this directory could get wiped out and forced to start from zero. How would your users like it if they entered their payment details, clicked "Purchase", and suddenly found themselves back on the login screen as if they'd never done anything? That's just not good UX. Two rebuilds occur at once? That's really going to end badly for everyone involved (including your data integrity).

# Other Concerns

From the link the OP posted, security concerns arise as the system will try to apply security policies in a very "CSS-like" fashion where a lower container could have access to an outer container's resources unless explicitly prohibited. Remember when you could access web server resources by doing something like "mywebsite.com/../another\_folder/private\_resource.txt"? Also, sometimes filesystems just don't play well with each other when they're nested in this way.

# The Fix

Thankfully, the blog post in the OP has a good solution for the issue. Unless your needs are not met by "build/run/push Docker containers from your CI system itself running on Docker", you can use [`-v` mode](https://docs.docker.com/engine/tutorials/dockervolumes/#adding-a-data-volume) (add a data volume to your container) on the Docker socket (usually `/var/run/docker.sock:/var/run/docker.sock`) to allow the kind of access you need to the "shared" data volume. These containers will be started alongside the parent, instead of underneath, forcing synchronous IO. Now you have the same thing (almost) as dind but without the downsides that come with Docker not being build for concurrency.

Reference (from OP): [Using Docker-in-Docker for your CI or testing environment? Think twice.](http://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/)

> **Comments:**
> **username_5**: Here's one example of described approach (dood) for Jenkins, yet several issues reported while using it https://hub.docker.com/r/psharkey/jenkins-dood/  
> **username_6**: From this explanation I still can't really tell if dind should be avoided in my case...  My build agent runs in a docker container, and does the following: 1. `Checkout repo.` 2. `Start container & mount repo.` 3. `Run some build-/test script inside container.` Per agent, there is ever only one 'dind'-container running. Are there still issues with dind in this use-case?  
> **username_7**: -1 this answer answer has several problems. CI/CD systems are better if they are stateless. You're not storing session data or anything related to runtime operations in the filesystem of your container!  "Sometimes filesystems just don't play well with each other when they're nested in this way" <-- citation needed.  Using docker-in-docker for CICD is now a standard practice, in fact both GitLab and GitHub have this as an officially supported configuration for their CICD systems.  
> **username_8**: Is this answer still updated, or is there a real solution nowadays?  
> **username_9**: Oh great, now you can't run docker directly in k8s since 1.24 without a custom AMI, so I don't think so @username_8 in fact, trying to find the current best practice has been a nightmare for me  

---

### Answer by username_3 on 2020-07-31. Score: 0

Docker-in-docker has gotten better over the years and the docker dind image is getting more maturity. However there are still a number of caveats that make its use relatively difficult to get right.
Storage driver
The main challenge of getting docker-in-docker to work well is the storage driver of the docker installation. On your host you would have filesystems like ext4, which docker can run overlayfs2 on top of. However, within a docker container, docker is unable to use overlayfs2 as the backing filesystem of another overlayfs2. It doesn't work recursively. There are no easy setups to create a recursive docker-in-docker storage driver stack that works well.
The only storage driver that works on anything is vfs. However, vfs does not have copy-on-write capabilities, meaning that every docker build layer is actually a complete copy of the previous layers. In docker builds with many layers, high amounts of duplication cause performance issues as well as they quickly exhaust disk space. So it's usually not a real solution.
A possible workaround is to use a volume mount for the directory of the inner docker daemon's storage (/var/lib/docker). Then the inner docker daemon can use the host filesystem directly and can thus run overlayfs2 on it.
Another workaround would be to create a loop device formatted as ext4 which can then be used as the backing filesystem for the inner docker daemon. To avoid pre-allocating space on the disk, you can use a sparse file for the loop device. For example
dd if=/dev/zero of="$image_file" bs=1M count=0 seek="$sparse_loop_device_size_mb"
mkfs.ext4 "$image_file"
mount -n -o loop,noatime,nodiratime,noexec,noauto "$image_file" /var/lib/docker

(The size of the loop device here is the maximum size, not the initial size)
Note however that docker recommends that write-heavy containers should not use the container's writeable layer due to performance reasons. So it is anyway advisable to use a volume mount for the docker daemon directory.
Image cache
Another typical issue is the cache of docker builds or of docker pulls within docker-in-docker. If you spin up docker-in-docker daemon for a build every time, it will have zero cache. Or if you spin up a docker-in-docker daemon for an integration test, you will have to pull down everything all over again. This is obvious when you think about it, but it can be a key performance issue if not taken into account. There is currently no easy way to share cache between the host docker daemon and the docker-in-docker.
However, you may be able to mount a volume for the docker dir (/var/lib/docker), to use it across multiple runs. Just make sure you never mount it to two docker daemons at the same time as that's an easy way to get unpredictable behavior due to corruption. The docker dir was not designed to be accessed by more than one daemon concurrently.
Performance
I have seen no evidence in the community of any significant performance impact of using docker-in-docker (aside from the storage driver and cache considerations mentioned above).
Security
Docker-in-docker requires privileged mode, which can be an additional attack vector. There is ongoing work to support docker-in-docker in rootless mode, including using a fuse variant of overlayfs. This is less mature however.
Workaround: Mounting docker.sock
There is, of course, the possibility of mounting /var/run/docker.sock within a container and you can use the docker CLI within that container. This is not a real docker-in-docker, as the container merely uses a client to connect to the host's docker daemon. It is however a popular, hassle-free option for certain use cases. This is far easier to achieve, if your circumstances permit.


---

