<p>As this question is quite high-level, I give a high level answer.</p>

<p>Technically, if there is an incident, you would normally enforce a shutdown of a running container and a replacement with a newer image with an implementation of the change request resulting from the incident.</p>

<p>How much automation will be possible to get there, I think it really dependends on single cases or at best their classes.</p>

<p><strong>Best is therefore I think to assess different scenarios, group them into classes and derive solution workflows. Once a problem catalogue and a required set of organization policies are there you will be able to sort out which tools can help to realize them and which automations grades are possible.</strong></p>

<p>For example:</p>

<ul>
<li>discovered system environment level vulnerabilities are normally patchable through operation system security upgrades. You could have batch job delivering daily updated base images.</li>
<li>application environment level changes like "let's migrate from Java 8 to Java 9" is an organizational decision which requires an organizational process</li>
<li>application code level vulnerabitites requiring updating to patched versions requires some ways of dependency management governance in your organization, tools for quality gates are available but without an organizational decision nothing will happen here.</li>
<li>default credentials in production - again where do they come from? This is very special case and possibly a design failure; not sure how solve this in an automated way.</li>
</ul>

<p>So I have to conclude that this question is more about culture than automation.</p>

CC BY-SA 3.0
2018-02-27T12:13:30.343Z
though your answer is high-level it provides some clarity. So this is purely DevOps - culture. I'd really like to understand the low level stuff also but probably with more research into the tooling approaches and best practices. Thanks :).
SyCode
CC BY-SA 3.0
2018-02-27T11:47:21.550Z
2018-02-27T11:47:21.550Z
Ta Mu
<p>Docker and other container images have become important building blocks especially for software production pipelines. Unfortunately, several researchers (e.g. <a href="https://banyanops.com/blog/analyzing-docker-hub/" rel="nofollow noreferrer">BanyanOps</a>) have shown how vulnerable docker images can be , including those from the official distributors. I was researching into the possibilities for automatic patching of such vulnerable images when discovered. According to <a href="https://blogs.msdn.microsoft.com/stevelasker/2017/12/20/os-framework-patching-with-docker-containers-paradigm-shift/" rel="nofollow noreferrer">this link</a>, container patching is very sensitive and error-prone. Infact, there seems to be no solution as all the papers/reports about container image vulnerabilities do not proffer patching approaches. I will be grateful if someone has some opinions, options or even a possible solution.</p>

<p><strong>Note</strong>: I am interested in how patching is conducted with or without update notifications.</p>

CC BY-SA 3.0
2018-02-26T23:16:47.927Z
Possible duplicate of [What are best and comprehensive practices to consider when running docker in production?](https://devops.stackexchange.com/questions/1943/what-are-best-and-comprehensive-practices-to-consider-when-running-docker-in-pro)
030
CC BY-SA 3.0
2018-02-26T23:35:26.467Z
@030 the issue of how to patch was not discussed there. The closest was getting notifications for patches. I am interested in how patching is conducted with or without update notifications.
SyCode
CC BY-SA 3.0
2018-02-27T07:59:23.757Z
wait but what does mean patching here? if this is about other libraries, you just exchange the image. Please elaborate more on scenarios which bring you in trouble.
Ta Mu
CC BY-SA 3.0
2018-02-27T10:17:43.577Z
@PeterMuryshkin yes exchanging the images i.e. updating the Dockerfiles to reflect the non-vulnerable packages. For example if the you update an image from Java 8 to Java 9, the application built into the eventual container might not be compliant with Java 9. Therefore there might be some extra steps e.g compatibility tests. So, this gets more complex for large pipelines, and automation becomes handy. Are there methods that deal with this kind of challenge or someone has some experience to share.
SyCode
CC BY-SA 3.0
2018-02-27T10:49:42.900Z
@SyCode oh I see I thought on more granular level like you find a vulnerable JAR and in a DevOps culture developers are able to fix it. Moving from Java 8 to Java 9 means to me rather an organizational migration decision which needs to be properly scheduled: this is more than just a patch.
Ta Mu
CC BY-SA 3.0
2018-02-27T11:32:22.910Z
@PeterMuryshkin thanks for your response. Fixing jars will be more like correcting the erroneous code and recompiling right? Yeah the Java example might be too high level. How about if an image containing openssh is discovered to have default credentials configured. How will fixing this be done, maybe the images have already been deployed in the production. Sorry, I have no experience hence my questions. Thanks
SyCode
<docker><continuous-integration><culture><build-pipeline>
Automated patching of docker images
SyCode
# Automated patching of docker images

(Asked by: username_0 on 2018-02-26)

Docker and other container images have become important building blocks especially for software production pipelines. Unfortunately, several researchers (e.g. [BanyanOps](https://banyanops.com/blog/analyzing-docker-hub/)) have shown how vulnerable docker images can be , including those from the official distributors. I was researching into the possibilities for automatic patching of such vulnerable images when discovered. According to [this link](https://blogs.msdn.microsoft.com/stevelasker/2017/12/20/os-framework-patching-with-docker-containers-paradigm-shift/), container patching is very sensitive and error-prone. Infact, there seems to be no solution as all the papers/reports about container image vulnerabilities do not proffer patching approaches. I will be grateful if someone has some opinions, options or even a possible solution.

**Note**: I am interested in how patching is conducted with or without update notifications.

**Tags:** docker, continuous-integration, culture, build-pipeline

> **Comments:**
> **username_1**: Possible duplicate of [What are best and comprehensive practices to consider when running docker in production?](https://devops.stackexchange.com/questions/1943/what-are-best-and-comprehensive-practices-to-consider-when-running-docker-in-pro)  
> **username_0**: @username_1 the issue of how to patch was not discussed there. The closest was getting notifications for patches. I am interested in how patching is conducted with or without update notifications.  
> **username_2**: wait but what does mean patching here? if this is about other libraries, you just exchange the image. Please elaborate more on scenarios which bring you in trouble.  
> **username_0**: @PeterMuryshkin yes exchanging the images i.e. updating the Dockerfiles to reflect the non-vulnerable packages. For example if the you update an image from Java 8 to Java 9, the application built into the eventual container might not be compliant with Java 9. Therefore there might be some extra steps e.g compatibility tests. So, this gets more complex for large pipelines, and automation becomes handy. Are there methods that deal with this kind of challenge or someone has some experience to share.  
> **username_2**: @username_0 oh I see I thought on more granular level like you find a vulnerable JAR and in a DevOps culture developers are able to fix it. Moving from Java 8 to Java 9 means to me rather an organizational migration decision which needs to be properly scheduled: this is more than just a patch.  
> **username_0**: @PeterMuryshkin thanks for your response. Fixing jars will be more like correcting the erroneous code and recompiling right? Yeah the Java example might be too high level. How about if an image containing openssh is discovered to have default credentials configured. How will fixing this be done, maybe the images have already been deployed in the production. Sorry, I have no experience hence my questions. Thanks  

---

## 1 answers

### Answer by username_2 on 2018-02-27. Score: 4 | Accepted answer

As this question is quite high-level, I give a high level answer.

Technically, if there is an incident, you would normally enforce a shutdown of a running container and a replacement with a newer image with an implementation of the change request resulting from the incident.

How much automation will be possible to get there, I think it really dependends on single cases or at best their classes.

**Best is therefore I think to assess different scenarios, group them into classes and derive solution workflows. Once a problem catalogue and a required set of organization policies are there you will be able to sort out which tools can help to realize them and which automations grades are possible.**

For example:

* discovered system environment level vulnerabilities are normally patchable through operation system security upgrades. You could have batch job delivering daily updated base images.
* application environment level changes like "let's migrate from Java 8 to Java 9" is an organizational decision which requires an organizational process
* application code level vulnerabitites requiring updating to patched versions requires some ways of dependency management governance in your organization, tools for quality gates are available but without an organizational decision nothing will happen here.
* default credentials in production - again where do they come from? This is very special case and possibly a design failure; not sure how solve this in an automated way.

So I have to conclude that this question is more about culture than automation.

> **Comments:**
> **username_0**: though your answer is high-level it provides some clarity. So this is purely DevOps - culture. I'd really like to understand the low level stuff also but probably with more research into the tooling approaches and best practices. Thanks :).  

---

