<p>A very interesting question. On the assumption that the goal of a Software Escrow process is to allow for a 3<sup>rd</sup> Party to take over or nominate an additional party to fulfil the responsibility of the software vendor, I would suggest the following elements of a DevOps operating model that would support software escrow:</p>

<ol>
<li><p><strong>Infrastructure-as-code</strong> - effectively documenting through an executable specification of dependant infrastructure, stored and versioned in source control provides the environments that the source code is developed in.  Unlike static documentation in text files because it is executed on a regular basis by the software vendor to build their own environments it does not go out of date of suffer from "bit rot".  This will always hold the most value when the entire development pipeline is built and maintained in source control applying Infrastructure-as-code principals.</p></li>
<li><p><strong>Continuous Integration</strong> - the goal of continuous integration is to execute a set of steps that integrate the solution on a regular basis, ideally upon each change.  Typically this means upon check-in and push to a central repository a set of tests are executed that validates the process.  From a software escrow perspective I would expect this to also push a working version to a secondary "backup" repository owned and operated by the 3<sup>rd</sup> parties.  It is important to note that this does need to be both legally and financially "unlinked" from the software vendor.</p></li>
<li><p><strong>Continuous Deployment</strong> - the goal of continuous deployment is to deliver software in a working state on a frequent basis.  In this sense the 3<sup>rd</sup> party is just another deployment target to deploy the outputs to, albeit probably not actively spinning up infrastructure on the 3<sup>rd</sup> parties fabric.</p></li>
</ol>

<p>Some other considerations to bring into the equation are:</p>

<ul>
<li>the move from static documentation to infrastructure as code significantly reduces the toil involved in updating documentation that describes how to install, configure and recover software.</li>
<li>don't forget secrets management, such as X.509 Certificates, symmetric keys, passwords and licence keys, these could be stored in source control however this does have it's own drawbacks.  </li>
</ul>

<p>From a tools perspective this will depend very much upon the development environment, I don't believe there are any tools specific to software escrow however:</p>

<ul>
<li><p><strong>.NET Development</strong></p>

<ul>
<li><a href="https://github.com" rel="nofollow noreferrer">GitHub</a> or <a href="https://bitbucket.com" rel="nofollow noreferrer">BitBucket</a> for source control</li>
<li><a href="https://appveyor.com/" rel="nofollow noreferrer">AppVeyor</a>, <a href="https://www.jetbrains.com/teamcity/" rel="nofollow noreferrer">TeamCity</a> or <a href="https://jenkins.io/" rel="nofollow noreferrer">Jenkins</a> for the build and test elements of the pipeline.</li>
<li><a href="https://www.ansible.com/" rel="nofollow noreferrer">Ansible</a>, <a href="https://puppet.com/" rel="nofollow noreferrer">Puppet</a> or <a href="https://www.chef.io/chef/" rel="nofollow noreferrer">Chef</a> for the configuration management elements of the pipeline.</li>
<li><a href="https://inedo.com/proget" rel="nofollow noreferrer">ProGet</a> for the package management elements of the pipeline</li>
<li><a href="https://octopus.com/" rel="nofollow noreferrer">Octopus Deploy</a> for the deployment aspects of the pipeline</li>
</ul></li>
<li><p><strong>Java Development, and to some extend Open Source platforms</strong></p>

<ul>
<li><a href="https://circleci.com/" rel="nofollow noreferrer">CircleCI</a>, <a href="https://travis-ci.org/" rel="nofollow noreferrer">TravisCI</a>, <a href="https://www.jetbrains.com/teamcity/" rel="nofollow noreferrer">TeamCity</a> or <a href="https://jenkins.io/" rel="nofollow noreferrer">Jenkins</a> for the build and test elements of the pipeline.</li>
<li><a href="https://www.ansible.com/" rel="nofollow noreferrer">Ansible</a>, <a href="https://puppet.com/" rel="nofollow noreferrer">Puppet</a> or <a href="https://www.chef.io/chef/" rel="nofollow noreferrer">Chef</a> for the configuration management elements of the pipeline.</li>
<li><a href="https://www.jfrog.com/artifactory/" rel="nofollow noreferrer">JFrog Artifactory</a> or <a href="http://www.sonatype.org/nexus/" rel="nofollow noreferrer">Sonartype Nexus</a> for the package management elements of the pipeline</li>
<li><a href="http://capistranorb.com/" rel="nofollow noreferrer">Capistrano</a> for the deployment aspects of the pipeline</li>
</ul></li>
</ul>

<p>At the end of the day if you are able to employ Infrastructure-as-code, Continuous Integration and Continuous Deployment principles to a software package it can be used to fulfil your obligations under a Software Escrow contract.</p>

CC BY-SA 3.0
2017-03-24T20:30:02.940Z
What would necessitate Capistrano on top of puppet or chef ?
Tensibai
CC BY-SA 3.0
2017-03-24T20:31:10.243Z
Impressive answer, I need to digest it a few more times. But for now aready this comments/thoughts: (1) who do you mean by "supplier" (can you try to use "software vendor" or "SE-agent" to clarify that)? (2) "secrets" in this case would be the license keys (consider the as pieces of code also, eg containing PU ids, etc) (3) As an ITer I do agree with the "continuous" recommendation. However "continuous deposits" would also imply "continuous invoices" (= not affordable), so I'm not sure (yet) how the "release" (like 1 every 6 months) fits in this
Pierre.Vriens
CC BY-SA 3.0
2017-03-25T00:26:08.200Z
@Tensibai: the way I look at it is this, [Capistrano](http://capistranorb.com/) is a deployment tool with a configuration management capability; [Ansible](https://www.ansible.com/) is a configuration management tool with a deployment capability... just because you can use one tool to do both tasks [doesn't mean you should](http://codefol.io/posts/why-do-we-need-both-capistrano-and-chef).
Richard Slater
CC BY-SA 3.0
2017-03-25T00:39:55.917Z
@Pierre.Vriens - I've attempted to address both your point **1** and point **2**.  Point **3**, however, may need a little more input, fundamentally the industry is changing if an SE-agent is suggesting that each deposit results in an invoice then, respectfully they need to re-visit their cost model.  The consequences of them not doing so is the difference between thriving in a DevOps world and fading into obsolescence.
Richard Slater
CC BY-SA 3.0
2017-03-25T07:08:14.590Z
This blog post is a little old and doesn't take into account some chef [ressource](https://docs.chef.io/resource_deploy.html), here deploy inspired by Capistrano. Still doesn't mean you should or should not, but did worth a word about it IMHO
Tensibai
CC BY-SA 3.0
2017-03-25T19:29:08.973Z
Richard, check the "note 2." I added, to clarify that "point 3" in my prior comment. Also, it seems to me that what your describing in your answer here is like a recommendation for the Software Vendor to use DevOps practices to improve their SDLC and related things. But looking at it from the SE-agent and/or licensed customer (who both don't really care about how the Software Vendor's SDLC looks like, they only care about those major new releases), I don't see how it would help for the fulfillment of the various parts of the SE-agreement.
Pierre.Vriens
CC BY-SA 3.0
2017-03-24T20:11:42.107Z
2017-03-25T00:28:31.853Z
2017-03-25T00:28:31.853Z
Richard Slater
<blockquote>
  <p>But to actually "process the SE-deposits" (by the SE-agents), all
  sorts of release management and/or software delivery tasks need to be
  performed by somebody or something (the poor SE-agent) who probably
  doesn't know at all what the licensed software is supposed to do ...
  fun guaranteed!</p>
</blockquote>

<p>I wouldn't do business with an SE provider which would allow such unfortunate situation to exist - they don't know what they're doing.</p>

<p>If the SE deposits are to be processed in any manner by the SE agent the exact and complete processing procedure needs to be <em>fully documented</em> in the agreements so that it is actually usable. </p>

<p>That should also include the exact environment specification (or procedure to reproduce it) and the toolchain used for such processing. In other words the choice is not really open unilaterally to the SE agent. Except maybe for the actual storage strategy (personally I'd include that choice in the agreement specification as well, even if the choice is done unilaterally by the SE agent).</p>

<p>A good SE agreement would also ensure that each and every SE deposit by the vendor goes through that processing and the customer always gets and <em>qualifies/signs off</em> the result of that particular processing, not an alternate result coming directly from the vendor - to validate that the procedure itself remains up2date and effective for each and every SE deposit.</p>

<p>Otherwise the ability to reproduce the "SE withdrawals" at a later time (if/when needed) is questionable, which practically voids the whole SE story.</p>

CC BY-SA 3.0
2017-03-26T14:56:04.570Z
Merci Dan for this answer. I do agree with (most of) what you wrote, but it seems that you've not taken into account my "note 1". Your answer here seems related to a (new) question like "What are the typical levels of verification offered in SE-agreements (and how to select one)?". FYI: IMO there are 3 levels ... Not sure if I should post that some day as a self answered question ...
Pierre.Vriens
CC BY-SA 3.0
2017-03-26T15:08:49.900Z
I did see note 1, but if I apply it then I fail to see how the question makes sense :) If there is no processing pipeline then what is the devops toolset used for? Except maybe for the CRM/storage (my answer touches that as well).
Dan Cornilescu
CC BY-SA 3.0
2017-03-26T18:33:06.420Z
Not sure what you mean with that CRM in your comment (can you retry?). Also, imagine the (poor) SE-agent accepts deposits in the format of CDs that get physically delivered to their office (eg 1 to 3 CDs per year for "a" software vendor, and there are a few dozens of such software vendors). If the SE-agent would hire you to modernize this outdated approach by introducing DevOps compliant procedures, then what would you recommend?
Pierre.Vriens
CC BY-SA 3.0
2017-03-26T21:21:45.303Z
@Pierre.Vriens Sorry, mixed up acronyms. Not CRM. I meant artifact repository/storage.
Dan Cornilescu
CC BY-SA 3.0
2017-03-26T21:58:59.097Z
@Pierre.Vriens If receiving CDs is what's in the agreement then storing and retrieving CDs (or CD images) is pretty much all there is to it. If it's more than that but it doesn't involve processing - I'm not sure what that is. If processing remains on the table - there's plenty that can be done. At the extreme you can consider all 3 parts of the SE agreement involved into the sw delivery pipeline which produces sw that is guaranteed to remain maintainable even in a catastrophic vendor event.
Dan Cornilescu
CC BY-SA 3.0
2017-03-26T14:42:57.193Z
2017-03-26T14:42:57.193Z
Dan Cornilescu
<p>Consider a software vendor and a licensed customer of some software of this vendor, wheres the software being licensed is either used on premise (at the customer's location), or in the format of SaaS solution (hosted by the vendor). However the customer only gets access to what's needed to use/run the software (executables and similar things), so not the source components and anything related to that to create the executables.</p>

<p>To protect the customer's business continuity in scenarios where something may go wrong with the vendor (eg: bankruptcy), both parties may agree to some sort of Software Escrow (SE ... 'also') agreement (also called <a href="https://en.wikipedia.org/wiki/Source_code_escrow" rel="noreferrer">Source code escrow</a>). With such agreement both parties agree to get a 3rd party involved (= a "Software Escrow agent"), trusted by both parties. These are the highlights of such SE-agreement (= specs of the actual SE-process):</p>

<ul>
<li>ALL pieces of software components (<a href="/questions/tagged/artifacts" class="post-tag" title="show questions tagged &#39;artifacts&#39;" rel="tag">artifacts</a> related to the licensed software) get deposited by the software vendor, at some agreed location related to the SE-agent. Such deposits include the executables, but also the source components and anything related to that to create the executables (even documentation, instructions, etc to create the executables).</li>
<li>Since the software vendor may create multiple releases for the duration of the software license, and the customer has the right to receive such new releases (as per the license agreement), part of such SE-agreement is that "with every major new release" (whatever "major" may mean ...), the deposit delivered to the SE-agent will also be updated/refreshed.</li>
<li>If specific conditions are satisfied (e.g: bankruptcy of the vendor), then the Software Escrow agent will deliver to the licensed customer, upon request of such customer, of copy of everything that was deposited, so that customer will be able to continue using the software, and where needed even adapt the source code to continue to use it for the customer's business.</li>
</ul>

<p>A common practice for such SE-agent to get involved, is some sort of a legal person/entity, such as a lawyer. But to actually "process the SE-deposits" (by the SE-agents), all sorts of release management and/or software delivery tasks need to be performed by somebody or something (the poor SE-agent) who probably doesn't know at all what the licensed software is supposed to do ... fun guaranteed!</p>

<p><strong>My question</strong>:</p>

<p>How can DevOps help to improve Software Escrow procedures as described above? Like what kind of <a href="/questions/tagged/toolchain" class="post-tag" title="show questions tagged &#39;toolchain&#39;" rel="tag">toolchain</a>-tools would you recommend to be used for the fulfillment of which part of the SE-agreement? And where appropriate using which (preferably open source) software solutions for it?</p>

<p><strong>Notes</strong>:</p>

<ol>
<li><p>To not further complicate things, just assume that it is agreed between all parties involved, that the SE-agent does NOT have to do any type of "<em>verifications</em>" about the deposits being done. That is: whatever is deposited is assumed to be complete, up-to-date, documented, etc.</p></li>
<li><p>About "major new release": assume there are between 1 to 3 every year, which means that the licensed customer only expects to be able to get access (via the SE-agent) to those releases. Even though if there have been intermediate deliveries (like fixes or beta versions) to the licensed customer, those types of deliveries are considered out-of-scope. Even if it was only because:</p>

<ul>
<li>the SE-agent charges "for each deposit to be processed by the SE-agent".</li>
<li>the licensed customer rarely changes releases, and is only interested in being able to use the SE-agreement if things go wrong, for the very release they are running at the moment things go wrong.</li>
</ul></li>
</ol>

<release-management>
How can DevOps help to improve Software Escrow procedures?
Pierre.Vriens
# How can DevOps help to improve Software Escrow procedures?

(Asked by: username_0 on 2017-03-24)

Consider a software vendor and a licensed customer of some software of this vendor, wheres the software being licensed is either used on premise (at the customer's location), or in the format of SaaS solution (hosted by the vendor). However the customer only gets access to what's needed to use/run the software (executables and similar things), so not the source components and anything related to that to create the executables.

To protect the customer's business continuity in scenarios where something may go wrong with the vendor (eg: bankruptcy), both parties may agree to some sort of Software Escrow (SE ... 'also') agreement (also called [Source code escrow](https://en.wikipedia.org/wiki/Source_code_escrow)). With such agreement both parties agree to get a 3rd party involved (= a "Software Escrow agent"), trusted by both parties. These are the highlights of such SE-agreement (= specs of the actual SE-process):

* ALL pieces of software components ([artifacts](/questions/tagged/artifacts "show questions tagged 'artifacts'") related to the licensed software) get deposited by the software vendor, at some agreed location related to the SE-agent. Such deposits include the executables, but also the source components and anything related to that to create the executables (even documentation, instructions, etc to create the executables).
* Since the software vendor may create multiple releases for the duration of the software license, and the customer has the right to receive such new releases (as per the license agreement), part of such SE-agreement is that "with every major new release" (whatever "major" may mean ...), the deposit delivered to the SE-agent will also be updated/refreshed.
* If specific conditions are satisfied (e.g: bankruptcy of the vendor), then the Software Escrow agent will deliver to the licensed customer, upon request of such customer, of copy of everything that was deposited, so that customer will be able to continue using the software, and where needed even adapt the source code to continue to use it for the customer's business.

A common practice for such SE-agent to get involved, is some sort of a legal person/entity, such as a lawyer. But to actually "process the SE-deposits" (by the SE-agents), all sorts of release management and/or software delivery tasks need to be performed by somebody or something (the poor SE-agent) who probably doesn't know at all what the licensed software is supposed to do ... fun guaranteed!

**My question**:

How can DevOps help to improve Software Escrow procedures as described above? Like what kind of [toolchain](/questions/tagged/toolchain "show questions tagged 'toolchain'")-tools would you recommend to be used for the fulfillment of which part of the SE-agreement? And where appropriate using which (preferably open source) software solutions for it?

**Notes**:

1. To not further complicate things, just assume that it is agreed between all parties involved, that the SE-agent does NOT have to do any type of "*verifications*" about the deposits being done. That is: whatever is deposited is assumed to be complete, up-to-date, documented, etc.
2. About "major new release": assume there are between 1 to 3 every year, which means that the licensed customer only expects to be able to get access (via the SE-agent) to those releases. Even though if there have been intermediate deliveries (like fixes or beta versions) to the licensed customer, those types of deliveries are considered out-of-scope. Even if it was only because:

   * the SE-agent charges "for each deposit to be processed by the SE-agent".
   * the licensed customer rarely changes releases, and is only interested in being able to use the SE-agreement if things go wrong, for the very release they are running at the moment things go wrong.

**Tags:** release-management


---

## 2 answers

### Answer by username_1 on 2017-03-24. Score: 5

A very interesting question. On the assumption that the goal of a Software Escrow process is to allow for a 3rd Party to take over or nominate an additional party to fulfil the responsibility of the software vendor, I would suggest the following elements of a DevOps operating model that would support software escrow:

1. **Infrastructure-as-code** - effectively documenting through an executable specification of dependant infrastructure, stored and versioned in source control provides the environments that the source code is developed in. Unlike static documentation in text files because it is executed on a regular basis by the software vendor to build their own environments it does not go out of date of suffer from "bit rot". This will always hold the most value when the entire development pipeline is built and maintained in source control applying Infrastructure-as-code principals.
2. **Continuous Integration** - the goal of continuous integration is to execute a set of steps that integrate the solution on a regular basis, ideally upon each change. Typically this means upon check-in and push to a central repository a set of tests are executed that validates the process. From a software escrow perspective I would expect this to also push a working version to a secondary "backup" repository owned and operated by the 3rd parties. It is important to note that this does need to be both legally and financially "unlinked" from the software vendor.
3. **Continuous Deployment** - the goal of continuous deployment is to deliver software in a working state on a frequent basis. In this sense the 3rd party is just another deployment target to deploy the outputs to, albeit probably not actively spinning up infrastructure on the 3rd parties fabric.

Some other considerations to bring into the equation are:

* the move from static documentation to infrastructure as code significantly reduces the toil involved in updating documentation that describes how to install, configure and recover software.
* don't forget secrets management, such as X.509 Certificates, symmetric keys, passwords and licence keys, these could be stored in source control however this does have it's own drawbacks.

From a tools perspective this will depend very much upon the development environment, I don't believe there are any tools specific to software escrow however:

* **.NET Development**

  + [GitHub](https://github.com) or [BitBucket](https://bitbucket.com) for source control
  + [AppVeyor](https://appveyor.com/), [TeamCity](https://www.jetbrains.com/teamcity/) or [Jenkins](https://jenkins.io/) for the build and test elements of the pipeline.
  + [Ansible](https://www.ansible.com/), [Puppet](https://puppet.com/) or [Chef](https://www.chef.io/chef/) for the configuration management elements of the pipeline.
  + [ProGet](https://inedo.com/proget) for the package management elements of the pipeline
  + [Octopus Deploy](https://octopus.com/) for the deployment aspects of the pipeline
* **Java Development, and to some extend Open Source platforms**

  + [CircleCI](https://circleci.com/), [TravisCI](https://travis-ci.org/), [TeamCity](https://www.jetbrains.com/teamcity/) or [Jenkins](https://jenkins.io/) for the build and test elements of the pipeline.
  + [Ansible](https://www.ansible.com/), [Puppet](https://puppet.com/) or [Chef](https://www.chef.io/chef/) for the configuration management elements of the pipeline.
  + [JFrog Artifactory](https://www.jfrog.com/artifactory/) or [Sonartype Nexus](http://www.sonatype.org/nexus/) for the package management elements of the pipeline
  + [Capistrano](http://capistranorb.com/) for the deployment aspects of the pipeline

At the end of the day if you are able to employ Infrastructure-as-code, Continuous Integration and Continuous Deployment principles to a software package it can be used to fulfil your obligations under a Software Escrow contract.

> **Comments:**
> **username_2**: What would necessitate Capistrano on top of puppet or chef ?  
> **username_0**: Impressive answer, I need to digest it a few more times. But for now aready this comments/thoughts: (1) who do you mean by "supplier" (can you try to use "software vendor" or "SE-agent" to clarify that)? (2) "secrets" in this case would be the license keys (consider the as pieces of code also, eg containing PU ids, etc) (3) As an ITer I do agree with the "continuous" recommendation. However "continuous deposits" would also imply "continuous invoices" (= not affordable), so I'm not sure (yet) how the "release" (like 1 every 6 months) fits in this  
> **username_1**: @username_2: the way I look at it is this, [Capistrano](http://capistranorb.com/) is a deployment tool with a configuration management capability; [Ansible](https://www.ansible.com/) is a configuration management tool with a deployment capability... just because you can use one tool to do both tasks [doesn't mean you should](http://codefol.io/posts/why-do-we-need-both-capistrano-and-chef).  
> **username_1**: @username_0 - I've attempted to address both your point **1** and point **2**.  Point **3**, however, may need a little more input, fundamentally the industry is changing if an SE-agent is suggesting that each deposit results in an invoice then, respectfully they need to re-visit their cost model.  The consequences of them not doing so is the difference between thriving in a DevOps world and fading into obsolescence.  
> **username_2**: This blog post is a little old and doesn't take into account some chef [ressource](https://docs.chef.io/resource_deploy.html), here deploy inspired by Capistrano. Still doesn't mean you should or should not, but did worth a word about it IMHO  
> **username_0**: Richard, check the "note 2." I added, to clarify that "point 3" in my prior comment. Also, it seems to me that what your describing in your answer here is like a recommendation for the Software Vendor to use DevOps practices to improve their SDLC and related things. But looking at it from the SE-agent and/or licensed customer (who both don't really care about how the Software Vendor's SDLC looks like, they only care about those major new releases), I don't see how it would help for the fulfillment of the various parts of the SE-agreement.  

---

### Answer by username_3 on 2017-03-26. Score: 1

> But to actually "process the SE-deposits" (by the SE-agents), all
> sorts of release management and/or software delivery tasks need to be
> performed by somebody or something (the poor SE-agent) who probably
> doesn't know at all what the licensed software is supposed to do ...
> fun guaranteed!

I wouldn't do business with an SE provider which would allow such unfortunate situation to exist - they don't know what they're doing.

If the SE deposits are to be processed in any manner by the SE agent the exact and complete processing procedure needs to be *fully documented* in the agreements so that it is actually usable.

That should also include the exact environment specification (or procedure to reproduce it) and the toolchain used for such processing. In other words the choice is not really open unilaterally to the SE agent. Except maybe for the actual storage strategy (personally I'd include that choice in the agreement specification as well, even if the choice is done unilaterally by the SE agent).

A good SE agreement would also ensure that each and every SE deposit by the vendor goes through that processing and the customer always gets and *qualifies/signs off* the result of that particular processing, not an alternate result coming directly from the vendor - to validate that the procedure itself remains up2date and effective for each and every SE deposit.

Otherwise the ability to reproduce the "SE withdrawals" at a later time (if/when needed) is questionable, which practically voids the whole SE story.

> **Comments:**
> **username_0**: Merci Dan for this answer. I do agree with (most of) what you wrote, but it seems that you've not taken into account my "note 1". Your answer here seems related to a (new) question like "What are the typical levels of verification offered in SE-agreements (and how to select one)?". FYI: IMO there are 3 levels ... Not sure if I should post that some day as a self answered question ...  
> **username_3**: I did see note 1, but if I apply it then I fail to see how the question makes sense :) If there is no processing pipeline then what is the devops toolset used for? Except maybe for the CRM/storage (my answer touches that as well).  
> **username_0**: Not sure what you mean with that CRM in your comment (can you retry?). Also, imagine the (poor) SE-agent accepts deposits in the format of CDs that get physically delivered to their office (eg 1 to 3 CDs per year for "a" software vendor, and there are a few dozens of such software vendors). If the SE-agent would hire you to modernize this outdated approach by introducing DevOps compliant procedures, then what would you recommend?  
> **username_3**: @username_0 Sorry, mixed up acronyms. Not CRM. I meant artifact repository/storage.  
> **username_3**: @username_0 If receiving CDs is what's in the agreement then storing and retrieving CDs (or CD images) is pretty much all there is to it. If it's more than that but it doesn't involve processing - I'm not sure what that is. If processing remains on the table - there's plenty that can be done. At the extreme you can consider all 3 parts of the SE agreement involved into the sw delivery pipeline which produces sw that is guaranteed to remain maintainable even in a catastrophic vendor event.  

---

