<p>Answer based on my knowledge of French "Internal controls" regulation, kind of  equivalent to the SEC regulations you point to, I assume linking here to a French legal text would not be really useful and I know of no good translation of it.</p>

<p>In an ideal 'You build it, you run it' model, everyone in the team will be accountable for the change. The risk assessment can't be enforced by a separation of duties and the only way I know to keep conforming with regulation is to have a periodic short cycle audit of transactions along with an unalterable action tracking to get back to the person who did the release.<br>
This mean all logs of transaction and actions are pushed to a restricted area the team doesn't have access to, a change in the what is logged "should" be catch by functional tests the team has no access to and at worse will be catch by the audit and tracked to its author.  </p>

<p>This is not applicable to all products, at time of writing in France any company allowed to emit money (mainly banks), has to ensure every transaction will be recorded and thus can't take the risk of missing a transaction.<br>
On the other hand, they have no legal obligation to track any commercial offer or risk evaluation when someone ask for a loan, and thus the products handling this customer selection and computing the fees which will be in the offer are easier to fit in a post-release audit model.</p>

<p>Main idea is that the release model has to be tweaked according to risk assessment obligations.</p>

<p>A related resource is the <a href="https://en.wikipedia.org/wiki/ISO/IEC_27001:2005" rel="nofollow noreferrer">ISO27001</a> norm.</p>

CC BY-SA 3.0
2017-03-21T07:51:30.733Z
Interesting answer and very relevant as many European banks do indeed operate in France.  Is there any chance you could expand on what 'Emit Money' means, i.e. is that just cash out of an ATM or does it include balance transfers.  In this case linking to the statutes would be valuable as it gives a pointer to the relevant laws, regardless of the language they are in.
Richard Slater
CC BY-SA 3.0
2017-03-21T08:28:09.347Z
@RichardSlater In brief, any company working with money, could be an investment only company as well as loan brokers along the traditional banks. Mostly anything which have a financial impact somewhere is concerned (out of few exception which the authority can give under come circonstances). The legal "list" in French is [here](https://www.legifrance.gouv.fr/affichTexte.do?cidTexte=JORFTEXT000029700770&dateTexte=20170320) but even in French it is not always obvious.
Tensibai
CC BY-SA 3.0
2017-03-21T18:17:26.010Z
I am making the assumption that the link to the ISO standard should actually be [ISO27001:2013](https://en.wikipedia.org/wiki/ISO/IEC_27001:2013)
Richard Slater
CC BY-SA 3.0
2017-03-21T18:30:33.987Z
@Richard indeed, seems the French to English link hasn't been updated on Wikipedia. I'll update later (or if you wish , feel free to edit)
Tensibai
CC BY-SA 3.0
2017-03-26T03:49:12.950Z
related: [How can I persuade developers on my team to embrace “You build it, you run it”?](http://devops.stackexchange.com/questions/484/how-can-i-persuade-developers-on-my-team-to-embrace-you-build-it-you-run-it)
Dan Cornilescu
CC BY-SA 3.0
2017-03-20T09:48:14.903Z
2017-03-21T12:26:12.090Z
2017-03-21T12:26:12.090Z
Tensibai
<p>IMHO, Developers &amp; Operations could be represented by nothing more than just two git repositories <strong>for the same codebase</strong>, with distinct permissions model each, so that teams will not interfer in the work of each other, at all.</p>

<p>Let's call them <strong>Dev.mygithub.co</strong> &amp; <strong>ops.mygithub.co</strong>, just as an example.</p>

<p>The idea is, that Developers are free to create/branch/merge to their heart's content -git is providing full traceability and that is what matters here- meanwhile, at the moments that regulatory framework implies review effort a <strong>Pull Request</strong> can be raised, for the merging to happen in a controlled manner.</p>

<p>Taking that concept to its next level, a <strong>develop</strong> branch can be propagated towards the remote Ops' <strong>production</strong> via yet another <strong>Pull Request</strong> act. 
That last part has to happen by operations' hands and eyes, since they have a responsibility to vet it into production and they choose the level of review.</p>

<p>Such scheme allows <em>infinite flexibility, full traceability, ability to catch issues early on via a variety of processes, separation of concerns and a very reasonable user experience in the process</em>!</p>

<p>N.B. The model described above may be followed even if Ops &amp; Dev overlap totally!</p>

CC BY-SA 3.0
2017-03-21T18:14:42.353Z
Surely, this same control could be achieved through pull requests and post-commit hooks that ensured that developers could commit freely, however, merge commits could only be made by an approved group of people.  Equally that same post-commit hook could ensure that of the authors of the commits that made up the pull request did not include the person making the pull request.
Richard Slater
CC BY-SA 3.0
2017-06-24T08:04:40.187Z
@RichardSlater: the reason you'd may wish to have two distinct repositories is that you have the dual need to both **allow** developers to merge -when they freely swap code in developer mode- as well as **block** most developers from merging code when it is to go towards production (modulo SysOps, i.e.. the so called "approved group of people").
fgeorgatos
CC BY-SA 3.0
2017-06-24T09:05:23.147Z
Again you can achieve that with post-commit hooks and pull requests, not to mention GitHub Enterprise allows protected branches.
Richard Slater
CC BY-SA 3.0
2017-03-21T18:06:52.737Z
2017-03-21T18:06:52.737Z
fgeorgatos
<p>in order, topmost is most expensive:</p>
<ul>
<li>distinct dev and ops sites &amp; methods to carry over work from one to the other =&gt; many sites</li>
<li>distinct dev and ops systems &amp; methods as above =&gt; 1 joint site</li>
<li>distinct dev and ops git/vcs repositories &amp; related methods =&gt; 1 system</li>
<li>distinct dev and ops git/vcs branches (protected) &amp; related methods =&gt; 1 repo, many branches</li>
</ul>
<p>Depending on what you do, some solutions are better than others; for example, if you have the need to serve two teams with distinct roles within them and each having ownership within and provide full traceability, you are hovering over first three.</p>
<p>In short, anything that enforces that one guy or gal cannot take the ball all alone and run with it, as s/he crosses a distinct explicit boundary among dev &amp; ops.</p>
<p>Now, depending on the subsequent level of risk, that boundary might be enforced or not. Choices, choices.</p>

CC BY-SA 4.0
2018-10-12T09:19:50.417Z
2021-08-16T23:00:13.847Z
2021-08-16T23:00:13.847Z
fgeorgatos
<p>Your question doesn't seem to make any assumption about the platform/OS it is about. Which is why it may make sense to add an answer about how this is typically done/addressed in a mainframe environment, where the "engineers" (as in your question title) are actually groups of people were dozens (possibly hundreds) of people are involved. My answer is based on using the SCM product where I'm most familiar with (not sure if it's needed to disclose the product name).</p>

<p><hr /></p>

<h3>1. Architecture</h3>

<hr />

<p>Here are the highlights of how I'd answer your question:</p>

<ul>
<li>All code (and related artifacts like executables, etc) is stored in files which, all together is what we call the <strong><em>library structure</em></strong>.</li>
<li>For each environment on each (possibly remote) target system, there is a server (a "started task" in mainframe speak), which takes care of ALL (repeat: ALL) updates to anything in the library structure. There are a few exceptions (like security people, or space management team), but apart from that, nobody (repeat: nobody) has authorization to apply updates to any file within that library structure. In other words: <strong><em>the server gets exclusive update authority to the entire library structure</em></strong>. Attention: OPS-people will go bonkers if you walk in to limit their access (at first they are going to resist ...), so make sure you're covered by upper management (CxO) to impose those access rules ...</li>
<li>The actual software changes my consist of a single component (a tiny code fix in the middle of the night ...), or it may also be hundreds or thousands of sources, executables, or whatever other artifacts (during a release weekend). To make them manageable, things that should be moved (more or less) together, at the same time, are bundled together in what's called a <strong><em>software change package</em></strong>.</li>
</ul>

<p>With the above in place, any kind of update to be applied by the server to the library structure, will only be possible via a well defined workflow, which we call the <strong><em>lifecycle of a software change package</em></strong> (SDLC if you prefer). To actually execute the various steps in that workflow, this is what it takes to make it happen:</p>

<ul>
<li>Only the server will execute the required (and preconfigured) steps.</li>
<li>The server will only do a specific step (= update something somewhere in the library structure), after the required approvals (from human beings) have been gathered to perform such step.</li>
<li>The approvals can only be given by users who have a <strong>role</strong> that allows them (= permission) to issue such approvals.</li>
</ul>

<p><hr /></p>

<h3>2. Roles and permissions</h3>

<p><hr />
The server will ensure that the user trying to make something happen (like 'approve something') will only be able to do so, if the user's permissions are appropriate. That part is easy. But you don't want to use the SCM system to administer all those permissions for all the users involved, that's what belongs in your security system (not the SCM system!), so that you can adapt your workflow (in your SCM system) to go check those permissions whenever appropriate. The steps below provide some more details on that.</p>

<p><strong>Step 1: Configure the permissions (in the security system)</strong></p>

<ul>
<li><p>Define <strong>security entities</strong> in your security system, with well defined names for those entities. A few samples (add as many similar ones to fit your own needs):</p>

<ul>
<li><code>PrmUnit</code>, used for getting permission to request a <em>Promote</em> to say <em>Unit</em>-testing.</li>
<li><code>PrmQA</code>, used for getting permission to request a <em>Promote</em> to say <em>Qa</em>-testing (let's assume this is the highest level of testing).</li>
<li><code>PrdEnduser</code>, used by end-users involved in some level of testing, to indicate that they are satisfied by the results produced by some kind of testing. And because of that, those end-users agree with the change moving forward in the library structure.</li>
<li><code>PrdRelmgnt</code>, used by release managers to authorize an <em>Activation</em> in production (= the last/highest level in the library structure).</li>
</ul></li>
<li><p>Define <strong>groups of users</strong> in your security system. A few samples (add as many similar ones to fit your own needs):</p>

<ul>
<li><code>GrpDevs</code>, which (say) corresponds to your developers (probably more then just 1).</li>
<li><code>GrpEnduser</code>, which (say) corresponds to your end-users (at least 1, preferably with more similar users).</li>
<li><code>GrpRelMgnt</code>, which (say) corresponds to your release managers (at least 1, preferably a few more users).</li>
</ul></li>
<li><p><strong>Grant permissions</strong>, also using your security system, to allow access to selected "<em>security entities</em>" for selected "<em>groups of users</em>". To continue the example above, here is what seems appropriate (adapt to fit your own needs):</p>

<ul>
<li>Group <code>GrpDevs</code> gets access to (only!) security entity <code>PrmUnit</code>.</li>
<li>Group <code>GrpEnduser</code> gets access to (only!) security entity <code>PrdEnduser</code>.</li>
<li>Group <code>GrpRelMgnt</code> gets access to (both!) security entity  <code>PrmQA</code> and <code>PrdRelmgnt</code>.</li>
</ul></li>
</ul>

<p><strong>Step 2: Configure the workflow (in the SCM system)</strong></p>

<p>After the permissions are configured in your security system (as in Step 1), all that's left to do in your SCM system is to configure how the various steps in the lifecyle match with the related security entities in your security system. That is, only those users who have the appropriate access to the required security entity, are allowed to request the server to perform the corresponding step in the workflow.</p>

<p>Here are some examples of how you'd configure your SCM system to make some magic happen:</p>

<ul>
<li>If a user has access to <code>PrmUnit</code>, then such user is allowed to request a <em>Promote</em> to <em>Unit</em>-testing. Obviously, the users in group <code>GrpDevs</code> are the users authorized for this (note: not, e.g, the users in group <code>GrpRelMgnt</code>).</li>
<li>If a user has access to <code>PrmQA</code>, then such user is allowed to request a <em>Promote</em> to <em>QA</em>-testing. Obviously, the users in group <code>GrpRelMgnt</code> are the users authorized for this (note: not, e.g, the users in group <code>GrpDevs</code>, or in group <code>GrpEnduser</code>).</li>
<li>If a user has access to <code>PrdEnduser</code>, then such user is allowed to authorize the change moving forward in the library structure (which is typically a prereq for users in group <code>GrpRelMgnt</code> to even be able to review a change). Obviously, the users in group <code>GrpEnduser</code> are the (only) users authorized for this.</li>
<li>If a user has access to <code>PrdRelmgnt</code>, then such user is allowed to authorize an <em>Activation</em> in production (= the last/highest level in the library structure).</li>
</ul>

<p><hr /></p>

<h3>3. Expect the unexpected, and be ready for it</h3>

<hr />

<p>The above is just a blueprint, which hopefully helps to understand how in the end it is the server that takes care of the segregation of duties ... provided you have the CxO cover you to impose some access rules that not everybody will like.</p>

<p>To complete the picture as explained above, the server creates an audit trail (logging) of anything that's happening in the system. So that at any point in time, it is always possible to answer questions like</p>

<blockquote>
  <p><strong><em>What happened when and why, and which authorized user actually approved it ... upfront?</em></strong></p>
</blockquote>

<p>But, probably the toughest part is to have adequate reporting tools available (and know how to use them). At least to (easily) satisfy requests from IT auditors (their questions can be very challenging). But also to point to relevant log records in your SCM system to answer all sorts of "What happened"-questions in crisis situations where (part of) production is down.</p>

<p><hr />
PS: I leave it to everybody's own judgement if my answer is yes or no DevOps-compliant.</p>

CC BY-SA 3.0
2017-03-20T20:36:30.347Z
That's sounds like a basic implementation of top-down risk assessment, I don't get how it address the question on how this could be implemented in a devops manner where devs would have the rights to trigger the ´deploy' switch. Is the idea that you can't do it in a devops organization?
Tensibai
CC BY-SA 3.0
2017-03-20T20:52:08.393Z
@Tensibai "if" devs would have the auth (role) of (eg) final approval for prod (which they typically do NOT have in such organisations), then such server (started task) would start the deployment. And as per the title of the question, I think this is "a" possible answer. One could question however if this is what we would call a DevOps organization, but I do know that auditors really like this kind of "configurable" segragation of duties (eg: four-eyes and variations of that). Maybe Richard can help us with his viewpoint on this?
Pierre.Vriens
CC BY-SA 3.0
2017-03-20T21:00:54.400Z
I agree auditors like it absolutely, I just missed how this relate/fit with the ´explosion' of access, which auditor usually doesn't like when the list contains more than 6 to 7 persons. Saying it doesn't fit is an absolutely valid answer IMHO.
Tensibai
CC BY-SA 3.0
2017-03-20T21:10:12.890Z
Aha, I smell something about those lists with 6/7 persons that need clarification. I'll try to add a sample to clarify/illustrate the segragation with the typical roles / workflow steps you could configure (tomorrow)
Pierre.Vriens
CC BY-SA 3.0
2017-03-20T21:20:25.367Z
:) you know as me bank auditors like the validators list being as restricted as possible, when the list grow there's a hard need for a strict habilitation process around along with a strict following of Human Resources in this list I think :)
Tensibai
CC BY-SA 3.0
2017-03-21T08:01:04.900Z
Thank you for putting so much time into an answer.  I am actually thinking of implementing a 3-person rule, in that, one developer writes the code, a different developer reviews the code and a third person presses the release button to  deploy the code.  The other consideration is because this is part of a company wide Agile/DevOps adoption the development teams are quite small, with the net effect of small groups of people having production access to thin slice of production, this seems to be favourable from a risk perspective.
Richard Slater
CC BY-SA 3.0
2017-03-21T17:00:34.223Z
@Pierre.Vriens Can't upvote twice, great extension :)
Tensibai
CC BY-SA 3.0
2017-03-21T17:00:42.537Z
@RichardSlater I just completed my improved version of my answer ...
Pierre.Vriens
CC BY-SA 3.0
2017-03-21T17:05:12.843Z
@Tensibai merci for the kudos, toughest part of it is to avoid it becomes a book (manual). I'd be interested to hear if this helps to address your prior comments such as about those lists with 6/7 persons. Referring to my updated answer, I trust you understand I'd say something like "well, it depends what you're up to, and what you have authorization for, which head you are wearing, etc". Does that makes sense? Also, what's your opinion about yes/no fitting in a DevOps-organization? BTW: what I wrote in my answer is not SF, that's how we use to do it for over 2 decades already ...
Pierre.Vriens
CC BY-SA 3.0
2017-03-21T17:07:14.223Z
@Pierre.Vriens at the end of the day, you address the concern with the last paragraph about reporting tools. This sounds a valid implementation of even if the number of people are allowed to push in production, the actions are tracked and the audit can't be altered by those persons to hide their actions.
Tensibai
CC BY-SA 3.0
2017-03-20T20:25:37.617Z
2017-03-21T16:58:25.837Z
2017-03-21T16:58:25.837Z
Pierre.Vriens
<p>In highly regulated environments, such as the Financial Services sector, <a href="https://en.wikipedia.org/wiki/Separation_of_duties" rel="noreferrer">Segregation of Duties</a> is an essential mechanism for avoiding collision amongst individuals with development responsibilities and production privileges.</p>
<p>Traditionally this has meant that <strong>Developers</strong> develop code and then hand it over to <strong>Operations</strong> however in many DevOps Operating Models the segregation between Development and Operations is, at a minimum, blurred:</p>
<ul>
<li><p>In Google's <a href="https://landing.google.com/sre/interview/ben-treynor.html" rel="noreferrer">Site Reliability Engineering</a>, or SRE, practice there is a separate SRE function within Google, however, Developers are brought in to backstop the SREs in times of high operational load.</p>
</li>
<li><p>In the <a href="http://queue.acm.org/detail.cfm?id=1142065" rel="noreferrer">&quot;You Build It, You Run It&quot;</a> model there is no separate operations function.</p>
</li>
</ul>
<p>After spending months drilling down to the root causes of a Segregation of Duties mechanism it seems it predominantly exists to satisfy <a href="http://sarbanes-oxley-101.com/SOX-404.htm" rel="noreferrer">Sarbanes Oxley Section 404</a>: Management Assessment of Internal Controls:</p>
<blockquote>
<p>(a) Rules Required.
The Commission shall prescribe rules requiring each annual report required by section 13(a) or 15(d) of the Securities Exchange Act of 1934 to contain an internal control report, which shall--</p>
<p>(1) state the responsibility of management for establishing and maintaining an adequate internal control structure and procedures for financial reporting; and</p>
<p>(2) contain an assessment, as of the end of the most recent fiscal year of the issuer, of the effectiveness of the internal control structure and procedures of the issuer for financial reporting.</p>
<p>(b) Internal Control Evaluation and Reporting. On the internal control assessment required by subsection (a), each registered public accounting firm that prepares or issues the audit report for the issuer shall attest to, and report on, the assessment made by the management of the issuer. An attestation made under this subsection shall be made in accordance with standards for attestation engagements issued or adopted by the Board. Any such attestation shall not be the subject of a separate engagement.</p>
</blockquote>
<p>Based upon the comments it is important to call out a couple of <strong>assumptions</strong> I am making:</p>
<ul>
<li>I am predominantly considering mass-market financial services, i.e. transaction volumes are high but relatively low value.  This would be as opposed to commercial financial services which has a different transaction value profile.</li>
<li>A financial institution's online offering will be made up of many components that have differing risk considerations:
<ul>
<li><strong>Move Money</strong> - Moving money between accounts or transfers between accounts of different owners.  An operation that has to consider Anti-Money Laundering, Fraud Protection, and Embargo'ed countries to name a few.</li>
<li><strong>Customer Acquisition</strong> - Less &quot;Risky&quot; as it has low transaction volumes compared to Move Money but still needs consideration.</li>
<li><strong>Internet Banking</strong> - Covers a wide range of services with varying levels of risk, Move Money would be considered part of this.</li>
</ul>
</li>
<li>Conceivably a different approach could be taken for each depending on the risk, however, in the interests of keeping it simple, I am working towards a solution that would apply to some of the riskiest operations.</li>
</ul>
<p><strong>TL;DR</strong>: It is the responsibility of the management to ensure that adequate internal controls are in place that comply with the <a href="https://en.wikipedia.org/wiki/United_States_Securities_and_Exchange_Commission" rel="noreferrer">Securities and Exchange Commission's</a> <a href="https://www.sec.gov/spotlight/soxcomp.htm" rel="noreferrer">regulations</a>.</p>
<p>Sarbanes Oxley 404 is normally satisfied through the completion of a Top-Down Risk Assessment, part of which will assess the risk of collusion and puts forward mitigation strategies.</p>
<p>Within a company which employs DevOps practice and culture, where Developers routinely have access to both source control and production how can Segregation of Duties be achieved, or more generally how can the risk of collusion be mitigated.</p>

CC BY-SA 3.0
2017-03-18T13:29:44.210Z
Main idea behind a devops organization beingto put everyone in the Team accountable for what happens in production, there can't be separation of duties. This mainly means this kind of organization can't be really used when there's regulatory needs for this separation.
Tensibai
CC BY-SA 3.0
2017-03-18T14:16:43.900Z
@Tensbai I fundementally disagree with the assertion that DevOps is incompatible with Segregation of Duties.  The laws are not prescriptive as to the manner of the controls, nor are the regulators imposing a predefined process on banks and financial services.  It is largely down to the organisation to identify what appropriate is and be totally transparent with the regulators and their appointed auditors.  As an example both ING and Barclays have adopted DevOps practices to allow them to accelerate their ability to deliver value to their customers.
Richard Slater
CC BY-SA 3.0
2017-03-18T14:26:08.780Z
Yes, devops on subjects not bound to regulatory separation, and they took advantage of the automation on a traditional silo based org for subjects restricted (which are in fact very few). They just have two kind of orgs depending on which kind of operations the  software will do
Tensibai
CC BY-SA 3.0
2017-03-18T14:49:16.263Z
There is no such thing as "Regulatory Separation" the statutes/laws and regulatory bodies don't impose separation upon financial institutions they impose a management responsibility to have "Appropriate Controls" to manage financial risk.  In the same way that Agile took software development from long cycles to small cycles, DevOps is taking operations into small cycles, DevOps in Financial Services needs to find a way to take Segregation of Duties into small cycles, by for example creating a CD pipeline that enforces "appropriate controls" such as peer review and approval based promotion.
Richard Slater
CC BY-SA 3.0
2017-03-18T15:34:02.993Z
You took barcklays as an example, the concerns about the main website (not customer space) are not the same than those about the loan management, that is why I say there's some products (software) not concerned. But well, just a point of view
Tensibai
CC BY-SA 3.0
2017-03-18T17:24:17.940Z
I adore these kind of questions, especially when 'Sarbanes Oxley' is part of it (since about a decade or so I believe, no?). Based on my SCM-background I could write a book to answer/address these challenges. Am I right that your actual question is as in your title (some sites don't consider the title as part of the actual question ...)? Also, should "roles" not be part of your question also? Or is it OK to just add "roles" as part of a possible answer?
Pierre.Vriens
CC BY-SA 3.0
2017-03-18T19:14:29.753Z
@Tensibai agreed "Public Web Site" is broadly marketing material and thus doesn't need the same level of rigour, interestingly some financial institutions do apply the same level of rigour to the "Public Website" and for example "Internet Banking" - that I believe is more out of a desire to have a single global standard.  Interestingly Barclays seem from an external perspective to be following a You Build It, You Run It approach to their mobile application which includes the ability to "move money" something which certainly requires closer scrutiny than the Public Web Site.
Richard Slater
CC BY-SA 3.0
2017-03-18T19:18:01.487Z
Not that much, the regulation enforcement is more on investment handling (benefits, loans) than on individuals accounts (for what I know of US rules, French one are a little different, U.K. One fairly different and I'm not even sure which one Barclays has to follow at all)
Tensibai
CC BY-SA 3.0
2017-03-18T19:27:40.757Z
@Tensibai in some cases it becomes all of the above, Barclays has to comply with 220 individual statutes from 17 countries.  Other banks with a large global footprint will have to comply with a far greater number, some banks operate in 50+ countries.
Richard Slater
CC BY-SA 3.0
2017-03-18T19:30:14.307Z
I'll try to expand the first comment in an answer Monday :)
Tensibai
CC BY-SA 3.0
2017-03-18T19:31:27.160Z
@Pierre.Vriens yes the broad question is in the title, I have tried to expand upon it by making some assumptions.  Roles are likely to be part of the solution as are things like Break-Glass and Privileged Account Management.  Roles and Responsibilities are an interesting concept in DevOps/Agile as where once upon a time you had a Java Developer, F/E Developer, Designer, PM, Build Engineer, Release Manager and Ops Engineer - now you have a group of people who can wear multiple hats - cross-functional teams made up of "Engineers" who may specialise but ultimately share responsibility,.
Richard Slater
<sre><financial-services><operating-model>
What processes or tools enable Segregation of Duties when engineers both deploy and run code?
Richard Slater
# What processes or tools enable Segregation of Duties when engineers both deploy and run code?

(Asked by: username_0 on 2017-03-18)

In highly regulated environments, such as the Financial Services sector, [Segregation of Duties](https://en.wikipedia.org/wiki/Separation_of_duties) is an essential mechanism for avoiding collision amongst individuals with development responsibilities and production privileges.

Traditionally this has meant that **Developers** develop code and then hand it over to **Operations** however in many DevOps Operating Models the segregation between Development and Operations is, at a minimum, blurred:

* In Google's [Site Reliability Engineering](https://landing.google.com/sre/interview/ben-treynor.html), or SRE, practice there is a separate SRE function within Google, however, Developers are brought in to backstop the SREs in times of high operational load.
* In the ["You Build It, You Run It"](http://queue.acm.org/detail.cfm?id=1142065) model there is no separate operations function.

After spending months drilling down to the root causes of a Segregation of Duties mechanism it seems it predominantly exists to satisfy [Sarbanes Oxley Section 404](http://sarbanes-oxley-101.com/SOX-404.htm): Management Assessment of Internal Controls:

> (a) Rules Required.
> The Commission shall prescribe rules requiring each annual report required by section 13(a) or 15(d) of the Securities Exchange Act of 1934 to contain an internal control report, which shall--
>
> (1) state the responsibility of management for establishing and maintaining an adequate internal control structure and procedures for financial reporting; and
>
> (2) contain an assessment, as of the end of the most recent fiscal year of the issuer, of the effectiveness of the internal control structure and procedures of the issuer for financial reporting.
>
> (b) Internal Control Evaluation and Reporting. On the internal control assessment required by subsection (a), each registered public accounting firm that prepares or issues the audit report for the issuer shall attest to, and report on, the assessment made by the management of the issuer. An attestation made under this subsection shall be made in accordance with standards for attestation engagements issued or adopted by the Board. Any such attestation shall not be the subject of a separate engagement.

Based upon the comments it is important to call out a couple of **assumptions** I am making:

* I am predominantly considering mass-market financial services, i.e. transaction volumes are high but relatively low value. This would be as opposed to commercial financial services which has a different transaction value profile.
* A financial institution's online offering will be made up of many components that have differing risk considerations:
  + **Move Money** - Moving money between accounts or transfers between accounts of different owners. An operation that has to consider Anti-Money Laundering, Fraud Protection, and Embargo'ed countries to name a few.
  + **Customer Acquisition** - Less "Risky" as it has low transaction volumes compared to Move Money but still needs consideration.
  + **Internet Banking** - Covers a wide range of services with varying levels of risk, Move Money would be considered part of this.
* Conceivably a different approach could be taken for each depending on the risk, however, in the interests of keeping it simple, I am working towards a solution that would apply to some of the riskiest operations.

**TL;DR**: It is the responsibility of the management to ensure that adequate internal controls are in place that comply with the [Securities and Exchange Commission's](https://en.wikipedia.org/wiki/United_States_Securities_and_Exchange_Commission) [regulations](https://www.sec.gov/spotlight/soxcomp.htm).

Sarbanes Oxley 404 is normally satisfied through the completion of a Top-Down Risk Assessment, part of which will assess the risk of collusion and puts forward mitigation strategies.

Within a company which employs DevOps practice and culture, where Developers routinely have access to both source control and production how can Segregation of Duties be achieved, or more generally how can the risk of collusion be mitigated.

**Tags:** sre, financial-services, operating-model

> **Comments:**
> **username_1**: Main idea behind a devops organization beingto put everyone in the Team accountable for what happens in production, there can't be separation of duties. This mainly means this kind of organization can't be really used when there's regulatory needs for this separation.  
> **username_0**: @Tensbai I fundementally disagree with the assertion that DevOps is incompatible with Segregation of Duties.  The laws are not prescriptive as to the manner of the controls, nor are the regulators imposing a predefined process on banks and financial services.  It is largely down to the organisation to identify what appropriate is and be totally transparent with the regulators and their appointed auditors.  As an example both ING and Barclays have adopted DevOps practices to allow them to accelerate their ability to deliver value to their customers.  
> **username_1**: Yes, devops on subjects not bound to regulatory separation, and they took advantage of the automation on a traditional silo based org for subjects restricted (which are in fact very few). They just have two kind of orgs depending on which kind of operations the  software will do  
> **username_0**: There is no such thing as "Regulatory Separation" the statutes/laws and regulatory bodies don't impose separation upon financial institutions they impose a management responsibility to have "Appropriate Controls" to manage financial risk.  In the same way that Agile took software development from long cycles to small cycles, DevOps is taking operations into small cycles, DevOps in Financial Services needs to find a way to take Segregation of Duties into small cycles, by for example creating a CD pipeline that enforces "appropriate controls" such as peer review and approval based promotion.  
> **username_1**: You took barcklays as an example, the concerns about the main website (not customer space) are not the same than those about the loan management, that is why I say there's some products (software) not concerned. But well, just a point of view  
> **username_2**: I adore these kind of questions, especially when 'Sarbanes Oxley' is part of it (since about a decade or so I believe, no?). Based on my SCM-background I could write a book to answer/address these challenges. Am I right that your actual question is as in your title (some sites don't consider the title as part of the actual question ...)? Also, should "roles" not be part of your question also? Or is it OK to just add "roles" as part of a possible answer?  
> **username_0**: @username_1 agreed "Public Web Site" is broadly marketing material and thus doesn't need the same level of rigour, interestingly some financial institutions do apply the same level of rigour to the "Public Website" and for example "Internet Banking" - that I believe is more out of a desire to have a single global standard.  Interestingly Barclays seem from an external perspective to be following a You Build It, You Run It approach to their mobile application which includes the ability to "move money" something which certainly requires closer scrutiny than the Public Web Site.  
> **username_1**: Not that much, the regulation enforcement is more on investment handling (benefits, loans) than on individuals accounts (for what I know of US rules, French one are a little different, U.K. One fairly different and I'm not even sure which one Barclays has to follow at all)  
> **username_0**: @username_1 in some cases it becomes all of the above, Barclays has to comply with 220 individual statutes from 17 countries.  Other banks with a large global footprint will have to comply with a far greater number, some banks operate in 50+ countries.  
> **username_1**: I'll try to expand the first comment in an answer Monday :)  
> **username_0**: @username_2 yes the broad question is in the title, I have tried to expand upon it by making some assumptions.  Roles are likely to be part of the solution as are things like Break-Glass and Privileged Account Management.  Roles and Responsibilities are an interesting concept in DevOps/Agile as where once upon a time you had a Java Developer, F/E Developer, Designer, PM, Build Engineer, Release Manager and Ops Engineer - now you have a group of people who can wear multiple hats - cross-functional teams made up of "Engineers" who may specialise but ultimately share responsibility,.  

---

## 4 answers

### Answer by username_2 on 2017-03-20. Score: 8 | Accepted answer

Your question doesn't seem to make any assumption about the platform/OS it is about. Which is why it may make sense to add an answer about how this is typically done/addressed in a mainframe environment, where the "engineers" (as in your question title) are actually groups of people were dozens (possibly hundreds) of people are involved. My answer is based on using the SCM product where I'm most familiar with (not sure if it's needed to disclose the product name).

---

### 1. Architecture

---

Here are the highlights of how I'd answer your question:

* All code (and related artifacts like executables, etc) is stored in files which, all together is what we call the ***library structure***.
* For each environment on each (possibly remote) target system, there is a server (a "started task" in mainframe speak), which takes care of ALL (repeat: ALL) updates to anything in the library structure. There are a few exceptions (like security people, or space management team), but apart from that, nobody (repeat: nobody) has authorization to apply updates to any file within that library structure. In other words: ***the server gets exclusive update authority to the entire library structure***. Attention: OPS-people will go bonkers if you walk in to limit their access (at first they are going to resist ...), so make sure you're covered by upper management (CxO) to impose those access rules ...
* The actual software changes my consist of a single component (a tiny code fix in the middle of the night ...), or it may also be hundreds or thousands of sources, executables, or whatever other artifacts (during a release weekend). To make them manageable, things that should be moved (more or less) together, at the same time, are bundled together in what's called a ***software change package***.

With the above in place, any kind of update to be applied by the server to the library structure, will only be possible via a well defined workflow, which we call the ***lifecycle of a software change package*** (SDLC if you prefer). To actually execute the various steps in that workflow, this is what it takes to make it happen:

* Only the server will execute the required (and preconfigured) steps.
* The server will only do a specific step (= update something somewhere in the library structure), after the required approvals (from human beings) have been gathered to perform such step.
* The approvals can only be given by users who have a **role** that allows them (= permission) to issue such approvals.

---

### 2. Roles and permissions

---

The server will ensure that the user trying to make something happen (like 'approve something') will only be able to do so, if the user's permissions are appropriate. That part is easy. But you don't want to use the SCM system to administer all those permissions for all the users involved, that's what belongs in your security system (not the SCM system!), so that you can adapt your workflow (in your SCM system) to go check those permissions whenever appropriate. The steps below provide some more details on that.

**Step 1: Configure the permissions (in the security system)**

* Define **security entities** in your security system, with well defined names for those entities. A few samples (add as many similar ones to fit your own needs):

  + `PrmUnit`, used for getting permission to request a *Promote* to say *Unit*-testing.
  + `PrmQA`, used for getting permission to request a *Promote* to say *Qa*-testing (let's assume this is the highest level of testing).
  + `PrdEnduser`, used by end-users involved in some level of testing, to indicate that they are satisfied by the results produced by some kind of testing. And because of that, those end-users agree with the change moving forward in the library structure.
  + `PrdRelmgnt`, used by release managers to authorize an *Activation* in production (= the last/highest level in the library structure).
* Define **groups of users** in your security system. A few samples (add as many similar ones to fit your own needs):

  + `GrpDevs`, which (say) corresponds to your developers (probably more then just 1).
  + `GrpEnduser`, which (say) corresponds to your end-users (at least 1, preferably with more similar users).
  + `GrpRelMgnt`, which (say) corresponds to your release managers (at least 1, preferably a few more users).
* **Grant permissions**, also using your security system, to allow access to selected "*security entities*" for selected "*groups of users*". To continue the example above, here is what seems appropriate (adapt to fit your own needs):

  + Group `GrpDevs` gets access to (only!) security entity `PrmUnit`.
  + Group `GrpEnduser` gets access to (only!) security entity `PrdEnduser`.
  + Group `GrpRelMgnt` gets access to (both!) security entity `PrmQA` and `PrdRelmgnt`.

**Step 2: Configure the workflow (in the SCM system)**

After the permissions are configured in your security system (as in Step 1), all that's left to do in your SCM system is to configure how the various steps in the lifecyle match with the related security entities in your security system. That is, only those users who have the appropriate access to the required security entity, are allowed to request the server to perform the corresponding step in the workflow.

Here are some examples of how you'd configure your SCM system to make some magic happen:

* If a user has access to `PrmUnit`, then such user is allowed to request a *Promote* to *Unit*-testing. Obviously, the users in group `GrpDevs` are the users authorized for this (note: not, e.g, the users in group `GrpRelMgnt`).
* If a user has access to `PrmQA`, then such user is allowed to request a *Promote* to *QA*-testing. Obviously, the users in group `GrpRelMgnt` are the users authorized for this (note: not, e.g, the users in group `GrpDevs`, or in group `GrpEnduser`).
* If a user has access to `PrdEnduser`, then such user is allowed to authorize the change moving forward in the library structure (which is typically a prereq for users in group `GrpRelMgnt` to even be able to review a change). Obviously, the users in group `GrpEnduser` are the (only) users authorized for this.
* If a user has access to `PrdRelmgnt`, then such user is allowed to authorize an *Activation* in production (= the last/highest level in the library structure).

---

### 3. Expect the unexpected, and be ready for it

---

The above is just a blueprint, which hopefully helps to understand how in the end it is the server that takes care of the segregation of duties ... provided you have the CxO cover you to impose some access rules that not everybody will like.

To complete the picture as explained above, the server creates an audit trail (logging) of anything that's happening in the system. So that at any point in time, it is always possible to answer questions like

> ***What happened when and why, and which authorized user actually approved it ... upfront?***

But, probably the toughest part is to have adequate reporting tools available (and know how to use them). At least to (easily) satisfy requests from IT auditors (their questions can be very challenging). But also to point to relevant log records in your SCM system to answer all sorts of "What happened"-questions in crisis situations where (part of) production is down.

---

PS: I leave it to everybody's own judgement if my answer is yes or no DevOps-compliant.

> **Comments:**
> **username_1**: That's sounds like a basic implementation of top-down risk assessment, I don't get how it address the question on how this could be implemented in a devops manner where devs would have the rights to trigger the ´deploy' switch. Is the idea that you can't do it in a devops organization?  
> **username_2**: @username_1 "if" devs would have the auth (role) of (eg) final approval for prod (which they typically do NOT have in such organisations), then such server (started task) would start the deployment. And as per the title of the question, I think this is "a" possible answer. One could question however if this is what we would call a DevOps organization, but I do know that auditors really like this kind of "configurable" segragation of duties (eg: four-eyes and variations of that). Maybe Richard can help us with his viewpoint on this?  
> **username_1**: I agree auditors like it absolutely, I just missed how this relate/fit with the ´explosion' of access, which auditor usually doesn't like when the list contains more than 6 to 7 persons. Saying it doesn't fit is an absolutely valid answer IMHO.  
> **username_2**: Aha, I smell something about those lists with 6/7 persons that need clarification. I'll try to add a sample to clarify/illustrate the segragation with the typical roles / workflow steps you could configure (tomorrow)  
> **username_1**: :) you know as me bank auditors like the validators list being as restricted as possible, when the list grow there's a hard need for a strict habilitation process around along with a strict following of Human Resources in this list I think :)  
> **username_0**: Thank you for putting so much time into an answer.  I am actually thinking of implementing a 3-person rule, in that, one developer writes the code, a different developer reviews the code and a third person presses the release button to  deploy the code.  The other consideration is because this is part of a company wide Agile/DevOps adoption the development teams are quite small, with the net effect of small groups of people having production access to thin slice of production, this seems to be favourable from a risk perspective.  
> **username_1**: @username_2 Can't upvote twice, great extension :)  
> **username_2**: @RichardSlater I just completed my improved version of my answer ...  
> **username_2**: @username_1 merci for the kudos, toughest part of it is to avoid it becomes a book (manual). I'd be interested to hear if this helps to address your prior comments such as about those lists with 6/7 persons. Referring to my updated answer, I trust you understand I'd say something like "well, it depends what you're up to, and what you have authorization for, which head you are wearing, etc". Does that makes sense? Also, what's your opinion about yes/no fitting in a DevOps-organization? BTW: what I wrote in my answer is not SF, that's how we use to do it for over 2 decades already ...  
> **username_1**: @username_2 at the end of the day, you address the concern with the last paragraph about reporting tools. This sounds a valid implementation of even if the number of people are allowed to push in production, the actions are tracked and the audit can't be altered by those persons to hide their actions.  

---

### Answer by username_1 on 2017-03-20. Score: 6

Answer based on my knowledge of French "Internal controls" regulation, kind of equivalent to the SEC regulations you point to, I assume linking here to a French legal text would not be really useful and I know of no good translation of it.

In an ideal 'You build it, you run it' model, everyone in the team will be accountable for the change. The risk assessment can't be enforced by a separation of duties and the only way I know to keep conforming with regulation is to have a periodic short cycle audit of transactions along with an unalterable action tracking to get back to the person who did the release.  
This mean all logs of transaction and actions are pushed to a restricted area the team doesn't have access to, a change in the what is logged "should" be catch by functional tests the team has no access to and at worse will be catch by the audit and tracked to its author.

This is not applicable to all products, at time of writing in France any company allowed to emit money (mainly banks), has to ensure every transaction will be recorded and thus can't take the risk of missing a transaction.  
On the other hand, they have no legal obligation to track any commercial offer or risk evaluation when someone ask for a loan, and thus the products handling this customer selection and computing the fees which will be in the offer are easier to fit in a post-release audit model.

Main idea is that the release model has to be tweaked according to risk assessment obligations.

A related resource is the [ISO27001](https://en.wikipedia.org/wiki/ISO/IEC_27001:2005) norm.

> **Comments:**
> **username_0**: Interesting answer and very relevant as many European banks do indeed operate in France.  Is there any chance you could expand on what 'Emit Money' means, i.e. is that just cash out of an ATM or does it include balance transfers.  In this case linking to the statutes would be valuable as it gives a pointer to the relevant laws, regardless of the language they are in.  
> **username_1**: @RichardSlater In brief, any company working with money, could be an investment only company as well as loan brokers along the traditional banks. Mostly anything which have a financial impact somewhere is concerned (out of few exception which the authority can give under come circonstances). The legal "list" in French is [here](https://www.legifrance.gouv.fr/affichTexte.do?cidTexte=JORFTEXT000029700770&dateTexte=20170320) but even in French it is not always obvious.  
> **username_0**: I am making the assumption that the link to the ISO standard should actually be [ISO27001:2013](https://en.wikipedia.org/wiki/ISO/IEC_27001:2013)  
> **username_1**: @Richard indeed, seems the French to English link hasn't been updated on Wikipedia. I'll update later (or if you wish , feel free to edit)  
> **username_3**: related: [How can I persuade developers on my team to embrace “You build it, you run it”?](http://devops.stackexchange.com/questions/484/how-can-i-persuade-developers-on-my-team-to-embrace-you-build-it-you-run-it)  

---

### Answer by username_4 on 2017-03-21. Score: 0

IMHO, Developers & Operations could be represented by nothing more than just two git repositories **for the same codebase**, with distinct permissions model each, so that teams will not interfer in the work of each other, at all.

Let's call them **Dev.mygithub.co** & **ops.mygithub.co**, just as an example.

The idea is, that Developers are free to create/branch/merge to their heart's content -git is providing full traceability and that is what matters here- meanwhile, at the moments that regulatory framework implies review effort a **Pull Request** can be raised, for the merging to happen in a controlled manner.

Taking that concept to its next level, a **develop** branch can be propagated towards the remote Ops' **production** via yet another **Pull Request** act.
That last part has to happen by operations' hands and eyes, since they have a responsibility to vet it into production and they choose the level of review.

Such scheme allows *infinite flexibility, full traceability, ability to catch issues early on via a variety of processes, separation of concerns and a very reasonable user experience in the process*!

N.B. The model described above may be followed even if Ops & Dev overlap totally!

> **Comments:**
> **username_0**: Surely, this same control could be achieved through pull requests and post-commit hooks that ensured that developers could commit freely, however, merge commits could only be made by an approved group of people.  Equally that same post-commit hook could ensure that of the authors of the commits that made up the pull request did not include the person making the pull request.  
> **username_4**: @RichardSlater: the reason you'd may wish to have two distinct repositories is that you have the dual need to both **allow** developers to merge -when they freely swap code in developer mode- as well as **block** most developers from merging code when it is to go towards production (modulo SysOps, i.e.. the so called "approved group of people").  
> **username_0**: Again you can achieve that with post-commit hooks and pull requests, not to mention GitHub Enterprise allows protected branches.  

---

### Answer by username_4 on 2018-10-12. Score: 0

in order, topmost is most expensive:

* distinct dev and ops sites & methods to carry over work from one to the other => many sites
* distinct dev and ops systems & methods as above => 1 joint site
* distinct dev and ops git/vcs repositories & related methods => 1 system
* distinct dev and ops git/vcs branches (protected) & related methods => 1 repo, many branches

Depending on what you do, some solutions are better than others; for example, if you have the need to serve two teams with distinct roles within them and each having ownership within and provide full traceability, you are hovering over first three.

In short, anything that enforces that one guy or gal cannot take the ball all alone and run with it, as s/he crosses a distinct explicit boundary among dev & ops.

Now, depending on the subsequent level of risk, that boundary might be enforced or not. Choices, choices.


---

