<p>it depends a bit what kind of cms you use, most cms offer a way to package data in to packages that you can check in to git and deploy to databases almost like code to webapps.</p>
<p>For example, we work with Sitecore and it lets you use TDS or Unicorn, other big CMS use different tools but mostly the same concept.</p>
<p>The tools let you pick items from your Database and serialise them in to code/text files that are packaged and check in to Git, lets say you have a copy of the database on your Development Workstation, you make some updates to the database and you serialise and package the change with TDS in to a package that you check in to your Development branch.</p>
<p>When you deploy the development branch to your Staging environment the TDS package is applied and makes the same updates to the database of your Staging environments.</p>

CC BY-SA 4.0
2022-09-26T20:01:04.517Z
Thanks! I read a bit into both tools for SiteCore and as I understand, developers make changes to their local Sitecore DB and sync the changes to disk, then commit the changes to Git. How would we go about doing this from Production to Staging? e.g. after each Production deployment, run this in an automated pipeline and commit the production changes to a repository that is then deployed to staging?
ggradnig
CC BY-SA 4.0
2022-09-26T20:12:37.213Z
we only use TDS from lower to higer environment, for the other way have a separate  pipeline that we run on demand that will take a backup of the production database(s) and recreate the database of the lower environments (Dev/QA, UAT), we can then apply the TDS packages again from Git if we have new changes that haven't been already on the prod database(s)
qBasicBoy
CC BY-SA 4.0
2022-09-26T20:28:46.607Z
makes completely sense - thank you for that insight!
I suppose we can also generalize this solution for other products: 
1) a tool to export changes from a local database to disk and have developers commit the changes to version control
2) a tool to apply changes from version control to a shared database, run by an automated pipeline
3) a tool to backup and restore a production database to a lower tier environment, run on demand or by an automated pipeline
ggradnig
CC BY-SA 4.0
2022-09-26T20:31:38.000Z
maybe just a last follow-up question: you mentioned "if we have new changes that haven't been already on the prod database"... does this mean that the changes are stored as (historical) differences with timestamps that can be applied onto each other, i.e. after each "sync to disk" a new file is generated, or does the repository only reflect the current state of the database and TDS somehow knows what's new and what's old?
ggradnig
CC BY-SA 4.0
2022-09-26T20:52:49.010Z
yeah so in the case of SiteCore and TDS, There always is an ItemID, TDS keeps the itemID in the package, you can also set a "Deploy Once" or "Always Deploy" property in TDS, so if you set it to always deploy it always overwrite whats in the DB with the same ID, if you set it to deploy once it will only create the item if it doesn't already exist.

There is also a time stamp so on your development machine it can show you which has been modified last, the item in the TDS (git) package or in the database.

PS. Liebe Gruesse nach Wien aus Birmingham :)
qBasicBoy
CC BY-SA 4.0
2022-09-26T21:03:20.813Z
Danke und liebe Grüße zurück :)
ggradnig
CC BY-SA 4.0
2022-09-26T18:53:49.740Z
2022-09-26T18:53:49.740Z
qBasicBoy
<p>One goal of DevOps is to create reproducible, production-like environments from source-code repositories. To achieve this, I believe it's necessary to also restore all the <strong>data</strong> that is needed to setup the environment, so that automated &amp; manual tests can be run there.</p>
<p>For example, let's take a CMS product that provides content (labels, images, rich-text) to a user-facing web application. We want to run integrations tests or explorative UI tests against that web application using production-like data from the CMS. Due to the nature of the product, the data itself can be manipulated at any time by users using the UI of the CMS.</p>
<p>There are two main use cases</p>
<ul>
<li>Developers create example content or add required label translations for new features</li>
<li>Content managers publish new content for production, which may include static content like label translations</li>
</ul>
<p>In a typical setup, there are at least two environments, let's say <code>staging</code> and <code>production</code>. Due to the use cases described above, the CMS databases where the contents are stored will deviate quite quickly between these environments.</p>
<p>Without taking any measures here, after a while, the two databases will be completely out-of-sync and the <code>staging</code> database will not be &quot;production-like&quot; anymore. Additionally, each release of a new feature will require manual steps to add data like label translations into the <code>production</code> database, a step that is neither verified nor tested before the release.</p>
<p>So my question is: What is a good way to bring new &quot;baseline&quot; data from a <code>staging</code> database into a <code>production</code> database without overriding the changes there. And was is a good way to bring back data from <code>production</code> into the <code>staging</code> environment, to provide production-like data for testing purposes. Ideally, all automated and all part of version-control.</p>
<p>Side Note: If we say &quot;different CMS content doesn't change anything about testing&quot;, let's think of another use-case where the data behaves even more like configuration. For example, payment or delivery options that are offered on an e-Commerce site.</p>

<versioning><environments><data>
How to put production-like data into version control
ggradnig
# How to put production-like data into version control

(Asked by: username_0 on 2022-09-26)

One goal of DevOps is to create reproducible, production-like environments from source-code repositories. To achieve this, I believe it's necessary to also restore all the **data** that is needed to setup the environment, so that automated & manual tests can be run there.

For example, let's take a CMS product that provides content (labels, images, rich-text) to a user-facing web application. We want to run integrations tests or explorative UI tests against that web application using production-like data from the CMS. Due to the nature of the product, the data itself can be manipulated at any time by users using the UI of the CMS.

There are two main use cases

* Developers create example content or add required label translations for new features
* Content managers publish new content for production, which may include static content like label translations

In a typical setup, there are at least two environments, let's say `staging` and `production`. Due to the use cases described above, the CMS databases where the contents are stored will deviate quite quickly between these environments.

Without taking any measures here, after a while, the two databases will be completely out-of-sync and the `staging` database will not be "production-like" anymore. Additionally, each release of a new feature will require manual steps to add data like label translations into the `production` database, a step that is neither verified nor tested before the release.

So my question is: What is a good way to bring new "baseline" data from a `staging` database into a `production` database without overriding the changes there. And was is a good way to bring back data from `production` into the `staging` environment, to provide production-like data for testing purposes. Ideally, all automated and all part of version-control.

Side Note: If we say "different CMS content doesn't change anything about testing", let's think of another use-case where the data behaves even more like configuration. For example, payment or delivery options that are offered on an e-Commerce site.

**Tags:** versioning, environments, data


---

## 1 answers

### Answer by username_1 on 2022-09-26. Score: 1 | Accepted answer

it depends a bit what kind of cms you use, most cms offer a way to package data in to packages that you can check in to git and deploy to databases almost like code to webapps.

For example, we work with Sitecore and it lets you use TDS or Unicorn, other big CMS use different tools but mostly the same concept.

The tools let you pick items from your Database and serialise them in to code/text files that are packaged and check in to Git, lets say you have a copy of the database on your Development Workstation, you make some updates to the database and you serialise and package the change with TDS in to a package that you check in to your Development branch.

When you deploy the development branch to your Staging environment the TDS package is applied and makes the same updates to the database of your Staging environments.

> **Comments:**
> **username_0**: Thanks! I read a bit into both tools for SiteCore and as I understand, developers make changes to their local Sitecore DB and sync the changes to disk, then commit the changes to Git. How would we go about doing this from Production to Staging? e.g. after each Production deployment, run this in an automated pipeline and commit the production changes to a repository that is then deployed to staging?  
> **username_1**: we only use TDS from lower to higer environment, for the other way have a separate  pipeline that we run on demand that will take a backup of the production database(s) and recreate the database of the lower environments (Dev/QA, UAT), we can then apply the TDS packages again from Git if we have new changes that haven't been already on the prod database(s)  
> **username_0**: makes completely sense - thank you for that insight! I suppose we can also generalize this solution for other products:  1) a tool to export changes from a local database to disk and have developers commit the changes to version control 2) a tool to apply changes from version control to a shared database, run by an automated pipeline 3) a tool to backup and restore a production database to a lower tier environment, run on demand or by an automated pipeline  
> **username_0**: maybe just a last follow-up question: you mentioned "if we have new changes that haven't been already on the prod database"... does this mean that the changes are stored as (historical) differences with timestamps that can be applied onto each other, i.e. after each "sync to disk" a new file is generated, or does the repository only reflect the current state of the database and TDS somehow knows what's new and what's old?  
> **username_1**: yeah so in the case of SiteCore and TDS, There always is an ItemID, TDS keeps the itemID in the package, you can also set a "Deploy Once" or "Always Deploy" property in TDS, so if you set it to always deploy it always overwrite whats in the DB with the same ID, if you set it to deploy once it will only create the item if it doesn't already exist.  There is also a time stamp so on your development machine it can show you which has been modified last, the item in the TDS (git) package or in the database.  PS. Liebe Gruesse nach Wien aus Birmingham :)  
> **username_0**: Danke und liebe Grüße zurück :)  

---

