<p>Would it be possible to segregate data by level of maturity/stability of its functionality? The same approach which is used to version the API might work here quite well. </p>

<p>Conceptually it would mean that data for <em>stable</em> functionality could be copied from build to build via snapshots of the underlying data store (maybe with few alterations depending on your context). </p>

<p>Data for <em>preview</em> (unstable, in development) functionality is best to generate - this way no matter how functionality is altered, it is always possible to get the fresh view of how the product is going to look like. Generation can happen (for example) as a pre-demo step, or straight in the app (auto-fill in the UI, it can even be made to look neat). </p>

<p>As soon as <em>preview</em> functionality graduates to <em>stable</em>, its data can be included into the snapshot (as an additional release process step)</p>

<p>It would be best to align with engineering on common way to generate data for new functionality and then ship generators for it’s data along with code itself. </p>

<p>This hybrid approach allows to achieve following benefits:</p>

<ul>
<li><p>Set up and align expectations between devops, engineering and product about how test data is obtained</p></li>
<li><p>No need to deal with exponential growth of time to generate <em>entire</em> data set</p></li>
<li><p>Enable quick iteration of features under development</p></li>
<li><p>Educate developers and make them more DevOps aware (they’d have to take last stable data snapshot and roll it out in their dev env when they start working on a feature)</p></li>
<li><p>Enable fast feedback loop between all the parties involved which leads to better product quality (if snapshot doesn’t rollout or app crashes, then the release process is broken)</p></li>
</ul>

<p>There aren’t many cons of this approach, but I’d highlight a few most important:</p>

<ul>
<li><p>It requires work. Main point of this approach is to fix (or actually define) the process, so it would require taking an extra mile to weigh, explain, prototype and build the alignment</p></li>
<li><p>It would not solve problems with depencies in preview functionality, eg if feature B depends on A and they <em>both</em> are under development, data for both have to be generated and A has to be stabilized before B.</p></li>
</ul>

<p>Overall, it’s not an easy problem you’re dealing with, and I would be appreciste if you could share your experience on how did it go, what path have your project had taken and reasoning behind it. Cheers!</p>

CC BY-SA 4.0
2020-02-10T21:17:07.280Z
2020-02-10T22:15:20.337Z
2020-02-10T22:15:20.337Z
Alexey Skolyarov
<p>Welcome to DevOps SE!</p>

<p>Is it possible to make data management and operation of data sets to be an asset on its own? It seems like data has become more important so a more <em>data-centric</em> approach could help.</p>

<blockquote>
  <p>Program managers have the insider info on what the demo data needs to
  look like</p>
</blockquote>

<p>This proves also organizational silo and lack of communication in the process: even before DevOps, in an agile team e.g. scrum you could have not only engineers but also a product owner and testers who take responsibility to provide not only useful tests but also useful testdata to the team.</p>

<p>That is, the fourth possible scenario could be that you take a data set from production and scramble/anonymize it. If it is too large, you need to reduce it to a representative mix. A DevOps way to do it would be to automate all these steps after they are well understood. => Working real life scenario.</p>

<p>There are some legal workaround methods though which require though considerable additional effort:</p>

<ul>
<li>data scrambling</li>
<li>data anonymization</li>
<li>synthetic data</li>
</ul>

<p>Depending on your domain, there could be also applicable and more or less flexible enterprise tools for master data management where product owners could manage and version various data profiles. </p>

<p>Since DevOps approach is more than just finding a tool to use, but also doing experiments to find out more, here are examples of recent academic research on these topics.</p>

<p>An example study: <em><a href="https://dl.acm.org/doi/abs/10.1145/3274335?casa_token=uI4NBUmHDIwAAAAA%3AheKYJO5RRmnDKsO_c5ieBjPQjnTS5_IumTWqWMrijRcIqxXwbJQl8xrtq1Z0RtHRZEcbYWzKNIcbjQ" rel="nofollow noreferrer">Going Beyond Obscurity: Organizational Approaches to Data Anonymization</a></em> (Hargitai et al. 2018)</p>

<blockquote>
  <p>Rather than being a purely technical question of applying the right
  algorithms, <strong>anonymization</strong> in practice is a complex socio-technical
  process that relies on multi-stakeholder collaborations.</p>
</blockquote>

<p>Or: <a href="https://www.worldscientific.com/doi/abs/10.1142/9789813273238_0160" rel="nofollow noreferrer"><em>Machine learning using synthetic and real data: Similarity of evaluation metrics for different healthcare datasets and for different algorithms</em></a>. (Heyburn et al., 2018)</p>

<blockquote>
  <p>In this paper, we carry out an experiment to study the validity of
  conducting machine learning on <strong>synthetic data</strong>.</p>
</blockquote>

CC BY-SA 4.0
2020-02-10T18:09:54.157Z
Can you elaborate what you mean by data management and operation of data sets as independent assets?     
I meant the PMs know what the demo data/configurations should look like for a specific customer. The work process between PMs and engineers are decoupled. This is fine. After tests and release, the PMs have the job to sell the product to the customer, this is where demo data is needed. However, the product can change and the old data that the PMs used can become irrelevant and the PMs have to recreate new demo data.     
I cannot use customer data, they will not allow that.
Joseph Luce
CC BY-SA 4.0
2020-02-10T20:05:46.190Z
therefore I've mentioned that enterprises do data scrambling and anonimizing to retain the data structure but remove its criticality. e.g. https://medium.com/district-data-labs/a-practical-guide-to-anonymizing-datasets-with-python-faker-ecf15114c9be
Ta Mu
CC BY-SA 4.0
2020-02-10T22:01:56.570Z
I am not allowed to touch/gather any customer data, that is the problem. I don't have a starting data set to scrub.
Joseph Luce
CC BY-SA 4.0
2020-02-10T22:22:11.183Z
"I am not allowed" this also shouldn't be necessarily your job, but you could raise the demand for a demo data set as preliminary item to do your work well. I have just shared what other enterprises typically do to support more flexible scenarios, long before DevOps. Also without giving you access to productive data, other teams could deliver demo data sets: they have the knowledge and access, they could deliver some input to support your work. Otherwise it does not feel like fair play in terms of DevOps culture. So, your problem is imho less technical but more of organizational nature.
Ta Mu
CC BY-SA 4.0
2020-02-12T18:56:20.830Z
Not allowed as its stepping into legal privacy laws.
Joseph Luce
CC BY-SA 4.0
2020-02-12T20:29:42.897Z
Yes, GDPR is why I am trying to establish a link to advanced methods. Exactly these methods have been developed, or are yet to develop, to comply with legal privacy laws. Thank you for mentioning this, indeed there is ongoing academic research on that which I've added to my answer.
Ta Mu
CC BY-SA 4.0
2020-02-07T12:08:09.993Z
2020-02-12T20:28:39.313Z
2020-02-12T20:28:39.313Z
Ta Mu
<p>Is there a good DevOps method to construct a workflow that creates demo data for a product? This situation is difficult when the app constantly changes. There are a few scenarios to consider.</p>

<ol>
<li><p>Either old demo data becomes obsolete in the new build or the old
data can be considered corrupted in the new build.</p></li>
<li><p>New business logic is introduced which requires interaction with the
    app for the new data to be formed. You cannot look at the new
    schema, static rules or old data to derivative/fill in such data.</p></li>
<li><p>Program managers have the insider info on what the demo data needs
    to look like, not the engineers. So the workflow needs to match up
    during the process, from (build -> test -> deploy) somehow.</p></li>
</ol>

<p>One approach is to run tests that have demo data in it. So after the tests were ran, we can extract the data for our demos. But I can see that the tests can be very long and not as instant as importing a csv file. Also the coordination between PMs and engineers seems to be tightly coupled.</p>

<p>Another approach is to start with the new scheme from the repository and generate data from there. But this requires overhead of creating rules for generation which may not be in sync with the logic of app itself.</p>

<p>It seems a top down approach is better, so you generate dynamic data sets per release. Instead of working with a backward compatibility mindset with older static data.</p>

<p>The problem is to keep the app and the data in sync and valid for each iteration, while allowing the PMs and engineers to work in parallel from one another without relying on each other.</p>

<azure-devops><workflow><data>
DevOps approach to creating demo data?
Joseph Luce
# DevOps approach to creating demo data?

(Asked by: username_0 on 2020-02-07)

Is there a good DevOps method to construct a workflow that creates demo data for a product? This situation is difficult when the app constantly changes. There are a few scenarios to consider.

1. Either old demo data becomes obsolete in the new build or the old
   data can be considered corrupted in the new build.
2. New business logic is introduced which requires interaction with the
   app for the new data to be formed. You cannot look at the new
   schema, static rules or old data to derivative/fill in such data.
3. Program managers have the insider info on what the demo data needs
   to look like, not the engineers. So the workflow needs to match up
   during the process, from (build -> test -> deploy) somehow.

One approach is to run tests that have demo data in it. So after the tests were ran, we can extract the data for our demos. But I can see that the tests can be very long and not as instant as importing a csv file. Also the coordination between PMs and engineers seems to be tightly coupled.

Another approach is to start with the new scheme from the repository and generate data from there. But this requires overhead of creating rules for generation which may not be in sync with the logic of app itself.

It seems a top down approach is better, so you generate dynamic data sets per release. Instead of working with a backward compatibility mindset with older static data.

The problem is to keep the app and the data in sync and valid for each iteration, while allowing the PMs and engineers to work in parallel from one another without relying on each other.

**Tags:** azure-devops, workflow, data


---

## 2 answers

### Answer by username_2 on 2020-02-07. Score: 1

Welcome to DevOps SE!

Is it possible to make data management and operation of data sets to be an asset on its own? It seems like data has become more important so a more *data-centric* approach could help.

> Program managers have the insider info on what the demo data needs to
> look like

This proves also organizational silo and lack of communication in the process: even before DevOps, in an agile team e.g. scrum you could have not only engineers but also a product owner and testers who take responsibility to provide not only useful tests but also useful testdata to the team.

That is, the fourth possible scenario could be that you take a data set from production and scramble/anonymize it. If it is too large, you need to reduce it to a representative mix. A DevOps way to do it would be to automate all these steps after they are well understood. => Working real life scenario.

There are some legal workaround methods though which require though considerable additional effort:

* data scrambling
* data anonymization
* synthetic data

Depending on your domain, there could be also applicable and more or less flexible enterprise tools for master data management where product owners could manage and version various data profiles.

Since DevOps approach is more than just finding a tool to use, but also doing experiments to find out more, here are examples of recent academic research on these topics.

An example study: *[Going Beyond Obscurity: Organizational Approaches to Data Anonymization](https://dl.acm.org/doi/abs/10.1145/3274335?casa_token=uI4NBUmHDIwAAAAA%3AheKYJO5RRmnDKsO_c5ieBjPQjnTS5_IumTWqWMrijRcIqxXwbJQl8xrtq1Z0RtHRZEcbYWzKNIcbjQ)* (Hargitai et al. 2018)

> Rather than being a purely technical question of applying the right
> algorithms, **anonymization** in practice is a complex socio-technical
> process that relies on multi-stakeholder collaborations.

Or: [*Machine learning using synthetic and real data: Similarity of evaluation metrics for different healthcare datasets and for different algorithms*](https://www.worldscientific.com/doi/abs/10.1142/9789813273238_0160). (Heyburn et al., 2018)

> In this paper, we carry out an experiment to study the validity of
> conducting machine learning on **synthetic data**.

> **Comments:**
> **username_0**: Can you elaborate what you mean by data management and operation of data sets as independent assets?      I meant the PMs know what the demo data/configurations should look like for a specific customer. The work process between PMs and engineers are decoupled. This is fine. After tests and release, the PMs have the job to sell the product to the customer, this is where demo data is needed. However, the product can change and the old data that the PMs used can become irrelevant and the PMs have to recreate new demo data.      I cannot use customer data, they will not allow that.  
> **username_2**: therefore I've mentioned that enterprises do data scrambling and anonimizing to retain the data structure but remove its criticality. e.g. https://medium.com/district-data-labs/a-practical-guide-to-anonymizing-datasets-with-python-faker-ecf15114c9be  
> **username_0**: I am not allowed to touch/gather any customer data, that is the problem. I don't have a starting data set to scrub.  
> **username_2**: "I am not allowed" this also shouldn't be necessarily your job, but you could raise the demand for a demo data set as preliminary item to do your work well. I have just shared what other enterprises typically do to support more flexible scenarios, long before DevOps. Also without giving you access to productive data, other teams could deliver demo data sets: they have the knowledge and access, they could deliver some input to support your work. Otherwise it does not feel like fair play in terms of DevOps culture. So, your problem is imho less technical but more of organizational nature.  
> **username_0**: Not allowed as its stepping into legal privacy laws.  
> **username_2**: Yes, GDPR is why I am trying to establish a link to advanced methods. Exactly these methods have been developed, or are yet to develop, to comply with legal privacy laws. Thank you for mentioning this, indeed there is ongoing academic research on that which I've added to my answer.  

---

### Answer by username_1 on 2020-02-10. Score: 0

Would it be possible to segregate data by level of maturity/stability of its functionality? The same approach which is used to version the API might work here quite well.

Conceptually it would mean that data for *stable* functionality could be copied from build to build via snapshots of the underlying data store (maybe with few alterations depending on your context).

Data for *preview* (unstable, in development) functionality is best to generate - this way no matter how functionality is altered, it is always possible to get the fresh view of how the product is going to look like. Generation can happen (for example) as a pre-demo step, or straight in the app (auto-fill in the UI, it can even be made to look neat).

As soon as *preview* functionality graduates to *stable*, its data can be included into the snapshot (as an additional release process step)

It would be best to align with engineering on common way to generate data for new functionality and then ship generators for it’s data along with code itself.

This hybrid approach allows to achieve following benefits:

* Set up and align expectations between devops, engineering and product about how test data is obtained
* No need to deal with exponential growth of time to generate *entire* data set
* Enable quick iteration of features under development
* Educate developers and make them more DevOps aware (they’d have to take last stable data snapshot and roll it out in their dev env when they start working on a feature)
* Enable fast feedback loop between all the parties involved which leads to better product quality (if snapshot doesn’t rollout or app crashes, then the release process is broken)

There aren’t many cons of this approach, but I’d highlight a few most important:

* It requires work. Main point of this approach is to fix (or actually define) the process, so it would require taking an extra mile to weigh, explain, prototype and build the alignment
* It would not solve problems with depencies in preview functionality, eg if feature B depends on A and they *both* are under development, data for both have to be generated and A has to be stabilized before B.

Overall, it’s not an easy problem you’re dealing with, and I would be appreciste if you could share your experience on how did it go, what path have your project had taken and reasoning behind it. Cheers!


---

