<p>Personally I don't see any reason for which an <a href="https://devops.stackexchange.com/questions/452/what-is-an-artifact-repository">Artefact Repository</a> - the recommeneded DevOps tool of managing artefacts - wouldn't be applicable to trained neural nets or other artefacts. </p>

<p>The artefact size <em>might</em> have some upper limit for a particular artefact repository, but in such case it would be a technical or policy limitation one, not a fundamental/principial one.</p>

<p>As for applying DevOps methodologies for the process producing these artefacts, I think most if not all of them can be applied equally well, as long as the artefacts:</p>

<ul>
<li>are produced from some sort of specification which supports change versioning (equivalent to software source code)</li>
<li>are built via a repeatable and automatable process</li>
<li>are validated using some sort of repeatable and automatable verification (similar to QA), eventually using some supporting data (training data in this case, equivalent to DB snapshots, for example)</li>
</ul>

<p>Side note: monolithic software code delivery is still a big deal and is perfectly maintainable with DevOps methodologies (with a bit of care), not everything can be split in microservices. Size doesn't matter enough to make DevOps not applicable.</p>

CC BY-SA 3.0
2017-09-14T02:41:47.497Z
Perfect answer.  I store all my heavy models in `git lfs` and pull them when necessary [serverless paradigm] :)
Dawny33
CC BY-SA 3.0
2017-09-14T07:38:50.773Z
@Dawny33 but would you now consider moving away from git lfs?
Ta Mu
CC BY-SA 3.0
2017-09-14T08:14:48.993Z
@J.Doe So far so good with lfs. Would probably move if I find a really good better alternative.
Dawny33
CC BY-SA 3.0
2017-09-14T08:16:34.530Z
then I do not get why you say the answer about is "perfect" while it suggests using an artefact repository?! @Dawny33
Ta Mu
CC BY-SA 3.0
2017-09-14T08:37:21.473Z
@J.Doe By the linked answer, I thought using `git lfs` is akin to the concept of using an artefact repo.  If I'm wrong, then my bad :D
Dawny33
CC BY-SA 3.0
2017-09-14T15:31:13.720Z
@J.Doe are you somehow suggesting that using `git lfs` and a dedicated artifact repository are not equally devopsy? "cuz they both can be used...
Dan Cornilescu
CC BY-SA 3.0
2017-09-15T07:11:36.227Z
@DanCornilescu hehe gotcha! Why then is that not listed as valid alternative in your answer? ;)
Ta Mu
CC BY-SA 3.0
2017-09-15T07:34:11.053Z
@J.Doe I'm not familiar with `git lfs`, I just read about it when I saw your comment. Also I didn't mean to list actual tools in the family.
Dan Cornilescu
CC BY-SA 4.0
2018-06-20T21:06:53.347Z
[DVC](https://dvc.org) can be considered as a better alternative to `git-lfs`
Shcheklein
CC BY-SA 3.0
2017-09-13T16:07:45.937Z
2017-09-13T16:12:58.623Z
2017-09-13T16:12:58.623Z
Dan Cornilescu
<p>I would recommend taking a look at <a href="https://dvc.org" rel="nofollow noreferrer">DVC</a> - an open source version control system for data science projects.</p>

<p>One of the basic things that it perfectly handles is managing data files (along with code) - inputs, outputs (models), intermediate results. Semantically it's similar to <code>git-lfs</code> but unlike <code>git-lfs</code> it is capable of managing files like 100GB and what is more important it does not rely on proprietary storage/format. It's completely open-source and is compatible with any network storage as a server to keep data files - S3, GCP cloud storage, SSH, FTP, etc.</p>

CC BY-SA 4.0
2021-04-09T09:53:41.330Z
DVC has some nice features, but I don't believe Git LFS uses proprietary storage - it's an open source project.
RichVel
CC BY-SA 4.0
2018-06-20T21:03:28.073Z
2018-06-20T21:03:28.073Z
Shcheklein
<p>Normally, one important topic in DevOps is how we take care of automated creation and delivery of software artefacts.</p>
<p>With the rise of data science there is a new type of artefact - monolithic binary blobs representing a trained neural net for example or other machine learning models. Such a blob can be many GB in size and its creation is not yet standardized AFAIK which brings organizations back to the pre-CI age. Nevertheless, such blobs have their version and associated collections of training data (corpora) which tend to grow rapidly as well.</p>
<p>What are best practices to address this new challenge using DevOps methods - if even possible?</p>

CC BY-SA 3.0
2017-09-13T12:24:41.423Z
I don't see the difference between a large blob and an uberjar in java context. Same practices applies, the size of an artefact has few reasons to come into play.
Tensibai
CC BY-SA 3.0
2017-09-13T13:39:28.503Z
Hi - I thought with uber-jars from 2gb upwards you would tell them the microservice architecture story, or?.. But model blobs just start there, 8gb will be soon not rare.
Ta Mu
CC BY-SA 3.0
2017-09-13T14:53:16.457Z
I just mean a dB snapshot of 350Go is not a different asset than a 5Mo jar, it has to be stored somewhere anyway and artifact repository can handle that
Tensibai
CC BY-SA 3.0
2017-09-13T15:40:52.013Z
I agree - just because the resulting program is large doesn't mean it isn't still compiled, versioned and stored like everything else (albeit with perhaps a few storage challenges), so I fail to see how this "brings organizations back to the pre-CI age" If an organization thinks that, I'm not sure that they actually understand DevOps/CI.
James Shewey
<continuous-integration><continuous-delivery><automation><methodology>
Data Science pipelines and monolithic model blobs
Ta Mu
# Data Science pipelines and monolithic model blobs

(Asked by: username_0 on 2017-09-13)

Normally, one important topic in DevOps is how we take care of automated creation and delivery of software artefacts.

With the rise of data science there is a new type of artefact - monolithic binary blobs representing a trained neural net for example or other machine learning models. Such a blob can be many GB in size and its creation is not yet standardized AFAIK which brings organizations back to the pre-CI age. Nevertheless, such blobs have their version and associated collections of training data (corpora) which tend to grow rapidly as well.

What are best practices to address this new challenge using DevOps methods - if even possible?

**Tags:** continuous-integration, continuous-delivery, automation, methodology

> **Comments:**
> **username_1**: I don't see the difference between a large blob and an uberjar in java context. Same practices applies, the size of an artefact has few reasons to come into play.  
> **username_0**: Hi - I thought with uber-jars from 2gb upwards you would tell them the microservice architecture story, or?.. But model blobs just start there, 8gb will be soon not rare.  
> **username_1**: I just mean a dB snapshot of 350Go is not a different asset than a 5Mo jar, it has to be stored somewhere anyway and artifact repository can handle that  
> **username_2**: I agree - just because the resulting program is large doesn't mean it isn't still compiled, versioned and stored like everything else (albeit with perhaps a few storage challenges), so I fail to see how this "brings organizations back to the pre-CI age" If an organization thinks that, I'm not sure that they actually understand DevOps/CI.  

---

## 2 answers

### Answer by username_3 on 2017-09-13. Score: 9 | Accepted answer

Personally I don't see any reason for which an [Artefact Repository](https://devops.stackexchange.com/questions/452/what-is-an-artifact-repository) - the recommeneded DevOps tool of managing artefacts - wouldn't be applicable to trained neural nets or other artefacts.

The artefact size *might* have some upper limit for a particular artefact repository, but in such case it would be a technical or policy limitation one, not a fundamental/principial one.

As for applying DevOps methodologies for the process producing these artefacts, I think most if not all of them can be applied equally well, as long as the artefacts:

* are produced from some sort of specification which supports change versioning (equivalent to software source code)
* are built via a repeatable and automatable process
* are validated using some sort of repeatable and automatable verification (similar to QA), eventually using some supporting data (training data in this case, equivalent to DB snapshots, for example)

Side note: monolithic software code delivery is still a big deal and is perfectly maintainable with DevOps methodologies (with a bit of care), not everything can be split in microservices. Size doesn't matter enough to make DevOps not applicable.

> **Comments:**
> **username_4**: Perfect answer.  I store all my heavy models in `git lfs` and pull them when necessary [serverless paradigm] :)  
> **username_0**: @username_4 but would you now consider moving away from git lfs?  
> **username_4**: @J.Doe So far so good with lfs. Would probably move if I find a really good better alternative.  
> **username_0**: then I do not get why you say the answer about is "perfect" while it suggests using an artefact repository?! @username_4  
> **username_4**: @J.Doe By the linked answer, I thought using `git lfs` is akin to the concept of using an artefact repo.  If I'm wrong, then my bad :D  
> **username_3**: @J.Doe are you somehow suggesting that using `git lfs` and a dedicated artifact repository are not equally devopsy? "cuz they both can be used...  
> **username_0**: @DanCornilescu hehe gotcha! Why then is that not listed as valid alternative in your answer? ;)  
> **username_3**: @J.Doe I'm not familiar with `git lfs`, I just read about it when I saw your comment. Also I didn't mean to list actual tools in the family.  
> **username_5**: [DVC](https://dvc.org) can be considered as a better alternative to `git-lfs`  

---

### Answer by username_5 on 2018-06-20. Score: 4

I would recommend taking a look at [DVC](https://dvc.org) - an open source version control system for data science projects.

One of the basic things that it perfectly handles is managing data files (along with code) - inputs, outputs (models), intermediate results. Semantically it's similar to `git-lfs` but unlike `git-lfs` it is capable of managing files like 100GB and what is more important it does not rely on proprietary storage/format. It's completely open-source and is compatible with any network storage as a server to keep data files - S3, GCP cloud storage, SSH, FTP, etc.

> **Comments:**
> **username_6**: DVC has some nice features, but I don't believe Git LFS uses proprietary storage - it's an open source project.  

---

