<p>I think one really nice example right now is the <a href="https://github.com/spacetelescope/jwst" rel="noreferrer">James Webb pipeline</a>. I haven't dived into the internals yet but I think it does a good job of demonstrating the best-practices of software engineering that you mentioned.</p>

CC BY-SA 4.0
2023-04-05T06:41:15.243Z
I just looked it up, and indeed it is an excellent example well-engineered pipelines.
nsimplex
CC BY-SA 4.0
2023-04-04T17:42:36.090Z
2023-04-04T17:49:24.713Z
2023-04-04T17:49:24.713Z
Roy Smart
<p>Not a necessarily overly authoritative answer because I don't know first hand, but beyond <a href="https://astronomy.stackexchange.com/a/53310/7982">RoySart's JWST pipeline recommendation</a>, another example of something really massive might be that of the <a href="https://en.wikipedia.org/wiki/Vera_C._Rubin_Observatory" rel="nofollow noreferrer">Vera C. Rubin Observatory</a> &quot;...that will photograph the entire available sky every few nights.&quot;</p>
<p>The reason I think so is the massive, firehose-like amount of data it will generate. The high cadence of re-observation will make it particularly sensitive as a <a href="/questions/tagged/time-domain-astronomy" class="post-tag" title="show questions tagged &#39;time-domain-astronomy&#39;" aria-label="show questions tagged &#39;time-domain-astronomy&#39;" rel="tag" aria-labelledby="tag-time-domain-astronomy-tooltip-container">time-domain-astronomy</a> instrument.</p>
<p>It will be setting off all kinds of alarms</p>
<ul>
<li><a href="https://astronomy.stackexchange.com/q/32777/7982">Will the Vera C. Rubin Observatory (formerly the LSST) make a significant increase in the rate of astronomical event alerts?</a></li>
<li><a href="https://astronomy.stackexchange.com/q/32778/7982">Is there a standardized &quot;Astronomical Alert&quot; system?</a></li>
</ul>
<p>much more than Zwicky</p>
<ul>
<li><a href="https://astronomy.stackexchange.com/q/31761/7982">Does the Zwicky Transient Facility only run when triggered by a cell phone?</a></li>
</ul>
<p>and so those data pipelines will need to be carefully vetted and tested and regularly tweaked and fine-tuned, and the only way for that to work efficiently is if they are well understood by a sizable group of astronomical programmers (programming astronomers? astro-engineers?). And the only way that's going to happen is if the whole thing is very well engineered <em><strong>and documented!</strong></em></p>
<p>So while it may be too big to serve as an easy example for you, there might be some explanations you can digest and use in your course.</p>

CC BY-SA 4.0
2023-04-05T08:16:56.900Z
2023-04-05T08:16:56.900Z
uhoh
<p>I am a software engineer and I am preparing a presentation to aspiring astronomy PhDs on how to use best-practice software engineering when publishing code (such as include documentation, modular design, include tests, ...). In particular my presentation will be focused on publishing &quot;pipeline&quot; code, that is, code that is mainly focused on extracting and transforming data from a particular dataset to a suitable shape for analysis.</p>
<p>I am trying to find good example of published astronomy pipelines that I can point students to, but as I am not a astronomer I am struggling to do so, so would like your help. It doesn't matter if the published pipeline is super-niche or not very popular so long as you think it is engineered well so that I can look it up.</p>

CC BY-SA 4.0
2023-04-04T17:59:46.423Z
I'm working on the 4MOST pipeline at the moment, but it is very much "in progress".
ProfRob
<data-analysis><data-pipeline>
What's a good example of published astronomy pipeline coding libraries
nsimplex
# What's a good example of published astronomy pipeline coding libraries

(Asked by: username_0 on 2023-04-03)

I am a software engineer and I am preparing a presentation to aspiring astronomy PhDs on how to use best-practice software engineering when publishing code (such as include documentation, modular design, include tests, ...). In particular my presentation will be focused on publishing "pipeline" code, that is, code that is mainly focused on extracting and transforming data from a particular dataset to a suitable shape for analysis.

I am trying to find good example of published astronomy pipelines that I can point students to, but as I am not a astronomer I am struggling to do so, so would like your help. It doesn't matter if the published pipeline is super-niche or not very popular so long as you think it is engineered well so that I can look it up.

**Tags:** data-analysis, data-pipeline

> **Comments:**
> **username_1**: I'm working on the 4MOST pipeline at the moment, but it is very much "in progress".  

---

## 2 answers

### Answer by username_2 on 2023-04-04. Score: 8

I think one really nice example right now is the [James Webb pipeline](https://github.com/spacetelescope/jwst). I haven't dived into the internals yet but I think it does a good job of demonstrating the best-practices of software engineering that you mentioned.

> **Comments:**
> **username_0**: I just looked it up, and indeed it is an excellent example well-engineered pipelines.  

---

### Answer by username_3 on 2023-04-05. Score: 3

Not a necessarily overly authoritative answer because I don't know first hand, but beyond [RoySart's JWST pipeline recommendation](https://astronomy.stackexchange.com/a/53310/7982), another example of something really massive might be that of the [Vera C. Rubin Observatory](https://en.wikipedia.org/wiki/Vera_C._Rubin_Observatory) "...that will photograph the entire available sky every few nights."

The reason I think so is the massive, firehose-like amount of data it will generate. The high cadence of re-observation will make it particularly sensitive as a [time-domain-astronomy](/questions/tagged/time-domain-astronomy "show questions tagged 'time-domain-astronomy'") instrument.

It will be setting off all kinds of alarms

* [Will the Vera C. Rubin Observatory (formerly the LSST) make a significant increase in the rate of astronomical event alerts?](https://astronomy.stackexchange.com/q/32777/7982)
* [Is there a standardized "Astronomical Alert" system?](https://astronomy.stackexchange.com/q/32778/7982)

much more than Zwicky

* [Does the Zwicky Transient Facility only run when triggered by a cell phone?](https://astronomy.stackexchange.com/q/31761/7982)

and so those data pipelines will need to be carefully vetted and tested and regularly tweaked and fine-tuned, and the only way for that to work efficiently is if they are well understood by a sizable group of astronomical programmers (programming astronomers? astro-engineers?). And the only way that's going to happen is if the whole thing is very well engineered ***and documented!***

So while it may be too big to serve as an easy example for you, there might be some explanations you can digest and use in your course.


---

