<p>Looper does not generate any of these per-sample outputs.</p>

<p>Looper does not have any control over where your pipeline puts its outputs.</p>

<p>The problem is in your pipeline, not in looper. Assuming you're using a pypiper pipeline, I suggest reading the docs; for example, the "your first pipeline" tutorial: <a href="http://pypiper.readthedocs.io/en/latest/tutorials-first.html" rel="nofollow noreferrer">http://pypiper.readthedocs.io/en/latest/tutorials-first.html</a></p>

<p>For real-life example, see <a href="https://github.com/databio/dnameth_pipelines/blob/3e6abd03185f8e2d5ef1c72849afdf9e5033f269/src/rrbs.py" rel="nofollow noreferrer">this pipeline</a></p>

<p><code>
outfolder = os.path.abspath(os.path.join(args.output_parent, args.sample_name))
pm = pypiper.PipelineManager(name="RRBS", outfolder=outfolder, args=args, version=__version__)
</code></p>

CC BY-SA 3.0
2018-01-20T03:29:58.567Z
2018-01-20T03:29:58.567Z
nsheff
<p>I have multiple samples that I need to run one pipeline on. I am using Looper: <a href="https://github.com/pepkit/looper" rel="nofollow noreferrer">https://github.com/pepkit/looper</a></p>

<p>I have a real trouble specifying a separate location for each pipeline output files. The first time when I did not care about it thinking that <code>Looper</code> would just automatically do that, it lead to multiple files being overwritten. Then, trying to avoid it I tried (and it did not work):</p>

<blockquote>
  <ol>
  <li>Adding <code>{sample_name}</code> to the <code>output_dir</code> of the <code>metadata</code> section of <code>project_config.yaml</code> file:</li>
  </ol>
</blockquote>

<pre><code> metadata:
  sample_annotation: sandro_annotation.csv
  output_dir: /scratch/nv4e/sandro_looper_results/{sample_name}
  pipelines_dir: [../pipeline/pipeline_interface.yaml]
</code></pre>

<blockquote>
  <ol start="2">
  <li>Creating a <code>derived_column</code> in the <code>.csv</code> annotation file:</li>
  </ol>
</blockquote>

<pre><code>sample_name,library,sample_name_1_read,sample_name_2_read,output_folder
abeta_24h_1,RNA-seq,read1,read2,out
abeta_24h_2,RNA-seq,read1,read2,out
abeta_24h_3,RNA-seq,read1,read2,out
</code></pre>

<p><code>pipeline_interface.yaml</code>:</p>

<pre><code>...
pipelines:
  sandro_rna_seq:
    name: sandro_rna_seq
    path: sandro_rna_seq.py  # relative to this 
pipeline_interface.yaml file
    looper_args: True
    arguments:
      "-S": sample_name
      "--input": sample_name_1_read
      "-O": output_folder
...
</code></pre>

<p>and <code>project.config.yaml</code>:</p>

<pre><code>...
derived_columns: [sample_name_1_read,sample_name_2_read,output_folder]
data_sources:
  read1: /scratch/nv4e/sandro_looper/data/merged/{sample_name}_r1.fq.gz
  read2: /scratch/nv4e/sandro_looper/data/merged/{sample_name}_r2.fq.gz
  out: /scratch/nv4e/sandro_looper_results/results_pipeline/{sample_name}
...
</code></pre>

<p>In this particular case the parameter <code>-O</code> was formed correctly which I was able to see in <code>sandro_looper_results/submissions/</code> <code>.sub</code> file, however, it was supplied the second time, apparently from the <code>output_dir</code> section of the <code>project_config.yaml</code> and was overriding the first <code>-O</code> command. So, obviously I tried to delete <code>output_dir</code> line but it is a required setting for <code>Looper</code>, so it can not run without it.</p>

<blockquote>
  <ol start="3">
  <li>I have also tried supplying <code>-O</code> command in <code>project_config.yaml</code> in the <code>pipeline_args:</code> section with no success.</li>
  </ol>
</blockquote>

<p>I really do not know what I could do at this point. Any suggestions would be greatly appreciated,</p>

<scrnaseq><yaml><looper>
Looper: can't specify the individual path for one pipeline run
# Looper: can't specify the individual path for one pipeline run

(Asked by: username_0 on 2018-01-20)

I have multiple samples that I need to run one pipeline on. I am using Looper: https://github.com/pepkit/looper
I have a real trouble specifying a separate location for each pipeline output files. The first time when I did not care about it thinking that Looper would just automatically do that, it lead to multiple files being overwritten. Then, trying to avoid it I tried (and it did not work):


Adding {sample_name} to the output_dir of the metadata section of project_config.yaml file:


 metadata:
  sample_annotation: sandro_annotation.csv
  output_dir: /scratch/nv4e/sandro_looper_results/{sample_name}
  pipelines_dir: [../pipeline/pipeline_interface.yaml]



Creating a derived_column in the .csv annotation file:


sample_name,library,sample_name_1_read,sample_name_2_read,output_folder
abeta_24h_1,RNA-seq,read1,read2,out
abeta_24h_2,RNA-seq,read1,read2,out
abeta_24h_3,RNA-seq,read1,read2,out

pipeline_interface.yaml:
...
pipelines:
  sandro_rna_seq:
    name: sandro_rna_seq
    path: sandro_rna_seq.py  # relative to this 
pipeline_interface.yaml file
    looper_args: True
    arguments:
      "-S": sample_name
      "--input": sample_name_1_read
      "-O": output_folder
...

and project.config.yaml:
...
derived_columns: [sample_name_1_read,sample_name_2_read,output_folder]
data_sources:
  read1: /scratch/nv4e/sandro_looper/data/merged/{sample_name}_r1.fq.gz
  read2: /scratch/nv4e/sandro_looper/data/merged/{sample_name}_r2.fq.gz
  out: /scratch/nv4e/sandro_looper_results/results_pipeline/{sample_name}
...

In this particular case the parameter -O was formed correctly which I was able to see in sandro_looper_results/submissions/ .sub file, however, it was supplied the second time, apparently from the output_dir section of the project_config.yaml and was overriding the first -O command. So, obviously I tried to delete output_dir line but it is a required setting for Looper, so it can not run without it.


I have also tried supplying -O command in project_config.yaml in the pipeline_args: section with no success.


I really do not know what I could do at this point. Any suggestions would be greatly appreciated,

**Tags:** scrnaseq, yaml, looper

---

## 1 answers

### Answer by username_1 on 2018-01-20. Score: 1 | Accepted answer

Looper does not generate any of these per-sample outputs.

Looper does not have any control over where your pipeline puts its outputs.

The problem is in your pipeline, not in looper. Assuming you're using a pypiper pipeline, I suggest reading the docs; for example, the "your first pipeline" tutorial: <http://pypiper.readthedocs.io/en/latest/tutorials-first.html>

For real-life example, see [this pipeline](https://github.com/databio/dnameth_pipelines/blob/3e6abd03185f8e2d5ef1c72849afdf9e5033f269/src/rrbs.py)

`outfolder = os.path.abspath(os.path.join(args.output_parent, args.sample_name))
pm = pypiper.PipelineManager(name="RRBS", outfolder=outfolder, args=args, version=__version__)`


---

