<p>Could you give an example of names of input files and output files of <code>normalize.py</code> ?</p>

<p>It seems to me that your case could be similar to the one described about the <code>dynamic</code> output:</p>

<p><a href="https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#dynamic-files" rel="nofollow noreferrer">https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#dynamic-files</a></p>

<p>Here you would probably use it for the output of <code>normalization</code> and for the input of <code>all</code> (with a wildcard different from <code>hdf5_file</code>).</p>

<p>Disclaimer: I haven't had the opportunity to use this feature so far.</p>

CC BY-SA 4.0
2018-09-11T21:30:44.003Z
Sample name is `{hdf5_file}_{num}.hdf5`, where `num` is an integer.
Nikita Vlasenko
CC BY-SA 4.0
2018-09-07T09:30:14.340Z
2018-09-07T09:30:14.340Z
bli
<p>I have a folder where the <code>hdf5</code> files inputs are (matrices of <code>scRNAseq</code> raw <code>UMI</code> counts). There is a <code>python</code> script that loads them, splits them into multiple matrices, normalizes them, and saves as <code>csv</code> files. When the name pattern of the output files is known, things are easy, I know how to do that, but when one input (<code>hdf5</code> file) can produce various number of outputs (<code>csv</code> files) I am not sure how to approach the issue. Currently my <code>Snakemake</code> file relevant part looks the following way:</p>

<pre><code>hdf5_files,=glob_wildcards('/main/{hdf5_file}.hdf5') 

rule all:
  input:
---&gt;  expand('data_files/normalized/{hdf5_file}.csv', hdf5_file=hdf5_files)
    ...

rule normalization:
  input:
     script = 'python/normalize.py',
     path = '/main/{hdf5_file}.hdf5'
  output:
---&gt;  path = 'data_files/normalized/{hdf5_file}.csv'
  shell:
     "python {input.script} -data {input.path} -path {output.path}"
</code></pre>

<p>Above I highlighted the <code>2</code> lines that I am not sure now how I should write. Any suggestions would be greatly appreciated.</p>

CC BY-SA 4.0
2018-09-05T19:55:59.717Z
Have you considered using [flag files](https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#flag-files)? Or do you need the names for downstream processing too and can't use `glob` or specify a directory at that point?
Devon Ryan
CC BY-SA 4.0
2018-09-05T21:39:16.980Z
The `rule normalization` processes, say, `2` `hdf5` files and outputs `10` `csv` ones into `data_files/normalized` folder. Then, I want to use `glob_wildcards` to read in these `10` `csv` files and move on.  Further rules are, of course, fully dependent on the `rule normalization` because it generates the input `csv` files. I am not sure how can I use `flag files`, but will try. Thank you!
Nikita Vlasenko
CC BY-SA 4.0
2018-09-05T22:19:17.973Z
Ah, if you need to process each of the resulting csv files independently downstream then there are no snakemake tricks that I know of that will help (I presume `glob_wildcards()` is run during rule parsing, so the CSV files won't be there yet). I suspect you'll want to run `normalization` from a wrapper script beforehand in that case, since then globbing will be certain to work as desired.
Devon Ryan
CC BY-SA 4.0
2018-09-06T02:27:23.050Z
What do you mean from a `wrapper script`? Outside the pipeline, you mean?
Nikita Vlasenko
CC BY-SA 4.0
2018-09-06T07:04:22.620Z
Yes, exactly. A (probably python) script that runs two Snakefiles when you call it, with appropriate options. Then you have the flexibility you need while maintaining the convenience of snakemake.
Devon Ryan
<snakemake>
Output file names are not known beforehand
# Output file names are not known beforehand

(Asked by: username_0 on 2018-09-05)

I have a folder where the hdf5 files inputs are (matrices of scRNAseq raw UMI counts). There is a python script that loads them, splits them into multiple matrices, normalizes them, and saves as csv files. When the name pattern of the output files is known, things are easy, I know how to do that, but when one input (hdf5 file) can produce various number of outputs (csv files) I am not sure how to approach the issue. Currently my Snakemake file relevant part looks the following way:
hdf5_files,=glob_wildcards('/main/{hdf5_file}.hdf5') 

rule all:
  input:
--->  expand('data_files/normalized/{hdf5_file}.csv', hdf5_file=hdf5_files)
    ...

rule normalization:
  input:
     script = 'python/normalize.py',
     path = '/main/{hdf5_file}.hdf5'
  output:
--->  path = 'data_files/normalized/{hdf5_file}.csv'
  shell:
     "python {input.script} -data {input.path} -path {output.path}"

Above I highlighted the 2 lines that I am not sure now how I should write. Any suggestions would be greatly appreciated.

**Tags:** snakemake
> **Comments:**
> **username_1**: Have you considered using [flag files](https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#flag-files)? Or do you need the names for downstream processing too and can't use `glob` or specify a directory at that point?  
> **username_0**: The `rule normalization` processes, say, `2` `hdf5` files and outputs `10` `csv` ones into `data_files/normalized` folder. Then, I want to use `glob_wildcards` to read in these `10` `csv` files and move on.  Further rules are, of course, fully dependent on the `rule normalization` because it generates the input `csv` files. I am not sure how can I use `flag files`, but will try. Thank you!  
> **username_1**: Ah, if you need to process each of the resulting csv files independently downstream then there are no snakemake tricks that I know of that will help (I presume `glob_wildcards()` is run during rule parsing, so the CSV files won't be there yet). I suspect you'll want to run `normalization` from a wrapper script beforehand in that case, since then globbing will be certain to work as desired.  
> **username_0**: What do you mean from a `wrapper script`? Outside the pipeline, you mean?  
> **username_1**: Yes, exactly. A (probably python) script that runs two Snakefiles when you call it, with appropriate options. Then you have the flexibility you need while maintaining the convenience of snakemake.  

---

## 1 answers

### Answer by username_2 on 2018-09-07. Score: 4 | Accepted answer

Could you give an example of names of input files and output files of `normalize.py` ?

It seems to me that your case could be similar to the one described about the `dynamic` output:

<https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#dynamic-files>

Here you would probably use it for the output of `normalization` and for the input of `all` (with a wildcard different from `hdf5_file`).

Disclaimer: I haven't had the opportunity to use this feature so far.

> **Comments:**
> **username_0**: Sample name is `{hdf5_file}_{num}.hdf5`, where `num` is an integer.  

---

