<p>Using pointers provided by @bli, I made a few changes and the Snakefile started working as expected.</p>
<p>Instead of using <code>expand</code> for the rule's input, I switched to a function that accepts the wildcard sample as input and returns a dict with the actual inputs.</p>
<pre><code> # Could be a one-liner but I wrote it in a verbose
 # manner so it's easier to read
 def get_input_files(in_sample_id):
     fq1 = samples.loc[str(in_sample_id), 'FQ1_id']
     fq2 = samples.loc[str(in_sample_id), 'FQ2_id']
     logdir = &quot;{in_sample}.logs&quot;.format(in_sample=str(in_sample_id))
     return_dict = { 'fq1' : fq1, 'fq2' : fq2, 'logdir' : logdir }
     return(return_dict)
</code></pre>
<p>And the rule <code>my_job</code> is now changed (just the <code>input</code> part):</p>
<pre><code>rule my_job:
    input:
        unpack(get_input_files)
    output:
        &quot;{sample}_1.fastq.gz&quot;,
        &quot;{sample}_2.fastq.gz&quot;,
...
...
</code></pre>
<p>Hope this helps someone looking for a similar solution in the future!</p>

CC BY-SA 4.0
2021-12-28T20:09:36.070Z
2021-12-28T20:09:36.070Z
Ram RS
<p>I'm trying to run a command using snakemake and a TSV config file. Here is my sample config file:</p>
<pre><code>sample_id       FQ1_id  FQ2_id
MyCleanName-1       MyFqName-1_R1.fastq.gz        MyFqName-1_R2.fastq.gz
MyCleanName-2       MyFqName-2_R1.fastq.gz        MyFqName-2_R2.fastq.gz
</code></pre>
<p>Here is my snakefile:</p>
<pre><code>import pandas as pd
samples = pd.read_table(&quot;samples.tsv&quot;).set_index(&quot;sample_id&quot;, drop = False)

rule all:
    input:
        expand(&quot;{sample}_{num}.fastq.gz&quot;, sample = samples.sample_id, num = [1,2])

rule my_job:
    input:
        fq1 = expand(&quot;{fq1_id}&quot;, fq1_id = samples.FQ1_id),
        fq2 = expand(&quot;{fq2_id}&quot;, fq2_id = samples.FQ2_id),
        logdir = expand(&quot;{sample}.logs&quot;, sample = samples.sample_id)
    output:
        &quot;{sample}_1.fastq.gz&quot;,
        &quot;{sample}_2.fastq.gz&quot;,
    message:
        &quot;Running MyCmd on {wildcards.sample}&quot;
    shell:
        &quot;&quot;&quot;
        MyCmd --sample-name {wildcards.sample} -i1 {input.fq1} -i2 {input.fq2}
        &quot;&quot;&quot;


rule create_logdir:
    output:
        directory(&quot;{sample}.logs&quot;)
    message:
        &quot;Creating log directories&quot;
    shell:
        &quot;mkdir -pv {wildcards.sample}.logs/{{MyCmd,MyOtherCmd}}&quot;
</code></pre>
<p>When I run <code>snakemake -np MyCleanName-1.fastq.gz</code>, I get the following:</p>
<pre><code>Building DAG of jobs...
Job counts:
        count   jobs
        2       create_logdir
        1       my_job
        3

[Fri Dec 24 16:38:54 2021]
Job 1: Creating log directories

mkdir -pv MyCleanName-1.logs/{MyCmd,MyOtherCmd}

[Fri Dec 24 16:38:54 2021]
Job 2: Creating log directories

mkdir -pv MyCleanName-2.logs/{MyCmd,MyOtherCmd}

[Fri Dec 24 16:38:54 2021]
Job 0: Running MyCmd on MDA-IBC-3


        MyCmd         --sample-name MyCleanName-1 -i1 MyFqName-1_R1.fastq.gz MyFqName-2_R1.fastq.gz -i2 MyFqName-1_R2.fastq.gz MyFqName-2_R2.fastq.gz

Job counts:
        count   jobs
        2       create_logdir
        1       my_job
        3
This was a dry-run (flag -n). The order of jobs does not reflect the order of execution.
</code></pre>
<p>How can I restrict this so the run only creates log files for the Clean Name being used and uses only the corresponding FASTQs? I feel like I'm missing something trivial here. Should I not be using the config file variable <code>samples</code> directly in the rule <code>my_job</code>, but somehow wiring up the corresponding <code>sample_id</code> and <code>FQ_id</code>s on the <code>all</code> rule?</p>

CC BY-SA 4.0
2021-12-26T09:41:21.933Z
`expand` is a Snakemake help function that tends to create confusion for beginners. The goal is to generate lists of file names from patterns, usually to generate the input of a "gathering" / "summarizing" rule. If you want a single file name, I would recommend to use standard Python (>=3.6) f-strings.
bli
CC BY-SA 4.0
2021-12-26T09:45:57.040Z
However, in your case, I think that the use of a function as input will likely be part of the solution: https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html?#functions-as-input-files Maybe you can have a look at this answer for some ideas of how to proceed: https://stackoverflow.com/a/64660447/1878788
bli
CC BY-SA 4.0
2021-12-27T15:38:16.923Z
Thank you, @bli - I'll look into this - functions as input might make the most sense for my case.
Ram RS
<snakemake>
Pick matching entry from snakemake config table
# Pick matching entry from snakemake config table

(Asked by: username_0 on 2021-12-24)

I'm trying to run a command using snakemake and a TSV config file. Here is my sample config file:
sample_id       FQ1_id  FQ2_id
MyCleanName-1       MyFqName-1_R1.fastq.gz        MyFqName-1_R2.fastq.gz
MyCleanName-2       MyFqName-2_R1.fastq.gz        MyFqName-2_R2.fastq.gz

Here is my snakefile:
import pandas as pd
samples = pd.read_table("samples.tsv").set_index("sample_id", drop = False)

rule all:
    input:
        expand("{sample}_{num}.fastq.gz", sample = samples.sample_id, num = [1,2])

rule my_job:
    input:
        fq1 = expand("{fq1_id}", fq1_id = samples.FQ1_id),
        fq2 = expand("{fq2_id}", fq2_id = samples.FQ2_id),
        logdir = expand("{sample}.logs", sample = samples.sample_id)
    output:
        "{sample}_1.fastq.gz",
        "{sample}_2.fastq.gz",
    message:
        "Running MyCmd on {wildcards.sample}"
    shell:
        """
        MyCmd --sample-name {wildcards.sample} -i1 {input.fq1} -i2 {input.fq2}
        """


rule create_logdir:
    output:
        directory("{sample}.logs")
    message:
        "Creating log directories"
    shell:
        "mkdir -pv {wildcards.sample}.logs/{{MyCmd,MyOtherCmd}}"

When I run snakemake -np MyCleanName-1.fastq.gz, I get the following:
Building DAG of jobs...
Job counts:
        count   jobs
        2       create_logdir
        1       my_job
        3

[Fri Dec 24 16:38:54 2021]
Job 1: Creating log directories

mkdir -pv MyCleanName-1.logs/{MyCmd,MyOtherCmd}

[Fri Dec 24 16:38:54 2021]
Job 2: Creating log directories

mkdir -pv MyCleanName-2.logs/{MyCmd,MyOtherCmd}

[Fri Dec 24 16:38:54 2021]
Job 0: Running MyCmd on MDA-IBC-3


        MyCmd         --sample-name MyCleanName-1 -i1 MyFqName-1_R1.fastq.gz MyFqName-2_R1.fastq.gz -i2 MyFqName-1_R2.fastq.gz MyFqName-2_R2.fastq.gz

Job counts:
        count   jobs
        2       create_logdir
        1       my_job
        3
This was a dry-run (flag -n). The order of jobs does not reflect the order of execution.

How can I restrict this so the run only creates log files for the Clean Name being used and uses only the corresponding FASTQs? I feel like I'm missing something trivial here. Should I not be using the config file variable samples directly in the rule my_job, but somehow wiring up the corresponding sample_id and FQ_ids on the all rule?

**Tags:** snakemake
> **Comments:**
> **username_1**: `expand` is a Snakemake help function that tends to create confusion for beginners. The goal is to generate lists of file names from patterns, usually to generate the input of a "gathering" / "summarizing" rule. If you want a single file name, I would recommend to use standard Python (>=3.6) f-strings.  
> **username_1**: However, in your case, I think that the use of a function as input will likely be part of the solution: https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html?#functions-as-input-files Maybe you can have a look at this answer for some ideas of how to proceed: https://stackoverflow.com/a/64660447/1878788  
> **username_0**: Thank you, @bli - I'll look into this - functions as input might make the most sense for my case.  

---

## 1 answers

### Answer by username_0 on 2021-12-28. Score: 1 | Accepted answer

Using pointers provided by @bli, I made a few changes and the Snakefile started working as expected.
Instead of using expand for the rule's input, I switched to a function that accepts the wildcard sample as input and returns a dict with the actual inputs.
 # Could be a one-liner but I wrote it in a verbose
 # manner so it's easier to read
 def get_input_files(in_sample_id):
     fq1 = samples.loc[str(in_sample_id), 'FQ1_id']
     fq2 = samples.loc[str(in_sample_id), 'FQ2_id']
     logdir = "{in_sample}.logs".format(in_sample=str(in_sample_id))
     return_dict = { 'fq1' : fq1, 'fq2' : fq2, 'logdir' : logdir }
     return(return_dict)

And the rule my_job is now changed (just the input part):
rule my_job:
    input:
        unpack(get_input_files)
    output:
        "{sample}_1.fastq.gz",
        "{sample}_2.fastq.gz",
...
...

Hope this helps someone looking for a similar solution in the future!


---

