<p>Placing two wildcards directly behind each other will make it impossible for snakemake to determine where one wildcard ends and the other begins. So it is simply taking the longest possible match for the first wildcard (<code>sample</code>) and then one character for the second (<code>read</code>). This will also cause the <code>_</code> to be included in the wildcard value <code>sample</code>.</p>

<p>I'm assuming that this same pattern is used for the inputs of another rule to refer to the trim_galore_pe output. Hence why the <code>_</code> is included in the wildcards there.</p>

<p>You might be able to fix this, simply by adding a <code>_</code> in-between <code>{sample}</code> and <code>{readgroup}</code> in your first rule (and any other rules using this same pattern):</p>

<pre><code>rule fastqc:
    input:
        os.path.join(config['Directories']['fastq_raw_dir'], '{sample}_{read}' + fastq_file_extension)
    output:
        html=os.path.join(config['Directories']['fastqc_dir'], '{sample}{read}_fastqc.html'),
        zip=os.path.join(config['Directories']['fastqc_dir'], '{sample}{read}_fastqc.zip')
    wrapper:
        "0.49.0/bio/fastqc"
</code></pre>

CC BY-SA 4.0
2020-02-26T07:37:06.687Z
I did this but Snakemake conjures up a new _ out of nowhere:
Building DAG of jobs...
MissingInputException in line 2 of /rst1/temp//RNASeqPairedEnd/rules/trim_galore_pe.smk:
Missing input files for rule trim_galore_pe:
fastq_raw/0053_P2017BB3S19R_S1__R1_001.fastq.gz
fastq_raw/0053_P2017BB3S19R_S1__R2_001.fastq.gz
Freek
CC BY-SA 4.0
2020-02-26T07:40:39.413Z
But why is is impossible for Snakemake to separate them? They are not in {} for nothing right? I don't understand why Snakemake would want to change my wildcards from anything it gets from the samples table. What would be the use? It seems to me that this is just a bug? Or am I really fundamentally misunderstanding Snakemake?
Freek
CC BY-SA 4.0
2020-02-26T07:46:04.657Z
Regarding my first comment, one might think that Snakemake now added and extra _ so I can remove the existing one perhaps? But no, it then removes both:
Building DAG of jobs...
MissingInputException in line 2 of /rst1/temp/RNASeqPairedEnd/rules/trim_galore_pe.smk:
Missing input files for rule trim_galore_pe:
fastq_raw/0053_P2017BB3S19R_S1R2_001.fastq.gz
fastq_raw/0053_P2017BB3S19R_S1R1_001.fastq.gz

So it's either 0 or 2, not 1, as one would expect.
Freek
CC BY-SA 4.0
2020-02-26T08:01:33.647Z
So I guess I am misunderstanding this completely, When I change the input of trim_galore_pe, it complains of missing files, but when I remove the input entirely it complains that the fastqc rule is missing input files, even though I did not touch that rule. Is there any way to more clearly see where errors originate exactly? I feel completely lost now.
Freek
CC BY-SA 4.0
2020-02-26T08:18:24.400Z
@Freek Could you share the entire Snakefile? It's quite hard to see what's going on exactly without it.
DavyCats
CC BY-SA 4.0
2020-02-26T12:15:45.773Z
Ok, I think I have found the reason, I was using code as well to create the samples_table from the same config and it was adding and removing the _ there as well. So this method does work, my apologies. It remains unhandy that I can't directly use wildcards together, But I could perhaps define a separator in the config file to solve this and keep flexibility..
Freek
CC BY-SA 4.0
2020-02-21T14:33:17.483Z
2020-02-21T14:33:17.483Z
DavyCats
<p>I think you want to prevent snakemake from interpreting your wildcard strings as regular expressions. For this, escape each wildcard value using <code>wildcard_constraints</code>. E.g., towards the top of your Snakefile add:</p>

<pre><code>wildcard_constraints:
    read= '|'.join([re.escape(x) for x in reads]),
    sample= '|'.join([re.escape(x) for x in samples])
</code></pre>

<p>(Personally, I use the <code>wildcard_constraints</code> directive as above quite liberally to constrain every wildcard)</p>

CC BY-SA 4.0
2020-02-26T07:39:14.943Z
Does this mess with how you can use rules.fastqc.output.html as the input for other rules? It gives me a lot of errors for a downstream step (multiqc).
Freek
CC BY-SA 4.0
2020-02-26T08:08:02.040Z
As far as I know, this doesn't cause problems. If you don't show the error it's difficult to tell what's going on.
dariober
CC BY-SA 4.0
2020-02-24T11:45:11.663Z
2020-02-24T11:45:11.663Z
dariober
<p>EDIT: I posted this as a bug, perhaps wait for the result of that? <a href="https://github.com/snakemake/snakemake/issues/240" rel="nofollow noreferrer">github</a></p>

<p>All,</p>

<p>I am using Snakemake to build a pipeline. To define input and output names, I constructed them based on a filename base (from the samples table), a read indicator (like R1, R2) and the fastq file extension, like so:</p>

<pre><code>samples = pd.read_csv('samples.tsv', sep='\t').set_index('samples', drop=False)
reads = ['R1', 'R2']
fastq_file_extension = _001.fastq.gz
</code></pre>

<pre><code>rule fastqc:
    input:
        os.path.join(config['Directories']['fastq_raw_dir'], '{sample}{read}' + fastq_file_extension)
    output:
        html=os.path.join(config['Directories']['fastqc_dir'], '{sample}{read}_fastqc.html'),
        zip=os.path.join(config['Directories']['fastqc_dir'], '{sample}{read}_fastqc.zip')
    wrapper:
        "0.49.0/bio/fastqc"
</code></pre>

<p>However, during pipeline execution already I can see strange things happening, for this rule it reports:</p>

<pre><code>[Fri Feb 21 12:39:57 2020]
rule fastqc:
input: fastq_raw/0053_P2017BB3S20R_S2_R1_001.fastq.gz
output: qc/fastqc/0053_P2017BB3S20R_S2_R1_fastqc.html, qc/fastqc/0053_P2017BB3S20R_S2_R1_fastqc.zip
jobid: 10
wildcards: sample=0053_P2017BB3S20R_S2_R, read=1
</code></pre>

<p>As you can see, Snakemake pulled the R part of the read to the sample and set the read to 1 (should be R1). But this still works because it produces output files as Snakemake expects.</p>

<p>But the next rule, trim_galore_pe:</p>

<pre><code>rule trim_galore_pe:
    input:
        [os.path.join(config['Directories']['fastq_raw_dir'], '{sample}' + read + fastq_file_extension) for read in reads]
    output:
        fastq_read1 = os.path.join(config['Directories']['fastq_trimmed_dir'], '{sample}.1_val_1.fq.gz'),
        fastq_read2 = os.path.join(config['Directories']['fastq_trimmed_dir'], '{sample}.2_val_2.fq.gz'),
        report_read1 = os.path.join(config['Directories']['trim_galore_qc_dir'], '{sample}.1.fastq.gz_trimming_report.txt'),
        report_read2 = os.path.join(config['Directories']['trim_galore_qc_dir'], '{sample}.2.fastq.gz_trimming_report.txt')
    params:
        extra="--retain_unpaired"
    wrapper:
        "0.49.0/bio/trim_galore/pe"
</code></pre>

<p>This goes wrong, during pipeline execution Snakemake reports:</p>

<pre><code>[Fri Feb 21 12:40:37 2020]
Error in rule trim_galore_pe:
jobid: 7
output: fastq_trimmed/0053_P2017BB3S20R_S2_.1_val_1.fq.gz, fastq_trimmed/0053_P2017BB3S20R_S2_.2_val_2.fq.gz, qc/trim_galore/0053_P2017BB3S20R_S2_.1.fastq.gz_trimming_report.txt, qc/trim_galore/0053_P2017BB3S20R_S2_.2.fastq.gz_trimming_report..txt
conda-env: /home/nlv24077/projects/snaqs/snakemake_envs/1b94430b
cluster_jobid: 11904662.osios
Error executing rule trim_galore_pe on cluster (jobid: 7, external: 11904662.osios, jobscript: /home/nlv24077/temp/RNASeqPairedEnd/.snakemake/tmp.ooypuklc/snakejob.trim_galore_pe.7.sh).
For error details see the cluster log and the log files of the involved rule(s).
</code></pre>

<p>This is understandeble because the file fastq_trimmed/0053_P2017BB3S20R_S2_.1_val_1.fq.gz does not exist, it should be fastq_trimmed/0053_P2017BB3S20R_S2.1_val_1.fq.gz, somehow Snakemake added an extra underscore, which you can see in the execution output, under "wildcards" (0053_P2017BB3S20R_S2_).</p>

<pre><code>[Fri Feb 21 12:39:57 2020]
rule trim_galore_pe:
input: fastq_raw/0053_P2017BB3S20R_S2_R1_001.fastq.gz, fastq_raw/0053_P2017BB3S20R_S2_R2_001.fastq.gz
output: fastq_trimmed/0053_P2017BB3S20R_S2_.1_val_1.fq.gz, fastq_trimmed/0053_P2017BB3S20R_S2_.2_val_2.fq.gz, qc/trim_galore/0053_P2017BB3S20R_S2_.1.fastq.gz_trimming_report.txt, qc/trim_galore/0053_P2017BB3S20R_S2_.2.fastq.gz_trimming_report..txt
jobid: 7
wildcards: sample=0053_P2017BB3S20R_S2_
</code></pre>

<p>So how can I prevent Snakemake from changing the samples and read wildcard in seemingly arbitrary ways?</p>

<snakemake>
Prevent Snakemake from changing wildcards: sample for every rule
# Prevent Snakemake from changing wildcards: sample for every rule

(Asked by: username_0 on 2020-02-21)

EDIT: I posted this as a bug, perhaps wait for the result of that? github
All,
I am using Snakemake to build a pipeline. To define input and output names, I constructed them based on a filename base (from the samples table), a read indicator (like R1, R2) and the fastq file extension, like so:
samples = pd.read_csv('samples.tsv', sep='\t').set_index('samples', drop=False)
reads = ['R1', 'R2']
fastq_file_extension = _001.fastq.gz

rule fastqc:
    input:
        os.path.join(config['Directories']['fastq_raw_dir'], '{sample}{read}' + fastq_file_extension)
    output:
        html=os.path.join(config['Directories']['fastqc_dir'], '{sample}{read}_fastqc.html'),
        zip=os.path.join(config['Directories']['fastqc_dir'], '{sample}{read}_fastqc.zip')
    wrapper:
        "0.49.0/bio/fastqc"

However, during pipeline execution already I can see strange things happening, for this rule it reports:
[Fri Feb 21 12:39:57 2020]
rule fastqc:
input: fastq_raw/0053_P2017BB3S20R_S2_R1_001.fastq.gz
output: qc/fastqc/0053_P2017BB3S20R_S2_R1_fastqc.html, qc/fastqc/0053_P2017BB3S20R_S2_R1_fastqc.zip
jobid: 10
wildcards: sample=0053_P2017BB3S20R_S2_R, read=1

As you can see, Snakemake pulled the R part of the read to the sample and set the read to 1 (should be R1). But this still works because it produces output files as Snakemake expects.
But the next rule, trim_galore_pe:
rule trim_galore_pe:
    input:
        [os.path.join(config['Directories']['fastq_raw_dir'], '{sample}' + read + fastq_file_extension) for read in reads]
    output:
        fastq_read1 = os.path.join(config['Directories']['fastq_trimmed_dir'], '{sample}.1_val_1.fq.gz'),
        fastq_read2 = os.path.join(config['Directories']['fastq_trimmed_dir'], '{sample}.2_val_2.fq.gz'),
        report_read1 = os.path.join(config['Directories']['trim_galore_qc_dir'], '{sample}.1.fastq.gz_trimming_report.txt'),
        report_read2 = os.path.join(config['Directories']['trim_galore_qc_dir'], '{sample}.2.fastq.gz_trimming_report.txt')
    params:
        extra="--retain_unpaired"
    wrapper:
        "0.49.0/bio/trim_galore/pe"

This goes wrong, during pipeline execution Snakemake reports:
[Fri Feb 21 12:40:37 2020]
Error in rule trim_galore_pe:
jobid: 7
output: fastq_trimmed/0053_P2017BB3S20R_S2_.1_val_1.fq.gz, fastq_trimmed/0053_P2017BB3S20R_S2_.2_val_2.fq.gz, qc/trim_galore/0053_P2017BB3S20R_S2_.1.fastq.gz_trimming_report.txt, qc/trim_galore/0053_P2017BB3S20R_S2_.2.fastq.gz_trimming_report..txt
conda-env: /home/nlv24077/projects/snaqs/snakemake_envs/1b94430b
cluster_jobid: 11904662.osios
Error executing rule trim_galore_pe on cluster (jobid: 7, external: 11904662.osios, jobscript: /home/nlv24077/temp/RNASeqPairedEnd/.snakemake/tmp.ooypuklc/snakejob.trim_galore_pe.7.sh).
For error details see the cluster log and the log files of the involved rule(s).

This is understandeble because the file fastq_trimmed/0053_P2017BB3S20R_S2_.1_val_1.fq.gz does not exist, it should be fastq_trimmed/0053_P2017BB3S20R_S2.1_val_1.fq.gz, somehow Snakemake added an extra underscore, which you can see in the execution output, under "wildcards" (0053_P2017BB3S20R_S2_).
[Fri Feb 21 12:39:57 2020]
rule trim_galore_pe:
input: fastq_raw/0053_P2017BB3S20R_S2_R1_001.fastq.gz, fastq_raw/0053_P2017BB3S20R_S2_R2_001.fastq.gz
output: fastq_trimmed/0053_P2017BB3S20R_S2_.1_val_1.fq.gz, fastq_trimmed/0053_P2017BB3S20R_S2_.2_val_2.fq.gz, qc/trim_galore/0053_P2017BB3S20R_S2_.1.fastq.gz_trimming_report.txt, qc/trim_galore/0053_P2017BB3S20R_S2_.2.fastq.gz_trimming_report..txt
jobid: 7
wildcards: sample=0053_P2017BB3S20R_S2_

So how can I prevent Snakemake from changing the samples and read wildcard in seemingly arbitrary ways?

**Tags:** snakemake

---

## 2 answers

### Answer by username_1 on 2020-02-21. Score: 1

Placing two wildcards directly behind each other will make it impossible for snakemake to determine where one wildcard ends and the other begins. So it is simply taking the longest possible match for the first wildcard (sample) and then one character for the second (read). This will also cause the _ to be included in the wildcard value sample.
I'm assuming that this same pattern is used for the inputs of another rule to refer to the trim_galore_pe output. Hence why the _ is included in the wildcards there.
You might be able to fix this, simply by adding a _ in-between {sample} and {readgroup} in your first rule (and any other rules using this same pattern):
rule fastqc:
    input:
        os.path.join(config['Directories']['fastq_raw_dir'], '{sample}_{read}' + fastq_file_extension)
    output:
        html=os.path.join(config['Directories']['fastqc_dir'], '{sample}{read}_fastqc.html'),
        zip=os.path.join(config['Directories']['fastqc_dir'], '{sample}{read}_fastqc.zip')
    wrapper:
        "0.49.0/bio/fastqc"

> **Comments:**
> **username_0**: I did this but Snakemake conjures up a new _ out of nowhere: Building DAG of jobs... MissingInputException in line 2 of /rst1/temp//RNASeqPairedEnd/rules/trim_galore_pe.smk: Missing input files for rule trim_galore_pe: fastq_raw/0053_P2017BB3S19R_S1__R1_001.fastq.gz fastq_raw/0053_P2017BB3S19R_S1__R2_001.fastq.gz  
> **username_0**: But why is is impossible for Snakemake to separate them? They are not in {} for nothing right? I don't understand why Snakemake would want to change my wildcards from anything it gets from the samples table. What would be the use? It seems to me that this is just a bug? Or am I really fundamentally misunderstanding Snakemake?  
> **username_0**: Regarding my first comment, one might think that Snakemake now added and extra _ so I can remove the existing one perhaps? But no, it then removes both: Building DAG of jobs... MissingInputException in line 2 of /rst1/temp/RNASeqPairedEnd/rules/trim_galore_pe.smk: Missing input files for rule trim_galore_pe: fastq_raw/0053_P2017BB3S19R_S1R2_001.fastq.gz fastq_raw/0053_P2017BB3S19R_S1R1_001.fastq.gz  So it's either 0 or 2, not 1, as one would expect.  
> **username_0**: So I guess I am misunderstanding this completely, When I change the input of trim_galore_pe, it complains of missing files, but when I remove the input entirely it complains that the fastqc rule is missing input files, even though I did not touch that rule. Is there any way to more clearly see where errors originate exactly? I feel completely lost now.  
> **username_1**: @Freek Could you share the entire Snakefile? It's quite hard to see what's going on exactly without it.  
> **username_0**: Ok, I think I have found the reason, I was using code as well to create the samples_table from the same config and it was adding and removing the _ there as well. So this method does work, my apologies. It remains unhandy that I can't directly use wildcards together, But I could perhaps define a separator in the config file to solve this and keep flexibility..  

---

### Answer by username_2 on 2020-02-24. Score: 1

I think you want to prevent snakemake from interpreting your wildcard strings as regular expressions. For this, escape each wildcard value using wildcard_constraints. E.g., towards the top of your Snakefile add:
wildcard_constraints:
    read= '|'.join([re.escape(x) for x in reads]),
    sample= '|'.join([re.escape(x) for x in samples])

(Personally, I use the wildcard_constraints directive as above quite liberally to constrain every wildcard)

> **Comments:**
> **username_0**: Does this mess with how you can use rules.fastqc.output.html as the input for other rules? It gives me a lot of errors for a downstream step (multiqc).  
> **username_2**: As far as I know, this doesn't cause problems. If you don't show the error it's difficult to tell what's going on.  

---

