<p>I suggest using package <a href="https://docs.python.org/3/library/glob.html" rel="nofollow noreferrer">glob</a>, which supports unix-style pattern search. For instance finding all .py files in my repo:</p>
<pre><code>import glob 
all_python_files = glob.glob(r'c:\personal\coderepo\**\*.py', recursive=True)
</code></pre>
<p>Which outputs full paths:</p>
<blockquote>
<p>'c:\personal\coderepo\SimpleHTR\data\checkDirs.py', <br>
'c:\personal\coderepo\....</p>
</blockquote>
<p>I typically like to input the glob.glob to <a href="https://pandas.pydata.org/docs/" rel="nofollow noreferrer">Pandas</a> dataframes:</p>
<pre><code>all_files = glob.glob(f'{base_folder}/*')
all_files = pd.DataFrame(all_files, columns=['full_path'])
all_files['my_filename'] = all_files['full_path'].apply(lambda x: os.path.basename(x))

#do whatever filtering is required:
all_files = all_files[all_files['my_file_name'].str.contains('regex_here')]
</code></pre>
<p>Then perform what ever I need to do in loop:</p>
<pre><code>for idx, _ in all_files.iterrows():
    do_something(_['full_path'])
</code></pre>

CC BY-SA 4.0
2023-10-02T07:09:28.767Z
2023-10-02T07:09:28.767Z
pinegulf
<p>I am a beginner in snakemake and I'm trying to develop a working flow.</p>
<p>Imagine I have multiple folders and in each folder I have a tumour and normal sample files:</p>
<pre><code>SRC120/ 
      SRC120-T1.fastq
      SRC120-N1.fastq

SRC121/ 
      SRC121-T1.fastq
      SRC121-N1.fastq
</code></pre>
<p>and so on (in total I have more than 100 SRC folders).</p>
<p>I am trying to make a snakemake rule to make a symlink of these fastq files.</p>
<p>What I have written so far is:</p>
<pre class="lang-py prettyprint-override"><code>import os

SAMPLES=os.listdir(&quot;/analysis/Anna/snakemake-demo/data/samples_fastq-test1/&quot;)                                               

def getFastqFilesForTissu(wildcards):
    fastqs = list()
    # Beware no other files than fastqs should be there
    for s in os.listdir(&quot;/analysis/Anna/snakemake-demo/data/samples_fastq-test1/&quot;+wildcards.sample):
        fastqs.append(os.path.join(&quot;/analysis/Anna/snakemake-demo/data/samples_fastq-test1/&quot;,wildcards.sample,s))
    return fastqs


rule all:                                                                        
    input:                                                                       
        expand(&quot;results/{sample}/{sample}.fastq&quot;, sample=SAMPLES)                   


rule foo:
    input:
        fastq_files = getFastqFilesForTissu 
    output:
        &quot;results/{sample}/{sample}.fastq&quot;
    shell:
        'ln -s {input} {output}'
</code></pre>
<p>This works if I have one file in each folder and when the file name matches with the folder's name (i.e., when there is no T1 or N1 suffix).</p>
<p>Is there any way I could modify my rule to get it working for all samples in SRC directories? Or a better way to handle this issue?</p>

CC BY-SA 4.0
2023-09-15T07:46:55.483Z
What is the output you expect? What values do you want your `{input}` and `{output}` to have for a given example? This looks like a simple shell operation, do you really need to use snakemake?
terdon
CC BY-SA 4.0
2023-09-15T16:54:40.903Z
This is just a very small part of a big workflow. the output is symlinked fastq files in the results folder
Anna1364
CC BY-SA 4.0
2023-09-15T16:57:01.427Z
What symlinked files? What name should the symlink have and what should its target be? The more detail you give, the better the chances someone will be able to answer. Please [edit] your question and add the expected result so we know what you need.
terdon
CC BY-SA 4.0
2023-09-15T19:16:48.467Z
All of these information is in the code if you read through the snakemake code. There is no need for any edition.
Anna1364
<python><snakemake>
How can I develop a snakemake rule for all files in multiple directories?
# How can I develop a snakemake rule for all files in multiple directories?

(Asked by: username_0 on 2023-09-15)

I am a beginner in snakemake and I'm trying to develop a working flow.
Imagine I have multiple folders and in each folder I have a tumour and normal sample files:
SRC120/ 
      SRC120-T1.fastq
      SRC120-N1.fastq

SRC121/ 
      SRC121-T1.fastq
      SRC121-N1.fastq

and so on (in total I have more than 100 SRC folders).
I am trying to make a snakemake rule to make a symlink of these fastq files.
What I have written so far is:
import os

SAMPLES=os.listdir("/analysis/Anna/snakemake-demo/data/samples_fastq-test1/")                                               

def getFastqFilesForTissu(wildcards):
    fastqs = list()
    # Beware no other files than fastqs should be there
    for s in os.listdir("/analysis/Anna/snakemake-demo/data/samples_fastq-test1/"+wildcards.sample):
        fastqs.append(os.path.join("/analysis/Anna/snakemake-demo/data/samples_fastq-test1/",wildcards.sample,s))
    return fastqs


rule all:                                                                        
    input:                                                                       
        expand("results/{sample}/{sample}.fastq", sample=SAMPLES)                   


rule foo:
    input:
        fastq_files = getFastqFilesForTissu 
    output:
        "results/{sample}/{sample}.fastq"
    shell:
        'ln -s {input} {output}'

This works if I have one file in each folder and when the file name matches with the folder's name (i.e., when there is no T1 or N1 suffix).
Is there any way I could modify my rule to get it working for all samples in SRC directories? Or a better way to handle this issue?

**Tags:** python, snakemake
> **Comments:**
> **username_1**: What is the output you expect? What values do you want your `{input}` and `{output}` to have for a given example? This looks like a simple shell operation, do you really need to use snakemake?  
> **username_0**: This is just a very small part of a big workflow. the output is symlinked fastq files in the results folder  
> **username_1**: What symlinked files? What name should the symlink have and what should its target be? The more detail you give, the better the chances someone will be able to answer. Please [edit] your question and add the expected result so we know what you need.  
> **username_0**: All of these information is in the code if you read through the snakemake code. There is no need for any edition.  

---

## 1 answers

### Answer by username_2 on 2023-10-02. Score: 2

I suggest using package glob, which supports unix-style pattern search. For instance finding all .py files in my repo:
import glob 
all_python_files = glob.glob(r'c:\personal\coderepo\**\*.py', recursive=True)

Which outputs full paths:

'c:\personal\coderepo\SimpleHTR\data\checkDirs.py', 
'c:\personal\coderepo\....

I typically like to input the glob.glob to Pandas dataframes:
all_files = glob.glob(f'{base_folder}/*')
all_files = pd.DataFrame(all_files, columns=['full_path'])
all_files['my_filename'] = all_files['full_path'].apply(lambda x: os.path.basename(x))

#do whatever filtering is required:
all_files = all_files[all_files['my_file_name'].str.contains('regex_here')]

Then perform what ever I need to do in loop:
for idx, _ in all_files.iterrows():
    do_something(_['full_path'])


---

