<p>A very simple approach, but one which will work with the names you show, is to remove any occurrence of <code>00N</code> (where <code>N</code> is any number) from right before the <code>.gz</code>. To do so, edit your script to:</p>

<pre><code>for i in <span class="math-container">$(ls *.fastq*.gz | sed 's/00[0-9]\.gz/.gz/' | rev | cut -c 16- | rev | uniq); do 
  bowtie2 -q -p20 -x /run/media/punit/data1/BowtieIndex/hg38 \
    -1 $</span>{i}R1_001.fastq.gz \
    -2 <span class="math-container">${i}R2_001.fastq.gz | 
        samtools view -bS - &gt;$</span>{i%}.bam"
done
</code></pre>

CC BY-SA 4.0
2019-03-25T15:53:44.790Z
2019-03-25T15:53:44.790Z
terdon
<p>I have numerous gzip-compressed paired-end Fastq files with ChIP-seq data that I would like to align with bowtie2. I confirmed that bowtie2 takes <code>.gz</code> files, although I do see this warning when I run for two samples:</p>

<blockquote>
  <p>Warning: gzbuffer added in zlib v1.2.3.5. Unable to change buffer size
  from default of 8192.</p>
</blockquote>

<p>My files are named like so.</p>

<pre><code>10_S10_L002_R1_001.fastq.gz  12_S12_L002_R1_001.fastq.gz  2_S2_L001_R1_001.fastq.gz  4_S4_L001_R1_001.fastq-003.gz  6_S6_L001_R1_001.fastq-002.gz  8_S8_L002_R1_001.fastq.gz  
10_S10_L002_R2_001.fastq.gz  12_S12_L002_R2_001.fastq.gz  2_S2_L001_R2_001.fastq.gz  4_S4_L001_R2_001.fastq-002.gz  6_S6_L001_R2_001.fastq-003.gz  8_S8_L002_R2_001.fastq.gz  
11_S11_L002_R1_001.fastq.gz  1_S1_L001_R1_001.fastq.gz    3_S3_L001_R1_001.fastq.gz  5_S5_L001_R1_001.fastq-002.gz  7_S7_L002_R1_001.fastq.gz      9_S9_L002_R1_001.fastq.gz
11_S11_L002_R2_001.fastq.gz  1_S1_L001_R2_001.fastq.gz    3_S3_L001_R2_001.fastq.gz  5_S5_L001_R2_001.fastq-003.gz  7_S7_L002_R2_001.fastq.gz      9_S9_L002_R2_001.fastq.gz
</code></pre>

<p>The bash commands I tried are below. When I run this test command:</p>

<pre><code>for i in $(ls *.fastq*.gz | grep -v 6 | rev | cut -c 16- | rev | uniq); do 
    echo $i; 
done
</code></pre>

<p>...I get this output.</p>

<pre><code>12_S12_L002_R1_001.fastq.gz  12_S12_L002_R2_001.fastq.gz 
</code></pre>

<p>When I run this:</p>

<pre><code>for i in $(ls *.fastq*.gz | rev | cut -c 16- | rev | uniq); do 
    echo $i 
    bowtie2 -q -p20 -x /run/media/punit/data1/BowtieIndex/hg38 -1 ${i}R1_001.fastq.gz -2 ${i}R2_001.fastq.gz | samtools view -bS - &gt;${i%}.bam
done
</code></pre>

<p>I do see one BAM for my paired end files. But as I see the second set of files which are named:</p>

<pre><code>6_S6_L001_R1_001.fastq-002.gz  6_S6_L001_R2_001.fastq-003.gz
</code></pre>

<p>if I run the same script over them, I would get an error as the loop can't read the paired end file.</p>

<p>So the tedious way is I can make a separate folder for these sets of data with <code>fastq-002.gz</code> and <code>fastq-003.gz</code> and can run the script by tweaking the <code>cut</code> parameter.</p>

<p>But I would like to know what is the programmatic way of resolving the issue.</p>

CC BY-SA 4.0
2019-03-25T14:24:22.263Z
I strongly encourage you to use something like `snakeMake` rather than making a shell script. It's both much easier to ensure the proper files are paired and it scales to use in clusters and the cloud.
Devon Ryan
CC BY-SA 4.0
2019-03-25T15:06:23.757Z
Can't you just rename the files?
terdon
CC BY-SA 4.0
2019-03-25T15:13:48.730Z
@terdon that's how our collaborator gave us and i dont want to mess up with the file name yes i can if i dont find any other way..
kcm
CC BY-SA 4.0
2019-03-25T15:14:31.607Z
" strongly encourage you to use something like snakeMake rather than making a shell script. It's both much easier to ensure the proper files are paired and it scales to use in clusters and the cloud." i read a bit about it never tried it , i need to learn and give it a try
kcm
CC BY-SA 4.0
2019-03-26T18:31:33.363Z
It's good practice to treat files from collaborators as read-only, so not changing the filenames is a good idea. But if the files are inconsistently named, it's a very good idea to create consistently named symlinks to those files. Well organized filenames and directory structures will make it SO NICE AND EASY to automate batch data processing.
Daniel Standage
CC BY-SA 4.0
2019-03-27T04:37:36.497Z
yes i did that finally
kcm
<read-mapping><fastq><shell>
Batch alignment of inconsistently named Fastq files
# Batch alignment of inconsistently named Fastq files

(Asked by: username_0 on 2019-03-25)

I have numerous gzip-compressed paired-end Fastq files with ChIP-seq data that I would like to align with bowtie2. I confirmed that bowtie2 takes .gz files, although I do see this warning when I run for two samples:

Warning: gzbuffer added in zlib v1.2.3.5. Unable to change buffer size
  from default of 8192.

My files are named like so.
10_S10_L002_R1_001.fastq.gz  12_S12_L002_R1_001.fastq.gz  2_S2_L001_R1_001.fastq.gz  4_S4_L001_R1_001.fastq-003.gz  6_S6_L001_R1_001.fastq-002.gz  8_S8_L002_R1_001.fastq.gz  
10_S10_L002_R2_001.fastq.gz  12_S12_L002_R2_001.fastq.gz  2_S2_L001_R2_001.fastq.gz  4_S4_L001_R2_001.fastq-002.gz  6_S6_L001_R2_001.fastq-003.gz  8_S8_L002_R2_001.fastq.gz  
11_S11_L002_R1_001.fastq.gz  1_S1_L001_R1_001.fastq.gz    3_S3_L001_R1_001.fastq.gz  5_S5_L001_R1_001.fastq-002.gz  7_S7_L002_R1_001.fastq.gz      9_S9_L002_R1_001.fastq.gz
11_S11_L002_R2_001.fastq.gz  1_S1_L001_R2_001.fastq.gz    3_S3_L001_R2_001.fastq.gz  5_S5_L001_R2_001.fastq-003.gz  7_S7_L002_R2_001.fastq.gz      9_S9_L002_R2_001.fastq.gz

The bash commands I tried are below. When I run this test command:
for i in $(ls *.fastq*.gz | grep -v 6 | rev | cut -c 16- | rev | uniq); do 
    echo $i; 
done

...I get this output.
12_S12_L002_R1_001.fastq.gz  12_S12_L002_R2_001.fastq.gz 

When I run this:
for i in $(ls *.fastq*.gz | rev | cut -c 16- | rev | uniq); do 
    echo $i 
    bowtie2 -q -p20 -x /run/media/punit/data1/BowtieIndex/hg38 -1 ${i}R1_001.fastq.gz -2 ${i}R2_001.fastq.gz | samtools view -bS - >${i%}.bam
done

I do see one BAM for my paired end files. But as I see the second set of files which are named:
6_S6_L001_R1_001.fastq-002.gz  6_S6_L001_R2_001.fastq-003.gz

if I run the same script over them, I would get an error as the loop can't read the paired end file.
So the tedious way is I can make a separate folder for these sets of data with fastq-002.gz and fastq-003.gz and can run the script by tweaking the cut parameter.
But I would like to know what is the programmatic way of resolving the issue.

**Tags:** read-mapping, fastq, shell
> **Comments:**
> **username_1**: I strongly encourage you to use something like `snakeMake` rather than making a shell script. It's both much easier to ensure the proper files are paired and it scales to use in clusters and the cloud.  
> **username_2**: Can't you just rename the files?  
> **username_0**: @terdon that's how our collaborator gave us and i dont want to mess up with the file name yes i can if i dont find any other way..  
> **username_0**: " strongly encourage you to use something like snakeMake rather than making a shell script. It's both much easier to ensure the proper files are paired and it scales to use in clusters and the cloud." i read a bit about it never tried it , i need to learn and give it a try  
> **username_3**: It's good practice to treat files from collaborators as read-only, so not changing the filenames is a good idea. But if the files are inconsistently named, it's a very good idea to create consistently named symlinks to those files. Well organized filenames and directory structures will make it SO NICE AND EASY to automate batch data processing.  
> **username_0**: yes i did that finally  

---

## 1 answers

### Answer by username_2 on 2019-03-25. Score: 3 | Accepted answer

A very simple approach, but one which will work with the names you show, is to remove any occurrence of 00N (where N is any number) from right before the .gz. To do so, edit your script to:
for i in $(ls *.fastq*.gz | sed 's/00[0-9]\.gz/.gz/' | rev | cut -c 16- | rev | uniq); do 
  bowtie2 -q -p20 -x /run/media/punit/data1/BowtieIndex/hg38 \
    -1 ${i}R1_001.fastq.gz \
    -2 ${i}R2_001.fastq.gz | 
        samtools view -bS - >${i%}.bam"
done


---

