<p>Your samples end with <code>_1.fastq</code> and <code>_2.fastq</code>, so you need to change this:</p>

<pre><code>-1 $TMPDIR/${base}.1.fastq -2 $TMPDIR/${base}.2.fastq
</code></pre>

<p>to this:</p>

<pre><code>-1 $TMPDIR/${base}_1.fastq -2 $TMPDIR/${base}_2.fastq
</code></pre>

CC BY-SA 3.0
2018-04-23T11:51:14.670Z
Thanq. ya this was the only error in my code.
stack_learner
CC BY-SA 3.0
2018-04-30T06:27:14.307Z
Hi, The above code uses one after another tar.gz file. It is taking huge time. How should I make the code work for all the tar.gz files at a time?
stack_learner
CC BY-SA 3.0
2018-04-30T06:36:21.350Z
Untar the files and use a workflow manager like snakemake. BTW, make sure you have sufficient resources to process many samples in parallel.
Devon Ryan
CC BY-SA 3.0
2018-04-30T06:43:09.847Z
But as soon as Untar done for one file it should be the input for hisat2. Yes, I'm using 20 cpu cores, 1 week time and tmp 500G, memory usage 4G.
stack_learner
CC BY-SA 3.0
2018-04-30T06:44:05.930Z
Then put the untar command into something like Snakemake too.
Devon Ryan
CC BY-SA 3.0
2018-04-30T07:40:15.860Z
Never used a snakemake till now. May I know how I can give that?
stack_learner
CC BY-SA 3.0
2018-04-30T07:40:51.380Z
Please google such things and post a new question if you have further questions.
Devon Ryan
CC BY-SA 3.0
2018-04-23T09:31:54.723Z
2018-04-23T09:31:54.723Z
Devon Ryan
<p>This should work for you:</p>

<pre><code>for i in *.tar.gz
do
    dir2="/destination/directory"
    tar xvzf $i -C $TMPDIR
    files=($TMPDIR/*)
    module load HISAT2/2.0.4-goolf-1.7.20; module load SAMtools/1.3.1-goolf-1.7.20; hisat2 -p 8 --dta --rna-strandness RF -x /genome_index -1 $files[0] -2 $files[1]| samtools view -Sb - &gt; ${dir2}/$i.bam
done
</code></pre>

<p>(If this is the sort of thing you'll be doing often, it would be worth investing the time to learn a more robust approach such as Snakemake or Nextflow).</p>

<p>Edit:</p>

<p>If you want to run on all files at the same time, the easiest way is <code>find</code> / <code>xargs</code></p>

<p>Change you script to this:</p>

<pre><code>for i in $@
do
    dir2="/destination/directory"
    tar xvzf $i -C $TMPDIR
    files=($TMPDIR/*)
    module load HISAT2/2.0.4-goolf-1.7.20; module load SAMtools/1.3.1-goolf-1.7.20; hisat2 -p 8 --dta --rna-strandness RF -x /genome_index -1 $files[0] -2 $files[1]| samtools view -Sb - &gt; ${dir2}/$i.bam
done
</code></pre>

<p>Then run it like this:</p>

<pre><code>find . -d 1 -name '*.tar.gz' | xargs -n 1 bash NAMEOFSCRIPT
</code></pre>

<p>This works for SGE. I'm not sure about the details of SLURM. You might have to figure out how to run each job in the background or use <code>gnu parallel</code> or something. As <code>@Ian Sudbury</code> says, pipeline software can take care of all these details for you.</p>

CC BY-SA 3.0
2018-04-23T09:49:18.773Z
Definately +1 for putting all this in some sort of pipeline. Many of these  will now handle automated distribution of many files to your SLURM cluster. Job tracking. Restarts after errors etc.
Ian Sudbery
CC BY-SA 3.0
2018-04-30T06:26:52.177Z
@heathobrien Hi, The above code uses one after another tar.gz file. It is taking huge time. How should I make the code work for all the tar.gz files at a time?
stack_learner
CC BY-SA 3.0
2018-04-30T07:12:27.417Z
see updated answer
heathobrien
CC BY-SA 3.0
2018-04-30T11:48:32.263Z
Even I haven't worked on SLURM before. Do you think if I give #SBATCH --array=1-6%3 will work? lets say I have 6 tar.gz files so 3 jobs at a time.
stack_learner
CC BY-SA 3.0
2018-04-23T09:33:31.583Z
2018-04-30T07:12:16.343Z
2018-04-30T07:12:16.343Z
heathobrien
<p>Original tar.gz files look like below:</p>

<pre><code>UNCID_2668056.27d4c881-e906-437f-921d-467eb107798a.150108_UNC15-SN850_0402_AC5FTHACXX_2_GTTTCG.tar.gz
UNCID_2206882.58e94be5-2da2-4a96-944a-060f20216572.150108_UNC15-SN850_0402_AC5FTHACXX_3_CTTGTA.tar.gz
UNCID_2205121.2b3d596a-68ca-4c60-bb16-8c0cbd8fb190.140828_UNC11-SN627_0378_AC5FAWACXX_4_TAGCTT.tar.gz
UNCID_2209082.76a625e9-22b3-42b8-8813-27c133d0b248.110608_UNC11-SN627_0103_BD0CVVABXX_7.tar.gz
</code></pre>

<p>Before extracting the tar.gz files, I changed their names based on names.text file as below and using the command:</p>

<pre><code>xargs -n2 mv -i &lt;&lt;&lt;"$(sed '1d; s/$/.tar.gz/' names.txt)"

Tar.gz_filenames       Sample_Names
UNCID_2668056.27d4c881-e906-437f-921d-467eb107798a.150108_UNC15-SN850_0402_AC5FTHACXX_2_GTTTCG.tar.gz                Sample2
UNCID_2206882.58e94be5-2da2-4a96-944a-060f20216572.150108_UNC15-SN850_0402_AC5FTHACXX_3_CTTGTA.tar.gz                Sample1
UNCID_2205121.2b3d596a-68ca-4c60-bb16-8c0cbd8fb190.140828_UNC11-SN627_0378_AC5FAWACXX_4_TAGCTT.tar.gz                Sample3
UNCID_2209082.76a625e9-22b3-42b8-8813-27c133d0b248.110608_UNC11-SN627_0103_BD0CVVABXX_7.tar.gz                       Sample4
</code></pre>

<p>After changing the names, looks like below:</p>

<pre><code>Sample1.tar.gz
Sample2.tar.gz
Sample3.tar.gz
Sample4.tar.gz
</code></pre>

<p>Using the tar.gz files after changing names I used the below code:</p>

<pre><code>for i in /doc/*.tar.gz
do
    tar xvzf $i -C $TMPDIR
    for sample in $TMPDIR/*1.fastq
    do
        dir2="/destination/directory"
        base=$(basename $sample ".1.fastq")
        module load HISAT2/2.0.4-goolf-1.7.20; module load SAMtools/1.3.1-goolf-1.7.20; hisat2 -p 8 --dta --rna-strandness RF -x /genome_index -1 $TMPDIR/${base}.1.fastq -2 $TMPDIR/${base}.2.fastq | samtools view -Sb - &gt; ${dir2}/${base}.bam
    done
done
</code></pre>

<p>When I run the above code, the output is like below:</p>

<pre><code>150108_UNC15-SN850_0402_AC5FTHACXX_GTTTCG_L002_1.fastq
150108_UNC15-SN850_0402_AC5FTHACXX_GTTTCG_L002_2.fastq
150108_UNC15-SN850_0402_AC5FTHACXX_CTTGTA_L003_1.fastq
150108_UNC15-SN850_0402_AC5FTHACXX_CTTGTA_L003_2.fastq
140828_UNC11-SN627_0378_AC5FAWACXX_TAGCTT_L004_2.fastq
140828_UNC11-SN627_0378_AC5FAWACXX_TAGCTT_L004_1.fastq
110608_UNC11-SN627_0103_BD0CVVABXX.7_1.fastq
110608_UNC11-SN627_0103_BD0CVVABXX.7_2.fastq
Warning: Could not open read file "/scratch/username/slurm-job.36320423/140828_UNC11-SN627_0378_AC5FAWACXX_TAGCTT_L004_1.fastq.1.fastq" for reading; skipping...
Error: No input read files were valid
(ERR): hisat2-align exited with value 1
Warning: Could not open read file "/scratch/username/slurm-job.36320423/150108_UNC15-SN850_0402_AC5FTHACXX_CTTGTA_L003_1.fastq.1.fastq" for reading; skipping...
Error: No input read files were valid
(ERR): hisat2-align exited with value 1
Warning: Could not open read file "/scratch/username/slurm-job.36320423/150108_UNC15-SN850_0402_AC5FTHACXX_GTTTCG_L002_1.fastq.1.fastq" for reading; skipping...
Error: No input read files were valid
(ERR): hisat2-align exited with value 1
</code></pre>

<p>I dont mins whatever the name of fastq's in <code>$TMPDIR</code> but from <code>$TMPDIR</code> hisat2 should use those fastq's and give output bam files in /destination like below:</p>

<pre><code>Sample1.bam
Sample2.bam
Sample3.bam
Sample4.bam
</code></pre>

<p>I would like to run a script doing tar.gz files -> fastq -> bam (everything in loop)</p>

<rna-seq><file-formats><fastq><linux>
Script to run everything in a loop for extracting tar.gz files into fastq and to bam with alignment?
# Script to run everything in a loop for extracting tar.gz files into fastq and to bam with alignment?

(Asked by: username_0 on 2018-04-23)

Original tar.gz files look like below:
UNCID_2668056.27d4c881-e906-437f-921d-467eb107798a.150108_UNC15-SN850_0402_AC5FTHACXX_2_GTTTCG.tar.gz
UNCID_2206882.58e94be5-2da2-4a96-944a-060f20216572.150108_UNC15-SN850_0402_AC5FTHACXX_3_CTTGTA.tar.gz
UNCID_2205121.2b3d596a-68ca-4c60-bb16-8c0cbd8fb190.140828_UNC11-SN627_0378_AC5FAWACXX_4_TAGCTT.tar.gz
UNCID_2209082.76a625e9-22b3-42b8-8813-27c133d0b248.110608_UNC11-SN627_0103_BD0CVVABXX_7.tar.gz

Before extracting the tar.gz files, I changed their names based on names.text file as below and using the command:
xargs -n2 mv -i <<<"$(sed '1d; s/$/.tar.gz/' names.txt)"

Tar.gz_filenames       Sample_Names
UNCID_2668056.27d4c881-e906-437f-921d-467eb107798a.150108_UNC15-SN850_0402_AC5FTHACXX_2_GTTTCG.tar.gz                Sample2
UNCID_2206882.58e94be5-2da2-4a96-944a-060f20216572.150108_UNC15-SN850_0402_AC5FTHACXX_3_CTTGTA.tar.gz                Sample1
UNCID_2205121.2b3d596a-68ca-4c60-bb16-8c0cbd8fb190.140828_UNC11-SN627_0378_AC5FAWACXX_4_TAGCTT.tar.gz                Sample3
UNCID_2209082.76a625e9-22b3-42b8-8813-27c133d0b248.110608_UNC11-SN627_0103_BD0CVVABXX_7.tar.gz                       Sample4

After changing the names, looks like below:
Sample1.tar.gz
Sample2.tar.gz
Sample3.tar.gz
Sample4.tar.gz

Using the tar.gz files after changing names I used the below code:
for i in /doc/*.tar.gz
do
    tar xvzf $i -C $TMPDIR
    for sample in $TMPDIR/*1.fastq
    do
        dir2="/destination/directory"
        base=$(basename $sample ".1.fastq")
        module load HISAT2/2.0.4-goolf-1.7.20; module load SAMtools/1.3.1-goolf-1.7.20; hisat2 -p 8 --dta --rna-strandness RF -x /genome_index -1 $TMPDIR/${base}.1.fastq -2 $TMPDIR/${base}.2.fastq | samtools view -Sb - > ${dir2}/${base}.bam
    done
done

When I run the above code, the output is like below:
150108_UNC15-SN850_0402_AC5FTHACXX_GTTTCG_L002_1.fastq
150108_UNC15-SN850_0402_AC5FTHACXX_GTTTCG_L002_2.fastq
150108_UNC15-SN850_0402_AC5FTHACXX_CTTGTA_L003_1.fastq
150108_UNC15-SN850_0402_AC5FTHACXX_CTTGTA_L003_2.fastq
140828_UNC11-SN627_0378_AC5FAWACXX_TAGCTT_L004_2.fastq
140828_UNC11-SN627_0378_AC5FAWACXX_TAGCTT_L004_1.fastq
110608_UNC11-SN627_0103_BD0CVVABXX.7_1.fastq
110608_UNC11-SN627_0103_BD0CVVABXX.7_2.fastq
Warning: Could not open read file "/scratch/username/slurm-job.36320423/140828_UNC11-SN627_0378_AC5FAWACXX_TAGCTT_L004_1.fastq.1.fastq" for reading; skipping...
Error: No input read files were valid
(ERR): hisat2-align exited with value 1
Warning: Could not open read file "/scratch/username/slurm-job.36320423/150108_UNC15-SN850_0402_AC5FTHACXX_CTTGTA_L003_1.fastq.1.fastq" for reading; skipping...
Error: No input read files were valid
(ERR): hisat2-align exited with value 1
Warning: Could not open read file "/scratch/username/slurm-job.36320423/150108_UNC15-SN850_0402_AC5FTHACXX_GTTTCG_L002_1.fastq.1.fastq" for reading; skipping...
Error: No input read files were valid
(ERR): hisat2-align exited with value 1

I dont mins whatever the name of fastq's in $TMPDIR but from $TMPDIR hisat2 should use those fastq's and give output bam files in /destination like below:
Sample1.bam
Sample2.bam
Sample3.bam
Sample4.bam

I would like to run a script doing tar.gz files -> fastq -> bam (everything in loop)

**Tags:** rna-seq, file-formats, fastq, linux

---

## 2 answers

### Answer by username_2 on 2018-04-23. Score: 7 | Accepted answer

This should work for you:
for i in *.tar.gz
do
    dir2="/destination/directory"
    tar xvzf $i -C $TMPDIR
    files=($TMPDIR/*)
    module load HISAT2/2.0.4-goolf-1.7.20; module load SAMtools/1.3.1-goolf-1.7.20; hisat2 -p 8 --dta --rna-strandness RF -x /genome_index -1 $files[0] -2 $files[1]| samtools view -Sb - > ${dir2}/$i.bam
done

(If this is the sort of thing you'll be doing often, it would be worth investing the time to learn a more robust approach such as Snakemake or Nextflow).
Edit:
If you want to run on all files at the same time, the easiest way is find / xargs
Change you script to this:
for i in $@
do
    dir2="/destination/directory"
    tar xvzf $i -C $TMPDIR
    files=($TMPDIR/*)
    module load HISAT2/2.0.4-goolf-1.7.20; module load SAMtools/1.3.1-goolf-1.7.20; hisat2 -p 8 --dta --rna-strandness RF -x /genome_index -1 $files[0] -2 $files[1]| samtools view -Sb - > ${dir2}/$i.bam
done

Then run it like this:
find . -d 1 -name '*.tar.gz' | xargs -n 1 bash NAMEOFSCRIPT

This works for SGE. I'm not sure about the details of SLURM. You might have to figure out how to run each job in the background or use gnu parallel or something. As @Ian Sudbury says, pipeline software can take care of all these details for you.

> **Comments:**
> **username_3**: Definately +1 for putting all this in some sort of pipeline. Many of these  will now handle automated distribution of many files to your SLURM cluster. Job tracking. Restarts after errors etc.  
> **username_0**: @heathobrien Hi, The above code uses one after another tar.gz file. It is taking huge time. How should I make the code work for all the tar.gz files at a time?  
> **username_2**: see updated answer  
> **username_0**: Even I haven't worked on SLURM before. Do you think if I give #SBATCH --array=1-6%3 will work? lets say I have 6 tar.gz files so 3 jobs at a time.  

---

### Answer by username_1 on 2018-04-23. Score: 1

Your samples end with _1.fastq and _2.fastq, so you need to change this:
-1 $TMPDIR/${base}.1.fastq -2 $TMPDIR/${base}.2.fastq

to this:
-1 $TMPDIR/${base}_1.fastq -2 $TMPDIR/${base}_2.fastq

> **Comments:**
> **username_0**: Thanq. ya this was the only error in my code.  
> **username_0**: Hi, The above code uses one after another tar.gz file. It is taking huge time. How should I make the code work for all the tar.gz files at a time?  
> **username_1**: Untar the files and use a workflow manager like snakemake. BTW, make sure you have sufficient resources to process many samples in parallel.  
> **username_0**: But as soon as Untar done for one file it should be the input for hisat2. Yes, I'm using 20 cpu cores, 1 week time and tmp 500G, memory usage 4G.  
> **username_1**: Then put the untar command into something like Snakemake too.  
> **username_0**: Never used a snakemake till now. May I know how I can give that?  
> **username_1**: Please google such things and post a new question if you have further questions.  

---

