<p>Something like this might do it:</p>

<pre><code>#!/bin/bash

# make an array to hold your list of samples/base filenames
samples=( S1 S2 S3 S4 S5 )

# make a loop to go through the sample IDs
for sample in "<span class="math-container">${samples[@]}"
do
  :
  # do something with each sample/file, e.g. make a directory for it
  mkdir "$</span>{sample}"_home

  # do something else with the sample, e.g. unzip the fastq.gz file
  gunzip "<span class="math-container">${sample}".fq.gz &gt; "$</span>{sample}"_home/"${sample}".fq
done
</code></pre>

<p>One disadvantage with this is that it will process the files one after the other.  If you have a cluster, you might prefer to run them all at the same time to make the whole thing quicker.  If you have a SLURM cluster, you can use job array to do something similar.</p>

<p>If you have your list of samples in a text file, you could try replacing the samples array command in the script above with something like this:</p>

<pre><code>samples=( `cat samples_list.txt` )
</code></pre>

<p>This will only work if the samples are listed on a single line, space-separated. You may need to process your samples file further to get them in this format, or use the bash readarray command.</p>

CC BY-SA 4.0
2018-12-11T13:14:57.370Z
I have a list of sample ids store in a txt so is it possible that it make a list using that file?
Safina A.R
CC BY-SA 4.0
2018-12-11T13:26:29.720Z
@SafinaA.R please remember to actually show your data. Edit your question and show us this text file. That way, we will know the format and can use it to give you example answers.
terdon
CC BY-SA 4.0
2018-12-11T16:55:07.897Z
This solution requires saving the sample names directly into the bash script. This sounds a little difficult since OP said she has a file with 4000 sample names.
conchoecia
CC BY-SA 4.0
2018-12-12T04:16:27.970Z
hi @terdon
The data is same as shown by fin swimmer in the above example. It is a text file tab separated having two col. Sample Ids and Library Ids
Safina A.R
CC BY-SA 4.0
2019-01-04T16:30:54.430Z
I haven't tried this, but I added a comment to my script above about how to get the sample IDs out of a text file.
Jay Moore
CC BY-SA 4.0
2018-12-11T12:29:47.003Z
2019-01-04T16:39:30.057Z
2019-01-04T16:39:30.057Z
Jay Moore
<p>if you have a repeating workflow, I strongly recommend to have a look at workflow management systems like <a href="https://snakemake.readthedocs.io/en/stable/tutorial/basics.html" rel="noreferrer">snakemake</a>. I've also wrote a little tutorial on <a href="https://www.biostars.org/p/335903/" rel="noreferrer">biostars</a> about this topic, which might be useful for you.</p>

<p>fin swimmer</p>

CC BY-SA 4.0
2018-12-11T11:04:34.193Z
That's a bit complicated can there be simple for loop trick?
Safina A.R
CC BY-SA 4.0
2018-12-11T10:51:24.073Z
2018-12-11T10:51:24.073Z
finswimmer
<p>You can create a script to pass arguments to as follows:</p>

<pre><code>#! /bin/bash

bwa mem -M -R "@RG\tID:<span class="math-container">${1}\tLB:$</span>{2}\tPL:ILLUMINA\tPM:HiSeq2000\tSM:<span class="math-container">${1}" ../ref/M._tuberculosis_H37Rv_2015-11-13.fasta $</span>{1}_* &gt; ${1}.sam
</code></pre>

<p>Now if you run <code>bash script.sh sample lib</code> then <code>sample</code> and <code>lib</code> will be  the script as arguments <code><span class="math-container">${1}</code> and <code>$</span>{2}</code> respectively. Then all you need is a loop to pass arguments to the script:</p>

<pre><code>for sample in sample1 sample2 sample3
   do
   for lib in lib1 lib2 lib3
     do
     bash script.sh <span class="math-container">$sample $</span>lib
     done
   done
</code></pre>

<p>You can use <code>nohup</code> to run each sample in the background and proceed with next one. This will in essence, submit each command as a separate job to run on a different core. These will print to separate log files sample.out instead of to the terminal. It will also continue to run, even if you disconnect from SSH or close the terminal.</p>

<pre><code>for sample in sample1 sample2 sample3
   do
   for lib in lib1 lib2 lib3
     do
     nohup bash script.sh <span class="math-container">$sample $</span>lib &gt; ${sample}.out &amp;
     done
   done
</code></pre>

<p>You should only do this for single-threaded processes. For multi-threaded processes or memory-intensive jobs, this will demand a lot from the machine. Even if this isn’t a shared server, it will perform poorly if you run out of memory and have to “swap” memory by writing to disk. So it’s best to submit them one-by-one (above) or use a cluster with a job scheduler. </p>

CC BY-SA 4.0
2019-01-05T01:57:30.633Z
2019-01-05T01:57:30.633Z
Tom Kelly ケリー・トム
<p>Another solution for repeating one command with different parameters is to use <code>parallel</code>.</p>

<p>Create a tab-delimited file <code>samples.txt</code>, which contains the sample names and the information for your read groups, e.g.:</p>

<pre><code>Sample1 Lib1
Sample2 Lib2
Sample3 Lib3
</code></pre>

<p>Then you can use <code>parallel</code> like this:</p>

<pre><code>parallel --colsep "\t" 'bwa mem -M -R "@RG\tID:{1}\tLB:{2}\tPL:ILLUMINA\tPM:HiSeq2000\tSM:{1}" ../ref/M._tuberculosis_H37Rv_2015-11-13.fasta {1}_* &gt; {1}.sam' :::: samples.txt
</code></pre>

<p>For each line in <code>samples.txt</code> a new process is started, replacing <code>{1}</code> and <code>{2}</code> with the values in column 1 and 2 of this line.</p>

<p>fin swimmer</p>

CC BY-SA 4.0
2018-12-11T13:15:43.897Z
wow that's look much easier i will try and update you.
Safina A.R
CC BY-SA 4.0
2018-12-12T07:57:33.827Z
swimmer thank you for this simplest code. I want to know whether this parallel produce log? because i cant see anything on the screen as the bwa command produce screen messages while running. Secondly, if i don't want to run these commands parallel than what should i used instead of parallel as while running it parallel my computer got stuck.
Safina A.R
CC BY-SA 4.0
2018-12-12T08:57:51.387Z
By default `parallel` prints to the screen after a job has finished. Doing this makes sure that you can e.g. concatenate outputs of parallel jobs without mixing them. You can use the option `--ungroup` to get the output immediately. But use this with care! Without any parameter the number of parallel jobs is the same as the number of CPU cores. You can set it to an explicit number using `-j`, e.g. `-j 1`
finswimmer
CC BY-SA 4.0
2018-12-12T12:35:26.113Z
Ok. Thank you. It is running for this command i will modified it accordingly for other steps and if i got stuck i will update here.
Safina A.R
CC BY-SA 4.0
2018-12-11T13:08:13.020Z
2018-12-11T13:08:13.020Z
finswimmer
<p>Hi im doing Variant calling on fastq files for which i have 4000 fq files and the variant calling are done in different 9 steps. Each step generate different files that are output to next steps.</p>
<p>So can anybody help me with this to do it in a loop it pick up names from previous steps for all files. Step 1 command is here:</p>
<pre><code>bwa mem -M -R '@RG\tID:ERR040140\tLB:2517707\tPL:ILLUMINA\tPM:HiSeq2000\tSM:ERR040140' ../ref/M._tuberculosis_H37Rv_2015-11-13.fasta ERR040140_* &gt; ERR040140.sam
</code></pre>
<p>So i made an excel file to break the command in columns so that i can change the file names keeping remaining command same just like this:</p>
<p><a href="https://i.sstatic.net/uj6vl.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/uj6vl.png" alt="command_column_wise" /></a></p>
<p>So changes are occurring 5th, 7th and 9th column. which are:</p>
<pre><code>tID:ERR040140
tLB:2517707
tSM:ERR040140
ERR040140_*
ERR040140.sam
</code></pre>
<p>Can anyone help me with loop or something?</p>
<p>Thank you!</p>

CC BY-SA 4.0
2018-12-11T11:13:29.083Z
Please [edit] your question and replace the image of the excel file with a copy/paste of the same file exported to tsv (save as 'tab separated text file'). That way we can actually copy it and use it to test our examples.
terdon
CC BY-SA 4.0
2018-12-11T18:38:08.243Z
Please remember to confirm an answer once you've received one.
Stephopolis
CC BY-SA 4.0
2018-12-12T04:23:15.190Z
I have already given the excel file command above. And im unable to paste it here the page is not letting me paste it that is why i quoted 1 example and than added a picture with all other repeating commands with different file names.
Safina A.R
<variant-calling><linux>
How to run same command on multiple files?
# How to run same command on multiple files?

(Asked by: username_0 on 2018-12-11)

Hi im doing Variant calling on fastq files for which i have 4000 fq files and the variant calling are done in different 9 steps. Each step generate different files that are output to next steps.
So can anybody help me with this to do it in a loop it pick up names from previous steps for all files. Step 1 command is here:
bwa mem -M -R '@RG\tID:ERR040140\tLB:2517707\tPL:ILLUMINA\tPM:HiSeq2000\tSM:ERR040140' ../ref/M._tuberculosis_H37Rv_2015-11-13.fasta ERR040140_* > ERR040140.sam

So i made an excel file to break the command in columns so that i can change the file names keeping remaining command same just like this:

So changes are occurring 5th, 7th and 9th column. which are:
tID:ERR040140
tLB:2517707
tSM:ERR040140
ERR040140_*
ERR040140.sam

Can anyone help me with loop or something?
Thank you!

**Tags:** variant-calling, linux
> **Comments:**
> **username_1**: Please [edit] your question and replace the image of the excel file with a copy/paste of the same file exported to tsv (save as 'tab separated text file'). That way we can actually copy it and use it to test our examples.  
> **username_2**: Please remember to confirm an answer once you've received one.  
> **username_0**: I have already given the excel file command above. And im unable to paste it here the page is not letting me paste it that is why i quoted 1 example and than added a picture with all other repeating commands with different file names.  

---

## 4 answers

### Answer by username_5 on 2018-12-11. Score: 10 | Accepted answer

Another solution for repeating one command with different parameters is to use parallel.
Create a tab-delimited file samples.txt, which contains the sample names and the information for your read groups, e.g.:
Sample1 Lib1
Sample2 Lib2
Sample3 Lib3

Then you can use parallel like this:
parallel --colsep "\t" 'bwa mem -M -R "@RG\tID:{1}\tLB:{2}\tPL:ILLUMINA\tPM:HiSeq2000\tSM:{1}" ../ref/M._tuberculosis_H37Rv_2015-11-13.fasta {1}_* > {1}.sam' :::: samples.txt

For each line in samples.txt a new process is started, replacing {1} and {2} with the values in column 1 and 2 of this line.
fin swimmer

> **Comments:**
> **username_0**: wow that's look much easier i will try and update you.  
> **username_0**: swimmer thank you for this simplest code. I want to know whether this parallel produce log? because i cant see anything on the screen as the bwa command produce screen messages while running. Secondly, if i don't want to run these commands parallel than what should i used instead of parallel as while running it parallel my computer got stuck.  
> **username_5**: By default `parallel` prints to the screen after a job has finished. Doing this makes sure that you can e.g. concatenate outputs of parallel jobs without mixing them. You can use the option `--ungroup` to get the output immediately. But use this with care! Without any parameter the number of parallel jobs is the same as the number of CPU cores. You can set it to an explicit number using `-j`, e.g. `-j 1`  
> **username_0**: Ok. Thank you. It is running for this command i will modified it accordingly for other steps and if i got stuck i will update here.  

---

### Answer by username_3 on 2018-12-11. Score: 8

Something like this might do it:
#!/bin/bash

# make an array to hold your list of samples/base filenames
samples=( S1 S2 S3 S4 S5 )

# make a loop to go through the sample IDs
for sample in "${samples[@]}"
do
  :
  # do something with each sample/file, e.g. make a directory for it
  mkdir "${sample}"_home

  # do something else with the sample, e.g. unzip the fastq.gz file
  gunzip "${sample}".fq.gz > "${sample}"_home/"${sample}".fq
done

One disadvantage with this is that it will process the files one after the other.  If you have a cluster, you might prefer to run them all at the same time to make the whole thing quicker.  If you have a SLURM cluster, you can use job array to do something similar.
If you have your list of samples in a text file, you could try replacing the samples array command in the script above with something like this:
samples=( `cat samples_list.txt` )

This will only work if the samples are listed on a single line, space-separated. You may need to process your samples file further to get them in this format, or use the bash readarray command.

> **Comments:**
> **username_0**: I have a list of sample ids store in a txt so is it possible that it make a list using that file?  
> **username_1**: @SafinaA.R please remember to actually show your data. Edit your question and show us this text file. That way, we will know the format and can use it to give you example answers.  
> **username_4**: This solution requires saving the sample names directly into the bash script. This sounds a little difficult since OP said she has a file with 4000 sample names.  
> **username_0**: hi @terdon The data is same as shown by fin swimmer in the above example. It is a text file tab separated having two col. Sample Ids and Library Ids  
> **username_3**: I haven't tried this, but I added a comment to my script above about how to get the sample IDs out of a text file.  

---

### Answer by username_5 on 2018-12-11. Score: 5

if you have a repeating workflow, I strongly recommend to have a look at workflow management systems like [snakemake](https://snakemake.readthedocs.io/en/stable/tutorial/basics.html). I've also wrote a little tutorial on [biostars](https://www.biostars.org/p/335903/) about this topic, which might be useful for you.

fin swimmer

> **Comments:**
> **username_0**: That's a bit complicated can there be simple for loop trick?  

---

### Answer by username_6 on 2019-01-05. Score: 0

You can create a script to pass arguments to as follows:
#! /bin/bash

bwa mem -M -R "@RG\tID:${1}\tLB:${2}\tPL:ILLUMINA\tPM:HiSeq2000\tSM:${1}" ../ref/M._tuberculosis_H37Rv_2015-11-13.fasta ${1}_* > ${1}.sam

Now if you run bash script.sh sample lib then sample and lib will be  the script as arguments ${1} and ${2} respectively. Then all you need is a loop to pass arguments to the script:
for sample in sample1 sample2 sample3
   do
   for lib in lib1 lib2 lib3
     do
     bash script.sh $sample $lib
     done
   done

You can use nohup to run each sample in the background and proceed with next one. This will in essence, submit each command as a separate job to run on a different core. These will print to separate log files sample.out instead of to the terminal. It will also continue to run, even if you disconnect from SSH or close the terminal.
for sample in sample1 sample2 sample3
   do
   for lib in lib1 lib2 lib3
     do
     nohup bash script.sh $sample $lib > ${sample}.out &
     done
   done

You should only do this for single-threaded processes. For multi-threaded processes or memory-intensive jobs, this will demand a lot from the machine. Even if this isn’t a shared server, it will perform poorly if you run out of memory and have to “swap” memory by writing to disk. So it’s best to submit them one-by-one (above) or use a cluster with a job scheduler.


---

