<p>Your only option here is to first split the fastq files into small chunks, process each and then merge them back together. Alternatively, find something else that runs faster.</p>

CC BY-SA 3.0
2017-05-19T14:03:08.770Z
2017-05-19T14:03:08.770Z
Devon Ryan
<p>Here is a bash script and GNU <code>parallel</code> along with some other tools to split-apply-combine the data. It may or may not work out of the box, but it's not far off and it should give the general idea of how one way of how to approach this.</p>



<pre><code>#!/bin/bash

# ====== USER PROVIDED PARAMETERS ========
N=20 # number of cores to use
ngsfilter="rawdata_scandinavia.ngsfilter"
unidentified="unidentified_samples.fastq"

R1="rawdata_scandinavia_R1.fastq"
R2="rawdata_scandinavia_R2.fastq"
# ==== END USER PROVIDED PARAMETERS =====

# fastqutils available @ http://ngsutils.org/modules/fastqutils
fastqutils split $R1 to_illumina_R1_ $N &amp;
fastqutils split $R2 to_illumina_R2_ $N &amp;
wait

R1=$( ls | grep -P "to_illumina_R1_" )
R2=$( ls | grep -P "to_illumina_R2_" )

# Do alignment of two reads.
for i in $( seq $N )
do
    partR1=$( ls | grep -P "^to_illumina_R1_\\.$i\\.fastq$" )
    partR2=$( ls | grep -P "^to_illumina_R2_\\.$i\\.fastq$" )
    illuminapairedend -r $partR1 $partR2 | tee to_ngsfilter_$i.fastq | obiannotate -S goodAli:'"Align" if score&gt;40.00 else "Bad"' | obisplit -t goodAli -p to_ngsfilter_$i. &amp;
done

# Remove intermediate aligned files.
echo $R1 | xargs rm -r
echo $R2 | xargs rm -r
ls | grep -P "\\.seq$|\\.err$" | xargs rm -r

input=$( ls | grep -P "to_ngsfilter_[0-9]+\\.Align\\.fastq" )

# For some reasons, files need to be "touched" in order to work. Perhaps
# it's not closing them...
touch $unidentified

touchedbyangel=$( ls | grep -P "to_ngsfilter_"  )
for i in $touchedbyangel
do
    touch $i
done

parallel -j$N --result filtered_{} ngsfilter -t $ngsfilter -u $unidentified {} ::: $input

# Remove intermediate results.
ls | grep -P "\\.seq$|\\.err$" | xargs rm -r

ls | grep -P "filtered_.*\\.fastq$" | xargs cat &gt; filtered_data.fastq

# remove intermediate results to conserve space
rm to_ngsfilter*

# Split reads by locus.
filtered=$( ls | grep -P "^filtered_"  )
parallel -j$N --result to_split_{} obisplit -p MS.PCR_ -t experiment {} ::: $filtered

ls | grep -P "\\.err$|\\.seq$" | xargs rm -r
ls | grep -P "^to_split_" | xargs rm -r

# Wasn't able to parallelize this, so it's just pushing tasks into the background.
obiuniq -m sample MS.PCR_UA_MxRout1_03.fastq &gt; MS.PCR_UA_MxRout1_03.uniq.fasta &amp;
obiuniq -m sample MS.PCR_UA_MxRout1_06.fastq &gt; MS.PCR_UA_MxRout1_06.uniq.fasta &amp;
obiuniq -m sample MS.PCR_UA_MxRout1_14.fastq &gt; MS.PCR_UA_MxRout1_14.uniq.fasta &amp;
obiuniq -m sample MS.PCR_UA_MxRout1_16.fastq &gt; MS.PCR_UA_MxRout1_16.uniq.fasta &amp;
obiuniq -m sample MS.PCR_UA_MxRout1_17.fastq &gt; MS.PCR_UA_MxRout1_17.uniq.fasta &amp;
obiuniq -m sample MS.PCR_UA_MxRout1_25.fastq &gt; MS.PCR_UA_MxRout1_25.uniq.fasta &amp;
obiuniq -m sample MS.PCR_UA_MxRout1_51.fastq &gt; MS.PCR_UA_MxRout1_51.uniq.fasta &amp;
obiuniq -m sample MS.PCR_UA_MxRout1_57.fastq &gt; MS.PCR_UA_MxRout1_57.uniq.fasta &amp;
obiuniq -m sample MS.PCR_UA_MxRout1_63.fastq &gt; MS.PCR_UA_MxRout1_63.uniq.fasta &amp;
obiuniq -m sample MS.PCR_UA_MxRout1_64.fastq &gt; MS.PCR_UA_MxRout1_64.uniq.fasta &amp;
obiuniq -m sample MS.PCR_UA_MxRout1_65.fastq &gt; MS.PCR_UA_MxRout1_65.uniq.fasta &amp;
obiuniq -m sample MS.PCR_UA_MxRout1_67.fastq &gt; MS.PCR_UA_MxRout1_67.uniq.fasta &amp;
obiuniq -m sample MS.PCR_UA_MxRout1_68.fastq &gt; MS.PCR_UA_MxRout1_68.uniq.fasta &amp;
obiuniq -m sample MS.PCR_UA_MxRout1_ZF.fastq &gt; MS.PCR_UA_MxRout1_ZF.uniq.fasta &amp;
wait

uniqfiles=$( ls | grep -P ".{2}+\\.uniq\\.fasta$" )

for i in $uniqfiles
do
    # this section removes the file suffix
    # https://stackoverflow.com/questions/125281/how-do-i-remove-the-file-suffix-and-path-portion-from-a-path-string-in-bash
    tab=${i%.fasta}
    tab=${tab##*/}
    obigrep -p 'count&gt;1' $i | obiannotate -k merged_sample -k count | obiannotate --length | obisort -r -k seq_length | obitab --no-definition --output-seq &gt; $tab.tab
done

ls | grep -P "\\.uniq\\.fasta" | xargs rm -r

echo "Done. I think."
</code></pre>

CC BY-SA 3.0
2017-06-02T09:35:00.113Z
2017-06-02T13:18:13.977Z
2017-06-02T13:18:13.977Z
Roman Luštrik
<p>I have an <a href="http://datadryad.org/bitstream/handle/10255/dryad.122664/obitools_script.txt?sequence=1" rel="nofollow noreferrer">obitools script</a> (de Barba et al. 2016) that I would like to run faster. How would you run it in parallel to cut down on time?</p>

<pre><code>illuminapairedend -r rawdata_scandinavia_R2.fastq rawdata_scandinavia_R1.fastq | tee rawdata_scandinavia.fastq | obiannotate -S goodAli:'"Alignement" if score&gt;40.00 else "Bad"' | obisplit -t goodAli -p rawdata_scandinavia.
touch rawdata_scandinavia.Bad.fastq
touch rawdata_scandinavia.Alignement.fastq
touch rawdata_scandinavia.fastq
ngsfilter -t rawdata_scandinavia.ngsfilter -u rawdata_scandinavia.unidentified.fastq rawdata_scandinavia.Alignement.fastq &gt; rawdata_scandinavia.filtered.fastq
obisplit -p MICROSAT.PCR_ -t experiment rawdata_scandinavia.filtered.fastq
</code></pre>

<p>Can someone describe what tools and steps one needs to take to run this job in parallel, given that the functions are not designed to be run in parallel out of the box?</p>

<p>De Barba, M., Miquel, C., Lobréaux, S., Quenette, P. Y., Swenson, J. E., &amp; Taberlet, P. (2016). High-throughput microsatellite genotyping in ecology: improved accuracy, efficiency, standardization and success with low-quantity and degraded DNA. Molecular Ecology Resources, 1–16. <a href="https://doi.org/10.1111/1755-0998.12594" rel="nofollow noreferrer">https://doi.org/10.1111/1755-0998.12594</a></p>

CC BY-SA 3.0
2017-06-01T22:12:24.330Z
Bioinformatics encompasses biology, statistics, and computer science. Parallel processing is a common need in bioinformatics, and can be a challenging task to implement depending on the particular programs that are used. The programs used here in the example script are specific to bioinformatics, and answers in stack overflow might be lower quality due to the inexperience of the community in working with bioinformatics programs.
gringer
CC BY-SA 3.0
2017-06-02T08:15:25.103Z
@gringer this question was downvoted and closed because it's a programming question. Fast forward a few weeks later, [they're suddenly fine](https://bioinformatics.stackexchange.com/questions/361/what-is-the-fastest-way-to-calculate-the-number-of-unknown-nucleotides-in-fasta).
Roman Luštrik
CC BY-SA 3.0
2017-06-02T08:25:47.657Z
Pure programming questions (as was the impression from reading only the question title for this) are better placed in SO. However, this question is about a specific type of parallel processing that is associated with a particular bioinformatic tool. I can imagine that this question might be closed in SO for offtopic reasons.
gringer
CC BY-SA 3.0
2017-06-02T08:30:56.183Z
@gringer I hope we can get one more vote. I've solved the problem in the mean time and would like to offer my fruits of labor to anybody who might come up with this kind of a problem in the future.
Roman Luštrik
CC BY-SA 3.0
2017-06-02T08:50:51.697Z
@RomanLuštrik the FASTA/FASTQ question has now been updated with more context.
gringer
CC BY-SA 3.0
2017-05-19T14:02:01.897Z
I think this is a more general programming question and so would be more suited for stack overflow
Chris_Rands
CC BY-SA 3.0
2017-05-19T14:44:49.407Z
Divide & conquer!
Kamil S Jaron
<ngs><linux>
Parallel processing of scripts that use obitools
# Parallel processing of scripts that use obitools

(Asked by: username_0 on 2017-05-19)

I have an obitools script (de Barba et al. 2016) that I would like to run faster. How would you run it in parallel to cut down on time?
illuminapairedend -r rawdata_scandinavia_R2.fastq rawdata_scandinavia_R1.fastq | tee rawdata_scandinavia.fastq | obiannotate -S goodAli:'"Alignement" if score>40.00 else "Bad"' | obisplit -t goodAli -p rawdata_scandinavia.
touch rawdata_scandinavia.Bad.fastq
touch rawdata_scandinavia.Alignement.fastq
touch rawdata_scandinavia.fastq
ngsfilter -t rawdata_scandinavia.ngsfilter -u rawdata_scandinavia.unidentified.fastq rawdata_scandinavia.Alignement.fastq > rawdata_scandinavia.filtered.fastq
obisplit -p MICROSAT.PCR_ -t experiment rawdata_scandinavia.filtered.fastq

Can someone describe what tools and steps one needs to take to run this job in parallel, given that the functions are not designed to be run in parallel out of the box?
De Barba, M., Miquel, C., Lobréaux, S., Quenette, P. Y., Swenson, J. E., & Taberlet, P. (2016). High-throughput microsatellite genotyping in ecology: improved accuracy, efficiency, standardization and success with low-quantity and degraded DNA. Molecular Ecology Resources, 1–16. https://doi.org/10.1111/1755-0998.12594

**Tags:** ngs, linux
> **Comments:**
> **username_2**: I think this is a more general programming question and so would be more suited for stack overflow  
> **username_3**: Divide & conquer!  
> **username_1**: Bioinformatics encompasses biology, statistics, and computer science. Parallel processing is a common need in bioinformatics, and can be a challenging task to implement depending on the particular programs that are used. The programs used here in the example script are specific to bioinformatics, and answers in stack overflow might be lower quality due to the inexperience of the community in working with bioinformatics programs.  
> **username_0**: @gringer this question was downvoted and closed because it's a programming question. Fast forward a few weeks later, [they're suddenly fine](https://bioinformatics.stackexchange.com/questions/361/what-is-the-fastest-way-to-calculate-the-number-of-unknown-nucleotides-in-fasta).  
> **username_1**: Pure programming questions (as was the impression from reading only the question title for this) are better placed in SO. However, this question is about a specific type of parallel processing that is associated with a particular bioinformatic tool. I can imagine that this question might be closed in SO for offtopic reasons.  
> **username_0**: @gringer I hope we can get one more vote. I've solved the problem in the mean time and would like to offer my fruits of labor to anybody who might come up with this kind of a problem in the future.  
> **username_1**: @RomanLuštrik the FASTA/FASTQ question has now been updated with more context.  

---

## 2 answers

### Answer by username_4 on 2017-05-19. Score: 3

Your only option here is to first split the fastq files into small chunks, process each and then merge them back together. Alternatively, find something else that runs faster.


---

### Answer by username_0 on 2017-06-02. Score: 3

Here is a bash script and GNU parallel along with some other tools to split-apply-combine the data. It may or may not work out of the box, but it's not far off and it should give the general idea of how one way of how to approach this.
#!/bin/bash

# ====== USER PROVIDED PARAMETERS ========
N=20 # number of cores to use
ngsfilter="rawdata_scandinavia.ngsfilter"
unidentified="unidentified_samples.fastq"

R1="rawdata_scandinavia_R1.fastq"
R2="rawdata_scandinavia_R2.fastq"
# ==== END USER PROVIDED PARAMETERS =====

# fastqutils available @ http://ngsutils.org/modules/fastqutils
fastqutils split $R1 to_illumina_R1_ $N &
fastqutils split $R2 to_illumina_R2_ $N &
wait

R1=$( ls | grep -P "to_illumina_R1_" )
R2=$( ls | grep -P "to_illumina_R2_" )

# Do alignment of two reads.
for i in $( seq $N )
do
    partR1=$( ls | grep -P "^to_illumina_R1_\\.$i\\.fastq$" )
    partR2=$( ls | grep -P "^to_illumina_R2_\\.$i\\.fastq$" )
    illuminapairedend -r $partR1 $partR2 | tee to_ngsfilter_$i.fastq | obiannotate -S goodAli:'"Align" if score>40.00 else "Bad"' | obisplit -t goodAli -p to_ngsfilter_$i. &
done

# Remove intermediate aligned files.
echo $R1 | xargs rm -r
echo $R2 | xargs rm -r
ls | grep -P "\\.seq$|\\.err$" | xargs rm -r

input=$( ls | grep -P "to_ngsfilter_[0-9]+\\.Align\\.fastq" )

# For some reasons, files need to be "touched" in order to work. Perhaps
# it's not closing them...
touch $unidentified

touchedbyangel=$( ls | grep -P "to_ngsfilter_"  )
for i in $touchedbyangel
do
    touch $i
done

parallel -j$N --result filtered_{} ngsfilter -t $ngsfilter -u $unidentified {} ::: $input

# Remove intermediate results.
ls | grep -P "\\.seq$|\\.err$" | xargs rm -r

ls | grep -P "filtered_.*\\.fastq$" | xargs cat > filtered_data.fastq

# remove intermediate results to conserve space
rm to_ngsfilter*

# Split reads by locus.
filtered=$( ls | grep -P "^filtered_"  )
parallel -j$N --result to_split_{} obisplit -p MS.PCR_ -t experiment {} ::: $filtered

ls | grep -P "\\.err$|\\.seq$" | xargs rm -r
ls | grep -P "^to_split_" | xargs rm -r

# Wasn't able to parallelize this, so it's just pushing tasks into the background.
obiuniq -m sample MS.PCR_UA_MxRout1_03.fastq > MS.PCR_UA_MxRout1_03.uniq.fasta &
obiuniq -m sample MS.PCR_UA_MxRout1_06.fastq > MS.PCR_UA_MxRout1_06.uniq.fasta &
obiuniq -m sample MS.PCR_UA_MxRout1_14.fastq > MS.PCR_UA_MxRout1_14.uniq.fasta &
obiuniq -m sample MS.PCR_UA_MxRout1_16.fastq > MS.PCR_UA_MxRout1_16.uniq.fasta &
obiuniq -m sample MS.PCR_UA_MxRout1_17.fastq > MS.PCR_UA_MxRout1_17.uniq.fasta &
obiuniq -m sample MS.PCR_UA_MxRout1_25.fastq > MS.PCR_UA_MxRout1_25.uniq.fasta &
obiuniq -m sample MS.PCR_UA_MxRout1_51.fastq > MS.PCR_UA_MxRout1_51.uniq.fasta &
obiuniq -m sample MS.PCR_UA_MxRout1_57.fastq > MS.PCR_UA_MxRout1_57.uniq.fasta &
obiuniq -m sample MS.PCR_UA_MxRout1_63.fastq > MS.PCR_UA_MxRout1_63.uniq.fasta &
obiuniq -m sample MS.PCR_UA_MxRout1_64.fastq > MS.PCR_UA_MxRout1_64.uniq.fasta &
obiuniq -m sample MS.PCR_UA_MxRout1_65.fastq > MS.PCR_UA_MxRout1_65.uniq.fasta &
obiuniq -m sample MS.PCR_UA_MxRout1_67.fastq > MS.PCR_UA_MxRout1_67.uniq.fasta &
obiuniq -m sample MS.PCR_UA_MxRout1_68.fastq > MS.PCR_UA_MxRout1_68.uniq.fasta &
obiuniq -m sample MS.PCR_UA_MxRout1_ZF.fastq > MS.PCR_UA_MxRout1_ZF.uniq.fasta &
wait

uniqfiles=$( ls | grep -P ".{2}+\\.uniq\\.fasta$" )

for i in $uniqfiles
do
    # this section removes the file suffix
    # https://stackoverflow.com/questions/125281/how-do-i-remove-the-file-suffix-and-path-portion-from-a-path-string-in-bash
    tab=${i%.fasta}
    tab=${tab##*/}
    obigrep -p 'count>1' $i | obiannotate -k merged_sample -k count | obiannotate --length | obisort -r -k seq_length | obitab --no-definition --output-seq > $tab.tab
done

ls | grep -P "\\.uniq\\.fasta" | xargs rm -r

echo "Done. I think."


---

