<p>I see two probable sources of error. The general format of the bwa command you are trying to run is</p>
<pre><code>bwa mem genome.fa read1.fastq.gz read2.fastq.gz 
</code></pre>
<p>However, you are running (with some options removed and paths truncated for brevity):</p>
<pre><code>bwa mem *_contigs.fasta /groups/[...]/SRR* -r *_1.[...].fastq.gz *_2.[..].fastq.gz 
</code></pre>
<p>Now, I don't know what the <a href="https://en.wikipedia.org/wiki/Glob_(programming)" rel="nofollow noreferrer">globs</a> <code>*_contigs.fasta</code>, <code>/groups/[...]/SRR*</code>, <code>*_1.[...].fastq.gz</code> or <code>*_2.[..].fastq.gz</code> would expand to but I doubt the result of expanding so many globs will be just one genome file and two fastq files. In addition, you seem to be already running this once for every file ending in <code>_contigs.fasta</code> since you have it in the loop:</p>
<pre><code>for file in *_contigs.fasta; do ...; done
</code></pre>
<p>Assuming you actually have <em>one</em> file wth all of your contigs and this file has been indexed and that's what you want to use as a genome, then you would want</p>
<pre><code>bwa mem /path/to/contig/contigs.fasta read1.fastq.gz read2.fastq.gz
</code></pre>
<p>I don't quite understand what you have from your question though so I can't be more specific. Next, presumably you have various sequencing run outputs, so multiple pairs of fastq files. Those are the ones your want to iterate over, not the contigs. So you would want a loop like this (I removed the unnecessary <code>samtools view</code> step, since <code>samtools sort</code> can handle the sam to bam conversion, and the also unnecessary <code>-r</code> option; if you <em>know</em> you need the <code>-r</code>, you can put it back):</p>
<pre class="lang-sh prettyprint-override"><code>outBamsDir=/groups/lilianasalvador/noah_100/bwa/bwa_mem_output/
for file in *_1.fastq.00.0_0.cor.fastq.gz; do  
    sample_name=${file%_1*}
    file2=${file/_1/_2}  
    mkdir -p &quot;$outBamsDir&quot;/&quot;$sample_name&quot;
    bwa mem \ 
      -t 14 \
      /path/to/contig/contigs.fasta \
      &quot;$file&quot; &quot;$file2&quot; | 
         samtools sort -@14 \
          -o &quot;$outBamsDir/$sample_name/$sample_name&quot;.bam
done
</code></pre>
<p>Or, if you really have multiple contig files and want to treat each of them as the reference genome to align against, so presumably you only have 2 fastq files, then the loop would be:</p>
<pre class="lang-sh prettyprint-override"><code>outBamsDir=/groups/lilianasalvador/noah_100/bwa/bwa_mem_output/
for file in *_contigs.fasta; do  
  sample_name=${file%_contigs.fasta}  
  mkdir -p &quot;$outBamsDir&quot;/&quot;$sample_name&quot;
  bwa mem -t 14 /whatever/you/want/to/use/as/genome.fa \
   &quot;$file&quot; | 
   samtools sort -@14 -o &quot;$outBamsDir/$sample_name&quot;.bam 
done
</code></pre>
<p>That, of course, assumes single-end sequencing with one file per sample.</p>

CC BY-SA 4.0
2024-07-01T09:41:56.113Z
Good points. AFAIK neither `bwa`  nor `bwa-mem2` can take more than 2 FASTQ files, guess the paired or non-paired mode and align them. Besides, using a single computing node to align multiple FASTQ pairs sequentially is not optimal save for a single node "cluster".
darked89
CC BY-SA 4.0
2024-07-01T21:21:02.040Z
I have a contigs.fasta file for each sequence as well as two .cor.fastq.gz files (reverse and forward) for each sequence. In /groups/lilianasalvador/noah_100/bwa/bwa_output I have 100 files - each corresponding to a sequence (i.e. SRR8599992, SRR8599993...) and within each of those is the contigs.fasta and the .cor.fastq.gz files for the sequence. That's why I was saying *_contigs.fasta /groups/lilianasalvador/noah_100/bwa/bwa_output/SRR* to try to explain that the contigs.fasta files are within the corresponding directory that starts with SRR. Idk how to correctly write the path
Lucy Kelly
CC BY-SA 4.0
2024-07-01T21:22:10.890Z
Do I have too many files? I was told I should have a contigs.fasta file for every sequence, which is why they're separated in directories
Lucy Kelly
CC BY-SA 4.0
2024-07-02T14:09:12.510Z
You might or you.might not, but we cannot know since you never showed us the files or even explained what you are really trying to do. However if you found a solution, please post that as an answer and not as part of the question.
terdon
CC BY-SA 4.0
2024-06-29T12:31:22.137Z
2024-07-01T09:49:19.093Z
2024-07-01T09:49:19.093Z
terdon
<p>I note that you make a mistake in your command lines,it seems that you want to map reads to the ref , convert the sam to the bam file and sort the bam file in one command, actually, you need to get the unsorted bam file first and sort the bam file with samtools singly. Because <code>samtools sort</code> will sort the whole file,but if you use the pipe as input of the 'samtools sort', the result is not a whole file from the <code>samtools view -Sb</code>,it just is a part of the resulty,so the <code>samtools sort</code> can't sort it , and the result may be stored in the memory , when many parts of results into the memory,it will be killed as OOM. You can try to map and convert sam to bam first and do the sorting singly.</p>
<p>I find that it contains <code>/groups/lilianasalvador/noah_100/bwa/bwa_output/SRR</code> between the reference sequence and the <code>-r</code> , it seems don't need this parameter , you can confirm the manual of the bwa mem</p>

CC BY-SA 4.0
2024-06-29T11:33:03.727Z
This isn't true. Samtools sort is perfectly capable of sorting from a pipe. Try it, just run `samtools view -h foo.bam | samtools sort -o bar.bam`. And it can do this with humongous files, it doesn't store in memory, it creates temporary files while sorting precisely to avoid the problem you mention.
terdon
CC BY-SA 4.0
2024-06-30T05:53:13.467Z
Sorry about that,actually i have tried many times in my machine, but that's not hpc,it just has 64GB memory, and I indeed got that problem,when i use the bwa mem | samtools view | samtools sort my computer will be down, i will can't connect it by ssh, and after long time ,it will recover,but that is uncontrollable ,and i hava find out some oom events during the analysis when i can connect it again. I have resolved it by the way i hava said.But there is an other question in his commandline which has been mentioned in the second paragraph in my answer last post.It seems that you mention it too.
kkmonster
CC BY-SA 4.0
2024-06-30T14:59:07.893Z
That can be caused by various things, but I can assure you, I have been running `bwa mem | samtools sort` in a professional, production pipeline for almost a decade now. And also did so on my laptop, before posting, just to double check. I also confirmed in the [samtools manual\(https://www.htslib.org/doc/samtools-sort.html) that it creates temp files when memory is not enough. As for the last paragraph, I'm sorry but I don't understand it at all. Yes, the `-r` isn't needed, but it isn't causing the problem either.
terdon
CC BY-SA 4.0
2024-07-01T01:28:28.813Z
Thanks to reply, actually i have noted the directory which he should not use between the reference and the reads file too. But the words may be unclear.  On the other hand i major in computer science ,when i get the  problem i have mentioned and take account for the behaviour of my comuputer and the syslog, i think i could be the reason what i have said , and by split the command line i resolve it. So it makes me believe the reason more.  The version of the samtools and htslib is 1.20. I will try to do reproduction experiments to located the ture reason. Thank you for reminding me.
kkmonster
CC BY-SA 4.0
2024-06-29T08:45:02.570Z
2024-06-29T08:45:02.570Z
kkmonster
<p>I am trying to write a script that will run this line from <a href="https://github.com/kmceres/Mbovis_pangenome/blob/main/Parameter_file.md" rel="nofollow noreferrer">https://github.com/kmceres/Mbovis_pangenome/blob/main/Parameter_file.md</a>:</p>
<pre><code>bwa mem -t 14 /path/to/contig/contigs.fasta -r contig_1P.fq.gz contig_2P.fq.gz | 
   samtools view -Sb | 
      samtools sort -@14 -o /path/to/contig/contig.sorted.bam
</code></pre>
<p>All of the files are in directories under their sequence number within a <code>bwa_output</code> (from when I ran bwa index) directory (e.g. <code>/groups/lilianasalvador/noah_100/bwa/bwa_output/SRR8599992</code>)</p>
<p>Here is my script:</p>
<pre><code>#!/bin/bash     

#SBATCH --job-name=bwa_mem     
#SBATCH --account=lilianasalvador   
#SBATCH --partition=standard   
#SBATCH --nodes=1   
#SBATCH --ntasks=1   
#SBATCH --mem-per-cpu=16gb   
#SBATCH --time=99:00:00   

module load bwa/0.7.17    
module load samtools/1.19.2 

for file in *_contigs.fasta; do  
    sample_name=<span class="math-container">${file%_contigs.fasta}  
mkdir -p /groups/lilianasalvador/noah_100/bwa/bwa_mem_output/$</span>sample_name   

bwa mem -t 14 *_contigs.fasta /groups/lilianasalvador/noah_100/bwa/bwa_output/SRR* -r *_1.fastq.00.0_0.cor.fastq.gz *_2.fastq.00.0_0.cor.fastq.gz | samtools view -Sb | samtools sort -@14 -o /groups/lilianasalvador/noah_100/bwa/bwa_mem_output 

done
</code></pre>
<p>I'm getting this error:</p>
<pre><code>Usage: bwa mem [options] &lt;idxbase&gt; &lt;in1.fq&gt; [in2.fq] 
...
[main_samview] fail to read the header from &quot;-&quot;. 
[W::hts_set_opt] Cannot change block size for this format 
samtools sort: failed to read header from &quot;-&quot; 
</code></pre>
<p>I am relatively new to all of this and would appreciate some help editing my script</p>
<p>update 7/1: I tried running this</p>
<pre><code>module load bwa/0.7.17    
module load samtools/1.19.2 

outBamsDir=/groups/lilianasalvador/noah_100/bwa/bwa_mem_output/ 

for file in *_1.fastq.00.0_0.cor.fastq.gz; do   
    sample_name=<span class="math-container">${file%_1*} 
file2=$</span>{file/_1/_2}   
    mkdir -p &quot;<span class="math-container">$outBamsDir"/"$</span>sample_name&quot; 
    bwa mem \  
      -t 14 \ 
/groups/lilianasalvador/noah_100/bwa/bwa_output/SRR8599992/SRR8599992_contigs.fasta \ 
      &quot;<span class="math-container">$file" "$</span>file2&quot; |  
         samtools sort -@14 \ 
          -o &quot;<span class="math-container">$outBamsDir/$</span>sample_name/$sample_name&quot;.bam 
done
</code></pre>
<p>and I got this error</p>
<pre><code>/var/spool/slurm/d/job10075741/slurm_script: line 40: -t: command not found 
/var/spool/slurm/d/job10075741/slurm_script: line 42: /groups/lilianasalvador/noah_100/bwa/bwa_output/SRR8599992/SRR8599992_contigs.fasta: Permission denied 
/var/spool/slurm/d/job10075741/slurm_script: line 44: SRR8599992_1.fastq.00.0_0.cor.fastq.gz: command not found 
[E::hts_open_format] Failed to open file &quot; &quot; : No such file or directory 
samtools sort: can't open &quot; &quot;: No such file or directory 
/var/spool/slurm/d/job10075741/slurm_script: line 48: -o: command not found 
</code></pre>
<p>I also tried this:</p>
<pre><code>module load bwa/0.7.17    
module load samtools/1.19.2 

for file in *_contigs.fasta; do  
    sample_name=<span class="math-container">${file%_contigs.fasta}  
mkdir -p /groups/lilianasalvador/noah_100/bwa/bwa_mem_output/$</span>sample_name   

    bwa mem -t 14 /groups/lilianasalvador/noah_100/bwa/bwa_output/SRR8599992/SRR8599992_contigs.fasta -r SRR8599992_1.fastq.00.0_0.cor.fastq.gz SRR8599992_2.fastq.00.1_0.cor.fastq.gz | samtools view -Sb | samtools sort -@14 -o /groups/lilianasalvador/noah_100/bwa/bwa_mem_output/SRR8599992 

done
</code></pre>
<p>and got this error</p>
<pre><code>[M::bwa_idx_load_from_disk] read 0 ALT contigs 
[M::process] read 648167 sequences (95550856 bp)... 
[M::mem_process_seqs] Processed 648167 reads in 28.660 CPU sec, 28.709 real sec 
[main] Version: 0.7.17-r1188 
[main] CMD: bwa mem -t 14 -r SRR8599992_1.fastq.00.0_0.cor.fastq.gz /groups/lilianasalvador/noah_100/bwa/bwa_output/SRR8599992/SRR8599992_contigs.fasta SRR8599992_2.fastq.00.1_0.cor.fastq.gz 
[main] Real time: 42.903 sec; CPU: 30.392 sec 
[bam_sort_core] merging from 0 files and 14 in-memory blocks... 
[E::hts_open_format] Failed to open file &quot;/groups/lilianasalvador/noah_100/bwa/bwa_mem_output&quot; : Is a directory 
samtools sort: failed to create &quot;/groups/lilianasalvador/noah_100/bwa/bwa_mem_output&quot;: Is a directory 
</code></pre>
<p>This is what worked:</p>
<pre><code>module load bwa/0.7.17    
module load samtools/1.19.2 

bwa mem -t 14 /groups/lilianasalvador/noah_100/bwa/bwa_output/SRR8599992/SRR8599992_contigs.fasta -r SRR8599992_1.fastq.00.0_0.cor.fastq.gz SRR8599992_2.fastq.00.1_0.cor.fastq.gz | 
   samtools view -Sb | 
    samtools sort -@14 -o /groups/lilianasalvador/noah_100/bwa/bwa_mem_output/SRR8599992/SRR8599992.contig.sorted.bam

</code></pre>

CC BY-SA 4.0
2024-06-29T11:52:05.753Z
What exactly are you trying to do? Do you have _one_ set of fastq files and you want to align that against multiple genomes (contigs) or do you have many sets of fastq files, the output of multiple sequencing runs (or the same run but multiple sequenced samples) and you want to align each set of fastqs against the contigs?
terdon
CC BY-SA 4.0
2024-06-29T11:54:10.010Z
Please [edit] your question and explain what your files are, and then also show us the output of `ls *_contigs.fasta`, `ls /groups/lilianasalvador/noah_100/bwa/bwa_output/SRR*`, `ls *_1.fastq.00.0_0.cor.fastq.gz` and `ls  *_2.fastq.00.0_0.cor.fastq.gz`. I tried to answer but without knowing what the actual data and end objective are, I can only make guesses.
terdon
CC BY-SA 4.0
2024-07-02T10:16:51.220Z
Please read https://bioinformatics.stackexchange.com/help/formatting and format your code as code.
terdon
<samtools><bwa><hpc>
Running bwa mem and samtools in one script on a linux cluster
# Running bwa mem and samtools in one script on a linux cluster

(Asked by: username_0 on 2024-06-28)

I am trying to write a script that will run this line from https://github.com/kmceres/Mbovis_pangenome/blob/main/Parameter_file.md:
bwa mem -t 14 /path/to/contig/contigs.fasta -r contig_1P.fq.gz contig_2P.fq.gz | 
   samtools view -Sb | 
      samtools sort -@14 -o /path/to/contig/contig.sorted.bam

All of the files are in directories under their sequence number within a bwa_output (from when I ran bwa index) directory (e.g. /groups/lilianasalvador/noah_100/bwa/bwa_output/SRR8599992)
Here is my script:
#!/bin/bash     

#SBATCH --job-name=bwa_mem     
#SBATCH --account=lilianasalvador   
#SBATCH --partition=standard   
#SBATCH --nodes=1   
#SBATCH --ntasks=1   
#SBATCH --mem-per-cpu=16gb   
#SBATCH --time=99:00:00   

module load bwa/0.7.17    
module load samtools/1.19.2 

for file in *_contigs.fasta; do  
    sample_name=${file%_contigs.fasta}  
mkdir -p /groups/lilianasalvador/noah_100/bwa/bwa_mem_output/$sample_name   

bwa mem -t 14 *_contigs.fasta /groups/lilianasalvador/noah_100/bwa/bwa_output/SRR* -r *_1.fastq.00.0_0.cor.fastq.gz *_2.fastq.00.0_0.cor.fastq.gz | samtools view -Sb | samtools sort -@14 -o /groups/lilianasalvador/noah_100/bwa/bwa_mem_output 

done

I'm getting this error:
Usage: bwa mem [options]   [in2.fq] 
...
[main_samview] fail to read the header from "-". 
[W::hts_set_opt] Cannot change block size for this format 
samtools sort: failed to read header from "-" 

I am relatively new to all of this and would appreciate some help editing my script
update 7/1: I tried running this
module load bwa/0.7.17    
module load samtools/1.19.2 

outBamsDir=/groups/lilianasalvador/noah_100/bwa/bwa_mem_output/ 

for file in *_1.fastq.00.0_0.cor.fastq.gz; do   
    sample_name=${file%_1*} 
file2=${file/_1/_2}   
    mkdir -p "$outBamsDir"/"$sample_name" 
    bwa mem \  
      -t 14 \ 
/groups/lilianasalvador/noah_100/bwa/bwa_output/SRR8599992/SRR8599992_contigs.fasta \ 
      "$file" "$file2" |  
         samtools sort -@14 \ 
          -o "$outBamsDir/$sample_name/$sample_name".bam 
done

and I got this error
/var/spool/slurm/d/job10075741/slurm_script: line 40: -t: command not found 
/var/spool/slurm/d/job10075741/slurm_script: line 42: /groups/lilianasalvador/noah_100/bwa/bwa_output/SRR8599992/SRR8599992_contigs.fasta: Permission denied 
/var/spool/slurm/d/job10075741/slurm_script: line 44: SRR8599992_1.fastq.00.0_0.cor.fastq.gz: command not found 
[E::hts_open_format] Failed to open file " " : No such file or directory 
samtools sort: can't open " ": No such file or directory 
/var/spool/slurm/d/job10075741/slurm_script: line 48: -o: command not found 

I also tried this:
module load bwa/0.7.17    
module load samtools/1.19.2 

for file in *_contigs.fasta; do  
    sample_name=${file%_contigs.fasta}  
mkdir -p /groups/lilianasalvador/noah_100/bwa/bwa_mem_output/$sample_name   

    bwa mem -t 14 /groups/lilianasalvador/noah_100/bwa/bwa_output/SRR8599992/SRR8599992_contigs.fasta -r SRR8599992_1.fastq.00.0_0.cor.fastq.gz SRR8599992_2.fastq.00.1_0.cor.fastq.gz | samtools view -Sb | samtools sort -@14 -o /groups/lilianasalvador/noah_100/bwa/bwa_mem_output/SRR8599992 

done

and got this error
[M::bwa_idx_load_from_disk] read 0 ALT contigs 
[M::process] read 648167 sequences (95550856 bp)... 
[M::mem_process_seqs] Processed 648167 reads in 28.660 CPU sec, 28.709 real sec 
[main] Version: 0.7.17-r1188 
[main] CMD: bwa mem -t 14 -r SRR8599992_1.fastq.00.0_0.cor.fastq.gz /groups/lilianasalvador/noah_100/bwa/bwa_output/SRR8599992/SRR8599992_contigs.fasta SRR8599992_2.fastq.00.1_0.cor.fastq.gz 
[main] Real time: 42.903 sec; CPU: 30.392 sec 
[bam_sort_core] merging from 0 files and 14 in-memory blocks... 
[E::hts_open_format] Failed to open file "/groups/lilianasalvador/noah_100/bwa/bwa_mem_output" : Is a directory 
samtools sort: failed to create "/groups/lilianasalvador/noah_100/bwa/bwa_mem_output": Is a directory 

This is what worked:
module load bwa/0.7.17    
module load samtools/1.19.2 

bwa mem -t 14 /groups/lilianasalvador/noah_100/bwa/bwa_output/SRR8599992/SRR8599992_contigs.fasta -r SRR8599992_1.fastq.00.0_0.cor.fastq.gz SRR8599992_2.fastq.00.1_0.cor.fastq.gz | 
   samtools view -Sb | 
    samtools sort -@14 -o /groups/lilianasalvador/noah_100/bwa/bwa_mem_output/SRR8599992/SRR8599992.contig.sorted.bam

**Tags:** samtools, bwa, hpc
> **Comments:**
> **username_1**: What exactly are you trying to do? Do you have _one_ set of fastq files and you want to align that against multiple genomes (contigs) or do you have many sets of fastq files, the output of multiple sequencing runs (or the same run but multiple sequenced samples) and you want to align each set of fastqs against the contigs?  
> **username_1**: Please [edit] your question and explain what your files are, and then also show us the output of `ls *_contigs.fasta`, `ls /groups/lilianasalvador/noah_100/bwa/bwa_output/SRR*`, `ls *_1.fastq.00.0_0.cor.fastq.gz` and `ls  *_2.fastq.00.0_0.cor.fastq.gz`. I tried to answer but without knowing what the actual data and end objective are, I can only make guesses.  
> **username_1**: Please read https://bioinformatics.stackexchange.com/help/formatting and format your code as code.  

---

## 2 answers

### Answer by username_1 on 2024-06-29. Score: 3

I see two probable sources of error. The general format of the bwa command you are trying to run is
bwa mem genome.fa read1.fastq.gz read2.fastq.gz 

However, you are running (with some options removed and paths truncated for brevity):
bwa mem *_contigs.fasta /groups/[...]/SRR* -r *_1.[...].fastq.gz *_2.[..].fastq.gz 

Now, I don't know what the globs *_contigs.fasta, /groups/[...]/SRR*, *_1.[...].fastq.gz or *_2.[..].fastq.gz would expand to but I doubt the result of expanding so many globs will be just one genome file and two fastq files. In addition, you seem to be already running this once for every file ending in _contigs.fasta since you have it in the loop:
for file in *_contigs.fasta; do ...; done

Assuming you actually have one file wth all of your contigs and this file has been indexed and that's what you want to use as a genome, then you would want
bwa mem /path/to/contig/contigs.fasta read1.fastq.gz read2.fastq.gz

I don't quite understand what you have from your question though so I can't be more specific. Next, presumably you have various sequencing run outputs, so multiple pairs of fastq files. Those are the ones your want to iterate over, not the contigs. So you would want a loop like this (I removed the unnecessary samtools view step, since samtools sort can handle the sam to bam conversion, and the also unnecessary -r option; if you know you need the -r, you can put it back):
outBamsDir=/groups/lilianasalvador/noah_100/bwa/bwa_mem_output/
for file in *_1.fastq.00.0_0.cor.fastq.gz; do  
    sample_name=${file%_1*}
    file2=${file/_1/_2}  
    mkdir -p "$outBamsDir"/"$sample_name"
    bwa mem \ 
      -t 14 \
      /path/to/contig/contigs.fasta \
      "$file" "$file2" | 
         samtools sort -@14 \
          -o "$outBamsDir/$sample_name/$sample_name".bam
done

Or, if you really have multiple contig files and want to treat each of them as the reference genome to align against, so presumably you only have 2 fastq files, then the loop would be:
outBamsDir=/groups/lilianasalvador/noah_100/bwa/bwa_mem_output/
for file in *_contigs.fasta; do  
  sample_name=${file%_contigs.fasta}  
  mkdir -p "$outBamsDir"/"$sample_name"
  bwa mem -t 14 /whatever/you/want/to/use/as/genome.fa \
   "$file" | 
   samtools sort -@14 -o "$outBamsDir/$sample_name".bam 
done

That, of course, assumes single-end sequencing with one file per sample.

> **Comments:**
> **username_2**: Good points. AFAIK neither `bwa`  nor `bwa-mem2` can take more than 2 FASTQ files, guess the paired or non-paired mode and align them. Besides, using a single computing node to align multiple FASTQ pairs sequentially is not optimal save for a single node "cluster".  
> **username_0**: I have a contigs.fasta file for each sequence as well as two .cor.fastq.gz files (reverse and forward) for each sequence. In /groups/lilianasalvador/noah_100/bwa/bwa_output I have 100 files - each corresponding to a sequence (i.e. SRR8599992, SRR8599993...) and within each of those is the contigs.fasta and the .cor.fastq.gz files for the sequence. That's why I was saying *_contigs.fasta /groups/lilianasalvador/noah_100/bwa/bwa_output/SRR* to try to explain that the contigs.fasta files are within the corresponding directory that starts with SRR. Idk how to correctly write the path  
> **username_0**: Do I have too many files? I was told I should have a contigs.fasta file for every sequence, which is why they're separated in directories  
> **username_1**: You might or you.might not, but we cannot know since you never showed us the files or even explained what you are really trying to do. However if you found a solution, please post that as an answer and not as part of the question.  

---

### Answer by username_3 on 2024-06-29. Score: -2

I note that you make a mistake in your command lines,it seems that you want to map reads to the ref , convert the sam to the bam file and sort the bam file in one command, actually, you need to get the unsorted bam file first and sort the bam file with samtools singly. Because `samtools sort` will sort the whole file,but if you use the pipe as input of the 'samtools sort', the result is not a whole file from the `samtools view -Sb`,it just is a part of the resulty,so the `samtools sort` can't sort it , and the result may be stored in the memory , when many parts of results into the memory,it will be killed as OOM. You can try to map and convert sam to bam first and do the sorting singly.

I find that it contains `/groups/lilianasalvador/noah_100/bwa/bwa_output/SRR` between the reference sequence and the `-r` , it seems don't need this parameter , you can confirm the manual of the bwa mem

> **Comments:**
> **username_1**: This isn't true. Samtools sort is perfectly capable of sorting from a pipe. Try it, just run `samtools view -h foo.bam | samtools sort -o bar.bam`. And it can do this with humongous files, it doesn't store in memory, it creates temporary files while sorting precisely to avoid the problem you mention.  
> **username_3**: Sorry about that,actually i have tried many times in my machine, but that's not hpc,it just has 64GB memory, and I indeed got that problem,when i use the bwa mem | samtools view | samtools sort my computer will be down, i will can't connect it by ssh, and after long time ,it will recover,but that is uncontrollable ,and i hava find out some oom events during the analysis when i can connect it again. I have resolved it by the way i hava said.But there is an other question in his commandline which has been mentioned in the second paragraph in my answer last post.It seems that you mention it too.  
> **username_1**: That can be caused by various things, but I can assure you, I have been running `bwa mem | samtools sort` in a professional, production pipeline for almost a decade now. And also did so on my laptop, before posting, just to double check. I also confirmed in the [samtools manual\(https://www.htslib.org/doc/samtools-sort.html) that it creates temp files when memory is not enough. As for the last paragraph, I'm sorry but I don't understand it at all. Yes, the `-r` isn't needed, but it isn't causing the problem either.  
> **username_3**: Thanks to reply, actually i have noted the directory which he should not use between the reference and the reads file too. But the words may be unclear.  On the other hand i major in computer science ,when i get the  problem i have mentioned and take account for the behaviour of my comuputer and the syslog, i think i could be the reason what i have said , and by split the command line i resolve it. So it makes me believe the reason more.  The version of the samtools and htslib is 1.20. I will try to do reproduction experiments to located the ture reason. Thank you for reminding me.  

---

