<p>As part of its operations, <code>samtools sort</code> will split the input up into multiple temporary BAM files (which are each individually sorted) if the reads can't fit entirely within memory. Those files are merged for the final output. From the manpage:</p>
<blockquote>
<p>The  sorted output is written to standard output by default, or to the specified file (out.bam) when -o is used. This command will also create temporary files tmpprefix.%d.bam as needed when the entire alignment data cannot fit into  memory (as controlled via the -m option).</p>
</blockquote>
<p>Given this, I would expect that a concatenated, unsorted BAM will take more time to process than merging sorted BAM files, because in addition to creating the concatenated BAM file, the splitting would need to be done again (assuming the reads don't fit in memory) prior to the final merge carried out by <code>samtools sort</code>.</p>
<p>I appreciate that this is not a quantified answer; it's just my guess based on what I have observed to happen when doing my own analyses.</p>

CC BY-SA 4.0
2022-02-11T15:37:54.060Z
It's funny. Now that you mention that, it seems so obvious! Great point. I may still quantify it just to see the difference.
Mark Ebbert
CC BY-SA 4.0
2022-02-11T05:00:00.417Z
2022-02-11T05:00:00.417Z
gringer
<p>Related to my other question (<a href="https://bioinformatics.stackexchange.com/questions/18538/samtools-sort-most-efficient-memory-and-thread-settings-for-many-samples-on-a-c">Samtools sort: most efficient memory and thread settings for many samples on a cluster</a>), we need to optimize <code>samtools sort</code> as we prepare to analyze thousands of <code>.bam</code> files, beginning with re-alignment, sorting, etc.</p>
<p><strong>Ultimate question</strong>: what is the fastest way to sort a single sample after aligning the sample across many separate cluster jobs? i.e., after splitting <code>.fastq</code>s for a single sample and aligning as distinct jobs. I'm looking for recent and clear time/thread/memory comparisons.</p>
<p>Unless I'm missing something, <code>samtools merge</code> was designed exactly for the purpose of merging pre-sorted bams. An excerpt from the manpage (<a href="http://www.htslib.org/doc/samtools-merge.html" rel="nofollow noreferrer">http://www.htslib.org/doc/samtools-merge.html</a>): &quot;Merge multiple sorted alignment files, producing a single sorted output file that contains all the input records and maintains the existing sort order.&quot;</p>
<p>Is it fastest to:</p>
<ol>
<li><code>samtools sort</code> each 'mini-bam' and then <code>samtools merge</code>?</li>
<li><code>samtools cat</code> the 'mini-bams' together and then <code>samtools sort</code>?</li>
<li>Another option I haven't considered?</li>
</ol>
<p><strong>Note:</strong> I have seen the following similar questions on other forums, but they are outdated (~8 years ago) and do not answer this question directly:</p>
<ol>
<li><a href="https://www.biostars.org/p/9864/" rel="nofollow noreferrer">Best Way To Merge A Many Thousand Small Bam Files Into One Big Bam File?</a></li>
<li><a href="http://seqanswers.com/forums/showthread.php?t=47110" rel="nofollow noreferrer">Merging BAM files before sorting</a></li>
</ol>

<phylogenetics><sequence-analysis><samtools>
Samtools sort: most efficient approach to sort a single sample after aligning split .fastq files
# Samtools sort: most efficient approach to sort a single sample after aligning split .fastq files

(Asked by: username_0 on 2022-02-10)

Related to my other question ([Samtools sort: most efficient memory and thread settings for many samples on a cluster](https://bioinformatics.stackexchange.com/questions/18538/samtools-sort-most-efficient-memory-and-thread-settings-for-many-samples-on-a-c)), we need to optimize `samtools sort` as we prepare to analyze thousands of `.bam` files, beginning with re-alignment, sorting, etc.

**Ultimate question**: what is the fastest way to sort a single sample after aligning the sample across many separate cluster jobs? i.e., after splitting `.fastq`s for a single sample and aligning as distinct jobs. I'm looking for recent and clear time/thread/memory comparisons.

Unless I'm missing something, `samtools merge` was designed exactly for the purpose of merging pre-sorted bams. An excerpt from the manpage (<http://www.htslib.org/doc/samtools-merge.html>): "Merge multiple sorted alignment files, producing a single sorted output file that contains all the input records and maintains the existing sort order."

Is it fastest to:

1. `samtools sort` each 'mini-bam' and then `samtools merge`?
2. `samtools cat` the 'mini-bams' together and then `samtools sort`?
3. Another option I haven't considered?

**Note:** I have seen the following similar questions on other forums, but they are outdated (~8 years ago) and do not answer this question directly:

1. [Best Way To Merge A Many Thousand Small Bam Files Into One Big Bam File?](https://www.biostars.org/p/9864/)
2. [Merging BAM files before sorting](http://seqanswers.com/forums/showthread.php?t=47110)

**Tags:** phylogenetics, sequence-analysis, samtools

---

## 1 answers

### Answer by username_1 on 2022-02-11. Score: 1

As part of its operations, `samtools sort` will split the input up into multiple temporary BAM files (which are each individually sorted) if the reads can't fit entirely within memory. Those files are merged for the final output. From the manpage:

> The sorted output is written to standard output by default, or to the specified file (out.bam) when -o is used. This command will also create temporary files tmpprefix.%d.bam as needed when the entire alignment data cannot fit into memory (as controlled via the -m option).

Given this, I would expect that a concatenated, unsorted BAM will take more time to process than merging sorted BAM files, because in addition to creating the concatenated BAM file, the splitting would need to be done again (assuming the reads don't fit in memory) prior to the final merge carried out by `samtools sort`.

I appreciate that this is not a quantified answer; it's just my guess based on what I have observed to happen when doing my own analyses.

> **Comments:**
> **username_0**: It's funny. Now that you mention that, it seems so obvious! Great point. I may still quantify it just to see the difference.  

---

