<p>Merging sorted files is a linear operation, so any well-implemented tools that do it will do it with approximately the same efficiency.  So <code>samtools merge</code> (use the most up-to-date version, as there have been improvements in merge header handling in the 1.3.x and 1.4.x versions), <code>picard MergeSamFiles</code>, etc.</p>

<p>These tools need to hold all the input BAM files open simultaneously, so depending on how many hundred of input files you have you may run into the system limit on open file descriptors.  Use <code>ulimit</code> to maximise this first; if there are still too many, you may need to merge the first 500 files, then merge the next 500 into that, etc.  Samtools does not do this internally; I'm not sure whether any of the other merge implementations do.</p>

CC BY-SA 3.0
2017-05-19T07:23:06.240Z
Maybe merging stepwise as you mention in your second paragraph could speed up things as well as one doesn't hit IO limitations?
sargas
CC BY-SA 3.0
2017-05-19T07:26:34.097Z
No, some of the data is now being written and read more than once, so it must be slower.  With crafty use of pipes and process substitution (`samtools merge … <(samtools merge …) …`) this can be minimised and avoid hitting disk, so the slowdown can be trivial (at the cost of using a few more cores).
John Marshall
CC BY-SA 3.0
2017-05-19T07:18:49.330Z
2017-05-19T07:18:49.330Z
John Marshall
<p><code>samtools merge merged.bam *.bam</code> is efficient enough since the input files are sorted. You can get a bit faster with sambamba and/or biobambam, but they're not typically already installed and IO quickly becomes a bottleneck anyway.</p>

CC BY-SA 3.0
2017-05-19T07:18:44.133Z
Any idea of how this compares to bamtools merge?
Scott Gigante
CC BY-SA 3.0
2017-05-19T07:20:45.177Z
Samtools tends to be a bit slower than other tools since it's the reference implementation, so I'd expect bamtools to be a bit faster.
Devon Ryan
CC BY-SA 3.0
2017-05-19T11:29:01.210Z
Bamtools is usually slower because it is not optimized and now unmaintained. Sambamba etc are developed with a specific goal to be faster.
user172818
CC BY-SA 3.0
2017-05-19T07:06:35.520Z
2017-05-19T07:06:35.520Z
Devon Ryan
<p>I am working with over a million (long) reads, and aligning them to a large genome. I am considering running my alignment jobs in parallel, distributing horizontally across hundreds of nodes rather than trying to run a single job with dozens of cores.</p>

<p>I would like to merge the sorted BAM files together for further downstream analysis. What is the most efficient way to do so while maintaining a valid file header and taking advantage of the fact that the input bam files are already sorted?</p>

<phylogenetics><sam><samtools>
Merge hundreds of small BAM files into a single BAM file
# Merge hundreds of small BAM files into a single BAM file

(Asked by: username_0 on 2017-05-19)

I am working with over a million (long) reads, and aligning them to a large genome. I am considering running my alignment jobs in parallel, distributing horizontally across hundreds of nodes rather than trying to run a single job with dozens of cores.

I would like to merge the sorted BAM files together for further downstream analysis. What is the most efficient way to do so while maintaining a valid file header and taking advantage of the fact that the input bam files are already sorted?

**Tags:** phylogenetics, sam, samtools

---

## 2 answers

### Answer by username_3 on 2017-05-19. Score: 15 | Accepted answer

`samtools merge merged.bam *.bam` is efficient enough since the input files are sorted. You can get a bit faster with sambamba and/or biobambam, but they're not typically already installed and IO quickly becomes a bottleneck anyway.

> **Comments:**
> **username_0**: Any idea of how this compares to bamtools merge?  
> **username_3**: Samtools tends to be a bit slower than other tools since it's the reference implementation, so I'd expect bamtools to be a bit faster.  
> **username_4**: Bamtools is usually slower because it is not optimized and now unmaintained. Sambamba etc are developed with a specific goal to be faster.  

---

### Answer by username_1 on 2017-05-19. Score: 8

Merging sorted files is a linear operation, so any well-implemented tools that do it will do it with approximately the same efficiency. So `samtools merge` (use the most up-to-date version, as there have been improvements in merge header handling in the 1.3.x and 1.4.x versions), `picard MergeSamFiles`, etc.

These tools need to hold all the input BAM files open simultaneously, so depending on how many hundred of input files you have you may run into the system limit on open file descriptors. Use `ulimit` to maximise this first; if there are still too many, you may need to merge the first 500 files, then merge the next 500 into that, etc. Samtools does not do this internally; I'm not sure whether any of the other merge implementations do.

> **Comments:**
> **username_2**: Maybe merging stepwise as you mention in your second paragraph could speed up things as well as one doesn't hit IO limitations?  
> **username_1**: No, some of the data is now being written and read more than once, so it must be slower.  With crafty use of pipes and process substitution (`samtools merge … <(samtools merge …) …`) this can be minimised and avoid hitting disk, so the slowdown can be trivial (at the cost of using a few more cores).  

---

