<p>The best tool for the job is probably <code>bcftools</code> as <a href="https://bioinformatics.stackexchange.com/a/3306/298">suggested</a> by Bioathlete, but you can also do this manually. You just need to collect all header lines from all vcf files, remove any duplicates, and then print all the headers + the actual data into the new one:</p>

<pre><code>grep '^##' chr*vcf | sort | uniq &gt; all.vcf
grep -m1 '^#CHR' chr1.vcf &gt;&gt; all.vcf  ## Get the chr header line 
grep -v '^#' chr*vcf &gt;&gt; all.vcf
</code></pre>

CC BY-SA 3.0
2018-01-16T01:57:16.013Z
I appear to get a malformed VCF header with the above commands
ShanZhengYang
CC BY-SA 3.0
2018-01-16T08:26:10.147Z
@ShanZhengYang malformed how? Could you edit your question and show us an example of your files?
terdon
CC BY-SA 3.0
2018-01-16T18:32:14.970Z
So, I see the above `grep` commands give the same behavior with all of my standard VCFs. I'll get something like this in `all.vcf`:
`chr10.vcf:##contig=<ID=chr10,length=133797422>
chr10.vcf:##fileDate=20121011
chr10.vcf:##fileformat=VCFv4.2
chr10.vcf:##reference=path/reference.fa
chr10.vcf:##source=foobar
chr11.vcf:##contig=<ID=chr10,length=133797422>
chr11.vcf:##fileDate=20121011
chr11.vcf:##fileformat=VCFv4.2
chr11.vcf:##reference=path/reference.fa
chr11.vcf:##source=foobar
....`
ShanZhengYang
CC BY-SA 3.0
2018-01-15T15:13:06.070Z
2018-01-15T15:13:06.070Z
terdon
<p>use picard <strong>GatherVcf</strong>: <a href="http://broadinstitute.github.io/picard/command-line-overview.html#GatherVcfs" rel="nofollow noreferrer">http://broadinstitute.github.io/picard/command-line-overview.html#GatherVcfs</a></p>

<blockquote>
  <p>Gathers multiple VCF files from a scatter operation into a single VCF file. Input files must be supplied in genomic order and must not have events at overlapping positions.</p>
</blockquote>

CC BY-SA 3.0
2018-01-16T01:53:58.560Z
The error I get here is `Exception in thread "main" picard.PicardException: In order to index the resulting VCF input VCFs must contain ##contig lines.`. It is true, my "by chromosome"-VCFs do not contain a `##contig` line in the header...but I'm not sure that would matter
ShanZhengYang
CC BY-SA 3.0
2018-01-16T02:06:14.400Z
Edit above: Apparently this issue has been discussed before: github.com/samtools/bcftools/issues/326 
Question: how could I add these contig lines if I don't know this information?I know that these VCFs are from hg38. Could I simply add the line `##contig=<ID=chr1,length=248956422>`, etc. to the VCFs in question? I would just take the chromosome lengths from hg38
ShanZhengYang
CC BY-SA 3.0
2018-01-16T16:38:38.303Z
"but I'm not sure that would matter" it checks the order of the contigs using the dict. use UpdateVcfSequenceDictionary https://broadinstitute.github.io/picard/command-line-overview.html
Pierre
CC BY-SA 3.0
2018-01-16T18:20:32.423Z
"it checks the order of the contigs using the dict. use UpdateVcfSequenceDictionary". Ah, I understand now. Thanks for the help!
ShanZhengYang
CC BY-SA 3.0
2018-01-15T18:30:09.480Z
2018-01-15T18:30:09.480Z
Pierre
<p>I would recommend <a href="https://samtools.github.io/bcftools/bcftools.html" rel="noreferrer">bcftools</a> concat.  You can't just <code>cat</code> them together because each file has a header section.  The bcftools command will handle all that for you.  Each vcf file must be sorted prior to calling <code>concat</code></p>

<p><code>bcftools concat -o total_chroms.vcf chr1.vcf chr2.vcf chr3.vcf ... chrX.vcf</code></p>

CC BY-SA 3.0
2018-01-15T03:27:21.287Z
It's a bit odd. `vcf-validator chr1.vcf` shows no problems. I believe the output of `bcftools concat -o chroms12.vcf chr1.vcf chr2.vcf` shows an error: `FIXME: sequence name chr1 in chr1.vcf`
ShanZhengYang
CC BY-SA 3.0
2018-01-15T15:09:32.983Z
@ShanZhengYang the errors seem to indicate that you have `chr1` in the CHROM field for some rows, but no corresponding line in the header. Can you check if that is true in the `chr1.vcf` file? Maybe the validator somehow is missing this error?
juod
CC BY-SA 3.0
2018-01-15T19:06:45.167Z
@juod The VCF header looks correct to me. `#CHROM POS ID REF ALT QUAL FILTER INFO`
ShanZhengYang
CC BY-SA 3.0
2018-01-15T23:41:00.307Z
@juod "but no corresponding line in the header" Maybe I don't understand this. Besides the code pasted above, there is no other "chrom" information in the header
ShanZhengYang
CC BY-SA 3.0
2018-01-16T00:58:10.817Z
According to the VCF spec https://samtools.github.io/hts-specs/VCFv4.1.pdf there should be header lines for each chromosome this is from the first example in the linked documents - `##contig=<ID=20,length=62435964,assembly=B36,md5=f126cdf8a6e0c7f379d618ff66beb2da,species="Homo sapiens",taxonomy=x>`
Bioathlete
CC BY-SA 3.0
2018-01-16T02:05:19.847Z
@Bioathlete I see. I think I understand now---this issue has been discussed before: https://github.com/samtools/bcftools/issues/326
I know that these VCFs are from hg38. Could I simply add the line `##contig=<ID=chr1,length=248956422>`, etc. to the VCFs in question? I would just take the chromosome lengths from hg38
ShanZhengYang
CC BY-SA 3.0
2018-01-16T09:34:19.263Z
@ShanZhengYang try that - it should work. I don't even think it's necessary to provide the length, but you could experiment and let us know.
juod
CC BY-SA 3.0
2018-01-16T18:20:12.587Z
@juod The above worked! Thanks for the help!
ShanZhengYang
CC BY-SA 3.0
2018-01-14T23:25:10.867Z
2018-01-14T23:25:10.867Z
Bioathlete
<p>I have a several VCFs which are VCF which only contain information by chromosome. That is, there's a chromosome 1 VCF (with only chr1), a chromosome 2 VCF (with only chr2), etc. </p>

<p>I checked to make sure that these VCFs were valid via <code>VCFtools</code>, i.e. </p>

<pre><code>$ vcf-validator chr1.vcf
</code></pre>

<p>which works---these are valid VCFs I was given. </p>

<p>Now, I would like to combine these VCFs into one VCF. </p>

<p>I naïvely tried the following <code>cat</code> operation:</p>

<pre><code>$ cat chr1.vcf chr2.vcf chr3.vcf ... chrX.vcf &gt; total_chroms.vcf
</code></pre>

<p>This doesn't work properly though</p>

<pre><code>$ vcf-validator total_chroms.vcf
The header tag 'contig' not present for CHROM=chr1. (Not required but highly recommended.)
Could not parse the line, wrong number of columns: [##fileformat=VCFv4.2\n]
 at /path/vcftools-0.1.14/perl/Vcf.pm line 172, &lt;__ANONIO__&gt; line 191016.
     Vcf::throw('Vcf4_2=HASH(0x1ae7208)', 'Could not parse the line, wrong number of columns: [##filefor...') called at  /path/vcftools-0.1.14/perl/Vcf.pm line 335
     Vcf::next_data_array('Vcf4_2=HASH(0x1ae7208)') called at  /path/vcftools-0.1.14/perl/Vcf.pm line 3457
     Vcf4_1::next_data_array('Vcf4_2=HASH(0x1ae7208)') called at  /path/vcftools-0.1.14/perl/Vcf.pm line 2574
     VcfReader::run_validation('Vcf4_2=HASH(0x1ae7208)') called at  /path/vcftools-0.1.14//bin/vcf-validator line 60
     main::do_validation('HASH(0x16ada68)') called at  /path/vcftools-0.1.14//bin/vcf-validator line 14
$
</code></pre>

<p>What tools are available to merge these VCF's together into a total VCF? </p>

<vcf><vcftools>
How to concatenate "by chromosome"-VCFs?
# How to concatenate "by chromosome"-VCFs?

(Asked by: username_0 on 2018-01-14)

I have a several VCFs which are VCF which only contain information by chromosome. That is, there's a chromosome 1 VCF (with only chr1), a chromosome 2 VCF (with only chr2), etc. 
I checked to make sure that these VCFs were valid via VCFtools, i.e. 
$ vcf-validator chr1.vcf

which works---these are valid VCFs I was given. 
Now, I would like to combine these VCFs into one VCF. 
I naïvely tried the following cat operation:
$ cat chr1.vcf chr2.vcf chr3.vcf ... chrX.vcf > total_chroms.vcf

This doesn't work properly though
$ vcf-validator total_chroms.vcf
The header tag 'contig' not present for CHROM=chr1. (Not required but highly recommended.)
Could not parse the line, wrong number of columns: [##fileformat=VCFv4.2\n]
 at /path/vcftools-0.1.14/perl/Vcf.pm line 172, <__ANONIO__> line 191016.
     Vcf::throw('Vcf4_2=HASH(0x1ae7208)', 'Could not parse the line, wrong number of columns: [##filefor...') called at  /path/vcftools-0.1.14/perl/Vcf.pm line 335
     Vcf::next_data_array('Vcf4_2=HASH(0x1ae7208)') called at  /path/vcftools-0.1.14/perl/Vcf.pm line 3457
     Vcf4_1::next_data_array('Vcf4_2=HASH(0x1ae7208)') called at  /path/vcftools-0.1.14/perl/Vcf.pm line 2574
     VcfReader::run_validation('Vcf4_2=HASH(0x1ae7208)') called at  /path/vcftools-0.1.14//bin/vcf-validator line 60
     main::do_validation('HASH(0x16ada68)') called at  /path/vcftools-0.1.14//bin/vcf-validator line 14
$

What tools are available to merge these VCF's together into a total VCF?

**Tags:** vcf, vcftools


---

## 3 answers

### Answer by username_3 on 2018-01-14. Score: 12 | Accepted answer

I would recommend [bcftools](https://samtools.github.io/bcftools/bcftools.html) concat. You can't just `cat` them together because each file has a header section. The bcftools command will handle all that for you. Each vcf file must be sorted prior to calling `concat`

`bcftools concat -o total_chroms.vcf chr1.vcf chr2.vcf chr3.vcf ... chrX.vcf`

> **Comments:**
> **username_0**: It's a bit odd. `vcf-validator chr1.vcf` shows no problems. I believe the output of `bcftools concat -o chroms12.vcf chr1.vcf chr2.vcf` shows an error: `FIXME: sequence name chr1 in chr1.vcf`  
> **username_4**: @username_0 the errors seem to indicate that you have `chr1` in the CHROM field for some rows, but no corresponding line in the header. Can you check if that is true in the `chr1.vcf` file? Maybe the validator somehow is missing this error?  
> **username_0**: @username_4 The VCF header looks correct to me. `#CHROM POS ID REF ALT QUAL FILTER INFO`  
> **username_0**: @username_4 "but no corresponding line in the header" Maybe I don't understand this. Besides the code pasted above, there is no other "chrom" information in the header  
> **username_3**: According to the VCF spec https://samtools.github.io/hts-specs/VCFv4.1.pdf there should be header lines for each chromosome this is from the first example in the linked documents - `##contig=<ID=20,length=62435964,assembly=B36,md5=f126cdf8a6e0c7f379d618ff66beb2da,species="Homo sapiens",taxonomy=x>`  
> **username_0**: @username_3 I see. I think I understand now---this issue has been discussed before: https://github.com/samtools/bcftools/issues/326 I know that these VCFs are from hg38. Could I simply add the line `##contig=<ID=chr1,length=248956422>`, etc. to the VCFs in question? I would just take the chromosome lengths from hg38  
> **username_4**: @username_0 try that - it should work. I don't even think it's necessary to provide the length, but you could experiment and let us know.  
> **username_0**: @username_4 The above worked! Thanks for the help!  

---

### Answer by username_1 on 2018-01-15. Score: 2

The best tool for the job is probably bcftools as suggested by username_3, but you can also do this manually. You just need to collect all header lines from all vcf files, remove any duplicates, and then print all the headers + the actual data into the new one:
grep '^##' chr*vcf | sort | uniq > all.vcf
grep -m1 '^#CHR' chr1.vcf >> all.vcf  ## Get the chr header line 
grep -v '^#' chr*vcf >> all.vcf

> **Comments:**
> **username_0**: I appear to get a malformed VCF header with the above commands  
> **username_1**: @username_0 malformed how? Could you edit your question and show us an example of your files?  
> **username_0**: So, I see the above `grep` commands give the same behavior with all of my standard VCFs. I'll get something like this in `all.vcf`: `chr10.vcf:##contig=<ID=chr10,length=133797422> chr10.vcf:##fileDate=20121011 chr10.vcf:##fileformat=VCFv4.2 chr10.vcf:##reference=path/reference.fa chr10.vcf:##source=foobar chr11.vcf:##contig=<ID=chr10,length=133797422> chr11.vcf:##fileDate=20121011 chr11.vcf:##fileformat=VCFv4.2 chr11.vcf:##reference=path/reference.fa chr11.vcf:##source=foobar ....`  

---

### Answer by username_2 on 2018-01-15. Score: 2

use picard **GatherVcf**: <http://broadinstitute.github.io/picard/command-line-overview.html#GatherVcfs>

> Gathers multiple VCF files from a scatter operation into a single VCF file. Input files must be supplied in genomic order and must not have events at overlapping positions.

> **Comments:**
> **username_0**: The error I get here is `Exception in thread "main" picard.PicardException: In order to index the resulting VCF input VCFs must contain ##contig lines.`. It is true, my "by chromosome"-VCFs do not contain a `##contig` line in the header...but I'm not sure that would matter  
> **username_0**: Edit above: Apparently this issue has been discussed before: github.com/samtools/bcftools/issues/326  Question: how could I add these contig lines if I don't know this information?I know that these VCFs are from hg38. Could I simply add the line `##contig=<ID=chr1,length=248956422>`, etc. to the VCFs in question? I would just take the chromosome lengths from hg38  
> **username_2**: "but I'm not sure that would matter" it checks the order of the contigs using the dict. use UpdateVcfSequenceDictionary https://broadinstitute.github.io/picard/command-line-overview.html  
> **username_0**: "it checks the order of the contigs using the dict. use UpdateVcfSequenceDictionary". Ah, I understand now. Thanks for the help!  

---

