<p>The best tool for the job is probably <code>bcftools</code> as <a href="https://bioinformatics.stackexchange.com/a/3306/298">suggested</a> by Bioathlete, but you can also do this manually. You just need to collect all header lines from all vcf files, remove any duplicates, and then print all the headers + the actual data into the new one:</p>

<pre><code>grep '^##' chr*vcf | sort | uniq &gt; all.vcf
grep -m1 '^#CHR' chr1.vcf &gt;&gt; all.vcf  ## Get the chr header line 
grep -v '^#' chr*vcf &gt;&gt; all.vcf
</code></pre>

CC BY-SA 3.0
2018-01-16T01:57:16.013Z
I appear to get a malformed VCF header with the above commands
ShanZhengYang
CC BY-SA 3.0
2018-01-16T08:26:10.147Z
@ShanZhengYang malformed how? Could you edit your question and show us an example of your files?
terdon
CC BY-SA 3.0
2018-01-16T18:32:14.970Z
So, I see the above `grep` commands give the same behavior with all of my standard VCFs. I'll get something like this in `all.vcf`:
`chr10.vcf:##contig=<ID=chr10,length=133797422>
chr10.vcf:##fileDate=20121011
chr10.vcf:##fileformat=VCFv4.2
chr10.vcf:##reference=path/reference.fa
chr10.vcf:##source=foobar
chr11.vcf:##contig=<ID=chr10,length=133797422>
chr11.vcf:##fileDate=20121011
chr11.vcf:##fileformat=VCFv4.2
chr11.vcf:##reference=path/reference.fa
chr11.vcf:##source=foobar
....`
ShanZhengYang
CC BY-SA 3.0
2018-01-15T15:13:06.070Z
2018-01-15T15:13:06.070Z
terdon
<p>use picard <strong>GatherVcf</strong>: <a href="http://broadinstitute.github.io/picard/command-line-overview.html#GatherVcfs" rel="nofollow noreferrer">http://broadinstitute.github.io/picard/command-line-overview.html#GatherVcfs</a></p>

<blockquote>
  <p>Gathers multiple VCF files from a scatter operation into a single VCF file. Input files must be supplied in genomic order and must not have events at overlapping positions.</p>
</blockquote>

CC BY-SA 3.0
2018-01-16T01:53:58.560Z
The error I get here is `Exception in thread "main" picard.PicardException: In order to index the resulting VCF input VCFs must contain ##contig lines.`. It is true, my "by chromosome"-VCFs do not contain a `##contig` line in the header...but I'm not sure that would matter
ShanZhengYang
CC BY-SA 3.0
2018-01-16T02:06:14.400Z
Edit above: Apparently this issue has been discussed before: github.com/samtools/bcftools/issues/326 
Question: how could I add these contig lines if I don't know this information?I know that these VCFs are from hg38. Could I simply add the line `##contig=<ID=chr1,length=248956422>`, etc. to the VCFs in question? I would just take the chromosome lengths from hg38
ShanZhengYang
CC BY-SA 3.0
2018-01-16T16:38:38.303Z
"but I'm not sure that would matter" it checks the order of the contigs using the dict. use UpdateVcfSequenceDictionary https://broadinstitute.github.io/picard/command-line-overview.html
Pierre
CC BY-SA 3.0
2018-01-16T18:20:32.423Z
"it checks the order of the contigs using the dict. use UpdateVcfSequenceDictionary". Ah, I understand now. Thanks for the help!
ShanZhengYang
CC BY-SA 3.0
2018-01-15T18:30:09.480Z
2018-01-15T18:30:09.480Z
Pierre
<p>I would recommend <a href="https://samtools.github.io/bcftools/bcftools.html" rel="noreferrer">bcftools</a> concat.  You can't just <code>cat</code> them together because each file has a header section.  The bcftools command will handle all that for you.  Each vcf file must be sorted prior to calling <code>concat</code></p>

<p><code>bcftools concat -o total_chroms.vcf chr1.vcf chr2.vcf chr3.vcf ... chrX.vcf</code></p>

CC BY-SA 3.0
2018-01-15T03:27:21.287Z
It's a bit odd. `vcf-validator chr1.vcf` shows no problems. I believe the output of `bcftools concat -o chroms12.vcf chr1.vcf chr2.vcf` shows an error: `FIXME: sequence name chr1 in chr1.vcf`
ShanZhengYang
CC BY-SA 3.0
2018-01-15T15:09:32.983Z
@ShanZhengYang the errors seem to indicate that you have `chr1` in the CHROM field for some rows, but no corresponding line in the header. Can you check if that is true in the `chr1.vcf` file? Maybe the validator somehow is missing this error?
juod
CC BY-SA 3.0
2018-01-15T19:06:45.167Z
@juod The VCF header looks correct to me. `#CHROM POS ID REF ALT QUAL FILTER INFO`
ShanZhengYang
CC BY-SA 3.0
2018-01-15T23:41:00.307Z
@juod "but no corresponding line in the header" Maybe I don't understand this. Besides the code pasted above, there is no other "chrom" information in the header
ShanZhengYang
CC BY-SA 3.0
2018-01-16T00:58:10.817Z
According to the VCF spec https://samtools.github.io/hts-specs/VCFv4.1.pdf there should be header lines for each chromosome this is from the first example in the linked documents - `##contig=<ID=20,length=62435964,assembly=B36,md5=f126cdf8a6e0c7f379d618ff66beb2da,species="Homo sapiens",taxonomy=x>`
Bioathlete
CC BY-SA 3.0
2018-01-16T02:05:19.847Z
@Bioathlete I see. I think I understand now---this issue has been discussed before: https://github.com/samtools/bcftools/issues/326
I know that these VCFs are from hg38. Could I simply add the line `##contig=<ID=chr1,length=248956422>`, etc. to the VCFs in question? I would just take the chromosome lengths from hg38
ShanZhengYang
CC BY-SA 3.0
2018-01-16T09:34:19.263Z
@ShanZhengYang try that - it should work. I don't even think it's necessary to provide the length, but you could experiment and let us know.
juod
CC BY-SA 3.0
2018-01-16T18:20:12.587Z
@juod The above worked! Thanks for the help!
ShanZhengYang
CC BY-SA 3.0
2018-01-14T23:25:10.867Z
2018-01-14T23:25:10.867Z
Bioathlete
<p>I have a several VCFs which are VCF which only contain information by chromosome. That is, there's a chromosome 1 VCF (with only chr1), a chromosome 2 VCF (with only chr2), etc. </p>

<p>I checked to make sure that these VCFs were valid via <code>VCFtools</code>, i.e. </p>

<pre><code>$ vcf-validator chr1.vcf
</code></pre>

<p>which works---these are valid VCFs I was given. </p>

<p>Now, I would like to combine these VCFs into one VCF. </p>

<p>I naïvely tried the following <code>cat</code> operation:</p>

<pre><code>$ cat chr1.vcf chr2.vcf chr3.vcf ... chrX.vcf &gt; total_chroms.vcf
</code></pre>

<p>This doesn't work properly though</p>

<pre><code>$ vcf-validator total_chroms.vcf
The header tag 'contig' not present for CHROM=chr1. (Not required but highly recommended.)
Could not parse the line, wrong number of columns: [##fileformat=VCFv4.2\n]
 at /path/vcftools-0.1.14/perl/Vcf.pm line 172, &lt;__ANONIO__&gt; line 191016.
     Vcf::throw('Vcf4_2=HASH(0x1ae7208)', 'Could not parse the line, wrong number of columns: [##filefor...') called at  /path/vcftools-0.1.14/perl/Vcf.pm line 335
     Vcf::next_data_array('Vcf4_2=HASH(0x1ae7208)') called at  /path/vcftools-0.1.14/perl/Vcf.pm line 3457
     Vcf4_1::next_data_array('Vcf4_2=HASH(0x1ae7208)') called at  /path/vcftools-0.1.14/perl/Vcf.pm line 2574
     VcfReader::run_validation('Vcf4_2=HASH(0x1ae7208)') called at  /path/vcftools-0.1.14//bin/vcf-validator line 60
     main::do_validation('HASH(0x16ada68)') called at  /path/vcftools-0.1.14//bin/vcf-validator line 14
$
</code></pre>

<p>What tools are available to merge these VCF's together into a total VCF? </p>

<vcf><vcftools>
How to concatenate "by chromosome"-VCFs?
# How to concatenate "by chromosome"-VCFs?

(Asked by: username_0 on 2018-01-14)

I have a several VCFs which are VCF which only contain information by chromosome. That is, there's a chromosome 1 VCF (with only chr1), a chromosome 2 VCF (with only chr2), etc. 
I checked to make sure that these VCFs were valid via VCFtools, i.e. 
$ vcf-validator chr1.vcf

which works---these are valid VCFs I was given. 
Now, I would like to combine these VCFs into one VCF. 
I naïvely tried the following cat operation:
$ cat chr1.vcf chr2.vcf chr3.vcf ... chrX.vcf > total_chroms.vcf

This doesn't work properly though
$ vcf-validator total_chroms.vcf
The header tag 'contig' not present for CHROM=chr1. (Not required but highly recommended.)
Could not parse the line, wrong number of columns: [##fileformat=VCFv4.2\n]
 at /path/vcftools-0.1.14/perl/Vcf.pm line 172, <__ANONIO__> line 191016.
     Vcf::throw('Vcf4_2=HASH(0x1ae7208)', 'Could not parse the line, wrong number of columns: [##filefor...') called at  /path/vcftools-0.1.14/perl/Vcf.pm line 335
     Vcf::next_data_array('Vcf4_2=HASH(0x1ae7208)') called at  /path/vcftools-0.1.14/perl/Vcf.pm line 3457
     Vcf4_1::next_data_array('Vcf4_2=HASH(0x1ae7208)') called at  /path/vcftools-0.1.14/perl/Vcf.pm line 2574
     VcfReader::run_validation('Vcf4_2=HASH(0x1ae7208)') called at  /path/vcftools-0.1.14//bin/vcf-validator line 60
     main::do_validation('HASH(0x16ada68)') called at  /path/vcftools-0.1.14//bin/vcf-validator line 14
$

What tools are available to merge these VCF's together into a total VCF?

**Tags:** vcf, vcftools

---

## 3 answers

### Answer by username_3 on 2018-01-14. Score: 12 | Accepted answer

I would recommend [bcftools](https://samtools.github.io/bcftools/bcftools.html) concat. You can't just `cat` them together because each file has a header section. The bcftools command will handle all that for you. Each vcf file must be sorted prior to calling `concat`

`bcftools concat -o total_chroms.vcf chr1.vcf chr2.vcf chr3.vcf ... chrX.vcf`

> **Comments:**
> **username_0**: It's a bit odd. `vcf-validator chr1.vcf` shows no problems. I believe the output of `bcftools concat -o chroms12.vcf chr1.vcf chr2.vcf` shows an error: `FIXME: sequence name chr1 in chr1.vcf`  
> **username_4**: @ShanZhengYang the errors seem to indicate that you have `chr1` in the CHROM field for some rows, but no corresponding line in the header. Can you check if that is true in the `chr1.vcf` file? Maybe the validator somehow is missing this error?  
> **username_0**: @juod The VCF header looks correct to me. `#CHROM POS ID REF ALT QUAL FILTER INFO`  
> **username_0**: @juod "but no corresponding line in the header" Maybe I don't understand this. Besides the code pasted above, there is no other "chrom" information in the header  
> **username_3**: According to the VCF spec https://samtools.github.io/hts-specs/VCFv4.1.pdf there should be header lines for each chromosome this is from the first example in the linked documents - `##contig=<ID=20,length=62435964,assembly=B36,md5=f126cdf8a6e0c7f379d618ff66beb2da,species="Homo sapiens",taxonomy=x>`  
> **username_0**: @Bioathlete I see. I think I understand now---this issue has been discussed before: https://github.com/samtools/bcftools/issues/326 I know that these VCFs are from hg38. Could I simply add the line `##contig=<ID=chr1,length=248956422>`, etc. to the VCFs in question? I would just take the chromosome lengths from hg38  
> **username_4**: @ShanZhengYang try that - it should work. I don't even think it's necessary to provide the length, but you could experiment and let us know.  
> **username_0**: @juod The above worked! Thanks for the help!  

---

### Answer by username_1 on 2018-01-15. Score: 2

The best tool for the job is probably bcftools as suggested by Bioathlete, but you can also do this manually. You just need to collect all header lines from all vcf files, remove any duplicates, and then print all the headers + the actual data into the new one:
grep '^##' chr*vcf | sort | uniq > all.vcf
grep -m1 '^#CHR' chr1.vcf >> all.vcf  ## Get the chr header line 
grep -v '^#' chr*vcf >> all.vcf

> **Comments:**
> **username_0**: I appear to get a malformed VCF header with the above commands  
> **username_1**: @ShanZhengYang malformed how? Could you edit your question and show us an example of your files?  
> **username_0**: So, I see the above `grep` commands give the same behavior with all of my standard VCFs. I'll get something like this in `all.vcf`: `chr10.vcf:##contig=<ID=chr10,length=133797422> chr10.vcf:##fileDate=20121011 chr10.vcf:##fileformat=VCFv4.2 chr10.vcf:##reference=path/reference.fa chr10.vcf:##source=foobar chr11.vcf:##contig=<ID=chr10,length=133797422> chr11.vcf:##fileDate=20121011 chr11.vcf:##fileformat=VCFv4.2 chr11.vcf:##reference=path/reference.fa chr11.vcf:##source=foobar ....`  

---

### Answer by username_2 on 2018-01-15. Score: 2

use picard **GatherVcf**: <http://broadinstitute.github.io/picard/command-line-overview.html#GatherVcfs>

> Gathers multiple VCF files from a scatter operation into a single VCF file. Input files must be supplied in genomic order and must not have events at overlapping positions.

> **Comments:**
> **username_0**: The error I get here is `Exception in thread "main" picard.PicardException: In order to index the resulting VCF input VCFs must contain ##contig lines.`. It is true, my "by chromosome"-VCFs do not contain a `##contig` line in the header...but I'm not sure that would matter  
> **username_0**: Edit above: Apparently this issue has been discussed before: github.com/samtools/bcftools/issues/326  Question: how could I add these contig lines if I don't know this information?I know that these VCFs are from hg38. Could I simply add the line `##contig=<ID=chr1,length=248956422>`, etc. to the VCFs in question? I would just take the chromosome lengths from hg38  
> **username_2**: "but I'm not sure that would matter" it checks the order of the contigs using the dict. use UpdateVcfSequenceDictionary https://broadinstitute.github.io/picard/command-line-overview.html  
> **username_0**: "it checks the order of the contigs using the dict. use UpdateVcfSequenceDictionary". Ah, I understand now. Thanks for the help!  

---

