<p><em>Answer from @user324810, converted from comment:</em></p>
<p>There is an alternative than doing it with Python unless it is absolutely necessary. If you want to merge multiple samples into one, use <a href="http://samtools.github.io/bcftools/bcftools.html#merge" rel="nofollow noreferrer">bcftools merge</a>. If you want simply to concatenate multiple VCFs, use <a href="https://samtools.github.io/bcftools/bcftools.html#concat" rel="nofollow noreferrer">bcftools concatenate</a>. Once you obtain the output file, you can load the dataframe and plot it as you wish.</p>

CC BY-SA 4.0
2022-01-23T13:00:13.923Z
2022-01-23T13:00:13.923Z
gringer
<p><em>This question has also been asked on <a href="https://www.biostars.org/p/9505729/" rel="nofollow noreferrer">Biostars</a></em></p>
<p>I have 2 directories that contains multiple VCF files.
<a href="https://i.sstatic.net/Irq3f.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Irq3f.png" alt="dir" /></a></p>
<p>I want to write code in python that will process all vcf files and create an AFD (allele frequency difference) plot for each VCF file present in a folder</p>
<p>I have started with this code, which attempts to convert the VCF files into a pandas dataframe:</p>
<pre><code># Import Module
import os
import pandas as pd
# Folder Path
path1 = &quot;C://Users//USER//Desktop//isiah/VCFs_1/&quot;
path2 = &quot;C://Users//USER//Desktop//isiah/VCFs_2/&quot;
#os.chdir(path1)

# Read text File


def read_text_file(file_path1,file_path2):
    with open(file_path, 'r') as f:
        print(f.read())


# iterate through all file
for file in os.listdir():
    # Check whether file is in text format or not
    if file.endswith(&quot;.vcf&quot;):
        file_path1 = f&quot;{path1}\{file}&quot;
        file_path2 = f&quot;{path2}\{file}&quot;
        print(file_path1,&quot;\n\n&quot;,file_path2)

# call read text file function
#data = read_text_file(path1,path2)
#print(read_text_file(path1,path2))
Final= pd.DataFrame(read_text_file(path1,path2))
print(Final)
</code></pre>
<p>and its giving me output without calling function and also gives me empty dataframe. Also i am not sure if result is showing all vcf files record or not.</p>
<p>Kindly help me to modify this code. I need my all result in dataframe so i can make plot easily</p>
<p>Folder 1 contains 3 test vcfs whereas folder 2 contains 4 test vcfs:</p>
<p><a href="https://i.sstatic.net/QHERb.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/QHERb.png" alt="directories" /></a></p>
<p>This is test 1 vcf file:</p>
<pre><code>##fileformat=VCFv4.2
##ALT=&lt;ID=NON_REF,Description=&quot;Represents any possible alternative allele at this location&quot;&gt;
##FILTER=&lt;ID=FS,Description=&quot;FS&gt;60.0&quot;&gt;
##FILTER=&lt;ID=HS,Description=&quot;HaplotypeScore&lt;13.0&quot;&gt;
##FILTER=&lt;ID=LowQual,Description=&quot;Low quality&quot;&gt;
##FILTER=&lt;ID=MQ,Description=&quot;MQ&lt;40.0&quot;&gt;
##FILTER=&lt;ID=MQRS,Description=&quot;MQRankSum &lt; -12.5&quot;&gt;
##FILTER=&lt;ID=Mask,Description=&quot;Doesn't overlap a user-input mask&quot;&gt;
##FILTER=&lt;ID=PASS,Description=&quot;All filters passed&quot;&gt;
##FILTER=&lt;ID=QD,Description=&quot;QD&lt;2.0&quot;&gt;
##FILTER=&lt;ID=RPRS,Description=&quot;ReadPosRankSum &lt; -8.0&quot;&gt;
##FORMAT=&lt;ID=AD,Number=R,Type=Integer,Description=&quot;Allelic depths for the ref and alt alleles in the order listed&quot;&gt;
##FORMAT=&lt;ID=DP,Number=1,Type=Integer,Description=&quot;Approximate read depth (reads with MQ=255 or with bad mates are filtered)&quot;&gt;
##FORMAT=&lt;ID=GQ,Number=1,Type=Integer,Description=&quot;Genotype Quality&quot;&gt;
##FORMAT=&lt;ID=GT,Number=1,Type=String,Description=&quot;Genotype&quot;&gt;
##FORMAT=&lt;ID=PGT,Number=1,Type=String,Description=&quot;Physical phasing haplotype information, describing how the alternate alleles are phased in relation to one another&quot;&gt;
##FORMAT=&lt;ID=PID,Number=1,Type=String,Description=&quot;Physical phasing ID information, where each unique ID within a given sample (but not across samples) connects records within a phasing group&quot;&gt;
##FORMAT=&lt;ID=PL,Number=G,Type=Integer,Description=&quot;Normalized, Phred-scaled likelihoods for genotypes as defined in the VCF specification&quot;&gt;
##FORMAT=&lt;ID=RGQ,Number=1,Type=Integer,Description=&quot;Unconditional reference genotype confidence, encoded as a phred quality -10*log10 p(genotype call is wrong)&quot;&gt;
##FORMAT=&lt;ID=SB,Number=4,Type=Integer,Description=&quot;Per-sample component statistics which comprise the Fisher's Exact Test to detect strand bias.&quot;&gt;
##INFO=&lt;ID=AC,Number=A,Type=Integer,Description=&quot;Allele count in genotypes, for each ALT allele, in the same order as listed&quot;&gt;
##INFO=&lt;ID=AF,Number=A,Type=Float,Description=&quot;Allele Frequency, for each ALT allele, in the same order as listed&quot;&gt;
##INFO=&lt;ID=AN,Number=1,Type=Integer,Description=&quot;Total number of alleles in called genotypes&quot;&gt;
##INFO=&lt;ID=BaseQRankSum,Number=1,Type=Float,Description=&quot;Z-score from Wilcoxon rank sum test of Alt Vs. Ref base qualities&quot;&gt;
##INFO=&lt;ID=ClippingRankSum,Number=1,Type=Float,Description=&quot;Z-score From Wilcoxon rank sum test of Alt vs. Ref number of hard clipped bases&quot;&gt;
##INFO=&lt;ID=DP,Number=1,Type=Integer,Description=&quot;Approximate read depth; some reads may have been filtered&quot;&gt;
##INFO=&lt;ID=DS,Number=0,Type=Flag,Description=&quot;Were any of the samples downsampled?&quot;&gt;
##INFO=&lt;ID=ExcessHet,Number=1,Type=Float,Description=&quot;Phred-scaled p-value for exact test of excess heterozygosity&quot;&gt;
##INFO=&lt;ID=FS,Number=1,Type=Float,Description=&quot;Phred-scaled p-value using Fisher's exact test to detect strand bias&quot;&gt;
##INFO=&lt;ID=HaplotypeScore,Number=1,Type=Float,Description=&quot;Consistency of the site with at most two segregating haplotypes&quot;&gt;
##INFO=&lt;ID=InbreedingCoeff,Number=1,Type=Float,Description=&quot;Inbreeding coefficient as estimated from the genotype likelihoods per-sample when compared against the Hardy-Weinberg expectation&quot;&gt;
##INFO=&lt;ID=MLEAC,Number=A,Type=Integer,Description=&quot;Maximum likelihood expectation (MLE) for the allele counts (not necessarily the same as the AC), for each ALT allele, in the same order as listed&quot;&gt;
##INFO=&lt;ID=MLEAF,Number=A,Type=Float,Description=&quot;Maximum likelihood expectation (MLE) for the allele frequency (not necessarily the same as the AF), for each ALT allele, in the same order as listed&quot;&gt;
##INFO=&lt;ID=MQ,Number=1,Type=Float,Description=&quot;RMS Mapping Quality&quot;&gt;
##INFO=&lt;ID=MQRankSum,Number=1,Type=Float,Description=&quot;Z-score From Wilcoxon rank sum test of Alt vs. Ref read mapping qualities&quot;&gt;
##INFO=&lt;ID=QD,Number=1,Type=Float,Description=&quot;Variant Confidence/Quality by Depth&quot;&gt;
##INFO=&lt;ID=RAW_MQ,Number=1,Type=Float,Description=&quot;Raw data for RMS Mapping Quality&quot;&gt;
##INFO=&lt;ID=ReadPosRankSum,Number=1,Type=Float,Description=&quot;Z-score from Wilcoxon rank sum test of Alt vs. Ref read position bias&quot;&gt;
##INFO=&lt;ID=SOR,Number=1,Type=Float,Description=&quot;Symmetric Odds Ratio of 2x2 contingency table to detect strand bias&quot;&gt;
##reference=file:///Users/sian_bray/Dropbox/Yant/Cochlearia/02_Data/References/pseudohap_cochlearia_135M_3kb.fasta
##source=SelectVariants
#CHROM  POS ID  REF ALT QUAL    FILTER  INFO    FORMAT  ALO_006 ALO_007 ALO_013 ALO_017 BEA_002 BEA_004 BEA_010 BRI_002 BRI_005 BRI_006 BRI_009 ELI_001 ELI_002 ELI_003 ELI_004 FRE_013 JON_001 KVA_002 KVA_003 KVA_009 KVA_010 LAB_004 LAB_300 LAB_400 LAB_500 LNL_001 LNL_002 LNL_003 LNL_008 MEL_001 MEL_002 NEN_001 NEN_003 NEN_200 NEN_300 ROT_004 ROT_006 ROT_007 ROT_013 RUZ_001 SAL_003 SAL_004 SAL_006 SKF_002 SKF_003 SKF_005 SKF_009 SKI_004 SKI_005 SKI_008 SKN_001 SKN_002 SKN_005 SKN_008 SPU_006 SPU_008 SPU_009 SPU_010 TET_002 TET_004 TET_006 TET_008 TRO_001 TRO_003 TRO_005 TRO_009 VAG_003 VAG_004 VAG_007 VAG_102 VEG_003 VEG_004 WOL_002 WOL_006 WOL_009 WOL_010
237 50041   .   G   T   4099.26 PASS    AC=35;AF=0.133;AN=264;BaseQRankSum=0.00;ClippingRankSum=0.00;DP=1228;ExcessHet=4.3562;FS=0.644;InbreedingCoeff=-0.0953;MQ=57.84;MQRankSum=0.00;QD=25.94;ReadPosRankSum=-1.570e-01;SOR=0.768 GT:AD:DP:GQ:PL  0/0/0/0:11,0:11:14:0,14,33,66,425   0/0/0/0:11,1:12:0:0,0,0,27,369  0/0/0/0:13,3:16:0:0,0,0,0,355   0/0/0/0:41,0:41:50:0,50,120,241,1800    0/0/1/1:5,8:13:1:293,15,0,1,170 0/0/0/1:5,3:8:0:104,0,0,10,191  0/0/0/0:8,0:8:10:0,10,24,48,272 0/1/1/1:1,8:9:11:318,33,11,0,26 ./././.:0,0 1/1/1/1:0,2:2:2:87,12,6,2,0 0/0/0/0:5,2:7:0:0,0,0,0,120 0/0/0/0:16,2:18:0:0,0,0,21,515  0/0/0/0:8,1:9:0:0,0,0,13,270    0/0/0/0:15,3:18:0:0,0,0,0,459   0/0/0/0:10,1:11:0:0,0,0,21,363  0/0/0/0/0/0:15,0:15:12:0,12,26,45,71,116,589    0/0/0/0/0/0:7,2:9:0:0,0,0,0,0,0,151 ./././.:0,0 0/0/0/0:14,0:14:14:0,14,33,66,495   0/0/0/0:1,0:1:1:0,1,3,6,42  0/0/0/0:9,0:9:10:0,10,24,48,360 0/0:21,2:23:14:0,14,726 0/0:14,2:16:11:0,11,452 0/0:14,2:16:0:0,0,467   0/0:23,9:32:0:0,0,527   0/0/0/0:10,0:10:12:0,12,30,60,359   0/0/0/0:11,1:12:0:0,0,0,27,404  0/0/0/0:13,0:13:16:0,16,39,78,526   0/0/0/0:20,3:23:0:0,0,0,2,655   1/1/1/1:0,4:4:5:155,24,12,5,0   0/0/0/0:2,3:5:0:0,0,0,0,8   0/0:23,2:25:0:0,0,798   0/0:12,2:14:0:0,0,381   0/0:14,0:14:36:0,36,540 0/0:24,0:24:63:0,63,945 0/0/0/0:8,1:9:0:0,0,0,12,245    0/0/0/0:27,0:27:32:0,32,78,157,1170 0/0/0/0:18,5:23:0:0,0,0,0,474   0/0/0/0:22,1:23:15:0,15,50,114,736  0/0:10,0:10:27:0,27,405 0/0:8,0:8:24:0,24,322   0/0:13,0:13:36:0,36,540 0/0:4,0:4:12:0,12,155   0/0/0/0/0/0:5,2:7:0:0,0,0,0,0,0,125 0/0/0/0/0/0:19,1:20:0:0,0,0,18,50,106,731   0/0/0/0/0/0:28,3:31:0:0,0,0,0,1,82,978  0/0/0/0/0/0:53,10:63:0:0,0,0,0,0,0,1843 0/0/0/0:1,1:2:0:0,0,0,0,4   1/1/1/1:0,22:22:27:917,132,66,27,0  ./././.:0,0 0/0/0/0:15,0:15:17:0,17,42,84,630   0/0/0/0:14,2:16:0:0,0,0,27,421  0/0/0/0:17,0:17:21:0,21,51,102,670  0/0/0/0:17,7:24:0:0,0,0,0,323   0/0:11,2:13:0:0,0,380
</code></pre>

<python><vcf>
How to process multiple VCF files present in a 2 folders and create afd plots
# How to process multiple VCF files present in a 2 folders and create afd plots

(Asked by: username_0 on 2022-01-15)

This question has also been asked on Biostars
I have 2 directories that contains multiple VCF files.

I want to write code in python that will process all vcf files and create an AFD (allele frequency difference) plot for each VCF file present in a folder
I have started with this code, which attempts to convert the VCF files into a pandas dataframe:
# Import Module
import os
import pandas as pd
# Folder Path
path1 = "C://Users//USER//Desktop//isiah/VCFs_1/"
path2 = "C://Users//USER//Desktop//isiah/VCFs_2/"
#os.chdir(path1)

# Read text File


def read_text_file(file_path1,file_path2):
    with open(file_path, 'r') as f:
        print(f.read())


# iterate through all file
for file in os.listdir():
    # Check whether file is in text format or not
    if file.endswith(".vcf"):
        file_path1 = f"{path1}\{file}"
        file_path2 = f"{path2}\{file}"
        print(file_path1,"\n\n",file_path2)

# call read text file function
#data = read_text_file(path1,path2)
#print(read_text_file(path1,path2))
Final= pd.DataFrame(read_text_file(path1,path2))
print(Final)

and its giving me output without calling function and also gives me empty dataframe. Also i am not sure if result is showing all vcf files record or not.
Kindly help me to modify this code. I need my all result in dataframe so i can make plot easily
Folder 1 contains 3 test vcfs whereas folder 2 contains 4 test vcfs:

This is test 1 vcf file:
##fileformat=VCFv4.2
##ALT=
##FILTER=60.0">
##FILTER=
##FILTER=
##FILTER=
##FILTER=
##FILTER=
##FILTER=
##FILTER=
##FILTER=
##FORMAT=
##FORMAT=
##FORMAT=
##FORMAT=
##FORMAT=
##FORMAT=
##FORMAT=
##FORMAT=
##FORMAT=
##INFO=
##INFO=
##INFO=
##INFO=
##INFO=
##INFO=
##INFO=
##INFO=
##INFO=
##INFO=
##INFO=
##INFO=
##INFO=
##INFO=
##INFO=
##INFO=
##INFO=
##INFO=
##INFO=
##reference=file:///Users/sian_bray/Dropbox/Yant/Cochlearia/02_Data/References/pseudohap_cochlearia_135M_3kb.fasta
##source=SelectVariants
#CHROM  POS ID  REF ALT QUAL    FILTER  INFO    FORMAT  ALO_006 ALO_007 ALO_013 ALO_017 BEA_002 BEA_004 BEA_010 BRI_002 BRI_005 BRI_006 BRI_009 ELI_001 ELI_002 ELI_003 ELI_004 FRE_013 JON_001 KVA_002 KVA_003 KVA_009 KVA_010 LAB_004 LAB_300 LAB_400 LAB_500 LNL_001 LNL_002 LNL_003 LNL_008 MEL_001 MEL_002 NEN_001 NEN_003 NEN_200 NEN_300 ROT_004 ROT_006 ROT_007 ROT_013 RUZ_001 SAL_003 SAL_004 SAL_006 SKF_002 SKF_003 SKF_005 SKF_009 SKI_004 SKI_005 SKI_008 SKN_001 SKN_002 SKN_005 SKN_008 SPU_006 SPU_008 SPU_009 SPU_010 TET_002 TET_004 TET_006 TET_008 TRO_001 TRO_003 TRO_005 TRO_009 VAG_003 VAG_004 VAG_007 VAG_102 VEG_003 VEG_004 WOL_002 WOL_006 WOL_009 WOL_010
237 50041   .   G   T   4099.26 PASS    AC=35;AF=0.133;AN=264;BaseQRankSum=0.00;ClippingRankSum=0.00;DP=1228;ExcessHet=4.3562;FS=0.644;InbreedingCoeff=-0.0953;MQ=57.84;MQRankSum=0.00;QD=25.94;ReadPosRankSum=-1.570e-01;SOR=0.768 GT:AD:DP:GQ:PL  0/0/0/0:11,0:11:14:0,14,33,66,425   0/0/0/0:11,1:12:0:0,0,0,27,369  0/0/0/0:13,3:16:0:0,0,0,0,355   0/0/0/0:41,0:41:50:0,50,120,241,1800    0/0/1/1:5,8:13:1:293,15,0,1,170 0/0/0/1:5,3:8:0:104,0,0,10,191  0/0/0/0:8,0:8:10:0,10,24,48,272 0/1/1/1:1,8:9:11:318,33,11,0,26 ./././.:0,0 1/1/1/1:0,2:2:2:87,12,6,2,0 0/0/0/0:5,2:7:0:0,0,0,0,120 0/0/0/0:16,2:18:0:0,0,0,21,515  0/0/0/0:8,1:9:0:0,0,0,13,270    0/0/0/0:15,3:18:0:0,0,0,0,459   0/0/0/0:10,1:11:0:0,0,0,21,363  0/0/0/0/0/0:15,0:15:12:0,12,26,45,71,116,589    0/0/0/0/0/0:7,2:9:0:0,0,0,0,0,0,151 ./././.:0,0 0/0/0/0:14,0:14:14:0,14,33,66,495   0/0/0/0:1,0:1:1:0,1,3,6,42  0/0/0/0:9,0:9:10:0,10,24,48,360 0/0:21,2:23:14:0,14,726 0/0:14,2:16:11:0,11,452 0/0:14,2:16:0:0,0,467   0/0:23,9:32:0:0,0,527   0/0/0/0:10,0:10:12:0,12,30,60,359   0/0/0/0:11,1:12:0:0,0,0,27,404  0/0/0/0:13,0:13:16:0,16,39,78,526   0/0/0/0:20,3:23:0:0,0,0,2,655   1/1/1/1:0,4:4:5:155,24,12,5,0   0/0/0/0:2,3:5:0:0,0,0,0,8   0/0:23,2:25:0:0,0,798   0/0:12,2:14:0:0,0,381   0/0:14,0:14:36:0,36,540 0/0:24,0:24:63:0,63,945 0/0/0/0:8,1:9:0:0,0,0,12,245    0/0/0/0:27,0:27:32:0,32,78,157,1170 0/0/0/0:18,5:23:0:0,0,0,0,474   0/0/0/0:22,1:23:15:0,15,50,114,736  0/0:10,0:10:27:0,27,405 0/0:8,0:8:24:0,24,322   0/0:13,0:13:36:0,36,540 0/0:4,0:4:12:0,12,155   0/0/0/0/0/0:5,2:7:0:0,0,0,0,0,0,125 0/0/0/0/0/0:19,1:20:0:0,0,0,18,50,106,731   0/0/0/0/0/0:28,3:31:0:0,0,0,0,1,82,978  0/0/0/0/0/0:53,10:63:0:0,0,0,0,0,0,1843 0/0/0/0:1,1:2:0:0,0,0,0,4   1/1/1/1:0,22:22:27:917,132,66,27,0  ./././.:0,0 0/0/0/0:15,0:15:17:0,17,42,84,630   0/0/0/0:14,2:16:0:0,0,0,27,421  0/0/0/0:17,0:17:21:0,21,51,102,670  0/0/0/0:17,7:24:0:0,0,0,0,323   0/0:11,2:13:0:0,0,380

**Tags:** python, vcf

---

## 1 answers

### Answer by username_1 on 2022-01-23. Score: 2

*Answer from @user324810, converted from comment:*

There is an alternative than doing it with Python unless it is absolutely necessary. If you want to merge multiple samples into one, use [bcftools merge](http://samtools.github.io/bcftools/bcftools.html#merge). If you want simply to concatenate multiple VCFs, use [bcftools concatenate](https://samtools.github.io/bcftools/bcftools.html#concat). Once you obtain the output file, you can load the dataframe and plot it as you wish.


---

