Q: merge pandas csv in different directories I have csv files with same names in different directories and i want to merge them as a single csv.

dir1  
abcd__diff.csv  
efgh__diff.csv  

dir2  
abcd_diffhere.csv  
efgh_diffhere.csv

operation  
dir1/abcd_diff.csv join dir2/abcd_diffhere.csv  
dir1/efgh_diff.csv join dir2/efgh_diffhere.csv

I want to merge them using a common field. I can use pandas join operator but what is the most efficient way to search and map the filenames across directories.
I split the filenames using character __  giving the list of files with same names in each directory. I can do two for loops and iterate but that would not be efficient as I have around 200 files.

A: Consider zip() on the two file name lists where a dictionary of dataframes are appended (avoiding 200 separate objects). Keys to dictionary would be unique filenames shared by each pair. Below assumes filename lists do not have directories just base names of files.
import os
...
dfDict = {}
for i, j in zip(dir1list, dir2list):
   temp1 = pd.read_csv(os.path.join(dir1, i))
   temp2 = pd.read_csv(os.path.join(dir2, j))

   key = i.replace('.csv','')
   dfDict[key] = pd.merge(temp1, temp2, on='commonfield')

Should lists be unordered and even of different lengths, consider a list comprehension comparing the two and creating a list of tuple pairs of items matched by first 4 characters: abcd, efgh, ... Then loop the list for the data frame merges
dir1list = ['abcd__diff.csv','efgh__diff.csv']
dir2list = ['abcd_diffhere.csv','efgh_diffhere.csv']

allfiles = [(i,j) for i in dir1list for j in dir2list if i[:4] == j[:4]]

dfDict = {}
for file in allfiles:
   temp1 = pd.read_csv(os.path.join(dir1, file[0]))
   temp2 = pd.read_csv(os.path.join(dir2, file[1]))

   key = i[:4]
   dfDict[key] = pd.merge(temp1, temp2, on='commonfield')


A: Situate files like this
files1 = []
files2 = []
dir1path = './dir1/'
dir2path = './dir2/'
dir1 = os.listdir(dir1path)
dir2 = os.listdir(dir2path)

for f in dir1:
    fmatch = f.split('.csv')[0] + 'here.csv'    
    if fmatch in dir2:
        files1.append(f)
        files2.append(fmatch)

files1 = [os.path.join(dir1path, f) for f in files1]
files2 = [os.path.join(dir2path, f) for f in files2]

fpairs = zip(files1, files2)

Then create list of dataframes
# edit this lambda function accroding to your needs
# it will have to be specific to your csv formatting
rf = lambda f: pd.read_csv(f)

dfs = [rf(fp[0]).merge(rf(fp[1]), on='Key') for fp in fpairs]

