Q: How to compare two files (~50MB) fast? I am using Python and want to compare two files and find the lines that are unique to each of them. I'm doing as shown below, but it is too slow.
f1 = open(file1)
text1Lines = f1.readlines()
f2 = open(file2)
text2Lines = f2.readlines()
diffInstance=difflib.Differ()
diffList = list(diffInstance.compare(text1Lines, text2Lines))

How can I speed it up considerably?

A: You might read and compare files simultaneously, instead of storing them in memory. The following snippet makes a lot of unrealistic assumptions (ie the files are of the same length and no line is present in the same file twice), but it illustrates the idea:
unique_1 = []
unique_2 = []
for line_1 in handle_1:
    #  Reading line from the 1st file and checking if we have already seen them in in the 2nd
    if line_1 in unique_2:
        unique_2.remove(line)
    #  If line was unique, remember it
    else:
        unique_1.append(line)
    #  The same, only files are the other way
    line_2 = handle_2.readline()
    if line_2 in unique_1:
        unique_1.remove(line)
    else:
        unique_2.append(line)

print('\n'.join(unique_1))
print('\n'.join(unique_2))

Sure, it smells of reinventing bicycle, but you might get better performance by using simple algorithm instead of fancy diff-building and distance calculations of difflib. Alternatively, if you are absolutely sure your files will never be too big to fit in memory (not the safest assumption, to be honest), you can just use the set difference:
set1=set()
set2=set()
for line in handle_1:
    set1.add(line)
for line in handle_2:
    set2.add(line)
set1_uniques = set1.difference(set2)
set2_uniques = set2.difference(set1)


A: your code may have some bugs... you can only find the difference in same line. if 2 files have different number of lines or the data is unsorted your code will have problem...next is my code:
f1 = open('a.txt')
text1Lines = f1.readlines()
f2 = open('b.txt')
text2Lines = f2.readlines()
set1 = set(text1Lines)
set2 = set(text2Lines)
diffList = (set1|set2)-(set1&set2)

