<p>I wrote the answer <a href="https://bioinformatics.stackexchange.com/a/13710/104">here</a> and while I'm not clear on all aspects of this question, I can offer two suggestions to reduce the memory footprint considerably while maintaining high speed.</p>
<p>First, rather than appending the records to a list you could use a generator that generates elements on-the-fly rather than storing them in memory, e.g.</p>
<pre><code>seen = set()

def yield_seqs():
    for record in SeqIO.parse(&quot;b4r2.fasta&quot;, &quot;fasta&quot;):  
        if record.seq not in seen:
            seen.add(record.seq)
            yield record

SeqIO.write(yield_seqs(), &quot;output.fasta&quot;, &quot;fasta&quot;)
</code></pre>
<p>Second you could save even more memory, <strong>but with the risk of false positives</strong>, by using a <a href="https://en.wikipedia.org/wiki/Bloom_filter" rel="nofollow noreferrer">bloom filter</a>, e.g. using <a href="https://github.com/hiway/python-bloom-filter" rel="nofollow noreferrer">python-bloom-filter</a>, just replace the set with <code>seen = BloomFilter(max_elements=10000, error_rate=0.001)</code></p>

CC BY-SA 4.0
2021-03-07T19:30:14.027Z
thanks for the generator hint.
pippo1980
CC BY-SA 4.0
2021-03-07T18:43:28.810Z
2021-03-07T18:43:28.810Z
Chris_Rands
<p>FIRST: I wasn't able to find a way to pass a SeqRecord object from a list to a file without saving it to a temp file first. That is without:</p>
<pre><code>for i , batch in enumerate(batch_iterator(data_input_iter , 1)):
        record_iter = batch 
......

SeqIO.write(record_iter, &quot;temp_file&quot;, &quot;fasta&quot;)
            
record_iter = SeqIO.read(&quot;temp_file&quot;, &quot;fasta&quot;)

</code></pre>
<p>the line:</p>
<pre><code> if record_iter.seq != record_output.seq  :

</code></pre>
<p>will throw:</p>
<pre><code>
AttributeError: 'list' object has no attribute 'seq'

</code></pre>
<p>After spending some timde on it I realized that:</p>
<pre><code>print('\n@@@@@@@@@@@@@@\n',type(record_iter),'\n',record_iter)
</code></pre>
<p>is returning:</p>
<pre><code>&lt;class 'list'&gt; 
</code></pre>
<p>but jus using :</p>
<pre><code>for i , batch in enumerate(batch_iterator(data_input_iter , 1)):
        record_iter = batch[0] 
</code></pre>
<p>now :</p>
<pre><code>print('\n@@@@@@@@@@@@@@\n',type(record_iter),'\n',record_iter)
</code></pre>
<p>is returning:</p>
<pre><code>&lt;class 'Bio.SeqRecord.SeqRecord'&gt; 
</code></pre>
<p>no need to save and read the Bio.SeqRecord.SeqRecord' anymore</p>
<p>so this is just half the question answered so no need to save and read the record on file</p>
<p>but even if speed increases the memory usage goes up :</p>
<pre><code>data_input_iter :  48 

Run time is 1.442115306854248
Maximum memory usage: 110.09765625
<span class="math-container">```</span>
</code></pre>

CC BY-SA 4.0
2021-04-06T14:09:06.083Z
2021-11-02T22:19:03.440Z
2021-11-02T22:19:03.440Z
pippo1980
<p>I need help with my Biopython code, I am kind of new to Python and Biopython so please don't be harsh on me.</p>
<p>Everything started from here: <a href="https://stackoverflow.com/questions/66462611/remove-duplicated-sequences-in-fasta-with-python/66463550#66463550">Remove duplicated sequences in FASTA with Python</a></p>
<p>I answered copying code from here: <a href="https://bioinformatics.stackexchange.com/questions/13708/remove-redundant-sequences-from-fasta-file-in-python">Remove Redundant Sequences from FASTA file in Python</a> but I am still bugged about the OP's first question: <a href="https://stackoverflow.com/questions/66462611/remove-duplicated-sequences-in-fasta-with-python/66463550#66463550">Remove duplicated sequences in FASTA with Python</a></p>
<p>So I decided to try to minimize the code footprint on RAM. I ended up copying from Biopython documentation <a href="https://biopython.org/wiki/Split_large_file" rel="nofollow noreferrer">Split large file</a> and producing:</p>
<pre><code>#!/usr/bin/env python3
# -*- coding: utf-8 -*-
&quot;&quot;&quot;
Created on Thu Nov  11 17:40:03 2020

@author: Pietro


&quot;&quot;&quot;

from memory_profiler import memory_usage

import memory_profiler 


def batch_iterator(iterator, batch_size):
    &quot;&quot;&quot; from https://biopython.org/wiki/Split_large_file
    &quot;&quot;&quot;
    entry = True  # Make sure we loop once
    while entry:
        batch = []
        while len(batch) &lt; batch_size:
            try:
                entry = iterator.__next__()
            except StopIteration:
                entry = None
            if entry is None:
                # End of file
                break
            batch.append(entry)
        if batch:
            yield batch


# @profile
def f():
    
    from Bio import SeqIO 
    
    import time
    
    import sys
   
    start = time.time() 

    data_input_iter = SeqIO.parse(&quot;Newtextfile-03&quot;, &quot;fasta&quot;)
    print('data_input_iter : ',sys.getsizeof(data_input_iter),'\n')
    

    c=0
    for i , batch in enumerate(batch_iterator(data_input_iter , 1)):
        record_iter = batch 

        val = True
        c += 1
        if c == 1:
            with open(&quot;removed-double_Newtextfile-03&quot;, 'w+') as output_file:
                    SeqIO.write(record_iter, output_file, &quot;fasta&quot;)
                    # print('copy   : ',(record_iter))
                    output_file.flush()
            output_file.close()
        
        else:
            
            SeqIO.write(record_iter, &quot;temp_file&quot;, &quot;fasta&quot;)
            
            record_iter = SeqIO.read(&quot;temp_file&quot;, &quot;fasta&quot;)
            
            data_output_iter = SeqIO.parse(&quot;removed-double_Newtextfile-03&quot;, &quot;fasta&quot;)
            
            d = 0
            
            try:
                for i , batch in enumerate(batch_iterator(data_output_iter  , 1)):
                        SeqIO.write(batch, &quot;temp_file2&quot;, &quot;fasta&quot;)
                    
                        record_output = SeqIO.read(&quot;temp_file2&quot;, &quot;fasta&quot;)
                    
                        d += 1
                        if record_iter.seq != record_output.seq  :
                            # print('pass   : ',record_iter.name)
                            pass
                        else:
                            val = False
                            # print('false')
                            break

                
            except:
                print('ERROR !!!!!!!!!!!')
                quit()

            if val:
                        # print('copy   : ',record_iter.name)
                        with open(&quot;removed-double_Newtextfile-03&quot;, 'a+') as output_file:
                            output_file.write(str(record_iter.format(&quot;fasta&quot;)))
                        output_file.close()
            del data_output_iter
    
    end = time.time()
    
    print(f&quot;Run time is {(end- start)}&quot;) 


mem_usage = memory_usage(f)
# print('Memory usage (in chunks of .1 seconds): %s' % mem_usage)
print('Maximum memory usage: %s' % max(mem_usage))


# f()


</code></pre>
<p>but I have to major problems :</p>
<p>FIRST: I wasn't able to find a way to pass a SeqRecord object from a list to a file without saving it to a temp file first. That is without:</p>
<pre><code>for i , batch in enumerate(batch_iterator(data_input_iter , 1)):
        record_iter = batch 
......

SeqIO.write(record_iter, &quot;temp_file&quot;, &quot;fasta&quot;)
            
record_iter = SeqIO.read(&quot;temp_file&quot;, &quot;fasta&quot;)

</code></pre>
<p>the line:</p>
<pre><code> if record_iter.seq != record_output.seq  :

</code></pre>
<p>will throw:</p>
<pre><code>AttributeError: 'list' object has no attribute 'format'

AttributeError: 'list' object has no attribute 'seq'

</code></pre>
<p>you get first error and the if remove format from:</p>
<pre><code>output_file.write(str(record_iter.format(&quot;fasta&quot;)))
</code></pre>
<p>and #quit() you get the second error
How should I cope with this ? I mean is there a way to bypass the save and read lines ?</p>
<p>SECOND : and more important. I am sure my code doesn't use a set and a list .... and that's why is extremely slow compared to the copied <a href="https://bioinformatics.stackexchange.com/questions/13708/remove-redundant-sequences-from-fasta-file-in-python">Remove Redundant Sequences from FASTA file in Python</a> posted below too. But how does it use even more RAM that the fastest code ? I tried to minimize the memory footprint but wasn't able !! Am I using a bad way to try to calculate the amount of RAM used ? And does Biopythom SeqIO.parse object already is the optimal choice for , lets say &quot;2/4 Gb size&quot; fasta files on a laptop with just 4Gb of RAM (remember the original OP post  <a href="https://stackoverflow.com/questions/66462611/remove-duplicated-sequences-in-fasta-with-python/66463550#66463550">Remove duplicated sequences in FASTA with Python</a>). Maybe my test file is not big enough ? Does anybody have a 2GB fasta file with a dozen of duplicates in it ?</p>
<p><a href="https://stackoverflow.com/questions/66462611/remove-duplicated-sequences-in-fasta-with-python/66463550#66463550">Remove duplicated sequences in FASTA with Python</a> memory usage:</p>
<pre><code>size of parser :  48
memory seen :  32984
memory records :  2888
Run time is 0.0008047382036844889
Maximum memory usage: 55.359375
</code></pre>
<p>my code memory usage and speed:</p>
<pre><code>data_input_iter :  48 

Run time is 37.781240463256836
Maximum memory usage: 56.40234375
</code></pre>
<p>My fasta test set is: <a href="https://easyupload.io/z2yss1" rel="nofollow noreferrer">https://easyupload.io/z2yss1</a></p>
<p>Biopython method to remove duplicated from  <a href="https://stackoverflow.com/questions/66462611/remove-duplicated-sequences-in-fasta-with-python/66463550#66463550">Remove duplicated sequences in FASTA with Python</a>:</p>
<pre><code>#!/usr/bin/env python3
# -*- coding: utf-8 -*-
&quot;&quot;&quot;
copied from https://bioinformatics.stackexchange.com/questions/13708/remove-redundant-sequences-from-fasta-file-in-python

uses set() instead of list for seen

should work faster for big files

&quot;&quot;&quot;

from memory_profiler import memory_usage

def f():
    
    import sys

    from Bio import SeqIO
    import time
    
    start = time.time() 
    
    seen = set()
    records = []
    
    
    parser = SeqIO.parse(&quot;Newtextfile-03&quot;, &quot;fasta&quot;)
    for record in parser:  
        # print('running')
        if str(record.seq) not in seen:
            seen.add(str(record.seq))
            records.append(record)
            # print('copying')
    
    
    print('size of parser : ',sys.getsizeof(parser) )
    print('memory seen : ',sys.getsizeof(seen))
    print('memory records : ',sys.getsizeof(records))
    
    #writing to a fasta file
    SeqIO.write(records, &quot;removed-double_Newtextfile-03-from-002&quot;, &quot;fasta&quot;)
    end = time.time()
    
    print(f&quot;Run time is {(end- start)/60}&quot;) 

mem_usage = memory_usage(f)
# print('Memory usage (in chunks of .1 seconds): %s' % mem_usage)
print('Maximum memory usage: %s' % max(mem_usage))

# f()
</code></pre>

<python><fasta><seqio><biopython>
Remove Redundant Sequences from FASTA file with Biopython reducing memory footprint
# Remove Redundant Sequences from FASTA file with Biopython reducing memory footprint

(Asked by: username_0 on 2021-03-07)

I need help with my Biopython code, I am kind of new to Python and Biopython so please don't be harsh on me.
Everything started from here: Remove duplicated sequences in FASTA with Python
I answered copying code from here: Remove Redundant Sequences from FASTA file in Python but I am still bugged about the OP's first question: Remove duplicated sequences in FASTA with Python
So I decided to try to minimize the code footprint on RAM. I ended up copying from Biopython documentation Split large file and producing:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Created on Thu Nov  11 17:40:03 2020

@author: Pietro


"""

from memory_profiler import memory_usage

import memory_profiler 


def batch_iterator(iterator, batch_size):
    """ from https://biopython.org/wiki/Split_large_file
    """
    entry = True  # Make sure we loop once
    while entry:
        batch = []
        while len(batch) < batch_size:
            try:
                entry = iterator.__next__()
            except StopIteration:
                entry = None
            if entry is None:
                # End of file
                break
            batch.append(entry)
        if batch:
            yield batch


# @profile
def f():
    
    from Bio import SeqIO 
    
    import time
    
    import sys
   
    start = time.time() 

    data_input_iter = SeqIO.parse("Newtextfile-03", "fasta")
    print('data_input_iter : ',sys.getsizeof(data_input_iter),'\n')
    

    c=0
    for i , batch in enumerate(batch_iterator(data_input_iter , 1)):
        record_iter = batch 

        val = True
        c += 1
        if c == 1:
            with open("removed-double_Newtextfile-03", 'w+') as output_file:
                    SeqIO.write(record_iter, output_file, "fasta")
                    # print('copy   : ',(record_iter))
                    output_file.flush()
            output_file.close()
        
        else:
            
            SeqIO.write(record_iter, "temp_file", "fasta")
            
            record_iter = SeqIO.read("temp_file", "fasta")
            
            data_output_iter = SeqIO.parse("removed-double_Newtextfile-03", "fasta")
            
            d = 0
            
            try:
                for i , batch in enumerate(batch_iterator(data_output_iter  , 1)):
                        SeqIO.write(batch, "temp_file2", "fasta")
                    
                        record_output = SeqIO.read("temp_file2", "fasta")
                    
                        d += 1
                        if record_iter.seq != record_output.seq  :
                            # print('pass   : ',record_iter.name)
                            pass
                        else:
                            val = False
                            # print('false')
                            break

                
            except:
                print('ERROR !!!!!!!!!!!')
                quit()

            if val:
                        # print('copy   : ',record_iter.name)
                        with open("removed-double_Newtextfile-03", 'a+') as output_file:
                            output_file.write(str(record_iter.format("fasta")))
                        output_file.close()
            del data_output_iter
    
    end = time.time()
    
    print(f"Run time is {(end- start)}") 


mem_usage = memory_usage(f)
# print('Memory usage (in chunks of .1 seconds): %s' % mem_usage)
print('Maximum memory usage: %s' % max(mem_usage))


# f()



but I have to major problems :
FIRST: I wasn't able to find a way to pass a SeqRecord object from a list to a file without saving it to a temp file first. That is without:
for i , batch in enumerate(batch_iterator(data_input_iter , 1)):
        record_iter = batch 
......

SeqIO.write(record_iter, "temp_file", "fasta")
            
record_iter = SeqIO.read("temp_file", "fasta")


the line:
 if record_iter.seq != record_output.seq  :


will throw:
AttributeError: 'list' object has no attribute 'format'

AttributeError: 'list' object has no attribute 'seq'


you get first error and the if remove format from:
output_file.write(str(record_iter.format("fasta")))

and #quit() you get the second error
How should I cope with this ? I mean is there a way to bypass the save and read lines ?
SECOND : and more important. I am sure my code doesn't use a set and a list .... and that's why is extremely slow compared to the copied Remove Redundant Sequences from FASTA file in Python posted below too. But how does it use even more RAM that the fastest code ? I tried to minimize the memory footprint but wasn't able !! Am I using a bad way to try to calculate the amount of RAM used ? And does Biopythom SeqIO.parse object already is the optimal choice for , lets say "2/4 Gb size" fasta files on a laptop with just 4Gb of RAM (remember the original OP post  Remove duplicated sequences in FASTA with Python). Maybe my test file is not big enough ? Does anybody have a 2GB fasta file with a dozen of duplicates in it ?
Remove duplicated sequences in FASTA with Python memory usage:
size of parser :  48
memory seen :  32984
memory records :  2888
Run time is 0.0008047382036844889
Maximum memory usage: 55.359375

my code memory usage and speed:
data_input_iter :  48 

Run time is 37.781240463256836
Maximum memory usage: 56.40234375

My fasta test set is: https://easyupload.io/z2yss1
Biopython method to remove duplicated from  Remove duplicated sequences in FASTA with Python:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
copied from https://bioinformatics.stackexchange.com/questions/13708/remove-redundant-sequences-from-fasta-file-in-python

uses set() instead of list for seen

should work faster for big files

"""

from memory_profiler import memory_usage

def f():
    
    import sys

    from Bio import SeqIO
    import time
    
    start = time.time() 
    
    seen = set()
    records = []
    
    
    parser = SeqIO.parse("Newtextfile-03", "fasta")
    for record in parser:  
        # print('running')
        if str(record.seq) not in seen:
            seen.add(str(record.seq))
            records.append(record)
            # print('copying')
    
    
    print('size of parser : ',sys.getsizeof(parser) )
    print('memory seen : ',sys.getsizeof(seen))
    print('memory records : ',sys.getsizeof(records))
    
    #writing to a fasta file
    SeqIO.write(records, "removed-double_Newtextfile-03-from-002", "fasta")
    end = time.time()
    
    print(f"Run time is {(end- start)/60}") 

mem_usage = memory_usage(f)
# print('Memory usage (in chunks of .1 seconds): %s' % mem_usage)
print('Maximum memory usage: %s' % max(mem_usage))

# f()

**Tags:** python, fasta, seqio, biopython

---

## 2 answers

### Answer by username_1 on 2021-03-07. Score: 2

I wrote the answer here and while I'm not clear on all aspects of this question, I can offer two suggestions to reduce the memory footprint considerably while maintaining high speed.
First, rather than appending the records to a list you could use a generator that generates elements on-the-fly rather than storing them in memory, e.g.
seen = set()

def yield_seqs():
    for record in SeqIO.parse("b4r2.fasta", "fasta"):  
        if record.seq not in seen:
            seen.add(record.seq)
            yield record

SeqIO.write(yield_seqs(), "output.fasta", "fasta")

Second you could save even more memory, but with the risk of false positives, by using a bloom filter, e.g. using python-bloom-filter, just replace the set with seen = BloomFilter(max_elements=10000, error_rate=0.001)

> **Comments:**
> **username_0**: thanks for the generator hint.  

---

### Answer by username_0 on 2021-04-06. Score: 0

FIRST: I wasn't able to find a way to pass a SeqRecord object from a list to a file without saving it to a temp file first. That is without:
for i , batch in enumerate(batch_iterator(data_input_iter , 1)):
        record_iter = batch 
......

SeqIO.write(record_iter, "temp_file", "fasta")
            
record_iter = SeqIO.read("temp_file", "fasta")


the line:
 if record_iter.seq != record_output.seq  :


will throw:

AttributeError: 'list' object has no attribute 'seq'


After spending some timde on it I realized that:
print('\n@@@@@@@@@@@@@@\n',type(record_iter),'\n',record_iter)

is returning:
 

but jus using :
for i , batch in enumerate(batch_iterator(data_input_iter , 1)):
        record_iter = batch[0] 

now :
print('\n@@@@@@@@@@@@@@\n',type(record_iter),'\n',record_iter)

is returning:
 

no need to save and read the Bio.SeqRecord.SeqRecord' anymore
so this is just half the question answered so no need to save and read the record on file
but even if speed increases the memory usage goes up :
data_input_iter :  48 

Run time is 1.442115306854248
Maximum memory usage: 110.09765625
```


---

