<p>I'd start by saying that Python is inherently slow and this feels like a case of perhaps unnecessary optimisation. With that in mind here are my suggested (untested) optimisations of @gaspanic code:</p>
<pre><code>pos = {2, 5, 10, 11, 14} # make pos a set for O(1) member testing, and don't define inside the loop
for seq in (seqE3,seqE4,seqE5): # Define seq here rather than inside the loop, and use an immutable tuple instead of a mutable list  
    new_seq = []
    count = 0
    for s in seq:
        if s == &quot;-&quot;:
            new_seq.append(s)
        else:
            count += 1
            new_seq.append('Z' if count in pos else s) # use ternary operator
</code></pre>

CC BY-SA 4.0
2022-08-04T11:42:54.017Z
Thanks, the tuple did add speed, particularly tuple the iterator output. Using a `set` (the `{}` brackets) was a good solution and reduced runtime by ~0.5 seconds, which I'd not considered. It was better than `tuple` in this example. Repositioning `pos` adds a tiny amount, but agree its better practice and depends on the size of `pos` (in this case its small). Happy to discuss `Python`'s slow speed in a summary.
M__
CC BY-SA 4.0
2022-08-03T21:34:59.983Z
2022-08-03T21:34:59.983Z
Chris_Rands
<p>Huge thanks to @Chris_Rands, this solution increases runtime by ~ 0.5 seconds uniformly across all coded solutions, so this code has a runtime of just over 3 seconds runtime on the benchmark machine. I incorporated my solution with @Chris_Rands solution and the final runtime I got was:</p>
<p>@gaspanic</p>
<blockquote>
<p>1.7 - 1.8 seconds</p>
</blockquote>
<p>This being twice the fastest speed in the question and four times the speed of my raw code.</p>
<p>My code</p>
<blockquote>
<p>4.4 - 4.5 seconds</p>
</blockquote>
<p>However a unique combination is needed:</p>
<ol>
<li><code>tuple</code> (Chris Rands)</li>
<li><code>set</code> - important (Chris Rands)</li>
<li>an <code>iterator</code></li>
<li>a <code>list comprehension</code></li>
</ol>
<p>In particular 3. and 4. are needed in combination. This is explained below.</p>
<pre><code>def iterit(x):
    seq = 'M----KFLAFLCLLGFANAQ-------------D-GKCG----TLSNKSPS--------------------K'
    return tuple(''.join(list(repeat(seq, x))))

def mIterit(seqE3,seqE4,seqE5):
    pos = {2, 5, 10, 11, 14}
    for seqbit in (seqE3,seqE4,seqE5):
        seq = seqbit
        count = 0
        # loc = 0
        for s in seq:
            if s == &quot;-&quot;:
                yield '-'
            else:
                count += 1
                if count in pos:
                    yield 'Z'
                else:
                    yield s

f __name__ == '__main__':
    
    seqE3 = iterit(2800) # length is 201600
    seqE4 = iterit(28000) # length is 2016000
    seqE5 = iterit(280000) # length is 20160000
    
    listComp = [i for i in mIterit(seqE3,seqE4,seqE5)]
    
</code></pre>
<p>The list comprehension (variable <code>listComp</code>) is the very last line and is crucial to the solution otherwise there is little increase in speed to @Chris_Rand's code. This alternative is to replace the last line with:</p>
<pre><code>newseq = list()
for i in mIterit(seqE3,seqE4,seqE5):
    newseq.append(i)   
</code></pre>
<p>A simple:</p>
<pre><code>for i in mIterit(seqE3,seqE4,seqE5):
    pass
</code></pre>
<p>Is the same speed as the list comprehension, but its of no value because the output isn't saved to anything.</p>
<p>I was impressed by the interaction between the <code>iterator</code> and <code>list comprehension</code>. The other notable influence on speed is the <code>set</code> described by @Chris_Rand.</p>
<p>How does it work? <code>tuples</code> do increase speed by removing some functional of a <code>list</code> (immutability). Here the gain is small. Placing the <code>pos</code> variable outside the loop speaks for itself (0.1 seconds gain). <code>set</code> is interesting.</p>
<p>The <code>iterator</code> spews out a stream of results via <code>yield</code> which is where it attains its speed. The problem is capturing that stream outside the <code>iterator</code> is expensive. However <code>list comprehensions</code> are faster than standard loops and in this scenerio carry little/no detectable overhead. I was impressed. Up to this point I thought <code>list comprehensions</code> were a bit faster and a lot 'cooler' than loops. I hadn't understood there was a synergy with an <code>iterator</code>.</p>
<p>Is it all worth it? <code>Python</code> can easily achieve faster run times than <code>Java</code> using Cython but I will place the discussion in the <em>chat</em> forum next week.</p>

CC BY-SA 4.0
2022-08-05T02:28:03.897Z
2022-08-05T05:29:05.230Z
2022-08-05T05:29:05.230Z
M__
<p>I discussed a question with @gaspanic <a href="https://bioinformatics.stackexchange.com/questions/14625/python-biopython-replace-amino-acid-residue-on-msa-with-z-from-a-list-of-una">Python/Biopython - Replace amino acid residue on MSA with &quot;z&quot; from a list of unaligned positions</a>
. The issue emerged was speeding up lists in Python. The solutions we discussed were</p>
<ol>
<li>Raw code</li>
<li>Predefining a list size</li>
<li><code>deque</code></li>
</ol>
<div class="s-table-container">
<table class="s-table">
<thead>
<tr>
<th style="text-align: left;">1. @gaspanic</th>
<th style="text-align: left;">my code</th>
<th style="text-align: left;">2. @gaspanic</th>
<th style="text-align: left;">my code</th>
<th style="text-align: left;">3. @gaspanic</th>
<th style="text-align: left;">my code</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: left;">3.45-3.48</td>
<td style="text-align: left;">7.17-7.2</td>
<td style="text-align: left;">4.38-4.85</td>
<td style="text-align: left;">7.09-7.23</td>
<td style="text-align: left;">3.44-3.53</td>
<td style="text-align: left;">7.21-7.39</td>
</tr>
</tbody>
</table>
</div>
<p>Run timings are seconds (3 replicated) above with @gaspanic's code being faster.</p>
<p>What improvements would speed this up?</p>
<p>Here I extended the data set</p>
<pre><code>from itertools import repeat
def iterit(x):
    seq = 'M----KFLAFLCLLGFANAQ-------------D-GKCG----TLSNKSPS--------------------K'
    return ''.join(list(repeat(seq, x)))

if __name__ == '__main__':
    seqE3 = iterit(2800) # length is 201600
    seqE4 = iterit(28000) # length is 2016000
    seqE5 = iterit(280000) # length is 20160000
</code></pre>
<p>@gaspanic code</p>
<pre><code>for seqbit in [seqE3,seqE4,seqE5]:
    seq = seqbit
    pos = [2, 5, 10, 11, 14]    
    new_seq = []
    count = 0
    for s in seq:
        if s == &quot;-&quot;:
            new_seq.append(s)
        else:
            count += 1
            if count in pos:
                new_seq.append('Z')
            else:
                new_seq.append(s)
   # print(''.join(new_seq))
</code></pre>
<p>My code:</p>
<pre><code># same variables and loop structure
newseq = list()
count = 0
for n,s in enumerate(prot):    
    if str(s) == '-':
        count += 1
        s = '-'
    else:
        if (n + 1 - count) in pos:
            s = 'Z'
    newseq.append(s)
# print(''.join(newseq))
</code></pre>
<p>Predefining lists code (additional counter needed for @gaspanic's code:</p>
<pre><code># same variables and loop structure
new_seq = [] * len(seq)
count = 0
for n,s in enumerate(prot):    
    if str(s) == '-':
        count += 1
        s = '-'
    else:
        if (n + 1 - count) in pos:
            s = 'Z'
    newseq[n] = s
</code></pre>
<p>Deque code simply any of the above substituting</p>
<pre><code>newseq = list() 
</code></pre>
<p>with</p>
<pre><code>from collections import deque 
newseq = deque()
</code></pre>
<p>Suggestions for efficiency improvements welcome.</p>
<p>A solution in <code>Cython</code> code would be cool. <code>PyPI</code> is not sought here.</p>

<python><phylogenetics><phylogeny><algorithms><benchmarking>
Improving list speed: glycosylation example
# Improving list speed: glycosylation example

(Asked by: username_0 on 2022-08-03)

I discussed a question with @gaspanic Python/Biopython - Replace amino acid residue on MSA with "z" from a list of unaligned positions
. The issue emerged was speeding up lists in Python. The solutions we discussed were

Raw code
Predefining a list size
deque





1. @gaspanic
my code
2. @gaspanic
my code
3. @gaspanic
my code




3.45-3.48
7.17-7.2
4.38-4.85
7.09-7.23
3.44-3.53
7.21-7.39




Run timings are seconds (3 replicated) above with @gaspanic's code being faster.
What improvements would speed this up?
Here I extended the data set
from itertools import repeat
def iterit(x):
    seq = 'M----KFLAFLCLLGFANAQ-------------D-GKCG----TLSNKSPS--------------------K'
    return ''.join(list(repeat(seq, x)))

if __name__ == '__main__':
    seqE3 = iterit(2800) # length is 201600
    seqE4 = iterit(28000) # length is 2016000
    seqE5 = iterit(280000) # length is 20160000

@gaspanic code
for seqbit in [seqE3,seqE4,seqE5]:
    seq = seqbit
    pos = [2, 5, 10, 11, 14]    
    new_seq = []
    count = 0
    for s in seq:
        if s == "-":
            new_seq.append(s)
        else:
            count += 1
            if count in pos:
                new_seq.append('Z')
            else:
                new_seq.append(s)
   # print(''.join(new_seq))

My code:
# same variables and loop structure
newseq = list()
count = 0
for n,s in enumerate(prot):    
    if str(s) == '-':
        count += 1
        s = '-'
    else:
        if (n + 1 - count) in pos:
            s = 'Z'
    newseq.append(s)
# print(''.join(newseq))

Predefining lists code (additional counter needed for @gaspanic's code:
# same variables and loop structure
new_seq = [] * len(seq)
count = 0
for n,s in enumerate(prot):    
    if str(s) == '-':
        count += 1
        s = '-'
    else:
        if (n + 1 - count) in pos:
            s = 'Z'
    newseq[n] = s

Deque code simply any of the above substituting
newseq = list() 

with
from collections import deque 
newseq = deque()

Suggestions for efficiency improvements welcome.
A solution in Cython code would be cool. PyPI is not sought here.

**Tags:** python, phylogenetics, phylogeny, algorithms, benchmarking

---

## 2 answers

### Answer by username_1 on 2022-08-03. Score: 2 | Accepted answer

I'd start by saying that Python is inherently slow and this feels like a case of perhaps unnecessary optimisation. With that in mind here are my suggested (untested) optimisations of @gaspanic code:
pos = {2, 5, 10, 11, 14} # make pos a set for O(1) member testing, and don't define inside the loop
for seq in (seqE3,seqE4,seqE5): # Define seq here rather than inside the loop, and use an immutable tuple instead of a mutable list  
    new_seq = []
    count = 0
    for s in seq:
        if s == "-":
            new_seq.append(s)
        else:
            count += 1
            new_seq.append('Z' if count in pos else s) # use ternary operator

> **Comments:**
> **username_0**: Thanks, the tuple did add speed, particularly tuple the iterator output. Using a `set` (the `{}` brackets) was a good solution and reduced runtime by ~0.5 seconds, which I'd not considered. It was better than `tuple` in this example. Repositioning `pos` adds a tiny amount, but agree its better practice and depends on the size of `pos` (in this case its small). Happy to discuss `Python`'s slow speed in a summary.  

---

### Answer by username_0 on 2022-08-05. Score: 1

Huge thanks to @Chris_Rands, this solution increases runtime by ~ 0.5 seconds uniformly across all coded solutions, so this code has a runtime of just over 3 seconds runtime on the benchmark machine. I incorporated my solution with @Chris_Rands solution and the final runtime I got was:
@gaspanic

1.7 - 1.8 seconds

This being twice the fastest speed in the question and four times the speed of my raw code.
My code

4.4 - 4.5 seconds

However a unique combination is needed:

tuple (Chris Rands)
set - important (Chris Rands)
an iterator
a list comprehension

In particular 3. and 4. are needed in combination. This is explained below.
def iterit(x):
    seq = 'M----KFLAFLCLLGFANAQ-------------D-GKCG----TLSNKSPS--------------------K'
    return tuple(''.join(list(repeat(seq, x))))

def mIterit(seqE3,seqE4,seqE5):
    pos = {2, 5, 10, 11, 14}
    for seqbit in (seqE3,seqE4,seqE5):
        seq = seqbit
        count = 0
        # loc = 0
        for s in seq:
            if s == "-":
                yield '-'
            else:
                count += 1
                if count in pos:
                    yield 'Z'
                else:
                    yield s

f __name__ == '__main__':
    
    seqE3 = iterit(2800) # length is 201600
    seqE4 = iterit(28000) # length is 2016000
    seqE5 = iterit(280000) # length is 20160000
    
    listComp = [i for i in mIterit(seqE3,seqE4,seqE5)]
    

The list comprehension (variable listComp) is the very last line and is crucial to the solution otherwise there is little increase in speed to @Chris_Rand's code. This alternative is to replace the last line with:
newseq = list()
for i in mIterit(seqE3,seqE4,seqE5):
    newseq.append(i)   

A simple:
for i in mIterit(seqE3,seqE4,seqE5):
    pass

Is the same speed as the list comprehension, but its of no value because the output isn't saved to anything.
I was impressed by the interaction between the iterator and list comprehension. The other notable influence on speed is the set described by @Chris_Rand.
How does it work? tuples do increase speed by removing some functional of a list (immutability). Here the gain is small. Placing the pos variable outside the loop speaks for itself (0.1 seconds gain). set is interesting.
The iterator spews out a stream of results via yield which is where it attains its speed. The problem is capturing that stream outside the iterator is expensive. However list comprehensions are faster than standard loops and in this scenerio carry little/no detectable overhead. I was impressed. Up to this point I thought list comprehensions were a bit faster and a lot 'cooler' than loops. I hadn't understood there was a synergy with an iterator.
Is it all worth it? Python can easily achieve faster run times than Java using Cython but I will place the discussion in the chat forum next week.


---

