<p>Alright, so there are a number of problematic patterns in your code - as far as I understand what you are trying to do. Next time, try to post a reproducible example that people can use and more people will be willing to help.</p>
<h3>L18-21:</h3>
<pre class="lang-py prettyprint-override"><code>combination_labels = []
combination_counts = []

for lineage in lineages:
</code></pre>
<p>Declaring these two lists <em>before</em> the loop, then appending all your combination labels inside the loop means that these two lists will contain all labels and counts for all runs (all your lineages). If I am understanding your intentions correctly this is probably not something you want. The simple fix here is to just move them into the for-loop.</p>
<h3>L29-30:</h3>
<pre class="lang-py prettyprint-override"><code>for mutation in combination:
    combination_df = combination_df[combination_df[mutation] == 1]
</code></pre>
<p>Here you are iteratively checking if the value for each mutation is set to '1', but you are also <em>overwriting</em> your <code>combination_df</code> variable at each iteration of the loop. After this loop, <code>combination_df</code> will be whatever mutation last was set to '1' in your current combination. I will get back to solving this in my solution below.</p>
<h3>L37-44:</h3>
<pre class="lang-py prettyprint-override"><code>combination_labels.append('_'.join(combination))
combination_counts.append(combination_df.shape[0])

out_df = pd.DataFrame({'combination':combination_labels,
                   'count':combination_counts})
out_df['percentage'] = (out_df['count'] / df.shape[0]) * 100
out_df = out_df.sort_values('percentage', ascending = False)   

out_df.to_csv('Results_2/' + lineage.replace(&quot;.csv&quot;, &quot;&quot;) + '_3.csv',
         header = True, 
         index = False)
</code></pre>
<p>Since this part is again scoped inside the for-loop, each iteration (each combination) will overwrite the same <code>'Results_2/' + lineage.replace(&quot;.csv&quot;, &quot;&quot;) + '_3.csv'</code> file. This you also want to move out of the <code>for combination in ...</code> loop.</p>
<hr />
<p>A big concern is also performance. You have 2000-something combinations and 100,000 lines in your file. As you loop over combinations, you also process the DataFrame each time, when this is a proble you can solve in one iteration. It's not much data so it's still fine, but these habits are still good to develop early.</p>
<p>Now here's an idea for a rewrite that should work better. I don't have a csv from you to test with, so there might (proabably are) still things to fix, but hopefully it'll give you a starting point:</p>
<pre class="lang-py prettyprint-override"><code>mutations = ['A222V', 'D614G', 'E484Q', 'E780Q', 'G476S', 'L18F', 'N439K',
             'S477', 'S477N', 'T478I', 'V483A']

combinations = []
for M in range(1, len(mutations)+1):
    for subset in itertools.combinations(mutations, M):
        combinations.append(subset)  
        
combinations = ['_'.join(sorted(x)) for x in combinations]

root = &quot;C:&quot;

os.chdir(root)

lineages = os.listdir('Results')

for lineage in lineages:

    out_df = pd.DataFrame({&quot;Count&quot;: 0}, index=combinations)
    
    df = pd.read_csv('Results/' + lineage, index_col=&quot;lineage&quot;).dropna()
    
    for index, row in df.iterrows():
        combination = '_'.join(sorted(df.columns[row==1]))
        out_df.loc[combination, &quot;Count&quot;] += 1

    out_df['Percentage'] = (out_df['Count'] / df.shape[0]) * 100
    out_df = out_df.sort_values('Percentage', ascending = False)   

    out_df.to_csv('Results_2/' + lineage.replace(&quot;.csv&quot;, &quot;&quot;) + '_3.csv', header = True, 
        index = True)
</code></pre>

CC BY-SA 4.0
2020-10-24T22:15:39.237Z
Absolutely amazingly detailed reply, thank you so much! It's worked exactly how I needed it! Not sure how to mark this as answered, as I'm very new to this forum - perhaps a mod could help here?
Thank you again, sincerely!
Theo Jones
CC BY-SA 4.0
2020-10-25T06:54:38.240Z
[This](https://stackoverflow.com/help/someone-answers) might help you with that
Bastian Schiffthaler
CC BY-SA 4.0
2020-10-24T14:42:32.840Z
2020-10-24T14:54:55.073Z
2020-10-24T14:54:55.073Z
Bastian Schiffthaler
<pre><code>mutations = ['A222V', 'D614G', 'E484Q', 'E780Q', 'G476S', 'L18F', 'N439K',
             'S477', 'S477N', 'T478I', 'V483A']

combinations = []
for M in range(1, len(mutations)+1):
    for subset in itertools.combinations(mutations, M):
        combinations.append(subset)
        
        combinations = ['_'.join(sorted(x)) for x in combinations]
        combinations = [x.split('_') for x in list(set(combinations))]

root = &quot;C:&quot;

os.chdir(root)

lineages = os.listdir('Results')

combination_labels = []
combination_counts = []

for lineage in lineages:
    
    df = pd.read_csv('Results/' + lineage).dropna()
    
    for combination in combinations:
    
            combination_df = df[list(df)]
        
            for mutation in combination:
                combination_df = combination_df[combination_df[mutation] == 1]
    
        #print(combination_df.shape[0])
        
            combination_labels.append('_'.join(combination))
            combination_counts.append(combination_df.shape[0])
        
            out_df = pd.DataFrame({'combination':combination_labels,
                               'count':combination_counts})
            out_df['percentage'] = (out_df['count'] / df.shape[0]) * 100
            out_df = out_df.sort_values('percentage', ascending = False)   
        
            out_df.to_csv('Results_2/' + lineage.replace(&quot;.csv&quot;, &quot;&quot;) + '_3.csv',
                     header = True, 
                     index = False)
</code></pre>
<p>The input CSV</p>
<pre><code>lineage,A222V,D614G,E484Q,E780Q,G476S,L18F,N439K,S477,S477N,T478I,V483A
417941,0,1,0,0,0,0,0,0,0,0,0
</code></pre>
<p>Output CSV</p>
<pre><code>combination,count,percentage
D614G,87355,90.7084929856806
</code></pre>
<p>My above code is used to count all occurences of combinations of spike protein mutations</p>
<ul>
<li>to determine the genotypes.
However, looking at the values processed into the .csv file, i can see ' D614G' is being counted even when
it is in combination with other combinations.</li>
</ul>
<p>My question is how do i ensure once something has been counted (a row in the imported .csv file)
it is not counted for a second time?</p>
<p>OR possibly how can I edit this code to prevent counting of singular mutations even when presented within a combination?</p>
<p>Thanks in advance</p>

CC BY-SA 4.0
2020-10-23T15:48:09.997Z
Nice to see a traditional python question based on pandas. The top of your code is chopped off. Could you inset it please?
M__
CC BY-SA 4.0
2020-10-23T16:53:44.513Z
Could you also add what a row in your CSV looks like?
Bastian Schiffthaler
CC BY-SA 4.0
2020-10-23T17:01:31.223Z
Input / output csv?
Theo Jones
CC BY-SA 4.0
2020-10-23T23:33:37.420Z
The input CSV in your Result folder is only two lines per file?
zorbax
CC BY-SA 4.0
2020-10-24T10:35:19.720Z
The input file has about 97,000 lines of data - each for a strain of SARSCOV2. The output file has around 54 lines, as genotypes but this should be quite a few less
Theo Jones
<python><genomics><covid-19><sars-cov-2><mutations>
>My counter is counting genotypic combination occurences more than once, how do I ensure it counts one combination and doesnt count it again?
# >My counter is counting genotypic combination occurences more than once, how do I ensure it counts one combination and doesnt count it again?

(Asked by: username_0 on 2020-10-23)

mutations = ['A222V', 'D614G', 'E484Q', 'E780Q', 'G476S', 'L18F', 'N439K',
             'S477', 'S477N', 'T478I', 'V483A']

combinations = []
for M in range(1, len(mutations)+1):
    for subset in itertools.combinations(mutations, M):
        combinations.append(subset)
        
        combinations = ['_'.join(sorted(x)) for x in combinations]
        combinations = [x.split('_') for x in list(set(combinations))]

root = "C:"

os.chdir(root)

lineages = os.listdir('Results')

combination_labels = []
combination_counts = []

for lineage in lineages:
    
    df = pd.read_csv('Results/' + lineage).dropna()
    
    for combination in combinations:
    
            combination_df = df[list(df)]
        
            for mutation in combination:
                combination_df = combination_df[combination_df[mutation] == 1]
    
        #print(combination_df.shape[0])
        
            combination_labels.append('_'.join(combination))
            combination_counts.append(combination_df.shape[0])
        
            out_df = pd.DataFrame({'combination':combination_labels,
                               'count':combination_counts})
            out_df['percentage'] = (out_df['count'] / df.shape[0]) * 100
            out_df = out_df.sort_values('percentage', ascending = False)   
        
            out_df.to_csv('Results_2/' + lineage.replace(".csv", "") + '_3.csv',
                     header = True, 
                     index = False)

The input CSV
lineage,A222V,D614G,E484Q,E780Q,G476S,L18F,N439K,S477,S477N,T478I,V483A
417941,0,1,0,0,0,0,0,0,0,0,0

Output CSV
combination,count,percentage
D614G,87355,90.7084929856806

My above code is used to count all occurences of combinations of spike protein mutations

to determine the genotypes.
However, looking at the values processed into the .csv file, i can see ' D614G' is being counted even when
it is in combination with other combinations.

My question is how do i ensure once something has been counted (a row in the imported .csv file)
it is not counted for a second time?
OR possibly how can I edit this code to prevent counting of singular mutations even when presented within a combination?
Thanks in advance

**Tags:** python, genomics, covid-19, sars-cov-2, mutations
> **Comments:**
> **username_1**: Nice to see a traditional python question based on pandas. The top of your code is chopped off. Could you inset it please?  
> **username_2**: Could you also add what a row in your CSV looks like?  
> **username_0**: Input / output csv?  
> **username_3**: The input CSV in your Result folder is only two lines per file?  
> **username_0**: The input file has about 97,000 lines of data - each for a strain of SARSCOV2. The output file has around 54 lines, as genotypes but this should be quite a few less  

---

## 1 answers

### Answer by username_2 on 2020-10-24. Score: 3 | Accepted answer

Alright, so there are a number of problematic patterns in your code - as far as I understand what you are trying to do. Next time, try to post a reproducible example that people can use and more people will be willing to help.
L18-21:
combination_labels = []
combination_counts = []

for lineage in lineages:

Declaring these two lists before the loop, then appending all your combination labels inside the loop means that these two lists will contain all labels and counts for all runs (all your lineages). If I am understanding your intentions correctly this is probably not something you want. The simple fix here is to just move them into the for-loop.
L29-30:
for mutation in combination:
    combination_df = combination_df[combination_df[mutation] == 1]

Here you are iteratively checking if the value for each mutation is set to '1', but you are also overwriting your combination_df variable at each iteration of the loop. After this loop, combination_df will be whatever mutation last was set to '1' in your current combination. I will get back to solving this in my solution below.
L37-44:
combination_labels.append('_'.join(combination))
combination_counts.append(combination_df.shape[0])

out_df = pd.DataFrame({'combination':combination_labels,
                   'count':combination_counts})
out_df['percentage'] = (out_df['count'] / df.shape[0]) * 100
out_df = out_df.sort_values('percentage', ascending = False)   

out_df.to_csv('Results_2/' + lineage.replace(".csv", "") + '_3.csv',
         header = True, 
         index = False)

Since this part is again scoped inside the for-loop, each iteration (each combination) will overwrite the same 'Results_2/' + lineage.replace(".csv", "") + '_3.csv' file. This you also want to move out of the for combination in ... loop.

A big concern is also performance. You have 2000-something combinations and 100,000 lines in your file. As you loop over combinations, you also process the DataFrame each time, when this is a proble you can solve in one iteration. It's not much data so it's still fine, but these habits are still good to develop early.
Now here's an idea for a rewrite that should work better. I don't have a csv from you to test with, so there might (proabably are) still things to fix, but hopefully it'll give you a starting point:
mutations = ['A222V', 'D614G', 'E484Q', 'E780Q', 'G476S', 'L18F', 'N439K',
             'S477', 'S477N', 'T478I', 'V483A']

combinations = []
for M in range(1, len(mutations)+1):
    for subset in itertools.combinations(mutations, M):
        combinations.append(subset)  
        
combinations = ['_'.join(sorted(x)) for x in combinations]

root = "C:"

os.chdir(root)

lineages = os.listdir('Results')

for lineage in lineages:

    out_df = pd.DataFrame({"Count": 0}, index=combinations)
    
    df = pd.read_csv('Results/' + lineage, index_col="lineage").dropna()
    
    for index, row in df.iterrows():
        combination = '_'.join(sorted(df.columns[row==1]))
        out_df.loc[combination, "Count"] += 1

    out_df['Percentage'] = (out_df['Count'] / df.shape[0]) * 100
    out_df = out_df.sort_values('Percentage', ascending = False)   

    out_df.to_csv('Results_2/' + lineage.replace(".csv", "") + '_3.csv', header = True, 
        index = True)

> **Comments:**
> **username_0**: Absolutely amazingly detailed reply, thank you so much! It's worked exactly how I needed it! Not sure how to mark this as answered, as I'm very new to this forum - perhaps a mod could help here? Thank you again, sincerely!  
> **username_2**: [This](https://stackoverflow.com/help/someone-answers) might help you with that  

---

