<p>What I'm not clear about is the definition of <code>y</code> - thats crucial to understanding what the code does. Just to note, although you've called the dataframe <code>data</code>, I've called it <code>df</code> (traditional shorthand). In terms of speed there will not be much in it between vectorised <code>pandas</code> and <code>numpy</code>. Explicitly using <code>iterrows</code> or else <code>apply</code> is when it slows down a lot.</p>
<p>Possibilities of the definition of <code>y</code>.</p>
<p>1.</p>
<pre><code>y = df.drop(['Model', 'foldchange'], axis=1)
</code></pre>
<p>I think the above is more likely, or ...</p>
<p>2.</p>
<pre><code>y = df['fold change']
</code></pre>
<p>What I think is occurring is <code>np.mean</code> is performing a boolean operation on the feature columns. What I didn't understand though was this ...</p>
<pre><code>abs(np.mean(y[x])[0] - np.mean(y[~x])[0])
</code></pre>
<p>.. this is because for example model 'IM00980' albeit the mean is 0 this will always be 1 under the above equation. This is because <code>y[~x]</code> will be 1 for all values and thus abs(0 - 1) is 1. Equally if all columns in a row were 1 the output would also be 1 because abs(1-0) is 1. Anyway, what I would do is explicitly perform the operation within <code>pandas</code> rather than coercing <code>np</code> into a boolean operation. This is</p>
<pre><code>import pandas as pd

df = pd.read_csv(r'/pathtofile/megha.csv', delimiter=&quot;\t&quot;)

feature = ['A1BG', 'A1CF', 'A2M', 'A2ML1', 'A3GALT2', 'A4GALT']
df=df.drop(['foldchange'],axis=1)
feature_level_partitions = ['L', 'G']
filtered = df[feature].isin(feature_level_partitions).eq(True).mul(1)
df2 = df['Model'].to_frame().join(filtered)
df2['mean'] = df2[feature].mean(axis=1)
print(df2)
</code></pre>
<p>Here ~x would simply <code>1 - mean</code> as a new pandas column, i.e. <code>df2['~mean'] = df2['mean'] - 1</code></p>
<p>The output of that code is:</p>
<div class="s-table-container">
<table class="s-table">
<thead>
<tr>
<th>-</th>
<th>Model</th>
<th>A1BG</th>
<th>A1CF</th>
<th>A2M</th>
<th>A2ML1</th>
<th>A3GALT2</th>
<th>A4GALT</th>
<th>mean.</th>
</tr>
</thead>
<tbody>
<tr>
<td>0</td>
<td>IM00499</td>
<td>0</td>
<td>1.</td>
<td>1.</td>
<td>1.</td>
<td>0.</td>
<td>0.</td>
<td>0.500000</td>
</tr>
<tr>
<td>1</td>
<td>IM00499.1</td>
<td>1</td>
<td>0</td>
<td>0</td>
<td>1</td>
<td>0</td>
<td>0</td>
<td>0.333333</td>
</tr>
<tr>
<td>2</td>
<td>IM00980</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0.000000</td>
</tr>
<tr>
<td>3</td>
<td>IM01529</td>
<td>0</td>
<td>0</td>
<td>1</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0.166667</td>
</tr>
<tr>
<td>4</td>
<td>IM01624</td>
<td>0</td>
<td>0</td>
<td>1</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0.166667</td>
</tr>
<tr>
<td>5</td>
<td>IM01240</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>1</td>
<td>0.166667</td>
</tr>
<tr>
<td>6</td>
<td>IM00807</td>
<td>0</td>
<td>1</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0.166667</td>
</tr>
<tr>
<td>7</td>
<td>IM00999</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0.000000</td>
</tr>
<tr>
<td>8</td>
<td>IM00983</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0.000000</td>
</tr>
<tr>
<td>9</td>
<td>IM01076</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0.000000</td>
</tr>
<tr>
<td>10</td>
<td>IM00803</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0.000000</td>
</tr>
<tr>
<td>11</td>
<td>IM00125</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0.000000</td>
</tr>
<tr>
<td>12</td>
<td>IM00998</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>1</td>
<td>0</td>
<td>0.166667</td>
</tr>
<tr>
<td>13</td>
<td>IM00997</td>
<td>0</td>
<td>1</td>
<td>1</td>
<td>1</td>
<td>0</td>
<td>1</td>
<td>0.666667</td>
</tr>
<tr>
<td>14</td>
<td>IM00996</td>
<td>0</td>
<td>1</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>1</td>
<td>0.333333</td>
</tr>
</tbody>
</table>
</div><hr />
<p>If this is correct it provides the basis to move forward for all other operations. Basically, what I'm saying is I think is good to break open the <code>numpy</code> into explicit <code>pandas</code> operations, what is happening is there's a mix of <code>pandas</code> and <code>numpy</code> and I would recommend going down one route or the other.</p>

CC BY-SA 4.0
2023-02-18T18:05:16.127Z
2023-02-18T18:16:48.997Z
2023-02-18T18:16:48.997Z
M__
<p>My data = <code>data</code></p>
<pre><code>Model   A1BG    A1CF    A2M A2ML1   A3GALT2 A4GALT  foldchange
IM00499 N   L   L   G   N   N   0.285744
IM00499.1   L   N   D   G   D   D   0.005046
IM00980 N   N   N   N   N   N   -0.068949
IM01529 N   N   L   N   N   N   0.065914
IM01624 A   N   L   N   N   N   0.302100
IM01240 N   D   N   D   N   G   -0.428324
IM00807 N   L   A   N   D   N   0.257222
IM00999 N   N   N   N   N   D   0.056268
IM00983 N   N   N   N   N   N   0.107456
IM01076 N   N   D   A   N   N   0.218301
IM00803 N   D   N   N   N   D   -0.479368
IM00125 N   N   A   N   N   A   0.270428
IM00998 D   N   N   N   G   N   0.142748
IM00997 N   L   G   G   N   G   -0.154540
IM00996 N   L   N   N   N   L   -1.302023
</code></pre>
<p>The above data frame is divided into 5 categories = A, N, L, G and D</p>
<p><code>feature = ['A1BG', 'A1CF', 'A2M', 'A2ML1', 'A3GALT2', 'A4GALT']</code></p>
<p>Feature_level_partitions are arrays of combination of classification values</p>
<p><code>feature_level_partitions = ['A', 'N' , 'L',  'G',  'D'], ['D'], ['L', 'G']</code></p>
<p>I was using the below <code>list comprehension</code> to calculate <code>Hedge's G</code>.</p>
<pre><code>        hedges_g = np.array(
            [
                Hedges_g(data[feature].isin(feature_level_partition), y)
                for feature_level_partition in feature_level_partitions
            ],
            dtype=np.float64,
        )
</code></pre>
<p><code>Hedges_g</code> is defined as follows:</p>
<pre><code>def Hedges_g(x, y):
    difference_of_means = abs(np.mean(y[x])[0] - np.mean(y[~x])[0])
    s_pooled_weighted = math.sqrt((((n1 - 1) * s1 ** 2) + ((n2 - 1) * s2 ** 2)) / (n1 + n2 - 2))
    return difference_of_means / s_pooled_weighted
</code></pre>
<p>I am interested in knowing how many models there are within each category. So for example if I divide up the data into [&quot;D&quot;,&quot;L&quot;] on one side, and the rest on the other, I want to know how many models have either &quot;D&quot; OR &quot;L&quot; and how many models have any of the other categories and then don't calculate <code>Hegde's G</code> if the number on either side is less than 2.</p>
<p>To answer the above query I tried the following code</p>
<pre><code>[data[feature].isin(feature_level_partition).sum()
    for feature_level_partition in feature_level_partitions]
</code></pre>
<p>The above code gave me the number of models with the <code>feature_level_partitions</code>
However I also want to know the number of models without the <code>feature_level_partitions</code></p>
<p>I also want to create a python function which calculates both above numbers in one line and fit this code as a <code>if statement</code> in my above <code>list comprehension</code> function</p>

CC BY-SA 4.0
2023-02-19T19:35:35.313Z
What you might think about @Megha is defining a new question in the context of both the comprehension and the models by defining a criteria for including 'foldchange' parameter.
M__
<python><pandas>
Python list comprehension to calculate the set of categories for each row
# Python list comprehension to calculate the set of categories for each row

(Asked by: username_0 on 2023-02-14)

My data = data
Model   A1BG    A1CF    A2M A2ML1   A3GALT2 A4GALT  foldchange
IM00499 N   L   L   G   N   N   0.285744
IM00499.1   L   N   D   G   D   D   0.005046
IM00980 N   N   N   N   N   N   -0.068949
IM01529 N   N   L   N   N   N   0.065914
IM01624 A   N   L   N   N   N   0.302100
IM01240 N   D   N   D   N   G   -0.428324
IM00807 N   L   A   N   D   N   0.257222
IM00999 N   N   N   N   N   D   0.056268
IM00983 N   N   N   N   N   N   0.107456
IM01076 N   N   D   A   N   N   0.218301
IM00803 N   D   N   N   N   D   -0.479368
IM00125 N   N   A   N   N   A   0.270428
IM00998 D   N   N   N   G   N   0.142748
IM00997 N   L   G   G   N   G   -0.154540
IM00996 N   L   N   N   N   L   -1.302023

The above data frame is divided into 5 categories = A, N, L, G and D
feature = ['A1BG', 'A1CF', 'A2M', 'A2ML1', 'A3GALT2', 'A4GALT']
Feature_level_partitions are arrays of combination of classification values
feature_level_partitions = ['A', 'N' , 'L',  'G',  'D'], ['D'], ['L', 'G']
I was using the below list comprehension to calculate Hedge's G.
        hedges_g = np.array(
            [
                Hedges_g(data[feature].isin(feature_level_partition), y)
                for feature_level_partition in feature_level_partitions
            ],
            dtype=np.float64,
        )

Hedges_g is defined as follows:
def Hedges_g(x, y):
    difference_of_means = abs(np.mean(y[x])[0] - np.mean(y[~x])[0])
    s_pooled_weighted = math.sqrt((((n1 - 1) * s1 ** 2) + ((n2 - 1) * s2 ** 2)) / (n1 + n2 - 2))
    return difference_of_means / s_pooled_weighted

I am interested in knowing how many models there are within each category. So for example if I divide up the data into ["D","L"] on one side, and the rest on the other, I want to know how many models have either "D" OR "L" and how many models have any of the other categories and then don't calculate Hegde's G if the number on either side is less than 2.
To answer the above query I tried the following code
[data[feature].isin(feature_level_partition).sum()
    for feature_level_partition in feature_level_partitions]

The above code gave me the number of models with the feature_level_partitions
However I also want to know the number of models without the feature_level_partitions
I also want to create a python function which calculates both above numbers in one line and fit this code as a if statement in my above list comprehension function

**Tags:** python, pandas
> **Comments:**
> **username_1**: What you might think about @Megha is defining a new question in the context of both the comprehension and the models by defining a criteria for including 'foldchange' parameter.  

---

## 1 answers

### Answer by username_1 on 2023-02-18. Score: 1 | Accepted answer

What I'm not clear about is the definition of y - thats crucial to understanding what the code does. Just to note, although you've called the dataframe data, I've called it df (traditional shorthand). In terms of speed there will not be much in it between vectorised pandas and numpy. Explicitly using iterrows or else apply is when it slows down a lot.
Possibilities of the definition of y.
1.
y = df.drop(['Model', 'foldchange'], axis=1)

I think the above is more likely, or ...
2.
y = df['fold change']

What I think is occurring is np.mean is performing a boolean operation on the feature columns. What I didn't understand though was this ...
abs(np.mean(y[x])[0] - np.mean(y[~x])[0])

.. this is because for example model 'IM00980' albeit the mean is 0 this will always be 1 under the above equation. This is because y[~x] will be 1 for all values and thus abs(0 - 1) is 1. Equally if all columns in a row were 1 the output would also be 1 because abs(1-0) is 1. Anyway, what I would do is explicitly perform the operation within pandas rather than coercing np into a boolean operation. This is
import pandas as pd

df = pd.read_csv(r'/pathtofile/megha.csv', delimiter="\t")

feature = ['A1BG', 'A1CF', 'A2M', 'A2ML1', 'A3GALT2', 'A4GALT']
df=df.drop(['foldchange'],axis=1)
feature_level_partitions = ['L', 'G']
filtered = df[feature].isin(feature_level_partitions).eq(True).mul(1)
df2 = df['Model'].to_frame().join(filtered)
df2['mean'] = df2[feature].mean(axis=1)
print(df2)

Here ~x would simply 1 - mean as a new pandas column, i.e. df2['~mean'] = df2['mean'] - 1
The output of that code is:




-
Model
A1BG
A1CF
A2M
A2ML1
A3GALT2
A4GALT
mean.




0
IM00499
0
1.
1.
1.
0.
0.
0.500000


1
IM00499.1
1
0
0
1
0
0
0.333333


2
IM00980
0
0
0
0
0
0
0.000000


3
IM01529
0
0
1
0
0
0
0.166667


4
IM01624
0
0
1
0
0
0
0.166667


5
IM01240
0
0
0
0
0
1
0.166667


6
IM00807
0
1
0
0
0
0
0.166667


7
IM00999
0
0
0
0
0
0
0.000000


8
IM00983
0
0
0
0
0
0
0.000000


9
IM01076
0
0
0
0
0
0
0.000000


10
IM00803
0
0
0
0
0
0
0.000000


11
IM00125
0
0
0
0
0
0
0.000000


12
IM00998
0
0
0
0
1
0
0.166667


13
IM00997
0
1
1
1
0
1
0.666667


14
IM00996
0
1
0
0
0
1
0.333333




If this is correct it provides the basis to move forward for all other operations. Basically, what I'm saying is I think is good to break open the numpy into explicit pandas operations, what is happening is there's a mix of pandas and numpy and I would recommend going down one route or the other.


---

