<p>Just to mention I personally think there is a higher loop above the loop stated above.</p>
<p>I've not looked at each bit of code in detail but
there's a bug is on line 4:</p>
<pre><code>results = results.append({&quot;Gene&quot;: gene, &quot;Model&quot;: model, &quot;Enriched Pathway&quot;: Pathway_name}, ignore_index=True)
</code></pre>
<p>Firstly, this is fashionable coding because you are dumping straight into a <code>pandas</code> dataframe (described below). The correction is simply ... declare <code>pandas</code> in advance</p>
<pre><code>import pandas as pd
results = pd.DataFrame()
p_value = # code missing
if p_value &lt; 0.05:
    for gene in list(pathway):
        results.append({&quot;Gene&quot;: gene, &quot;Model&quot;: model, &quot;Enriched Pathway&quot;: Pathway_name}, ignore_index=True)
</code></pre>
<p>The thing is if there is a higher loop above <code>list(pathway)</code> then <code>results = results.append(....</code> will over-ride the last set of results. This would explain why results 'disappear' because if the last data in the loop is a blank - or very little output, all previous stuff gets deleted.</p>
<p>I personally would use a dictionary here. Thus</p>
<pre><code># declare the dictionary before all loops     
results = {&quot;Gene&quot;: [], &quot;Model&quot;: [], &quot;Enriched Pathway&quot;: []} 
p_value = # code missing
if p_value &lt; 0.05:
    for gene in list(pathway):
       genedict = {&quot;Gene&quot;: gene, &quot;Model&quot;: model, &quot;Enriched Pathway&quot;: Pathway_name}
       for k,v in mydicti.values():
           results[v].append(v)
</code></pre>
<p>... you could then transfer this to <code>pandas</code> once the loops are complete via,</p>
<pre><code>df = pd.DataFrame(genedict)
</code></pre>
<p>I do suspect there's other issues, but as a general rule <code>results = results.append(...</code> is better declaring <code>results</code> first and then appending to <code>results</code>, unless you've a very specific reason (e.g. writing each loop into a separate file). The <code>pandas</code> thing is kinda clever for big data sets, because <code>pandas</code> is fast, whilst dictionaries are slow - for big data (like &gt;&gt;500 000 big). The interchangeability between <code>dictionaries</code> and <code>pandas</code> means it's kinda easier conceptually (and code-wise) to switch at the coders convenience in my personal opinion.</p>

CC BY-SA 4.0
2023-04-05T23:31:20.090Z
2023-04-05T23:42:48.990Z
2023-04-05T23:42:48.990Z
M__
<p>I was doing <code>pathway enrichment analysis</code> using the below code</p>
<pre><code>p_value = scipy.stats.hypergeom.sf(len(intersection) - 1, geneset_size, len(pathway), len(hits))
        if p_value &lt; 0.05:
            for gene in list(pathway):
                results = results.append({&quot;Gene&quot;: gene, &quot;Model&quot;: model, &quot;Enriched Pathway&quot;: Pathway_name}, ignore_index=True)
</code></pre>
<p>where,</p>
<pre><code>hits = df[&quot;Gene&quot;][np.logical_and(df[&quot;Model&quot;] == model, df[&quot;p_value_anova&quot;] &lt; 0.05)]
intersection = pathway.intersection(hits)
Model = model name (like A, B, C) in input file
Gene = Gene from the input gene
Enriched pathway=Pathways that are enriched from the Pathway input file provided
</code></pre>
<p>Now the problem is that earlier this checking pathway enrichment on the basis of p_value in <code>hits</code> was working but now using cutoff (0.05) is resulting in loss of all pathways. I am looking to test the enrichment using the statistical test that will work on continous measure than binarizing the P-value.</p>

CC BY-SA 4.0
2023-04-05T20:32:05.760Z
Hi @Megha, do you have a small amount of sample data?
M__
<python><r><pathway><go-enrichment><pandas>
How to get enriched pathways in the data using continous statistic measure?
# How to get enriched pathways in the data using continous statistic measure?

(Asked by: username_0 on 2023-04-05)

I was doing pathway enrichment analysis using the below code
p_value = scipy.stats.hypergeom.sf(len(intersection) - 1, geneset_size, len(pathway), len(hits))
        if p_value < 0.05:
            for gene in list(pathway):
                results = results.append({"Gene": gene, "Model": model, "Enriched Pathway": Pathway_name}, ignore_index=True)

where,
hits = df["Gene"][np.logical_and(df["Model"] == model, df["p_value_anova"] < 0.05)]
intersection = pathway.intersection(hits)
Model = model name (like A, B, C) in input file
Gene = Gene from the input gene
Enriched pathway=Pathways that are enriched from the Pathway input file provided

Now the problem is that earlier this checking pathway enrichment on the basis of p_value in hits was working but now using cutoff (0.05) is resulting in loss of all pathways. I am looking to test the enrichment using the statistical test that will work on continous measure than binarizing the P-value.

**Tags:** python, r, pathway, go-enrichment, pandas
> **Comments:**
> **username_1**: Hi @Megha, do you have a small amount of sample data?  

---

## 1 answers

### Answer by username_1 on 2023-04-05. Score: 1 | Accepted answer

Just to mention I personally think there is a higher loop above the loop stated above.
I've not looked at each bit of code in detail but
there's a bug is on line 4:
results = results.append({"Gene": gene, "Model": model, "Enriched Pathway": Pathway_name}, ignore_index=True)

Firstly, this is fashionable coding because you are dumping straight into a pandas dataframe (described below). The correction is simply ... declare pandas in advance
import pandas as pd
results = pd.DataFrame()
p_value = # code missing
if p_value < 0.05:
    for gene in list(pathway):
        results.append({"Gene": gene, "Model": model, "Enriched Pathway": Pathway_name}, ignore_index=True)

The thing is if there is a higher loop above list(pathway) then results = results.append(.... will over-ride the last set of results. This would explain why results 'disappear' because if the last data in the loop is a blank - or very little output, all previous stuff gets deleted.
I personally would use a dictionary here. Thus
# declare the dictionary before all loops     
results = {"Gene": [], "Model": [], "Enriched Pathway": []} 
p_value = # code missing
if p_value < 0.05:
    for gene in list(pathway):
       genedict = {"Gene": gene, "Model": model, "Enriched Pathway": Pathway_name}
       for k,v in mydicti.values():
           results[v].append(v)

... you could then transfer this to pandas once the loops are complete via,
df = pd.DataFrame(genedict)

I do suspect there's other issues, but as a general rule results = results.append(... is better declaring results first and then appending to results, unless you've a very specific reason (e.g. writing each loop into a separate file). The pandas thing is kinda clever for big data sets, because pandas is fast, whilst dictionaries are slow - for big data (like >>500 000 big). The interchangeability between dictionaries and pandas means it's kinda easier conceptually (and code-wise) to switch at the coders convenience in my personal opinion.


---

