Q: How to print significantly correlating candidates using Scipy I have a large data frame (15000,3000) that look like below:
row_name,col1,col2,col3
row1,22,33,44
row2,46,87,90
row3,23,51,12

I perform spearman correction on rows using Scipy as below:
import pandas as pd
from scipy import stats

df =pd.read_csv('my_dataframe.csv',index_col=0)
r, p =stats.spearmanr(df.T)

I have access to correlation values and their relative p-values but what I want to have is that - by giving specific cut off for r and p values I want to have the pairs get printed as below:
e.g. r > 0 and p < 0.7
row1,row2,r-value,p-value
row1,row3,r-value,p-value

Considering that I look at different Stackoverflow questions regarding this (like this one) I still have not been able to find a simple solution for it. What could be your suggestions?
Thank you in advance.

A: I wrote this script myself so that helps to get the output as I want. The output needs to be post-processed.
import pandas as pd
from scipy import stats

df =pd.read_csv('my_dataframe.csv',index_col=0)

lst1=[]
lst2=[]
for i in df.T.columns:
    lst1.append(i)
    lst2.append(i)

for i in lst1:
    for j in lst2:
        if i!=j:
            r, p = stats.spearmanr(df.T[i], df.T[j])
            #if r > 0 and p < 0.7: condition can be added here
            Data=str(i) + ',' + str(j) + ',' + str(r) + ',' + str(p)
            print(Data)

Output:
row1,row2,1.0,0.0
row1,row3,-0.5,0.6666666666666667
row2,row1,1.0,0.0
row2,row3,-0.5,0.6666666666666667
row3,row1,-0.5,0.6666666666666667
row3,row2,-0.5,0.6666666666666667

