<p>An <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4720993/" rel="nofollow noreferrer">UpSet Plot</a> is a useful way of visualising these kind of multi-dimensional overlaps. There's an <a href="https://cran.r-project.org/web/packages/UpSetR/README.html" rel="nofollow noreferrer">R package</a> for producing them, with a reasonable set of examples. </p>

<p>The input is a binary matrix indicating set membership:</p>

<pre><code>         h2    h4    h6...
GeneA     0     1     0
GeneB     1     1     0
GeneC     0     1     1
...
</code></pre>

<p>And the output is a pair of bar charts, showing the size of various sets, with a dot matrix showing the set membership. An example from the UpSetR documentation:</p>

<pre><code>library(UpSetR)
mutations &lt;- read.csv(system.file("extdata", "mutations.csv", 
                      package = "UpSetR"), header=T, sep = ",")
upset(mutations, sets = c("PTEN", "TP53", "EGFR", "PIK3R1", "RB1"),
      sets.bar.color = "#56B4E9", order.by = "freq", 
      empty.intersections = "on")
</code></pre>

<p><a href="https://i.sstatic.net/wSfK4.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/wSfK4.png" alt="Example UpSet Plot"></a></p>

CC BY-SA 4.0
2018-06-26T14:41:21.180Z
2018-06-26T14:41:21.180Z
sjcockell
<p>I would simply do this...
</p>

<pre><code># sepcify your timepoints, each timepoint is a list of genes!!
timepoints &lt;- list(h2,h4,h6,h8, h10,h12,h14,h16)
# initialize an empty array that you will fill with your timepoints
vect = vector()
# go over all combinations and fill the array with the set intersection values you want (I guess...)
for (i in timepoints) {
  for (j in timepoints) {
    vect &lt;- c(vect, length(intersect(i,j))/length(i))}}
# reshape your array to a more ordered matrix  
your_matrix &lt;- matrix(vect, nrow=length(timepoints))
# assign columns and rows names
colnames(your_matrix) &lt;- c("h2","h4","h6","h8","h10","h12","h14","h16")
rownames(your_matrix) &lt;- c("h2","h4","h6","h8","h10","h12","h14","h16")
</code></pre>

CC BY-SA 4.0
2018-06-24T18:42:58.863Z
2018-06-24T18:42:58.863Z
aerijman
<p>EDIT: Sorry, I did not see the R tag and wrote the code in bash. I still left the bash example below and added an R version.</p>

<p>Depends on what you want to do, but there are two common ways in which you could store this. Either as wide format: </p>

<pre><code>      h2  h4  h6 ...
Gene1 1   1   0
Gene2 0   1   1
Gene3 1   1   1
...
</code></pre>

<p>Where 1 denotes presence and 0 absence, or as long format:</p>

<pre><code>      timepoint
Gene1 h2
Gene1 h4
Gene2 h4
Gene2 h6
Gene3 h2
Gene3 h4
Gene3 h6
...
</code></pre>

<p>The best format probably depends on what you want to do. Since you did not specify a method/language, I took the liberty to write a few lines of bash to go from the gene lists to the matrices. 
Bash examples assume your files are named h2, h4 etc and are tab separated with gene names in the first column. R examples assume <code>tables</code> is a list of your data frames and <code>timepoints</code> is a vector <code>c("h2", "h4", ...)</code>.</p>

<p><strong>Generate table in wide format:</strong></p>

<ol>
<li>Concatenate gene names of all tables to a separate file</li>
<li>Extract unique values in this file</li>
<li>Perform outer join between the separate file and each table</li>
</ol>

<p><em>R:</em></p>

<pre><code>all_names &lt;- c()
for (tp in timepoints){
  all_names &lt;- append(all_names, tables[[tp]]$genes)
}
all_names &lt;- unique(all_names)

wide_table &lt;- matrix(nrow=length(all_names), ncol=(1+length(timepoints)))
wide_table &lt;- as.data.frame(wide_table)
colnames(wide_table) &lt;- append("genes", timepoints)

for (tp in timepoints){
 tmp &lt;- tables[[tp]]
 tmp$tp &lt;- 1
 tmp &lt;- merge(wide_table, tmp, by="genes", all=T)
 wide_table$tp &lt;- tmp$tp
}
</code></pre>

<p><em>Bash:</em></p>

<p>Note: you will might need to add headers to this table</p>

<pre><code>for timefile in $yourfiles;do
  # make a file with all gene names
  awk -v tp=$timefile '{print $1}' &gt;&gt; names_file
done

sort names_file | uniq &gt; tmp &amp;&amp; mv tmp names_file

for timefile in $yourfiles;do
  join -a1 -a2  -j 1 -o 0,1.2,2.2 -e 0 $names_file &lt;(awk '{print$1,1}' $timefile | sort)
done
</code></pre>

<p><strong>Generate table in long format:</strong></p>

<ol>
<li>For each table, add a column with timepoint</li>
<li>Concatenate all tables</li>
</ol>

<p><em>R:</em></p>

<pre><code>df &lt;- data.frame()
for (tp in timepoints){
  tables[[tp]][[timepoint]] &lt;- tp
  df &lt;- rbind(df, tables[[tp]] )
}
</code></pre>

<p><em>Bash:</em></p>

<pre><code>   for timefile in $yourfiles;do
      awk -v tp=$timefile '{print $1,timefile}' &gt;&gt; long_table
    done 
</code></pre>

<p>EDIT2: To follow up on this, here is a simple way (using the package <a href="https://www.tidyverse.org/" rel="nofollow noreferrer">dplyr</a>) to count the number of common genes between all possible combination of time points. Assuming you have your data in long format:</p>

<pre><code>df &lt;- data.frame(genes=c("G1","G1","G2","G3","G3"), tp=c("h2","h4","h2","h4","h6"))

  genes tp
1    G1 h2
2    G1 h4
3    G2 h2
4    G3 h4
5    G3 h6
</code></pre>

<p>You can get the co-occurrences using :</p>

<pre><code>library(dplyr)
# Make timepoint ordinal factor
df &lt;- mutate(df, tp = factor(tp, ordered=T))
df &lt;- df %&gt;% 
  # Compute all pairwise co-occurrences
  full_join(df, by="genes") %&gt;% 
  group_by(tp.x, tp.y) %&gt;% 
  summarise(N_genes = length(unique(genes))) %&gt;%
  # Exclude self pairs (gene in h2 and h2)
  filter(tp.x != tp.y) %&gt;% 
  # Check for duplicate pairs in reverse order (h2-h4 and h4-h2)
  mutate(comb=ifelse(tp.x &lt; tp.y, 
                     yes = paste(tp.x, tp.y), 
                     no  = paste(tp.y, tp.x)))
# Remove the duplicate pairs
df &lt;- df[which(!duplicated(df$comb)),]
</code></pre>

<p>And visualise it using for example a heatmap:
(ofc it looks stupid because I have a tiny sample where not all combinations are even present)</p>

<pre><code># Visualize number of genes present in each combination
library(ggplot2)
ggplot(df, aes(x=tp.x, y=tp.y, fill=N_genes)) + geom_tile()
</code></pre>

CC BY-SA 4.0
2018-06-21T17:41:29.120Z
excuse me, i did not quite understand. i have 8 separate lists of genes so how i know about 0 and 1 (absence and presence of a genes among the lists)?
Zizogolu
CC BY-SA 4.0
2018-06-21T17:45:49.877Z
@FereshTeh 0 and 1 were just placeholder values to give the info "geneY is present (1) / absent (0) at timepoint hX" . You could have expression values, ratios, pvalues or whatever variable instead of 1, and NA instead of 0, for example. Does it make sense ?
cmdoret
CC BY-SA 4.0
2018-06-21T18:10:39.173Z
thanks a lot, actually i just have 8 lists of genes, lets say that each list has one column of genes. i want a pairwise comparison for common genes for any possible combinations of these time points.
Zizogolu
CC BY-SA 4.0
2018-06-21T18:28:31.267Z
Oh sorry, i did not understand the last part about pairwise comparisons. First, if you want to filter common genes, you could just filter rows (genes) which have values available for more than x timepoints. For pairwise comparison, maybe that'd be easier if you provided more details/background. (E.g. do you want to compute a single summary stat per condition and compute it, or is it per gene ?)
cmdoret
CC BY-SA 4.0
2018-06-21T20:36:07.313Z
thanks a lot, i think per condition. actually i need common genes. for instance i have 630 common genes between h2 and h4 time points, while i have 1143 genes in h2 and 768 genes in h4 itself. so, i have two ways comparison; 630/1143 and 630/768. that would be great if i could do the same for all of my time points. something like having a matrix of percentage or a graph of them.
Zizogolu
CC BY-SA 4.0
2018-06-21T21:21:57.033Z
Oh I think I understand. Could you update your original question with the exact comparisons you want to make (e.g. Compute number of available genes between all possible pairs of combination). Then I can make sure I got it right and update my response.
cmdoret
CC BY-SA 4.0
2018-06-21T22:49:24.987Z
I updated my solution, please mark it as accepted if it answers your question.
cmdoret
CC BY-SA 4.0
2018-06-22T10:45:53.337Z
excuse me, i quite did not understand if i have 9 files each with one column of genes, how i can have df (long format). :(:(
Zizogolu
CC BY-SA 4.0
2018-06-22T11:25:58.760Z
I slightly edited my answer. hopefully it is more clear. To generate df, you need to generate the table in long format. I made a section with instructions (bash and R) to generate such table. If you chose R, you might need to edit the code to suite your data. If you chose bash, you just need to read the "long_table" file into R using read.table().
cmdoret
CC BY-SA 4.0
2018-06-22T11:53:04.160Z
sorry for sure your code will work. i am really sorry but I'm too new in coding. for example for producing long format n r, should i have a df for each time point? i mean 9 df files? i don't know what is tp or should i put my 9 columns in  one file with its own headers for example h2, h4 in header of each column. i really could not figure out. however definitely your code is so practical
Zizogolu
CC BY-SA 4.0
2018-06-22T12:55:42.927Z
No problem with being new to programming. I am afraid I cannot help you anymore than that, but if you get a bit more into programming first, I think it will make sense. There are great free resources out there. Here is a general purpose list:https://github.com/Michael0x2a/curated-programming-resources/blob/master/resources.md 
For R I would maybe try: https://www.datacamp.com/
For python in bioinformatics: http://rosalind.info/problems/locations/
For bash: https://www.codecademy.com
Hopefully some of these will be helpful to you :) Best of luck
cmdoret
CC BY-SA 4.0
2018-06-21T16:09:13.220Z
2018-06-25T07:35:24.700Z
2018-06-25T07:35:24.700Z
cmdoret
<p>I have 8 lists of differentially expressed genes for 8 time points (2, 4, 6, 8, 10, 12, 14, 16 hours), the number of genes in each of these lists are not equal.</p>

<p>For example there are 630 common genes between h2 and h4 time points but I have 1143 genes in h2 list and 768 genes in h4 list , so common genes in h2 vs h4 would be 630/1143 and common genes h4 vs h2 would be 630/768.</p>

<p>That would would be great if I have a code to produce a matrix or plot of all possible pairwise combinations of my 8 time points.</p>

CC BY-SA 4.0
2018-06-21T10:03:48.153Z
cross posted https://www.biostars.org/p/321861/
Pierre
<r>
Multiway intersection of 8 lists of genes
# Multiway intersection of 8 lists of genes

(Asked by: username_0 on 2018-06-21)

I have 8 lists of differentially expressed genes for 8 time points (2, 4, 6, 8, 10, 12, 14, 16 hours), the number of genes in each of these lists are not equal.

For example there are 630 common genes between h2 and h4 time points but I have 1143 genes in h2 list and 768 genes in h4 list , so common genes in h2 vs h4 would be 630/1143 and common genes h4 vs h2 would be 630/768.

That would would be great if I have a code to produce a matrix or plot of all possible pairwise combinations of my 8 time points.

**Tags:** r
> **Comments:**
> **username_1**: cross posted https://www.biostars.org/p/321861/  

---

## 3 answers

### Answer by username_4 on 2018-06-21. Score: 3 | Accepted answer

EDIT: Sorry, I did not see the R tag and wrote the code in bash. I still left the bash example below and added an R version.
Depends on what you want to do, but there are two common ways in which you could store this. Either as wide format: 
      h2  h4  h6 ...
Gene1 1   1   0
Gene2 0   1   1
Gene3 1   1   1
...

Where 1 denotes presence and 0 absence, or as long format:
      timepoint
Gene1 h2
Gene1 h4
Gene2 h4
Gene2 h6
Gene3 h2
Gene3 h4
Gene3 h6
...

The best format probably depends on what you want to do. Since you did not specify a method/language, I took the liberty to write a few lines of bash to go from the gene lists to the matrices. 
Bash examples assume your files are named h2, h4 etc and are tab separated with gene names in the first column. R examples assume tables is a list of your data frames and timepoints is a vector c("h2", "h4", ...).
Generate table in wide format:

Concatenate gene names of all tables to a separate file
Extract unique values in this file
Perform outer join between the separate file and each table

R:
all_names <- c()
for (tp in timepoints){
  all_names <- append(all_names, tables[[tp]]$genes)
}
all_names <- unique(all_names)

wide_table <- matrix(nrow=length(all_names), ncol=(1+length(timepoints)))
wide_table <- as.data.frame(wide_table)
colnames(wide_table) <- append("genes", timepoints)

for (tp in timepoints){
 tmp <- tables[[tp]]
 tmp$tp <- 1
 tmp <- merge(wide_table, tmp, by="genes", all=T)
 wide_table$tp <- tmp$tp
}

Bash:
Note: you will might need to add headers to this table
for timefile in $yourfiles;do
  # make a file with all gene names
  awk -v tp=$timefile '{print $1}' >> names_file
done

sort names_file | uniq > tmp && mv tmp names_file

for timefile in $yourfiles;do
  join -a1 -a2  -j 1 -o 0,1.2,2.2 -e 0 $names_file <(awk '{print$1,1}' $timefile | sort)
done

Generate table in long format:

For each table, add a column with timepoint
Concatenate all tables

R:
df <- data.frame()
for (tp in timepoints){
  tables[[tp]][[timepoint]] <- tp
  df <- rbind(df, tables[[tp]] )
}

Bash:
   for timefile in $yourfiles;do
      awk -v tp=$timefile '{print $1,timefile}' >> long_table
    done 

EDIT2: To follow up on this, here is a simple way (using the package dplyr) to count the number of common genes between all possible combination of time points. Assuming you have your data in long format:
df <- data.frame(genes=c("G1","G1","G2","G3","G3"), tp=c("h2","h4","h2","h4","h6"))

  genes tp
1    G1 h2
2    G1 h4
3    G2 h2
4    G3 h4
5    G3 h6

You can get the co-occurrences using :
library(dplyr)
# Make timepoint ordinal factor
df <- mutate(df, tp = factor(tp, ordered=T))
df <- df %>% 
  # Compute all pairwise co-occurrences
  full_join(df, by="genes") %>% 
  group_by(tp.x, tp.y) %>% 
  summarise(N_genes = length(unique(genes))) %>%
  # Exclude self pairs (gene in h2 and h2)
  filter(tp.x != tp.y) %>% 
  # Check for duplicate pairs in reverse order (h2-h4 and h4-h2)
  mutate(comb=ifelse(tp.x < tp.y, 
                     yes = paste(tp.x, tp.y), 
                     no  = paste(tp.y, tp.x)))
# Remove the duplicate pairs
df <- df[which(!duplicated(df$comb)),]

And visualise it using for example a heatmap:
(ofc it looks stupid because I have a tiny sample where not all combinations are even present)
# Visualize number of genes present in each combination
library(ggplot2)
ggplot(df, aes(x=tp.x, y=tp.y, fill=N_genes)) + geom_tile()

> **Comments:**
> **username_0**: excuse me, i did not quite understand. i have 8 separate lists of genes so how i know about 0 and 1 (absence and presence of a genes among the lists)?  
> **username_4**: @FereshTeh 0 and 1 were just placeholder values to give the info "geneY is present (1) / absent (0) at timepoint hX" . You could have expression values, ratios, pvalues or whatever variable instead of 1, and NA instead of 0, for example. Does it make sense ?  
> **username_0**: thanks a lot, actually i just have 8 lists of genes, lets say that each list has one column of genes. i want a pairwise comparison for common genes for any possible combinations of these time points.  
> **username_4**: Oh sorry, i did not understand the last part about pairwise comparisons. First, if you want to filter common genes, you could just filter rows (genes) which have values available for more than x timepoints. For pairwise comparison, maybe that'd be easier if you provided more details/background. (E.g. do you want to compute a single summary stat per condition and compute it, or is it per gene ?)  
> **username_0**: thanks a lot, i think per condition. actually i need common genes. for instance i have 630 common genes between h2 and h4 time points, while i have 1143 genes in h2 and 768 genes in h4 itself. so, i have two ways comparison; 630/1143 and 630/768. that would be great if i could do the same for all of my time points. something like having a matrix of percentage or a graph of them.  
> **username_4**: Oh I think I understand. Could you update your original question with the exact comparisons you want to make (e.g. Compute number of available genes between all possible pairs of combination). Then I can make sure I got it right and update my response.  
> **username_4**: I updated my solution, please mark it as accepted if it answers your question.  
> **username_0**: excuse me, i quite did not understand if i have 9 files each with one column of genes, how i can have df (long format). :(:(  
> **username_4**: I slightly edited my answer. hopefully it is more clear. To generate df, you need to generate the table in long format. I made a section with instructions (bash and R) to generate such table. If you chose R, you might need to edit the code to suite your data. If you chose bash, you just need to read the "long_table" file into R using read.table().  
> **username_0**: sorry for sure your code will work. i am really sorry but I'm too new in coding. for example for producing long format n r, should i have a df for each time point? i mean 9 df files? i don't know what is tp or should i put my 9 columns in  one file with its own headers for example h2, h4 in header of each column. i really could not figure out. however definitely your code is so practical  
> **username_4**: No problem with being new to programming. I am afraid I cannot help you anymore than that, but if you get a bit more into programming first, I think it will make sense. There are great free resources out there. Here is a general purpose list:https://github.com/Michael0x2a/curated-programming-resources/blob/master/resources.md  For R I would maybe try: https://www.datacamp.com/ For python in bioinformatics: http://rosalind.info/problems/locations/ For bash: https://www.codecademy.com Hopefully some of these will be helpful to you :) Best of luck  

---

### Answer by username_2 on 2018-06-26. Score: 3

An UpSet Plot is a useful way of visualising these kind of multi-dimensional overlaps. There's an R package for producing them, with a reasonable set of examples. 
The input is a binary matrix indicating set membership:
         h2    h4    h6...
GeneA     0     1     0
GeneB     1     1     0
GeneC     0     1     1
...

And the output is a pair of bar charts, showing the size of various sets, with a dot matrix showing the set membership. An example from the UpSetR documentation:
library(UpSetR)
mutations <- read.csv(system.file("extdata", "mutations.csv", 
                      package = "UpSetR"), header=T, sep = ",")
upset(mutations, sets = c("PTEN", "TP53", "EGFR", "PIK3R1", "RB1"),
      sets.bar.color = "#56B4E9", order.by = "freq", 
      empty.intersections = "on")


---

### Answer by username_3 on 2018-06-24. Score: 1

I would simply do this...

# sepcify your timepoints, each timepoint is a list of genes!!
timepoints <- list(h2,h4,h6,h8, h10,h12,h14,h16)
# initialize an empty array that you will fill with your timepoints
vect = vector()
# go over all combinations and fill the array with the set intersection values you want (I guess...)
for (i in timepoints) {
  for (j in timepoints) {
    vect <- c(vect, length(intersect(i,j))/length(i))}}
# reshape your array to a more ordered matrix  
your_matrix <- matrix(vect, nrow=length(timepoints))
# assign columns and rows names
colnames(your_matrix) <- c("h2","h4","h6","h8","h10","h12","h14","h16")
rownames(your_matrix) <- c("h2","h4","h6","h8","h10","h12","h14","h16")


---

