<p>If you want to separate each set, you can do something like this</p>

<pre><code>dat &lt;- readLines("~/Desktop/scsig.v1.0.metadata.txt")
dat &lt;- gsub("\t"," ",dat)

# get the indeces of each type
standardName &lt;- grep("STANDARD_NAME",dat)
organism &lt;- grep("ORGANISM",dat)
organSystem &lt;- grep("ORGAN_SYSTEM",dat)
pmid &lt;- grep("PMID",dat)

# return the values 
standardName &lt;- gsub(pattern = "STANDARD_NAME",replacement = "",dat[standardName])
organism &lt;- gsub(pattern = "ORGANISM",replacement = "",dat[organism])
organSystem &lt;- gsub(pattern = "ORGAN_SYSTEM",replacement = "",dat[organSystem])
pmid &lt;- gsub(pattern = "PMID",replacement = "",dat[pmid])

df &lt;- data.frame(standardName,organism,organSystem,pmid,stringsAsFactors = FALSE)

 head(df)
                                     standardName      organism             organSystem      pmid
1      Fan_Embryonic_CTX_Big_Groups_Cajal_Retzius  Homo sapiens  Central Nervous System  29867213
2  Fan_Embryonic_CTX_Big_Groups_Brain_Endothelial  Homo sapiens  Central Nervous System  29867213
3  Fan_Embryonic_CTX_Big_Groups_Excitatory_Neuron  Homo sapiens  Central Nervous System  29867213
4              Fan_Embryonic_CTX_Big_Groups_Glial  Homo sapiens  Central Nervous System  29867213
5       Fan_Embryonic_CTX_Big_Groups_Brain_Immune  Homo sapiens  Central Nervous System  29867213
6          Fan_Embryonic_CTX_Big_Groups_Microglia  Homo sapiens  Central Nervous System  29867213
</code></pre>

<p>This method is very verbose, but it's a start and it can be easily improved with a loop. </p>

<pre><code> #---- EDIT
</code></pre>

<p>Not knowing how the format works, there may be two issues:</p>

<p>1) gene sets with missing values only</p>

<pre><code>STANDARD_NAME Aizarani_Liver_C33_Stellate_cells_2
ORGANISM 
ORGAN_SYSTEM 
PMID 31292543
</code></pre>

<p>2) gene sets with missing value and type</p>

<pre><code>STANDARD_NAME Aizarani_Liver_C33_Stellate_cells_2
PMID 31292543
</code></pre>

<p><code>Case 1</code> is not an issue, as the code above will simply add an empty cell and go to the next line.
For <code>Case 2</code> we can use <code>unique</code> to get a list of all define types, create an empty <code>data.frame</code> with those <code>types</code> and have a loop (or <code>sapply</code>) to add the values in the right columns.</p>

<p>We can check the nature of <code>metadata.txt</code>:</p>

<pre><code> types &lt;- gsub("^(.*)\t.*$","\\1",dat)
 sapply(unique(types), function(t) length(grep(t,types)))

     STANDARD_NAME                ORGANISM            ORGAN_SYSTEM                    PMID       PUBLICATION_TITLE 
                257                     257                     257                     257                     257 
            AUTHORS                   GEOID            EXACT_SOURCE    EXTERNAL_DETAILS_URL                    CHIP 
                257                     257                     257                     257                     257 
      CATEGORY_CODE             CONTRIBUTOR         CONTRIBUTOR_ORG       DESCRIPTION_BRIEF RAW_PUBLICATION_MEMBERS 
                257                     514                     257                     257                     257 
</code></pre>

<p>Your are lucky and you don't have to worry about shifting because of missing values. 
There is a fixable issue with <code>CONTRIBUTORS</code>, but it is related to the <code>regex</code>, not the file format. </p>

CC BY-SA 4.0
2019-10-31T18:52:00.947Z
That should work, but if any of the records are missing a field, then they will be shifted.
burger
CC BY-SA 4.0
2019-11-04T10:47:40.863Z
I didn't think of that. To avoid this issue, you could get the list of all the types (`STANDARD_NAME`, `ORGANISM`,...), use `unique` to avoid repetition and create an empty `data.frame` with those columns. When you return the values, you can check the type and fill the right column. I will try to update my answer when I have some time.
fra
CC BY-SA 4.0
2019-10-31T14:17:31.537Z
2019-11-04T12:17:23.300Z
2019-11-04T12:17:23.300Z
fra
<p>MSigDB recently added <a href="http://www.gsea-msigdb.org/gsea/msigdb/supplementary_genesets.jsp" rel="nofollow noreferrer">Supplementary Collections</a>. The formatting is similar to the original MSigDB gene sets, but there is an additional <code>metadata.txt</code> file (<a href="http://software.broadinstitute.org/gsea/msigdb/supplemental/scsig.v1.0.metadata.txt" rel="nofollow noreferrer">link</a>).</p>

<p>It looks like (condensed for easier reading):</p>

<pre><code>STANDARD_NAME   Fan_Embryonic_CTX_Big_Groups_Cajal_Retzius
ORGANISM    Homo sapiens
ORGAN_SYSTEM    Central Nervous System
PMID    29867213
PUBLICATION_TITLE   Spatial transcriptomic survey of human embryonic cerebral cortex by single-cell RNA-seq analysis
AUTHORS Fan X,Dong J,Zhong S,Wei Y,Wu Q,Yan L,Yong J,Sun L,Wang X,Zhao Y,Wang W,Yan J,Wang X,Qiao J,Tang F,
GEOID   GSE103723
...
STANDARD_NAME   Fan_Embryonic_CTX_Big_Groups_Brain_Endothelial
ORGANISM    Homo sapiens
ORGAN_SYSTEM    Central Nervous System
PMID    29867213
PUBLICATION_TITLE   Spatial transcriptomic survey of human embryonic cerebral cortex by single-cell RNA-seq analysis
AUTHORS Fan X,Dong J,Zhong S,Wei Y,Wu Q,Yan L,Yong J,Sun L,Wang X,Zhao Y,Wang W,Yan J,Wang X,Qiao J,Tang F,
GEOID   GSE103723
...
STANDARD_NAME   Fan_Embryonic_CTX_Big_Groups_Excitatory_Neuron
ORGANISM    Homo sapiens
ORGAN_SYSTEM    Central Nervous System
PMID    29867213
PUBLICATION_TITLE   Spatial transcriptomic survey of human embryonic cerebral cortex by single-cell RNA-seq analysis
AUTHORS Fan X,Dong J,Zhong S,Wei Y,Wu Q,Yan L,Yong J,Sun L,Wang X,Zhao Y,Wang W,Yan J,Wang X,Qiao J,Tang F,
GEOID   GSE103723
...
</code></pre>

<p>I could not find any description of the formatting. Is this a standard format? Is there a parser for it (preferably in R)? It looks like I can just transpose each record to get it in a table format, but I don't know if I can trust all records to be consistent.</p>

<r><file-formats><format-conversion><pathway>
Parsing MSigDB Supplementary Collections
# Parsing MSigDB Supplementary Collections

(Asked by: username_0 on 2019-10-31)

MSigDB recently added Supplementary Collections. The formatting is similar to the original MSigDB gene sets, but there is an additional metadata.txt file (link).
It looks like (condensed for easier reading):
STANDARD_NAME   Fan_Embryonic_CTX_Big_Groups_Cajal_Retzius
ORGANISM    Homo sapiens
ORGAN_SYSTEM    Central Nervous System
PMID    29867213
PUBLICATION_TITLE   Spatial transcriptomic survey of human embryonic cerebral cortex by single-cell RNA-seq analysis
AUTHORS Fan X,Dong J,Zhong S,Wei Y,Wu Q,Yan L,Yong J,Sun L,Wang X,Zhao Y,Wang W,Yan J,Wang X,Qiao J,Tang F,
GEOID   GSE103723
...
STANDARD_NAME   Fan_Embryonic_CTX_Big_Groups_Brain_Endothelial
ORGANISM    Homo sapiens
ORGAN_SYSTEM    Central Nervous System
PMID    29867213
PUBLICATION_TITLE   Spatial transcriptomic survey of human embryonic cerebral cortex by single-cell RNA-seq analysis
AUTHORS Fan X,Dong J,Zhong S,Wei Y,Wu Q,Yan L,Yong J,Sun L,Wang X,Zhao Y,Wang W,Yan J,Wang X,Qiao J,Tang F,
GEOID   GSE103723
...
STANDARD_NAME   Fan_Embryonic_CTX_Big_Groups_Excitatory_Neuron
ORGANISM    Homo sapiens
ORGAN_SYSTEM    Central Nervous System
PMID    29867213
PUBLICATION_TITLE   Spatial transcriptomic survey of human embryonic cerebral cortex by single-cell RNA-seq analysis
AUTHORS Fan X,Dong J,Zhong S,Wei Y,Wu Q,Yan L,Yong J,Sun L,Wang X,Zhao Y,Wang W,Yan J,Wang X,Qiao J,Tang F,
GEOID   GSE103723
...

I could not find any description of the formatting. Is this a standard format? Is there a parser for it (preferably in R)? It looks like I can just transpose each record to get it in a table format, but I don't know if I can trust all records to be consistent.

**Tags:** r, file-formats, format-conversion, pathway

---

## 1 answers

### Answer by username_1 on 2019-10-31. Score: 2

If you want to separate each set, you can do something like this
dat <- readLines("~/Desktop/scsig.v1.0.metadata.txt")
dat <- gsub("\t"," ",dat)

# get the indeces of each type
standardName <- grep("STANDARD_NAME",dat)
organism <- grep("ORGANISM",dat)
organSystem <- grep("ORGAN_SYSTEM",dat)
pmid <- grep("PMID",dat)

# return the values 
standardName <- gsub(pattern = "STANDARD_NAME",replacement = "",dat[standardName])
organism <- gsub(pattern = "ORGANISM",replacement = "",dat[organism])
organSystem <- gsub(pattern = "ORGAN_SYSTEM",replacement = "",dat[organSystem])
pmid <- gsub(pattern = "PMID",replacement = "",dat[pmid])

df <- data.frame(standardName,organism,organSystem,pmid,stringsAsFactors = FALSE)

 head(df)
                                     standardName      organism             organSystem      pmid
1      Fan_Embryonic_CTX_Big_Groups_Cajal_Retzius  Homo sapiens  Central Nervous System  29867213
2  Fan_Embryonic_CTX_Big_Groups_Brain_Endothelial  Homo sapiens  Central Nervous System  29867213
3  Fan_Embryonic_CTX_Big_Groups_Excitatory_Neuron  Homo sapiens  Central Nervous System  29867213
4              Fan_Embryonic_CTX_Big_Groups_Glial  Homo sapiens  Central Nervous System  29867213
5       Fan_Embryonic_CTX_Big_Groups_Brain_Immune  Homo sapiens  Central Nervous System  29867213
6          Fan_Embryonic_CTX_Big_Groups_Microglia  Homo sapiens  Central Nervous System  29867213

This method is very verbose, but it's a start and it can be easily improved with a loop. 
 #---- EDIT

Not knowing how the format works, there may be two issues:
1) gene sets with missing values only
STANDARD_NAME Aizarani_Liver_C33_Stellate_cells_2
ORGANISM 
ORGAN_SYSTEM 
PMID 31292543

2) gene sets with missing value and type
STANDARD_NAME Aizarani_Liver_C33_Stellate_cells_2
PMID 31292543

Case 1 is not an issue, as the code above will simply add an empty cell and go to the next line.
For Case 2 we can use unique to get a list of all define types, create an empty data.frame with those types and have a loop (or sapply) to add the values in the right columns.
We can check the nature of metadata.txt:
 types <- gsub("^(.*)\t.*$","\\1",dat)
 sapply(unique(types), function(t) length(grep(t,types)))

     STANDARD_NAME                ORGANISM            ORGAN_SYSTEM                    PMID       PUBLICATION_TITLE 
                257                     257                     257                     257                     257 
            AUTHORS                   GEOID            EXACT_SOURCE    EXTERNAL_DETAILS_URL                    CHIP 
                257                     257                     257                     257                     257 
      CATEGORY_CODE             CONTRIBUTOR         CONTRIBUTOR_ORG       DESCRIPTION_BRIEF RAW_PUBLICATION_MEMBERS 
                257                     514                     257                     257                     257 

Your are lucky and you don't have to worry about shifting because of missing values. 
There is a fixable issue with CONTRIBUTORS, but it is related to the regex, not the file format.

> **Comments:**
> **username_0**: That should work, but if any of the records are missing a field, then they will be shifted.  
> **username_1**: I didn't think of that. To avoid this issue, you could get the list of all the types (`STANDARD_NAME`, `ORGANISM`,...), use `unique` to avoid repetition and create an empty `data.frame` with those columns. When you return the values, you can check the type and fill the right column. I will try to update my answer when I have some time.  

---

