<p>Okay, its done. What you are looking for is as follows:</p>
<pre><code>import requests
from pathlib import Path

# Test code    
def get_all_keys(d):
    for key, value in d.items():
        yield key
        if isinstance(value, dict):
            yield from get_all_keys(value)

def requestIt(links): 
    cancer_all = requests.get(links)
    return cancer_all.json() 

if __name__ == '__main__':

    path = Path('/PathToDir/test', 'link.html')
    with open (path, 'r') as fin:
        links = fin.readlines()
        
        
    data = list(map(requestIt, links))

    # Test code
    for x in get_all_keys(data[0]['_embedded']['terms'][0]):
        print(x)

    # Answer code
    for i,d in enumerate(data):
        print (f'Link {i + 1} ') 
        for obo_id in d['_embedded']['terms']:
            print (obo_id['obo_id'])
</code></pre>
<p><strong>Output</strong></p>
<blockquote>
<p><strong>Link 1</strong>
NCIT:C8263
NCIT:C9163
NCIT:C5630</p>
</blockquote>
<blockquote>
<p><strong>Link 2</strong>
NCIT:C7175
NCIT:C68700
NCIT:C68696
NCIT:C68697
NCIT:C68698
NCIT:C68699
NCIT:C9155
NCIT:C9288
NCIT:C9287
NCIT:C9019
NCIT:C9018
NCIT:C36055
NCIT:C36058
NCIT:C36057
NCIT:C36056
NCIT:C38377
NCIT:C122688
NCIT:C156718
NCIT:C156719
NCIT:C169107</p>
</blockquote>
<blockquote>
<p><strong>Link 3</strong>
NCIT:C7056
NCIT:C131911
NCIT:C165799
NCIT:C68684
NCIT:C4998
NCIT:C5089
NCIT:C5095
NCIT:C8154
NCIT:C8604
NCIT:C127840
NCIT:C8489
NCIT:C8852
NCIT:C69144
NCIT:C115349
NCIT:C115367
NCIT:C165240
NCIT:C8926
NCIT:C160230
NCIT:C160231
NCIT:C182035</p>
</blockquote>
<p>Where <strong>Link 1</strong> is ...</p>
<pre><code>https://www.ebi.ac.uk/ols/api/ontologies/ncit/terms/http%253A%252F%252Fpurl.obolibrary.org%252Fobo%252FNCIT_C4861/descendants
</code></pre>
<p>... and so on.</p>
<p>If the code is listed <code># Test code</code> that is what is used to interrogate the data structure. This doesn't contribute to your answer.</p>
<p>However, given the way I've done it, it's important to understand the data structure. Thus output of <code># Test code</code> is at the base of this post. If you need another part of the data structure you simply switch <code>id_odo</code> to whatever else you are looking for (at the base of this post).</p>
<p>There will be an easy to use OOP method based out the <code>data</code> object. I did't want to spend time investigating the object ... the question is specifically about <code>obo_id</code> so it was easier to go for that.</p>
<p><strong>Test code output</strong></p>
<pre><code>iri
lang
description
synonyms
annotation
Contributing_Source
Display_Name
Legacy Concept Name
Neoplastic_Status
Preferred_Name
Semantic_Type
UMLS_CUI
code
label
ontology_name
ontology_prefix
ontology_iri
is_obsolete
term_replaced_by
is_defining_ontology
has_children
is_root
short_form
obo_id  # &lt;- its here
in_subset
obo_definition_citation
obo_xref
obo_synonym
is_preferred_root
_links
self
href
parents
href
ancestors
href
hierarchicalParents
href
hierarchicalAncestors
href
jstree
href
graph
href
</code></pre>

CC BY-SA 4.0
2023-03-11T18:20:12.357Z
2023-03-11T19:41:02.550Z
2023-03-11T19:41:02.550Z
M__
<p>@Megha
The requestIt(link) function returns a list of dictionary objects.
So your &quot;data&quot; variable is a list of dictionary objects.</p>
<p>Every item on this list (a dict) has a key named &quot;_embedded&quot; whose value is also a dictionary.
This dictionary (the _embedded) has a key named &quot;terms&quot; which is a list of dictionaries.
The obo_ids you need are inside the dictionaries of the &quot;terms&quot; list.</p>
<p>The problem with your code was that you were trying to access the &quot;obo_id&quot; key on the top level dictionary.</p>
<p>So a simple solution would be like this:</p>
<pre><code>with open ('urls.txt', 'r') as fin:
    links = fin.readlines()
resps = [requestIt(link) for link in links] # this one holds all the data
obo_ids = []
for resp in resps:                          # for every response we got
    terms = resp['_embedded']['terms']      # choose the terms list
    for term in terms:                      # for every dict in this list
        obo_ids.append(term['obo_id'])      # extract obo_id and append it to our answer
print(obo_ids)                              # do stuff with obo_ids
</code></pre>
<p>You should probably perform some exception handling inside the requestIt() function to handle broken links etc.</p>

CC BY-SA 4.0
2023-03-09T17:24:21.197Z
Welcome @punxx and this looks cool
M__
CC BY-SA 4.0
2023-03-09T17:17:50.883Z
2023-03-09T17:17:50.883Z
punxx
<p>@Megha, I have not looked through the code in too much detail, however.</p>
<pre><code>obo = obj[&quot;obo_id&quot;] for obj in data
</code></pre>
<p>This isn't formal Python code. The correction would simply be to place this in a comprehension ...</p>
<pre><code>obo = [obj[&quot;obo_id&quot;] for obj in data]
</code></pre>
<p>... unless you want an iterator.</p>
<p>I think there is a deep issue with <code>obj[&quot;obo_id&quot;]</code>. If the syntax still throws an exception could you provide a print out of either <code>data</code> please, or else ....</p>
<pre><code>obj for obj in data:
    print (obj)
    print (obj.keys)
</code></pre>
<p>The <code>data</code> data structure for the output of <code>get.requests</code> followed by <code>.json()</code> is kinda key here</p>

CC BY-SA 4.0
2023-03-07T10:08:33.247Z
Thank you for the answer. Got the same error with obo = [obj["obo_id"] for obj in data]. Then I tried `obj for obj in data:
    print (obj)
    print (obj.keys)`. Got an error `obj for obj in data:
        ^
SyntaxError: invalid syntax
Megha
CC BY-SA 4.0
2023-03-07T10:10:16.980Z
Is there any way I can run the code (the cancer_id [ ] one)that worked perfectly for one link in a for loop for all the links?
Megha
CC BY-SA 4.0
2023-03-07T13:46:11.230Z
Thanks @Megha. Did you get a print out for `print (obj)`? Then an error for `print (obj.keys)`? The idea is to check the output before the error.
M__
CC BY-SA 4.0
2023-03-07T13:51:20.590Z
Hi got the same error for both print statements?and there was no output, just this error
Megha
CC BY-SA 4.0
2023-03-07T15:44:19.070Z
Hi @Megha over the weekend I will replicate your code and investigate, the object, json, dictionary ... (whatever it is) output. I should get I definitive answer, but I'm afraid I can't do that during this week due to work stuff.
M__
CC BY-SA 4.0
2023-03-08T04:12:18.737Z
@Megha `cancer_all.json()` is an object so its behaviour is very different. `obj["obo_id"]` is not going to work because you'll need to reference the instance. It's possible something like `str(obj)['repo_url']` might work. I'll need to check and examine all instances etc ...
M__
CC BY-SA 4.0
2023-03-06T19:04:56.693Z
2023-03-06T19:04:56.693Z
M__
<p>Hi I am trying to get the cancer ontologies (obo_id and label) from EBI-OLS. Earlier I have used the below code to get the <code>obo_id</code> terms and <code>labels</code></p>
<pre><code>descendants = requests.get(
    &quot;https://www.ebi.ac.uk/ols/api/ontologies/efo/terms/http%253A%252F%252Fpurl.obolibrary.org%252Fobo%252FMONDO_0004992/descendants&quot;
)
</code></pre>
<p>As there are more than 1 page so I created <code>total_pages</code></p>
<pre><code>total_pages = descendants.json()[&quot;page&quot;][&quot;totalElements&quot;] // 1000 + 1
</code></pre>
<pre><code>cancer_id = []

for i in range(total_pages):
    ## Check the number of id's present in 'totalElements'
    des = {&quot;page&quot;: i, &quot;size&quot;: descendants.json()[&quot;page&quot;][&quot;totalElements&quot;]}
    ## Add the number of 'totalElements' and 'pages' in the new python request for descendants
    cancer_efo = requests.get(
        &quot;https://www.ebi.ac.uk/ols/api/ontologies/efo/terms/http%253A%252F%252Fpurl.obolibrary.org%252Fobo%252FMONDO_0004992/descendants&quot;,
        params=des,
    )
    ## Get the number of obo_id's present in the cancer_efo.json
    obo = [obj[&quot;obo_id&quot;] for obj in cancer_efo.json()[&quot;_embedded&quot;][&quot;terms&quot;]]
    ## Get the number of cancer labels present in the cancer_efo.json
    label = [obj[&quot;label&quot;] for obj in cancer_efo.json()[&quot;_embedded&quot;][&quot;terms&quot;]]
    data = zip(obo, label)
    cancer_id.extend(data)
</code></pre>
<p>The above code worked pretty well with one link
Now the issue is that I want to get ontologies for 37 descendent links and maximum links have more than 1 page. I want to get one file (cancer_id) containing all obo_id's and labels from these all 37 links.</p>
<p>I made a separate file called <code>list.txt</code> (Right now I am copying only 3 here)</p>
<p><code>list.txt</code></p>
<pre><code>https://www.ebi.ac.uk/ols/api/ontologies/ncit/terms/http%253A%252F%252Fpurl.obolibrary.org%252Fobo%252FNCIT_C4861/descendants
https://www.ebi.ac.uk/ols/api/ontologies/ncit/terms/http%253A%252F%252Fpurl.obolibrary.org%252Fobo%252FNCIT_C3171/descendants
https://www.ebi.ac.uk/ols/api/ontologies/ncit/terms/http%253A%252F%252Fpurl.obolibrary.org%252Fobo%252FNCIT_C3457/descendants
</code></pre>
<p>I tried the below code but it is not working</p>
<pre><code>import requests

with open ('list.txt', 'r') as fin:
    links = fin.readlines()

def requestIt(links): 
    cancer_all = requests.get(links)
    return cancer_all.json() 

data = list(map(requestIt, links))

obo = obj[&quot;obo_id&quot;] for obj in data
</code></pre>
<p>The error is</p>
<pre><code>obo = obj[&quot;obo_id&quot;] for obj in data
                        ^
SyntaxError: invalid syntax
</code></pre>

<python><api><pandas><ontology>
How to run python request for list of url's with multiple page numbers?
# How to run python request for list of url's with multiple page numbers?

(Asked by: username_0 on 2023-03-06)

Hi I am trying to get the cancer ontologies (obo_id and label) from EBI-OLS. Earlier I have used the below code to get the obo_id terms and labels
descendants = requests.get(
    "https://www.ebi.ac.uk/ols/api/ontologies/efo/terms/http%253A%252F%252Fpurl.obolibrary.org%252Fobo%252FMONDO_0004992/descendants"
)

As there are more than 1 page so I created total_pages
total_pages = descendants.json()["page"]["totalElements"] // 1000 + 1

cancer_id = []

for i in range(total_pages):
    ## Check the number of id's present in 'totalElements'
    des = {"page": i, "size": descendants.json()["page"]["totalElements"]}
    ## Add the number of 'totalElements' and 'pages' in the new python request for descendants
    cancer_efo = requests.get(
        "https://www.ebi.ac.uk/ols/api/ontologies/efo/terms/http%253A%252F%252Fpurl.obolibrary.org%252Fobo%252FMONDO_0004992/descendants",
        params=des,
    )
    ## Get the number of obo_id's present in the cancer_efo.json
    obo = [obj["obo_id"] for obj in cancer_efo.json()["_embedded"]["terms"]]
    ## Get the number of cancer labels present in the cancer_efo.json
    label = [obj["label"] for obj in cancer_efo.json()["_embedded"]["terms"]]
    data = zip(obo, label)
    cancer_id.extend(data)

The above code worked pretty well with one link
Now the issue is that I want to get ontologies for 37 descendent links and maximum links have more than 1 page. I want to get one file (cancer_id) containing all obo_id's and labels from these all 37 links.
I made a separate file called list.txt (Right now I am copying only 3 here)
list.txt
https://www.ebi.ac.uk/ols/api/ontologies/ncit/terms/http%253A%252F%252Fpurl.obolibrary.org%252Fobo%252FNCIT_C4861/descendants
https://www.ebi.ac.uk/ols/api/ontologies/ncit/terms/http%253A%252F%252Fpurl.obolibrary.org%252Fobo%252FNCIT_C3171/descendants
https://www.ebi.ac.uk/ols/api/ontologies/ncit/terms/http%253A%252F%252Fpurl.obolibrary.org%252Fobo%252FNCIT_C3457/descendants

I tried the below code but it is not working
import requests

with open ('list.txt', 'r') as fin:
    links = fin.readlines()

def requestIt(links): 
    cancer_all = requests.get(links)
    return cancer_all.json() 

data = list(map(requestIt, links))

obo = obj["obo_id"] for obj in data

The error is
obo = obj["obo_id"] for obj in data
                        ^
SyntaxError: invalid syntax

**Tags:** python, api, pandas, ontology

---

## 3 answers

### Answer by username_1 on 2023-03-11. Score: 1 | Accepted answer

Okay, its done. What you are looking for is as follows:
import requests
from pathlib import Path

# Test code    
def get_all_keys(d):
    for key, value in d.items():
        yield key
        if isinstance(value, dict):
            yield from get_all_keys(value)

def requestIt(links): 
    cancer_all = requests.get(links)
    return cancer_all.json() 

if __name__ == '__main__':

    path = Path('/PathToDir/test', 'link.html')
    with open (path, 'r') as fin:
        links = fin.readlines()
        
        
    data = list(map(requestIt, links))

    # Test code
    for x in get_all_keys(data[0]['_embedded']['terms'][0]):
        print(x)

    # Answer code
    for i,d in enumerate(data):
        print (f'Link {i + 1} ') 
        for obo_id in d['_embedded']['terms']:
            print (obo_id['obo_id'])

Output

Link 1
NCIT:C8263
NCIT:C9163
NCIT:C5630


Link 2
NCIT:C7175
NCIT:C68700
NCIT:C68696
NCIT:C68697
NCIT:C68698
NCIT:C68699
NCIT:C9155
NCIT:C9288
NCIT:C9287
NCIT:C9019
NCIT:C9018
NCIT:C36055
NCIT:C36058
NCIT:C36057
NCIT:C36056
NCIT:C38377
NCIT:C122688
NCIT:C156718
NCIT:C156719
NCIT:C169107


Link 3
NCIT:C7056
NCIT:C131911
NCIT:C165799
NCIT:C68684
NCIT:C4998
NCIT:C5089
NCIT:C5095
NCIT:C8154
NCIT:C8604
NCIT:C127840
NCIT:C8489
NCIT:C8852
NCIT:C69144
NCIT:C115349
NCIT:C115367
NCIT:C165240
NCIT:C8926
NCIT:C160230
NCIT:C160231
NCIT:C182035

Where Link 1 is ...
https://www.ebi.ac.uk/ols/api/ontologies/ncit/terms/http%253A%252F%252Fpurl.obolibrary.org%252Fobo%252FNCIT_C4861/descendants

... and so on.
If the code is listed # Test code that is what is used to interrogate the data structure. This doesn't contribute to your answer.
However, given the way I've done it, it's important to understand the data structure. Thus output of # Test code is at the base of this post. If you need another part of the data structure you simply switch id_odo to whatever else you are looking for (at the base of this post).
There will be an easy to use OOP method based out the data object. I did't want to spend time investigating the object ... the question is specifically about obo_id so it was easier to go for that.
Test code output
iri
lang
description
synonyms
annotation
Contributing_Source
Display_Name
Legacy Concept Name
Neoplastic_Status
Preferred_Name
Semantic_Type
UMLS_CUI
code
label
ontology_name
ontology_prefix
ontology_iri
is_obsolete
term_replaced_by
is_defining_ontology
has_children
is_root
short_form
obo_id  # <- its here
in_subset
obo_definition_citation
obo_xref
obo_synonym
is_preferred_root
_links
self
href
parents
href
ancestors
href
hierarchicalParents
href
hierarchicalAncestors
href
jstree
href
graph
href


---

### Answer by username_2 on 2023-03-09. Score: 2

@Megha
The requestIt(link) function returns a list of dictionary objects.
So your "data" variable is a list of dictionary objects.
Every item on this list (a dict) has a key named "_embedded" whose value is also a dictionary.
This dictionary (the _embedded) has a key named "terms" which is a list of dictionaries.
The obo_ids you need are inside the dictionaries of the "terms" list.
The problem with your code was that you were trying to access the "obo_id" key on the top level dictionary.
So a simple solution would be like this:
with open ('urls.txt', 'r') as fin:
    links = fin.readlines()
resps = [requestIt(link) for link in links] # this one holds all the data
obo_ids = []
for resp in resps:                          # for every response we got
    terms = resp['_embedded']['terms']      # choose the terms list
    for term in terms:                      # for every dict in this list
        obo_ids.append(term['obo_id'])      # extract obo_id and append it to our answer
print(obo_ids)                              # do stuff with obo_ids

You should probably perform some exception handling inside the requestIt() function to handle broken links etc.

> **Comments:**
> **username_1**: Welcome @punxx and this looks cool  

---

### Answer by username_1 on 2023-03-06. Score: 1

@Megha, I have not looked through the code in too much detail, however.
obo = obj["obo_id"] for obj in data

This isn't formal Python code. The correction would simply be to place this in a comprehension ...
obo = [obj["obo_id"] for obj in data]

... unless you want an iterator.
I think there is a deep issue with obj["obo_id"]. If the syntax still throws an exception could you provide a print out of either data please, or else ....
obj for obj in data:
    print (obj)
    print (obj.keys)

The data data structure for the output of get.requests followed by .json() is kinda key here

> **Comments:**
> **username_0**: Thank you for the answer. Got the same error with obo = [obj["obo_id"] for obj in data]. Then I tried `obj for obj in data:     print (obj)     print (obj.keys)`. Got an error `obj for obj in data:         ^ SyntaxError: invalid syntax  
> **username_0**: Is there any way I can run the code (the cancer_id [ ] one)that worked perfectly for one link in a for loop for all the links?  
> **username_1**: Thanks @Megha. Did you get a print out for `print (obj)`? Then an error for `print (obj.keys)`? The idea is to check the output before the error.  
> **username_0**: Hi got the same error for both print statements?and there was no output, just this error  
> **username_1**: Hi @Megha over the weekend I will replicate your code and investigate, the object, json, dictionary ... (whatever it is) output. I should get I definitive answer, but I'm afraid I can't do that during this week due to work stuff.  
> **username_1**: @Megha `cancer_all.json()` is an object so its behaviour is very different. `obj["obo_id"]` is not going to work because you'll need to reference the instance. It's possible something like `str(obj)['repo_url']` might work. I'll need to check and examine all instances etc ...  

---

