<p>It looks like the issue comes from one of the fields I was filtering for. When I remove that filter the code works again. Still need to investigate why that happened.</p>

CC BY-SA 4.0
2023-07-24T17:34:20.107Z
Could you kindly expand on the answer please, specifically which field?
M__
CC BY-SA 4.0
2023-07-25T19:23:29.907Z
It really isn't clear. Possibly something in self-reported data, field 20002. I ended up using a workaround by filtering for fields separately and joining them with df.union(df2)
eclarette
CC BY-SA 4.0
2023-07-24T17:01:36.297Z
2023-07-24T17:01:36.297Z
eclarette
<p>Generally remote biological servers, for example UCSC genome server, will not perform SQL queries for a free account. They'll allow downloads, but not queries. That is probably the same reason here.</p>
<p>The reason is simply these are often huge databases and queries can take up a lot of their computational resources, particularly if everyone is doing them. I don't know BioBank UK, but suggest its common practice.</p>
<p>If thats true the only difference with Biobank and UCSC is the UCSC server refusals are stated very clearly, &quot;this operation ain't permitted for free accounts&quot; (paraphrasing). What might be happening in an API error is not being picked up and relayed to you. Depends how the API is written so it's just reporting a timeout.</p>

CC BY-SA 4.0
2023-07-23T22:11:00.073Z
The account is paid, not free
eclarette
CC BY-SA 4.0
2023-07-23T22:14:05.727Z
@eclarette If you login directly you will receive a more informative message than going through an API, i.e. perform the SQL query there. The API isn't reporting the error message.
M__
CC BY-SA 4.0
2023-07-23T04:34:20.253Z
2023-07-23T12:38:09.787Z
2023-07-23T12:38:09.787Z
M__
<p>I'm working with UK Biobank data. I made a cohort with the Cohort Browser and tried to extract the data with this code  <code>df = participant.retrieve_fields(names=field_names, filter_sql = cohort.sql, coding_values = &quot;replace&quot;, engine=dxdata.connect())</code>.</p>
<p>This is part of the text of the error: <code>Py4JJavaError: An error occurred while calling 04100.showString. org.apache.spark. SparkException: Exception thrown in awaitResult:</code>. The full error message is cut off.</p>
<p>However, if I remove the filter - <code>df = participant.retrieve_fields(names=field_names, coding_values = &quot;replace&quot;, engine=dxdata.connect())</code> – there is no issue.</p>
<p>What kind of error is this and why would filtering for a cohort help?</p>

CC BY-SA 4.0
2023-07-22T01:36:15.627Z
Thanks for the question and the edit. This is Python API for a Biobank - looks very interesting. Just to clarify the implication is you can connect to Biobank ok, but when you add a filter this stops the operation. However the code that connects okay has not been shown. Is that all correct?
M__
CC BY-SA 4.0
2023-07-23T03:23:21.120Z
Yes, that's correct
eclarette
CC BY-SA 4.0
2023-07-23T04:27:58.500Z
Okay, it's a bit hard without knowing the underlying code.
M__
<python>
Filtering cohorts in PySpark returns Py4JJavaError
# Filtering cohorts in PySpark returns Py4JJavaError

(Asked by: username_0 on 2023-07-21)

I'm working with UK Biobank data. I made a cohort with the Cohort Browser and tried to extract the data with this code `df = participant.retrieve_fields(names=field_names, filter_sql = cohort.sql, coding_values = "replace", engine=dxdata.connect())`.

This is part of the text of the error: `Py4JJavaError: An error occurred while calling 04100.showString. org.apache.spark. SparkException: Exception thrown in awaitResult:`. The full error message is cut off.

However, if I remove the filter - `df = participant.retrieve_fields(names=field_names, coding_values = "replace", engine=dxdata.connect())` – there is no issue.

What kind of error is this and why would filtering for a cohort help?

**Tags:** python
> **Comments:**
> **username_1**: Thanks for the question and the edit. This is Python API for a Biobank - looks very interesting. Just to clarify the implication is you can connect to Biobank ok, but when you add a filter this stops the operation. However the code that connects okay has not been shown. Is that all correct?  
> **username_0**: Yes, that's correct  
> **username_1**: Okay, it's a bit hard without knowing the underlying code.  

---

## 2 answers

### Answer by username_0 on 2023-07-24. Score: 1

It looks like the issue comes from one of the fields I was filtering for. When I remove that filter the code works again. Still need to investigate why that happened.

> **Comments:**
> **username_1**: Could you kindly expand on the answer please, specifically which field?  
> **username_0**: It really isn't clear. Possibly something in self-reported data, field 20002. I ended up using a workaround by filtering for fields separately and joining them with df.union(df2)  

---

### Answer by username_1 on 2023-07-23. Score: 0

Generally remote biological servers, for example UCSC genome server, will not perform SQL queries for a free account. They'll allow downloads, but not queries. That is probably the same reason here.

The reason is simply these are often huge databases and queries can take up a lot of their computational resources, particularly if everyone is doing them. I don't know BioBank UK, but suggest its common practice.

If thats true the only difference with Biobank and UCSC is the UCSC server refusals are stated very clearly, "this operation ain't permitted for free accounts" (paraphrasing). What might be happening in an API error is not being picked up and relayed to you. Depends how the API is written so it's just reporting a timeout.

> **Comments:**
> **username_0**: The account is paid, not free  
> **username_1**: @eclarette If you login directly you will receive a more informative message than going through an API, i.e. perform the SQL query there. The API isn't reporting the error message.  

---

