<p>For an API that deals with a large amount of data, you want to avoid loading all objects into memory. Bumping up the memory limit is only a stop-gap measure, as you said. A couple of approaches to handle this:</p>
<h2>Pagination</h2>
<p>Simple, but effective: Limit your API to return X objects per call with a parameter to control offset/current page. The Element API plugin supports this by default with the <a href="https://github.com/craftcms/element-api#paginate" rel="nofollow noreferrer">paginate parameter</a>.</p>
<p>If you still need all objects at once on the client side, you can use multiple parallel requests to get all pages and then glue them back together with JavaScript.</p>
<h2>Limiting scope</h2>
<p>Do you <em>really</em> need 1300 elements at once? Of course it depends on what you're doing with those. Since you said the entries contain geojson, I'm assuming they're related to locations or areas on a map? In this case, a simple way to limit the amount of elements you return per request would be to add mininum and maximum latitude and longitude parameters to your request and only return elements within those bounds. Then use JavaScript to detect map panning and dynamically load new elements as they come into view.</p>
<p>Of course, this isn't applicable to all situations, but its a good question to ask. If you're building some kind of store locator - nobody needs a forest of 1300 map markers.</p>
<h2>Streaming responses</h2>
<p>If you really need to serialize and send thousands of elements in a single request, one way to avoid hitting memory limits is to process the elements one by one. Instead of loading all serialized elements into a huge array and then encoding it as JSON, load one element at a time, serialize and send it to the client before loading the next element.</p>
<p>I don't think this is supported by Element API by default, and it doesn't work well with JSON. You could think about using something like <a href="http://ndjson.org/" rel="nofollow noreferrer">ND JSON</a>, though this would require some special care on the client-side and probably a custom serializer as well. At this point, it might be easier to build a <a href="https://craftcms.com/docs/3.x/extend/controllers.html" rel="nofollow noreferrer">custom controller</a> for your API.</p>

CC BY-SA 4.0
2021-10-04T11:39:18.427Z
2021-10-04T11:39:18.427Z
MoritzLost
<p>I have the element api pulling back 1300 entries. One of the fields is storing geojson data and I had to change the type from text to mediumtext to accommodate some of the data stored.
When I moved from local Dev to staging server  I got memory allocation issues falling over on jsondecode of the field.
Json_decode($entry-&gt;geojson)</p>
<p>Because it's my server I was able to bump php memory from 128M to 512M which has solved it for now. Could try handling the JSON in the js map app where Im calling API to render polygons but thought PHP would be more performant.</p>
<p>I have more data coming and wondered if I could get a steer on how to guard against further issues.</p>

<plugin-elementapi><json>
Jsondecode in element API causing memory issue
# Jsondecode in element API causing memory issue

(Asked by: username_0 on 2021-10-03)

I have the element api pulling back 1300 entries. One of the fields is storing geojson data and I had to change the type from text to mediumtext to accommodate some of the data stored.
When I moved from local Dev to staging server I got memory allocation issues falling over on jsondecode of the field.
Json\_decode($entry->geojson)

Because it's my server I was able to bump php memory from 128M to 512M which has solved it for now. Could try handling the JSON in the js map app where Im calling API to render polygons but thought PHP would be more performant.

I have more data coming and wondered if I could get a steer on how to guard against further issues.

**Tags:** plugin-elementapi, json


---

## 1 answers

### Answer by username_1 on 2021-10-04. Score: 3

For an API that deals with a large amount of data, you want to avoid loading all objects into memory. Bumping up the memory limit is only a stop-gap measure, as you said. A couple of approaches to handle this:

## Pagination

Simple, but effective: Limit your API to return X objects per call with a parameter to control offset/current page. The Element API plugin supports this by default with the [paginate parameter](https://github.com/craftcms/element-api#paginate).

If you still need all objects at once on the client side, you can use multiple parallel requests to get all pages and then glue them back together with JavaScript.

## Limiting scope

Do you *really* need 1300 elements at once? Of course it depends on what you're doing with those. Since you said the entries contain geojson, I'm assuming they're related to locations or areas on a map? In this case, a simple way to limit the amount of elements you return per request would be to add mininum and maximum latitude and longitude parameters to your request and only return elements within those bounds. Then use JavaScript to detect map panning and dynamically load new elements as they come into view.

Of course, this isn't applicable to all situations, but its a good question to ask. If you're building some kind of store locator - nobody needs a forest of 1300 map markers.

## Streaming responses

If you really need to serialize and send thousands of elements in a single request, one way to avoid hitting memory limits is to process the elements one by one. Instead of loading all serialized elements into a huge array and then encoding it as JSON, load one element at a time, serialize and send it to the client before loading the next element.

I don't think this is supported by Element API by default, and it doesn't work well with JSON. You could think about using something like [ND JSON](http://ndjson.org/), though this would require some special care on the client-side and probably a custom serializer as well. At this point, it might be easier to build a [custom controller](https://craftcms.com/docs/3.x/extend/controllers.html) for your API.


---

