<p>After looking through this, it would probably be better off as a <a href="https://github.com/craftcms/cms/issues" rel="nofollow noreferrer">Github issue</a> than a Stack Exchange post, but answering here for posterity.</p>

<p>In short, this comes down to a <a href="https://github.com/craftcms/cms/commit/95f62074ebbcdd328b5403320be22e9c9756e3fd" rel="nofollow noreferrer">change</a> we had to make in Craft 3 to fix some other caching issues where we started to serialize element queries when saving them, so that when retrieving them later to determine if a cache should be busted, we would unserialize and execute the saved element query.</p>

<p>It turns out that serialization requires significantly more memory than not serializing.</p>

<p>In the case of your database, with your custom fields and plugins on both <code>shows</code> and <code>movies</code> sections, it takes about 1.1MB of memory to save each element query inside of your <code>{% cache %}</code> tags.</p>

<p>Given your current template code, with the nested <code>for</code> loops and the number of elements you're accessing in the cache tag, you're looking at about 7,000 element queries that Craft has to track, or 7.7GB of memory - hence the error you're hitting.</p>

<p>It's on our list to completely revamp the template caching engine to help mitigate scenarios like this (as well as other template caching issues people have <a href="https://github.com/craftcms/cms/issues?utf8=%E2%9C%93&amp;q=is%3Aissue+is%3Aopen+cache" rel="nofollow noreferrer">come across</a>)</p>

<p>In the meantime, you've got a few options.</p>

<ol>
<li><p>Revamp your template logic so you're not executing <a href="https://rob-bell.net/2009/06/a-beginners-guide-to-big-o-notation/" rel="nofollow noreferrer">O(n2)</a> database queries.  That could involve pure Twig/templates or a custom plugin where you can grab exactly the data your template needs from PHP/Querybuilder.</p></li>
<li><p>Set the <a href="https://docs.craftcms.com/v3/config/config-settings.html#cacheelementqueries" rel="nofollow noreferrer">cacheElementQueries</a> config setting to <code>false</code>. Craft will no long attempt to record/bust template caches, so you can come up with your own logic for busting them (cron job, plugin, etc.).</p></li>
<li><p>Use a plugin like <a href="https://github.com/mmikkel/CacheFlag-Craft3" rel="nofollow noreferrer">Cache Flag</a> (or a similar one).  It also doesn't attempt to save element queries, but provides a way to bust caches based on keywords.</p></li>
</ol>

CC BY-SA 4.0
2018-07-22T14:59:58.070Z
Thanks a lot, Brad, and sorry the late reply. I had no idea that it could rack up like that! I will create a GitHub issue for it.

In the meantime it seems that cacheFlag does the trick.

Out of curiosity – am I going at the structure wrong, are the queries themselves inefficient, or is it just a problem for caching?
KSP
CC BY-SA 4.0
2018-07-22T15:00:57.410Z
As I'm basically looping through a cinema program that I load from an external XML, I keep the Movies and the actual Shows as different entry types, and relate the shows to the movies. That way I can update the movie information in only one place.

But as I both list the shows in a daily program (fetching titles from the movies they are related to), and list the shows for a movie in a movie view, it's useful to have them separate. This also allows me to override information on specific shows (e.g. if there's Q&A or similar).
KSP
CC BY-SA 4.0
2020-02-18T19:10:01.463Z
I just updated a theater site to Craft 3 where we use a very similar setup and ran into the same cache issues.  

We have a temporary solution utilizing Cache Flag, but it's far from ideal.I'm in the process of exploring a major rework to how we handle cache on this site to resolve these issues and get things working like before, but I don't want to dig in too far if some changes are coming soon.  

Searched through Github issues and milestones and couldn't find anything directly related to an update on this. Any ETA on cache engine revamp?
jsun
CC BY-SA 4.0
2018-07-17T23:16:28.390Z
2018-07-17T23:16:28.390Z
Brad Bell
<p>I have a problem with using <code>{% cache %}</code>in Craft 3, although the code runs fine without caching, and it works in Craft 2.</p>

<p>I have narrowed the problem down to the following code:</p>

<pre><code>{% cache %}
  {% set upcomingShows = craft.entries.section('shows').orderBy('showTime asc').limit(null).all() %}
  {% for date, shows in upcomingShows|group("showTime|date_modify('-3 hours')|date('Y-m-d')") %}
    {% for show in shows %}
      {% set movie = craft.entries.relatedTo({ targetElement: show }).one() %}
    {% endfor %}
  {% endfor %}
{% endcache %}
</code></pre>

<p>It gives me the following fatal error:</p>

<pre><code>Allowed memory size of 268435456 bytes exhausted (tried to allocate 71002178 bytes)
</code></pre>

<p>If I skip the Twig <code>|group</code>filter, the cache works again:</p>

<pre><code>{% cache %}
  {% set upcomingShows = craft.entries.section('shows').orderBy('showTime asc').limit(null).all() %}
  {% for shows in upcomingShows %}
    {% for show in shows %}
      {% set movie = craft.entries.relatedTo({ targetElement: show }).one() %}
    {% endfor %}
  {% endfor %}
{% endcache %}
</code></pre>

<p>(I first thought the problem was the <code>relatedTo()</code> function, but it turns out it's the <code>|group</code> is what triggers it, without it it works also with the  <code>relatedTo()</code>.)</p>

<p>Is this a bug in the cache, or is there a way around it?</p>

<p><strong>UPDATE:</strong> After some more testing, it seems like it works <em>either</em> with the <code>|group</code> filter or the <code>relatedTo()</code> function, but not both at the same time.</p>

CC BY-SA 4.0
2018-07-14T20:27:01.277Z
How many entries are returned by the `upcomingShows` query?
Mats Mikkel Rummelhoff
CC BY-SA 4.0
2018-07-15T10:25:16.467Z
Approximately 50 or so, nothing massive
KSP
CC BY-SA 4.0
2018-07-16T23:12:02.433Z
If you want to send a database dump over to support@craftcms.com and reference this SE post, we can look into it.
Brad Bell
CC BY-SA 4.0
2018-07-16T23:24:19.427Z
@BradBell Thanks a million, I’ll send it over tomorrow.
KSP
CC BY-SA 4.0
2018-07-17T08:57:00.993Z
@BradBell I get an error trying to back up the database from the admin. Will a dump from phpMyAdmin do?
KSP
CC BY-SA 4.0
2018-07-17T18:47:12.453Z
@KSP That's actually the same
Robin Schambach
CC BY-SA 4.0
2018-07-17T19:28:10.793Z
Thanks, @RobinSchambach, figured it out and sent both versions + sample output from the part I’m trying to cache.
KSP
<templating><craft3><cache>
Caching in Craft 3 causes out of memory when including grouped for loop
# Caching in Craft 3 causes out of memory when including grouped for loop

(Asked by: username_0 on 2018-07-14)

I have a problem with using {% cache %}in Craft 3, although the code runs fine without caching, and it works in Craft 2.
I have narrowed the problem down to the following code:
{% cache %}
  {% set upcomingShows = craft.entries.section('shows').orderBy('showTime asc').limit(null).all() %}
  {% for date, shows in upcomingShows|group("showTime|date_modify('-3 hours')|date('Y-m-d')") %}
    {% for show in shows %}
      {% set movie = craft.entries.relatedTo({ targetElement: show }).one() %}
    {% endfor %}
  {% endfor %}
{% endcache %}

It gives me the following fatal error:
Allowed memory size of 268435456 bytes exhausted (tried to allocate 71002178 bytes)

If I skip the Twig |groupfilter, the cache works again:
{% cache %}
  {% set upcomingShows = craft.entries.section('shows').orderBy('showTime asc').limit(null).all() %}
  {% for shows in upcomingShows %}
    {% for show in shows %}
      {% set movie = craft.entries.relatedTo({ targetElement: show }).one() %}
    {% endfor %}
  {% endfor %}
{% endcache %}

(I first thought the problem was the relatedTo() function, but it turns out it's the |group is what triggers it, without it it works also with the  relatedTo().)
Is this a bug in the cache, or is there a way around it?
UPDATE: After some more testing, it seems like it works either with the |group filter or the relatedTo() function, but not both at the same time.

**Tags:** templating, craft3, cache
> **Comments:**
> **username_1**: How many entries are returned by the `upcomingShows` query?  
> **username_0**: Approximately 50 or so, nothing massive  
> **username_2**: If you want to send a database dump over to support@craftcms.com and reference this SE post, we can look into it.  
> **username_0**: @BradBell Thanks a million, I’ll send it over tomorrow.  
> **username_0**: @BradBell I get an error trying to back up the database from the admin. Will a dump from phpMyAdmin do?  
> **username_3**: @KSP That's actually the same  
> **username_0**: Thanks, @RobinSchambach, figured it out and sent both versions + sample output from the part I’m trying to cache.  

---

## 1 answers

### Answer by username_2 on 2018-07-17. Score: 2 | Accepted answer

After looking through this, it would probably be better off as a [Github issue](https://github.com/craftcms/cms/issues) than a Stack Exchange post, but answering here for posterity.

In short, this comes down to a [change](https://github.com/craftcms/cms/commit/95f62074ebbcdd328b5403320be22e9c9756e3fd) we had to make in Craft 3 to fix some other caching issues where we started to serialize element queries when saving them, so that when retrieving them later to determine if a cache should be busted, we would unserialize and execute the saved element query.

It turns out that serialization requires significantly more memory than not serializing.

In the case of your database, with your custom fields and plugins on both `shows` and `movies` sections, it takes about 1.1MB of memory to save each element query inside of your `{% cache %}` tags.

Given your current template code, with the nested `for` loops and the number of elements you're accessing in the cache tag, you're looking at about 7,000 element queries that Craft has to track, or 7.7GB of memory - hence the error you're hitting.

It's on our list to completely revamp the template caching engine to help mitigate scenarios like this (as well as other template caching issues people have [come across](https://github.com/craftcms/cms/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+cache))

In the meantime, you've got a few options.

1. Revamp your template logic so you're not executing [O(n2)](https://rob-bell.net/2009/06/a-beginners-guide-to-big-o-notation/) database queries. That could involve pure Twig/templates or a custom plugin where you can grab exactly the data your template needs from PHP/Querybuilder.
2. Set the [cacheElementQueries](https://docs.craftcms.com/v3/config/config-settings.html#cacheelementqueries) config setting to `false`. Craft will no long attempt to record/bust template caches, so you can come up with your own logic for busting them (cron job, plugin, etc.).
3. Use a plugin like [Cache Flag](https://github.com/mmikkel/CacheFlag-Craft3) (or a similar one). It also doesn't attempt to save element queries, but provides a way to bust caches based on keywords.

> **Comments:**
> **username_0**: Thanks a lot, Brad, and sorry the late reply. I had no idea that it could rack up like that! I will create a GitHub issue for it.  In the meantime it seems that cacheFlag does the trick.  Out of curiosity – am I going at the structure wrong, are the queries themselves inefficient, or is it just a problem for caching?  
> **username_0**: As I'm basically looping through a cinema program that I load from an external XML, I keep the Movies and the actual Shows as different entry types, and relate the shows to the movies. That way I can update the movie information in only one place.  But as I both list the shows in a daily program (fetching titles from the movies they are related to), and list the shows for a movie in a movie view, it's useful to have them separate. This also allows me to override information on specific shows (e.g. if there's Q&A or similar).  
> **username_4**: I just updated a theater site to Craft 3 where we use a very similar setup and ran into the same cache issues.    We have a temporary solution utilizing Cache Flag, but it's far from ideal.I'm in the process of exploring a major rework to how we handle cache on this site to resolve these issues and get things working like before, but I don't want to dig in too far if some changes are coming soon.    Searched through Github issues and milestones and couldn't find anything directly related to an update on this. Any ETA on cache engine revamp?  

---

