<p>OK, finally answering my own question here as it seems to not be an issue with the latest Craft 2.5 builds when <code>devMode</code> is turned off, but is still an issue when <code>devMode</code> is on.</p>

<p>As far as I can tell its something to do with the Craft logging tool spiralling out of control, but thats as far as I got.</p>

CC BY-SA 3.0
2015-12-15T11:38:08.443Z
Just to say was having similar issues until disabling dev mode
Cole Henley
CC BY-SA 3.0
2015-12-15T11:39:09.707Z
Yeah took me ages to work out!
Josh Angell
CC BY-SA 3.0
2016-11-03T11:31:52.630Z
https://craftcms.com/support/dev-mode This should cast some light on dev mode. Explains what is being logged and more
Krzysztof Boduch
CC BY-SA 3.0
2015-12-28T19:03:15.387Z
Also want to echo these guys, disabling dev mode fixed my memory issue with my tasks plugin. @JoshAngell have you contacted P&T about this?
Chase Giunta
CC BY-SA 3.0
2015-12-28T19:14:25.160Z
@ChaseGiunta yeah I have an open ticket with Brad, I think as its not likely to creep into production (or shouldn't rather) and its a nightmare to test against it's not going to be sorted any time soon. I had to profile it to find it.
Josh Angell
CC BY-SA 3.0
2015-12-15T11:28:36.493Z
2015-12-15T11:28:36.493Z
Josh Angell
<p>I am writing a plugin that imports a large amount of data form an API into some custom element types. I have it all set up and working nicely with Tasks however I am running into a fairly big issue with memory.</p>

<p>The plugin runs in the following way:</p>

<ol>
<li>Controlling Task gets set off.</li>
<li>It sets off a SubTask twice that gets two different sets of data from the API and saves it to disk.</li>
<li>It then sets off another bunch of SubTasks that import that data into my custom element types one element at a time - the number of Tasks at this point depends on how large the saved API data is as I split it into chunks of 100.</li>
</ol>

<p>In my testing I am skipping point 2 from above, so it is only reading stuff from disk, importing it and repeating. I am finding that no matter what I do I can't stop the memory usage building up. I have used <code>unset()</code> in various places which has helped a little bit and have turned off the <code>cacheElementQueries</code> config parameter using <code>craft()-&gt;config-&gt;set('cacheElementQueries', false);</code> which has also helped but I'm still running into problems.</p>

<p><strong>Some numbers</strong></p>

<ul>
<li>Resulting files from the API are 2.5MB and 5.3MB of json.</li>
<li>Total elements to import from each are 1200 and 2000 respectively.</li>
<li>As mentioned above I'm splitting the sets of data in to chunks of 100 to try and conserve memory.</li>
<li>php.ini <code>memory_limit</code> setting is 512M</li>
</ul>

<p><strong>Conclusion</strong></p>

<p>I feel like I'm missing something obvious as php doesn't dump the memory until all the Tasks have run, or at least that is what it seems like. I kind of thought that after each Task or SubTask had completed the memory for that would be dumped.</p>

<p>I have to confess I'm still a bit new to serious programming and am only using <code>htop</code> to monitor the process and can see it going up continually until it reaches my php.ini <code>memory_limit</code> setting. I know I should probably use xdebug or something but that kind of scares me...</p>

<p>I've also read this: <a href="https://craftcms.stackexchange.com/questions/740/memory-issue-when-updating-all-entries-from-a-channel-allowed-memory-size-of-x">Memory issue when updating all entries from a channel (allowed memory size of X bytes exhausted (tried to allocate Y bytes))</a></p>

<p>If anyone has any ideas however wacky to help with plugging my leaky ship I'd be most grateful! I can post up code samples if it will help but I didn't want to make this post too lengthy.</p>

<hr>

<h2>Update</h2>

<p>Here is a cut-down set of functions that are used in the process, I've not included lots of little bits that are either just doing processing on non-large variables or other mundane tasks.</p>



<p><strong>Main Task</strong></p>

<pre class="lang-php prettyprint-override"><code>public function runStep($step)
{

    // Sync the stored API data to the Craft elements
    return $this-&gt;runSubTask('MyPlugin_Sync', null, array(
        'apiElement'         =&gt; $this-&gt;_apiElements[$step]['apiElement'],
        'apiElementSettings' =&gt; $this-&gt;_apiElements[$step]['apiElementSettings'],
        'apiService'         =&gt; $service,
        'offset'             =&gt; $this-&gt;_apiElements[$step]['offset']
    ));

}
</code></pre>

<p><strong>Sync Task</strong></p>

<pre class="lang-php prettyprint-override"><code>public function getTotalSteps()
{

    $this-&gt;_apiData = craft()-&gt;myPlugin_ApiService-&gt;getElementsFromFile($this-&gt;_apiElement, $this-&gt;_offset, 100);

    // Check we got something
    if ( is_array($this-&gt;_apiData) )
    {
        return count($this-&gt;_apiData);
    }
    else
    {
        return 0;
    }

}

public function runStep($step)
{

    return craft()-&gt;myPlugin_ImportService-&gt;importElement($this-&gt;_apiService, $this-&gt;_apiElementSettings, array_shift($this-&gt;_apiData));

}
</code></pre>

<p><strong>API Service</strong></p>

<pre class="lang-php prettyprint-override"><code>public function getElementsFromFile($apiElement, $offset = 0, $limit = null)
{

    // Get the file
    $filename = $this-&gt;_storage . $apiElement . 's.json';

    $file = IOHelper::getFile($filename);

    $data = json_decode($file-&gt;getContents(), true);

    // Try and help the memory
    unset($file);

    // Now select the right portion from it using our offset amount
    return array_slice($data, $offset, $limit);

}

public function mapData($data, $settings)
{

    $return = array();

    // Boring mapping code using $data

    unset($data);

    return $return;

}
</code></pre>

<p><strong>Import Service</strong></p>

<pre class="lang-php prettyprint-override"><code>public function importElement($apiService, $apiElementSettings, $apiData)
{

    // using the API service, map the data to our fields/attributes held in the plugin settings
    $mappedData = craft()-&gt;myPlugin_ApiService-&gt;mapData($apiData, $apiElementSettings);

    // We’re done with this now, save our memory
    unset($apiData);

    // Standard code for getting an existing element and updating it
    // or creating a new one if none exist

    // Save the thing
    return craft()-&gt;myPlugin_myElement-&gt;saveMyElement($element);

}
</code></pre>

CC BY-SA 3.0
2015-02-13T08:48:01.743Z
Hey @ChadW I've just put some code up there - again thanks for any help you can offer.
Josh Angell
CC BY-SA 3.0
2015-02-13T13:21:48.123Z
I'll dig into this more later, but one thing that stands out is unset() doesn't necessarily free up memory, it just frees up the variable, as I understand it anyway. Thanks for posting your code.
Chad W
CC BY-SA 3.0
2015-02-13T13:22:53.920Z
Yeah thats what I thought - it did make a small difference in my testing, but then so did just setting it to `null`.
Josh Angell
CC BY-SA 3.0
2015-02-12T16:04:53.447Z
If you wouldn't mind posting some code, it might help us see if there is something that is amiss.
Chad W
CC BY-SA 3.0
2015-02-12T16:06:39.430Z
Also, I don't know if you've tried working with the `phpMaxMemoryLimit` config setting, although that might just be a bandaid in this case... http://buildwithcraft.com/docs/config-settings#phpMaxMemoryLimit
Chad W
CC BY-SA 3.0
2015-02-12T16:21:00.810Z
Thanks @ChadW - yes I've set that config setting to the same as my php.ini one and I will try and get some pared down code samples up tonight / tomorrow. Thanks for taking a look!
Josh Angell
<plugin-development><memory>
Running out of memory in my Task-based plugin
# Running out of memory in my Task-based plugin

(Asked by: username_0 on 2015-02-12)

I am writing a plugin that imports a large amount of data form an API into some custom element types. I have it all set up and working nicely with Tasks however I am running into a fairly big issue with memory.
The plugin runs in the following way:

Controlling Task gets set off.
It sets off a SubTask twice that gets two different sets of data from the API and saves it to disk.
It then sets off another bunch of SubTasks that import that data into my custom element types one element at a time - the number of Tasks at this point depends on how large the saved API data is as I split it into chunks of 100.

In my testing I am skipping point 2 from above, so it is only reading stuff from disk, importing it and repeating. I am finding that no matter what I do I can't stop the memory usage building up. I have used unset() in various places which has helped a little bit and have turned off the cacheElementQueries config parameter using craft()->config->set('cacheElementQueries', false); which has also helped but I'm still running into problems.
Some numbers

Resulting files from the API are 2.5MB and 5.3MB of json.
Total elements to import from each are 1200 and 2000 respectively.
As mentioned above I'm splitting the sets of data in to chunks of 100 to try and conserve memory.
php.ini memory_limit setting is 512M

Conclusion
I feel like I'm missing something obvious as php doesn't dump the memory until all the Tasks have run, or at least that is what it seems like. I kind of thought that after each Task or SubTask had completed the memory for that would be dumped.
I have to confess I'm still a bit new to serious programming and am only using htop to monitor the process and can see it going up continually until it reaches my php.ini memory_limit setting. I know I should probably use xdebug or something but that kind of scares me...
I've also read this: Memory issue when updating all entries from a channel (allowed memory size of X bytes exhausted (tried to allocate Y bytes))
If anyone has any ideas however wacky to help with plugging my leaky ship I'd be most grateful! I can post up code samples if it will help but I didn't want to make this post too lengthy.

Update
Here is a cut-down set of functions that are used in the process, I've not included lots of little bits that are either just doing processing on non-large variables or other mundane tasks.
Main Task
public function runStep($step)
{

    // Sync the stored API data to the Craft elements
    return $this->runSubTask('MyPlugin_Sync', null, array(
        'apiElement'         => $this->_apiElements[$step]['apiElement'],
        'apiElementSettings' => $this->_apiElements[$step]['apiElementSettings'],
        'apiService'         => $service,
        'offset'             => $this->_apiElements[$step]['offset']
    ));

}

Sync Task
public function getTotalSteps()
{

    $this->_apiData = craft()->myPlugin_ApiService->getElementsFromFile($this->_apiElement, $this->_offset, 100);

    // Check we got something
    if ( is_array($this->_apiData) )
    {
        return count($this->_apiData);
    }
    else
    {
        return 0;
    }

}

public function runStep($step)
{

    return craft()->myPlugin_ImportService->importElement($this->_apiService, $this->_apiElementSettings, array_shift($this->_apiData));

}

API Service
public function getElementsFromFile($apiElement, $offset = 0, $limit = null)
{

    // Get the file
    $filename = $this->_storage . $apiElement . 's.json';

    $file = IOHelper::getFile($filename);

    $data = json_decode($file->getContents(), true);

    // Try and help the memory
    unset($file);

    // Now select the right portion from it using our offset amount
    return array_slice($data, $offset, $limit);

}

public function mapData($data, $settings)
{

    $return = array();

    // Boring mapping code using $data

    unset($data);

    return $return;

}

Import Service
public function importElement($apiService, $apiElementSettings, $apiData)
{

    // using the API service, map the data to our fields/attributes held in the plugin settings
    $mappedData = craft()->myPlugin_ApiService->mapData($apiData, $apiElementSettings);

    // We’re done with this now, save our memory
    unset($apiData);

    // Standard code for getting an existing element and updating it
    // or creating a new one if none exist

    // Save the thing
    return craft()->myPlugin_myElement->saveMyElement($element);

}

**Tags:** plugin-development, memory

> **Comments:**
> **username_1**: If you wouldn't mind posting some code, it might help us see if there is something that is amiss.  
> **username_1**: Also, I don't know if you've tried working with the `phpMaxMemoryLimit` config setting, although that might just be a bandaid in this case... http://buildwithcraft.com/docs/config-settings#phpMaxMemoryLimit  
> **username_0**: Thanks @ChadW - yes I've set that config setting to the same as my php.ini one and I will try and get some pared down code samples up tonight / tomorrow. Thanks for taking a look!  
> **username_0**: Hey @ChadW I've just put some code up there - again thanks for any help you can offer.  
> **username_1**: I'll dig into this more later, but one thing that stands out is unset() doesn't necessarily free up memory, it just frees up the variable, as I understand it anyway. Thanks for posting your code.  
> **username_0**: Yeah thats what I thought - it did make a small difference in my testing, but then so did just setting it to `null`.  

---

## 1 answers

### Answer by username_0 on 2015-12-15. Score: 6 | Accepted answer

OK, finally answering my own question here as it seems to not be an issue with the latest Craft 2.5 builds when `devMode` is turned off, but is still an issue when `devMode` is on.

As far as I can tell its something to do with the Craft logging tool spiralling out of control, but thats as far as I got.

> **Comments:**
> **username_2**: Just to say was having similar issues until disabling dev mode  
> **username_0**: Yeah took me ages to work out!  
> **username_4**: Also want to echo these guys, disabling dev mode fixed my memory issue with my tasks plugin. @JoshAngell have you contacted P&T about this?  
> **username_0**: @ChaseGiunta yeah I have an open ticket with Brad, I think as its not likely to creep into production (or shouldn't rather) and its a nightmare to test against it's not going to be sorted any time soon. I had to profile it to find it.  
> **username_3**: https://craftcms.com/support/dev-mode This should cast some light on dev mode. Explains what is being logged and more  

---

