<p>Remember that queue jobs are run in seperate processes from the code that enqueues them. So the 100 elements you're putting in the queue need to be serialized, saved to the database and then deserialized. I'm not sure if this will even work at all, but even if it does, it will cause memory issues as you have experienced.</p>
<p>Instead of passing entire element objects to the queue, give it just an array of IDs:</p>
<pre class="lang-php prettyprint-override"><code>$elementIds = Entry::find()
    -&gt;section('imdData')
    -&gt;orderBy('id')
    -&gt;ids();
$queue = Craft::$app-&gt;getQueue();

foreach (array_chunk($elementIds, 100) as $ids) {
    $job = $queue-&gt;push(new Importgeojson(['elementIds' =&gt; $ids]));
}
</code></pre>
<p>Then you can use the IDs to get the corresponding elements from the database. Instead of querying for all elements at once (which will cause memory issues once again), process them one by one in your <code>Importgeojson</code> class:</p>
<pre class="lang-php prettyprint-override"><code>$elementIds = $this-&gt;elementIds ?? [];
foreach ($elementIds as $id) {
    $element = Entry::find()-&gt;anyStatus()-&gt;id($id)-&gt;one();
    if (!$element) continue;
    // process the current element …
}
</code></pre>
<p>In general, when dealing with large amounts of entries/datasets, try to process them one by one to avoid memory issues.</p>

CC BY-SA 4.0
2021-12-06T16:40:38.917Z
Thanks @MoritzLost - gonna give this a whirl
joomkit
CC BY-SA 4.0
2021-12-06T23:02:42.517Z
This is a fantastic improvement. It looks like it is moving through the queue nicely. Cant thank you enough for pointing the way. More testing to do now. THANK YOU
joomkit
CC BY-SA 4.0
2021-12-07T10:10:33.270Z
@joomkit Glad this solution worked for you! If your issue is solved, would you mind marking my answer accepted? Thanks!
MoritzLost
CC BY-SA 4.0
2021-12-06T15:45:43.967Z
2021-12-06T15:45:43.967Z
MoritzLost
<p>I am trying to batch a large queue job in my custom plugin.
Here is my service that starts the queue.</p>
<pre><code>    public function doBackGroundJob()
    {

        $query = Entry::find();
        if (!empty($this-&gt;criteria)) {
            Craft::configure($query, $this-&gt;criteria);
        }
        $query
            -&gt;section('imdData')
            -&gt;orderBy('id');


        foreach ($query-&gt;batch('100') as $elements) {

            $queue = Craft::$app-&gt;getQueue();
            $jobId = $queue-&gt;push(new Importgeojson(['elements' =&gt; $elements]));
            Craft::debug(
                Craft::t(
                    'mapitapi',
                    'Started import geo json queue job id: {jobId}',
                    [
                        'jobId' =&gt; $jobId,
                    ]
                ),
                __METHOD__
            );
        }
    }
</code></pre>
<p>This calls a job importGeoJson</p>
<pre><code>    public function execute($queue)
    {

        $totalElements = count($this-&gt;elements);
        $currentElement = 0;


        try {
            foreach ($this-&gt;elements as $element) {
            //foreach ($query-&gt;each() as $element) {

                if(!$element-&gt;geojson){


                $this-&gt;setProgress($queue, $currentElement++ / $totalElements);
                // mapitOLF data contains mysocietiey mapit source where we can get the area id from the lsoa in order to get the geojson
                $match = Entry::find()-&gt;section('mapitOlfData')-&gt;ons($element-&gt;lsoacode)-&gt;one(); //one because we want one entry returned and lsoanames are unique

                if ($match) {
                    // if we have a match then call the mapit api with the area id
                    $geoData = $this-&gt;fetchGeoJson($match-&gt;mapitAreaId);

                    $element-&gt;setFieldValue('mapitAreaId', $match-&gt;mapitAreaId);
                    $element-&gt;setFieldValue('geojson', $geoData);

                    if (!Craft::$app-&gt;getElements()-&gt;saveElement($element)) {
                        Craft::error(
                            ' || Couldnt save funding entry: ' . $element-&gt;id . ':' . $element-&gt;title . '| lsoa:' . $element-&gt;lsoacode,
                            __METHOD__
                        );
                    }else{
                        Craft::debug(
                            Craft::t(
                                'mapitapi',
                                'Sleeping for 1 secs ..saved funding entry: ' . $element-&gt;id . ':' . $element-&gt;title . '| lsoa:' . $element-&gt;lsoacode
                            ),
                            __METHOD__
                        );
                        sleep(1);
                    }
                }

                }
            }
        } catch (Exception $e) {
            echo '[error]: '
                . $e-&gt;getMessage()
                . 'exception while processing '
                . $currentElement . '/' . $totalElements
                . ' - processing asset: ' . $element-&gt;title
                . ' from field: ' . $element-&gt;lsoacode . PHP_EOL;

        }
    }
</code></pre>
<p>This just bombs out with memory issues straight away. PHP memory_limit is set to 512M</p>
<p>Is there a better way to batch the job?
Originally i was trying to load all the records at once and loop through them, relying on the async plugin to help with the tasks. This worked better than batching like this but still had memory issues.
Any assistance appreciated :)</p>

<plugin-development><queue>
How to batch a large queue job?
# How to batch a large queue job?

(Asked by: username_0 on 2021-12-06)

I am trying to batch a large queue job in my custom plugin.
Here is my service that starts the queue.
    public function doBackGroundJob()
    {

        $query = Entry::find();
        if (!empty($this->criteria)) {
            Craft::configure($query, $this->criteria);
        }
        $query
            ->section('imdData')
            ->orderBy('id');


        foreach ($query->batch('100') as $elements) {

            $queue = Craft::$app->getQueue();
            $jobId = $queue->push(new Importgeojson(['elements' => $elements]));
            Craft::debug(
                Craft::t(
                    'mapitapi',
                    'Started import geo json queue job id: {jobId}',
                    [
                        'jobId' => $jobId,
                    ]
                ),
                __METHOD__
            );
        }
    }

This calls a job importGeoJson
    public function execute($queue)
    {

        $totalElements = count($this->elements);
        $currentElement = 0;


        try {
            foreach ($this->elements as $element) {
            //foreach ($query->each() as $element) {

                if(!$element->geojson){


                $this->setProgress($queue, $currentElement++ / $totalElements);
                // mapitOLF data contains mysocietiey mapit source where we can get the area id from the lsoa in order to get the geojson
                $match = Entry::find()->section('mapitOlfData')->ons($element->lsoacode)->one(); //one because we want one entry returned and lsoanames are unique

                if ($match) {
                    // if we have a match then call the mapit api with the area id
                    $geoData = $this->fetchGeoJson($match->mapitAreaId);

                    $element->setFieldValue('mapitAreaId', $match->mapitAreaId);
                    $element->setFieldValue('geojson', $geoData);

                    if (!Craft::$app->getElements()->saveElement($element)) {
                        Craft::error(
                            ' || Couldnt save funding entry: ' . $element->id . ':' . $element->title . '| lsoa:' . $element->lsoacode,
                            __METHOD__
                        );
                    }else{
                        Craft::debug(
                            Craft::t(
                                'mapitapi',
                                'Sleeping for 1 secs ..saved funding entry: ' . $element->id . ':' . $element->title . '| lsoa:' . $element->lsoacode
                            ),
                            __METHOD__
                        );
                        sleep(1);
                    }
                }

                }
            }
        } catch (Exception $e) {
            echo '[error]: '
                . $e->getMessage()
                . 'exception while processing '
                . $currentElement . '/' . $totalElements
                . ' - processing asset: ' . $element->title
                . ' from field: ' . $element->lsoacode . PHP_EOL;

        }
    }

This just bombs out with memory issues straight away. PHP memory_limit is set to 512M
Is there a better way to batch the job?
Originally i was trying to load all the records at once and loop through them, relying on the async plugin to help with the tasks. This worked better than batching like this but still had memory issues.
Any assistance appreciated :)

**Tags:** plugin-development, queue

---

## 1 answers

### Answer by username_1 on 2021-12-06. Score: 3 | Accepted answer

Remember that queue jobs are run in seperate processes from the code that enqueues them. So the 100 elements you're putting in the queue need to be serialized, saved to the database and then deserialized. I'm not sure if this will even work at all, but even if it does, it will cause memory issues as you have experienced.
Instead of passing entire element objects to the queue, give it just an array of IDs:
$elementIds = Entry::find()
    ->section('imdData')
    ->orderBy('id')
    ->ids();
$queue = Craft::$app->getQueue();

foreach (array_chunk($elementIds, 100) as $ids) {
    $job = $queue->push(new Importgeojson(['elementIds' => $ids]));
}

Then you can use the IDs to get the corresponding elements from the database. Instead of querying for all elements at once (which will cause memory issues once again), process them one by one in your Importgeojson class:
$elementIds = $this->elementIds ?? [];
foreach ($elementIds as $id) {
    $element = Entry::find()->anyStatus()->id($id)->one();
    if (!$element) continue;
    // process the current element …
}

In general, when dealing with large amounts of entries/datasets, try to process them one by one to avoid memory issues.

> **Comments:**
> **username_0**: Thanks @MoritzLost - gonna give this a whirl  
> **username_0**: This is a fantastic improvement. It looks like it is moving through the queue nicely. Cant thank you enough for pointing the way. More testing to do now. THANK YOU  
> **username_1**: @joomkit Glad this solution worked for you! If your issue is solved, would you mind marking my answer accepted? Thanks!  

---

