<p>I've encountered this issue myself when syncing asset indexes with S3 and after going through the Pixel &amp; Tonic support channels, they advised me to create a plugin.</p>

<p>Currently my plugin only indexes sources that are held on S3 but I may update it in future to also do local sources. Assets will be getting a big overhaul for Craft 3 though. If you wish to move your assets to S3 and cannot find any other solutions to get it working then here is my plugin to help the indexing service: <a href="https://github.com/a-digital/assetindexer" rel="nofollow noreferrer">https://github.com/a-digital/assetindexer</a></p>

<p>I hope you manage to get it sorted.</p>

CC BY-SA 3.0
2016-12-08T13:45:18.020Z
Hey Matt, that looks great, i'll give that a go!! Thanks!!
John
CC BY-SA 3.0
2016-12-08T12:51:44.717Z
2016-12-08T12:51:44.717Z
Matt Shearing
<p>I've done this before with a combination of a Craft console command that replicates the indexing process, and a external way that processes each row.</p>

<pre><code>&lt;?php
namespace Craft;

class BusinessLogicCommand extends BaseCommand
{
    /**
    * ./craft/app/etc/console/yiic businesslogic --offset=x --sourceId=y --sessionId=z
    */
    public function actionIndex ($offset = '', $sourceId = '', $sessionId = '')
    {
        if ( $offset ) {
            $index = craft()-&gt;assetIndexing-&gt;processIndexForSource($sessionId, $offset, $sourceId);

            if ( $index ) {
                echo sprintf("Processed offset: %s for source %s", $offset, $sourceId);

                return 1;
            }
        }

        return 0;
    }

    /**
    * ./craft/app/etc/console/yiic businesslogic getIndexOverview
    */
    public function actionGetIndexOverview ($sourceIds = '')
    {

        // Get batches
        $batches   = array();
        $sessionId = craft()-&gt;assetIndexing-&gt;getIndexingSessionId();

        // Selection of sources or all sources?
        if ( $sourceIds ) {
            $sourceIds = explode(',', $sourceIds);
        }
        else {
            $sourceIds = craft()-&gt;assetSources-&gt;getViewableSourceIds();
        }

        $missingFolders = array();
        $grandTotal     = 0;

        foreach ($sourceIds as $sourceId) {
            // Get the indexing list
            $indexList = craft()-&gt;assetIndexing-&gt;getIndexListForSource($sessionId, $sourceId);

            if ( !empty($indexList['error']) ) {
                echo json_encode($indexList);

                return 0;
            }

            if ( isset($indexList['missingFolders']) ) {
                $missingFolders += $indexList['missingFolders'];
            }

            $batch = array();

            for ($i = 0; $i &lt; $indexList['total']; $i++) {
                $batch[] = [
                    'sessionId' =&gt; $sessionId,
                    'sourceId'  =&gt; $sourceId,
                    'total'     =&gt; $indexList['total'],
                    'offset'    =&gt; $i,
                    'process'   =&gt; 1,
                ];
            }

            $batches[] = $batch;
        }


        // Overview

        $assetsSourcesBeingIndexed = $sourceIds;
        $assetsMissingFolders      = $missingFolders;

        $job = [
            'batches' =&gt; $batches,
            'total'   =&gt; $grandTotal,
        ];

        echo json_encode($job);

    }


    public function actionCleanUpIndex ()
    {
        // Get overview
        $sourceIds      = $assetsSourcesBeingIndexed;
        $missingFiles   = craft()-&gt;assetIndexing-&gt;getMissingFiles($sourceIds, $sessionId);
        $missingFolders = $assetsMissingFolders;

        $responseArray = array();

        if ( !empty($missingFiles) || !empty($missingFolders) ) {
            $responseArray['confirm'] = craft()-&gt;templates-&gt;render('assets/_missing_items', array( 'missingFiles' =&gt; $missingFiles, 'missingFolders' =&gt; $missingFolders ));
            $responseArray['params']  = array( 'finish' =&gt; 1 );
        }
        // Clean up stale indexing data (all sessions that have all recordIds set)
        $sessionsInProgress = craft()-&gt;db-&gt;createCommand()
                                        -&gt;select('sessionId')
                                        -&gt;from('assetindexdata')
                                        -&gt;where('recordId IS NULL')
                                        -&gt;group('sessionId')
                                        -&gt;queryScalar();

        if ( empty($sessionsInProgress) ) {
            craft()-&gt;db-&gt;createCommand()-&gt;delete('assetindexdata');
        }
        else {
            craft()-&gt;db-&gt;createCommand()-&gt;delete('assetindexdata', array( 'not in', 'sessionId', $sessionsInProgress ));
        }

    }

}
</code></pre>

<p>This is done in 3 steps:</p>

<ol>
<li>Trigger getIndexOverview, which returns an JSON object string that lists all your index rows.</li>
<li>Use the JSON object in some way, and create a job for each row, which should trigger step #3.</li>
<li>Trigger the index command with the parameters  --offset=x --sourceId=y --sessionId=z</li>
</ol>

<p>I have used Laravel + Queue backed by Redis for this. If you have huge files, or a lot of data to process, think about spinning up a big VPS somewhere.</p>

<p>That way you can run 10-12 jobs in parallel, export your database/file structure, and tear the VPS down after. With hourly pricing, you'd end up paying a few $.</p>

CC BY-SA 3.0
2016-12-08T10:40:12.800Z
2016-12-08T11:39:55.223Z
2016-12-08T11:39:55.223Z
Fred Carlsen
<p>I've got an asset folder where i need to bring over about 20Gb of assets, obviously trying to update asset indexes from the Craft Cpanel is causing a serious failure.</p>

<p>Other than adding my assets in smaller chunks is there any other option to make it less memory intensive?</p>

<p>If the assets were on S3 rather than local, would it be less intense?</p>

<assets>
Asset indexing failing with large asset source
# Asset indexing failing with large asset source

(Asked by: username_0 on 2016-12-08)

I've got an asset folder where i need to bring over about 20Gb of assets, obviously trying to update asset indexes from the Craft Cpanel is causing a serious failure.

Other than adding my assets in smaller chunks is there any other option to make it less memory intensive?

If the assets were on S3 rather than local, would it be less intense?

**Tags:** assets

---

## 2 answers

### Answer by username_2 on 2016-12-08. Score: 6 | Accepted answer

I've done this before with a combination of a Craft console command that replicates the indexing process, and a external way that processes each row.
assetIndexing->processIndexForSource($sessionId, $offset, $sourceId);

            if ( $index ) {
                echo sprintf("Processed offset: %s for source %s", $offset, $sourceId);

                return 1;
            }
        }

        return 0;
    }

    /**
    * ./craft/app/etc/console/yiic businesslogic getIndexOverview
    */
    public function actionGetIndexOverview ($sourceIds = '')
    {

        // Get batches
        $batches   = array();
        $sessionId = craft()->assetIndexing->getIndexingSessionId();

        // Selection of sources or all sources?
        if ( $sourceIds ) {
            $sourceIds = explode(',', $sourceIds);
        }
        else {
            $sourceIds = craft()->assetSources->getViewableSourceIds();
        }

        $missingFolders = array();
        $grandTotal     = 0;

        foreach ($sourceIds as $sourceId) {
            // Get the indexing list
            $indexList = craft()->assetIndexing->getIndexListForSource($sessionId, $sourceId);

            if ( !empty($indexList['error']) ) {
                echo json_encode($indexList);

                return 0;
            }

            if ( isset($indexList['missingFolders']) ) {
                $missingFolders += $indexList['missingFolders'];
            }

            $batch = array();

            for ($i = 0; $i < $indexList['total']; $i++) {
                $batch[] = [
                    'sessionId' => $sessionId,
                    'sourceId'  => $sourceId,
                    'total'     => $indexList['total'],
                    'offset'    => $i,
                    'process'   => 1,
                ];
            }

            $batches[] = $batch;
        }


        // Overview

        $assetsSourcesBeingIndexed = $sourceIds;
        $assetsMissingFolders      = $missingFolders;

        $job = [
            'batches' => $batches,
            'total'   => $grandTotal,
        ];

        echo json_encode($job);

    }


    public function actionCleanUpIndex ()
    {
        // Get overview
        $sourceIds      = $assetsSourcesBeingIndexed;
        $missingFiles   = craft()->assetIndexing->getMissingFiles($sourceIds, $sessionId);
        $missingFolders = $assetsMissingFolders;

        $responseArray = array();

        if ( !empty($missingFiles) || !empty($missingFolders) ) {
            $responseArray['confirm'] = craft()->templates->render('assets/_missing_items', array( 'missingFiles' => $missingFiles, 'missingFolders' => $missingFolders ));
            $responseArray['params']  = array( 'finish' => 1 );
        }
        // Clean up stale indexing data (all sessions that have all recordIds set)
        $sessionsInProgress = craft()->db->createCommand()
                                        ->select('sessionId')
                                        ->from('assetindexdata')
                                        ->where('recordId IS NULL')
                                        ->group('sessionId')
                                        ->queryScalar();

        if ( empty($sessionsInProgress) ) {
            craft()->db->createCommand()->delete('assetindexdata');
        }
        else {
            craft()->db->createCommand()->delete('assetindexdata', array( 'not in', 'sessionId', $sessionsInProgress ));
        }

    }

}

This is done in 3 steps:

Trigger getIndexOverview, which returns an JSON object string that lists all your index rows.
Use the JSON object in some way, and create a job for each row, which should trigger step #3.
Trigger the index command with the parameters  --offset=x --sourceId=y --sessionId=z

I have used Laravel + Queue backed by Redis for this. If you have huge files, or a lot of data to process, think about spinning up a big VPS somewhere.
That way you can run 10-12 jobs in parallel, export your database/file structure, and tear the VPS down after. With hourly pricing, you'd end up paying a few $.


---

### Answer by username_1 on 2016-12-08. Score: 3

I've encountered this issue myself when syncing asset indexes with S3 and after going through the Pixel & Tonic support channels, they advised me to create a plugin.

Currently my plugin only indexes sources that are held on S3 but I may update it in future to also do local sources. Assets will be getting a big overhaul for Craft 3 though. If you wish to move your assets to S3 and cannot find any other solutions to get it working then here is my plugin to help the indexing service: <https://github.com/a-digital/assetindexer>

I hope you manage to get it sorted.

> **Comments:**
> **username_0**: Hey Matt, that looks great, i'll give that a go!! Thanks!!  

---

