<p>Turns out the answer in my case is to use a <strong>JSON</strong> data source.</p>

<p>Importing users as <code>.json</code> instead of <code>.csv</code> appears to be about 10 times faster: </p>

<ul>
<li>With <code>.csv</code> importing 480 users took 2.5 hours</li>
<li>With <code>.json</code> importing 455 users took 15 minutes</li>
</ul>

<p>My JSON looked like this: </p>

<pre><code>{"user": [
    {
        "username":"jane.doe@example.com",
        "email":"jane.doe@example.com",
        "firstName":"Jane",
        "lastName":"Doe",
        "password":"User pass phrase",
        "groupName":"User group name"
    },
    {
        "username":"joe.bloggs@example.com",
        "email":"joe.bloggs@example.com",
        "firstName":"Joe",
        "lastName":"Bloggs",
        "password":"User pass phrase",
        "groupName":"User group name"
    }
]}
</code></pre>

CC BY-SA 4.0
2020-05-08T12:13:59.290Z
2020-05-08T12:13:59.290Z
Jonathan Schofield
<p>I'm aware of <a href="https://craftcms.stackexchange.com/questions/33048/craft-cms-feed-me-slow-performance">this thread on Feed Me slow performance</a>, but I'm raising it as a separate thread in case there is anything particular about my circumstance that is pertinent.</p>

<p>I'm running Craft 3.4.18 and trying to import a large number of <strong>users</strong> (about 1,500) into a specific user group using Feed Me: </p>

<ol>
<li>I'm using <code>.csv</code> sources on the local file system above web root</li>
<li>I'm importing:

<ul>
<li>Username (as email address string)</li>
<li>Email</li>
<li>First name</li>
<li>Last name</li>
<li>Shared password</li>
<li>User group string</li>
</ul></li>
<li>I'm batching imports in 500s but because things are so slow will now try smaller batches</li>
<li>I have read and attended to the <a href="https://docs.craftcms.com/feed-me/v4/troubleshooting.html#performance" rel="nofollow noreferrer">performance</a> aspects of the Feed Me troubleshooting docs</li>
<li>There is plenty of memory and execution time allocation to Craft and the instance is on a dedicated Linux server with 8Gb of RAM</li>
</ol>

<p>Imports are working without error, but paaaaaainfully slowly -- we're talking hours -- to the point of the import <em>process</em> being effectively useless*.</p>

<p>I have looked into <a href="https://nystudio107.com/blog/robust-queue-job-handling-in-craft-cms#solution-1-async-queue-plugin" rel="nofollow noreferrer">Andrew Welch's #1 solution</a> to running the queue as a background process, but have bumped into <a href="https://github.com/ostark/craft-async-queue/issues/32" rel="nofollow noreferrer">installation issues</a> with the <strong>Async Queue</strong> plugin.</p>

<p>Any suggestions much appreciated.</p>

<hr>

<p>I'd be happy in principle to do <strong>SQL inserts</strong> directly into the <code>craft_users</code> and <code>craft_usergroups_users</code> tables but I infer that's not an option since each hash of the shared password is unique and there is a unique UID to generate for each row too.</p>

<hr>

<p><strong>Update on #3 above:</strong></p>

<p>I'm currently running a batch of 100 instead of 500 and that makes no difference to speed whatsoever.</p>

<p>Next step is perhaps to try a JSON feed instead of CSV?</p>

<hr>

<p>*'Effectively useless' in the sense that although I have time to run extended imports right now, if it's going to generally take this long it's a non-starter for time-pressured imports.</p>

<hr>

<p><strong>Second update on #3 above:</strong></p>

<p>An import batch of 100 I started over 2 hours ago has crept along incrementally, with me periodically reviewing its status during that time and seeing barely any change, and then…</p>

<p>I have just revisited it and it suddenly went from 64% complete and unchanging to 100%. Now onto the next batch to see what happens.</p>

<plugin-feedme><queue>
Painfully slow Feed Me import of User accounts
# Painfully slow Feed Me import of User accounts

(Asked by: username_0 on 2020-05-07)

I'm aware of [this thread on Feed Me slow performance](https://craftcms.stackexchange.com/questions/33048/craft-cms-feed-me-slow-performance), but I'm raising it as a separate thread in case there is anything particular about my circumstance that is pertinent.

I'm running Craft 3.4.18 and trying to import a large number of **users** (about 1,500) into a specific user group using Feed Me:

1. I'm using `.csv` sources on the local file system above web root
2. I'm importing:
   * Username (as email address string)
   * Email
   * First name
   * Last name
   * Shared password
   * User group string
3. I'm batching imports in 500s but because things are so slow will now try smaller batches
4. I have read and attended to the [performance](https://docs.craftcms.com/feed-me/v4/troubleshooting.html#performance) aspects of the Feed Me troubleshooting docs
5. There is plenty of memory and execution time allocation to Craft and the instance is on a dedicated Linux server with 8Gb of RAM

Imports are working without error, but paaaaaainfully slowly -- we're talking hours -- to the point of the import *process* being effectively useless\*.

I have looked into [Andrew Welch's #1 solution](https://nystudio107.com/blog/robust-queue-job-handling-in-craft-cms#solution-1-async-queue-plugin) to running the queue as a background process, but have bumped into [installation issues](https://github.com/ostark/craft-async-queue/issues/32) with the **Async Queue** plugin.

Any suggestions much appreciated.

---

I'd be happy in principle to do **SQL inserts** directly into the `craft_users` and `craft_usergroups_users` tables but I infer that's not an option since each hash of the shared password is unique and there is a unique UID to generate for each row too.

---

**Update on #3 above:**

I'm currently running a batch of 100 instead of 500 and that makes no difference to speed whatsoever.

Next step is perhaps to try a JSON feed instead of CSV?

---

\*'Effectively useless' in the sense that although I have time to run extended imports right now, if it's going to generally take this long it's a non-starter for time-pressured imports.

---

**Second update on #3 above:**

An import batch of 100 I started over 2 hours ago has crept along incrementally, with me periodically reviewing its status during that time and seeing barely any change, and then…

I have just revisited it and it suddenly went from 64% complete and unchanging to 100%. Now onto the next batch to see what happens.

**Tags:** plugin-feedme, queue

---

## 1 answers

### Answer by username_0 on 2020-05-08. Score: 1

Turns out the answer in my case is to use a JSON data source.
Importing users as .json instead of .csv appears to be about 10 times faster: 

With .csv importing 480 users took 2.5 hours
With .json importing 455 users took 15 minutes

My JSON looked like this: 
{"user": [
    {
        "username":"jane.doe@example.com",
        "email":"jane.doe@example.com",
        "firstName":"Jane",
        "lastName":"Doe",
        "password":"User pass phrase",
        "groupName":"User group name"
    },
    {
        "username":"joe.bloggs@example.com",
        "email":"joe.bloggs@example.com",
        "firstName":"Joe",
        "lastName":"Bloggs",
        "password":"User pass phrase",
        "groupName":"User group name"
    }
]}


---

