<p>First you would need to figure out how to identify duplicates.</p>

<p>Craft doesn’t allow duplicate entry slugs; it will append “-1”, “-2”, etc. to them to keep them unique. So that would probably be a good starting point.</p>

<p>Use this query to get a list of all entries that have a number appended to the end of the slug. It’ll tell you the entry title, its section’s name, and the URL to edit it in the control panel. (Just update “example.com” with the actual host name.) Go through those and use your best judgement to decide which ones should be deleted.</p>

<pre><code>SELECT c.title, s.name AS sectionName, CONCAT('http://example.com/admin/entries/', s.handle, '/', el.elementId) AS cpUrl
FROM craft_elements_i18n el
INNER JOIN craft_content c ON c.elementId = el.elementId
INNER JOIN craft_entries en ON en.id = el.elementId
INNER JOIN craft_sections s ON s.id = en.sectionId
WHERE el.slug REGEXP '\-[0-9]+$'
</code></pre>

CC BY-SA 3.0
2014-06-30T18:38:44.207Z
Hi Brandon, thanks - that's a really useful piece of sql magic! However, I got two problems here: because my entries are imported with Bob Olde Hampsink's Import plugin I actually seem to have ended up with duplicate slugs. Secondly many of my entries have titles like this "Coop Obs Jessheim avd. 223000" with numbers in them. In lack of a better solution I'll try to manually remove the duplicates in the csv-files.

Thanks for a very useful code-snippet anyway!
alexbech
CC BY-SA 3.0
2014-06-30T18:57:44.667Z
The import plugin would have the same restrictions. Technically, slugs can be the same as long as the resulting URR would be different (e.g. the same slug across 2 different sections, or 2 different locales).
Brandon Kelly
CC BY-SA 3.0
2014-06-30T19:05:50.107Z
Ah I see, I've unchecked the "Entries in this section have their own URLs" - could that be it? Anyway, I modified your sql to just list duplicates based on title.

`SELECT c.title, count(c.title) as times, s.name AS sectionName
FROM craft_elements_i18n el
INNER JOIN craft_content c ON c.elementId = el.elementId
INNER JOIN craft_entries en ON en.id = el.elementId
INNER JOIN craft_sections s ON s.id = en.sectionId
GROUP BY c.title
HAVING times>1`

It turns out there are quite a few duplicates, now to figure out a safe way to delete them.
alexbech
CC BY-SA 3.0
2014-06-30T19:17:46.157Z
Ah yes -- the "-1"s only get appended when Craft is trying to come up with a unique URI. Good call.
Brandon Kelly
CC BY-SA 3.0
2014-06-30T10:21:35.757Z
2014-06-30T10:21:35.757Z
Brandon Kelly
<p>I have imported some data for a client, and it turns out there are some duplicates in the csv-files.</p>

<p>Is there an easy way to remove or filter out duplicates, either in my entries-query or in my template?</p>

<p>My query looks like this now</p>

<pre><code>{% set dealers = craft.entries({section:'dealers'}).order('dealerName') %}
</code></pre>

<p>Thanks!</p>

<entries><templating>
Is there a way to remove or filter out duplicate entries?
# Is there a way to remove or filter out duplicate entries?

(Asked by: username_0 on 2014-06-30)

I have imported some data for a client, and it turns out there are some duplicates in the csv-files.
Is there an easy way to remove or filter out duplicates, either in my entries-query or in my template?
My query looks like this now
{% set dealers = craft.entries({section:'dealers'}).order('dealerName') %}

Thanks!

**Tags:** entries, templating


---

## 1 answers

### Answer by username_1 on 2014-06-30. Score: 3 | Accepted answer

First you would need to figure out how to identify duplicates.
Craft doesn’t allow duplicate entry slugs; it will append “-1”, “-2”, etc. to them to keep them unique. So that would probably be a good starting point.
Use this query to get a list of all entries that have a number appended to the end of the slug. It’ll tell you the entry title, its section’s name, and the URL to edit it in the control panel. (Just update “example.com” with the actual host name.) Go through those and use your best judgement to decide which ones should be deleted.
SELECT c.title, s.name AS sectionName, CONCAT('http://example.com/admin/entries/', s.handle, '/', el.elementId) AS cpUrl
FROM craft_elements_i18n el
INNER JOIN craft_content c ON c.elementId = el.elementId
INNER JOIN craft_entries en ON en.id = el.elementId
INNER JOIN craft_sections s ON s.id = en.sectionId
WHERE el.slug REGEXP '\-[0-9]+$'

> **Comments:**
> **username_0**: Hi Brandon, thanks - that's a really useful piece of sql magic! However, I got two problems here: because my entries are imported with Bob Olde Hampsink's Import plugin I actually seem to have ended up with duplicate slugs. Secondly many of my entries have titles like this "Coop Obs Jessheim avd. 223000" with numbers in them. In lack of a better solution I'll try to manually remove the duplicates in the csv-files.  Thanks for a very useful code-snippet anyway!  
> **username_1**: The import plugin would have the same restrictions. Technically, slugs can be the same as long as the resulting URR would be different (e.g. the same slug across 2 different sections, or 2 different locales).  
> **username_0**: Ah I see, I've unchecked the "Entries in this section have their own URLs" - could that be it? Anyway, I modified your sql to just list duplicates based on title.  `SELECT c.title, count(c.title) as times, s.name AS sectionName FROM craft_elements_i18n el INNER JOIN craft_content c ON c.elementId = el.elementId INNER JOIN craft_entries en ON en.id = el.elementId INNER JOIN craft_sections s ON s.id = en.sectionId GROUP BY c.title HAVING times>1`  It turns out there are quite a few duplicates, now to figure out a safe way to delete them.  
> **username_1**: Ah yes -- the "-1"s only get appended when Craft is trying to come up with a unique URI. Good call.  

---

