<p>If you're used to another CMS such as WordPress, you're probably used to using a <code>Discourage search engines from indexing this site</code> option. As <a href="https://craftcms.stackexchange.com/a/23294/1016">Robin's answer</a> mentions, the training wheels are off. </p>

<p>However, that doesn't mean you can't replicate WP's functionality.</p>

<p>When you tick that box, there's a few things that happen behind the scenes:</p>

<ul>
<li>If <code>robots.txt</code> file is requested, WP auto generates a version that tells google to skip indexing the site.</li>
<li>A <code>&lt;meta name="robots" content="noindex,follow"&gt;</code> is added to each page.</li>
<li>XML-RPC pinging is disabled (so WP won't "ping" services like ping-o-matic when you update pages on the site).</li>
</ul>

<p>Craft doesn't automatically ping any services so you don't have to worry about the last option but the first 2 are pretty straightforward to setup.</p>

<h1>Global Lightswitch / templates</h1>

<p>Create a <a href="https://craftcms.com/docs/globals" rel="noreferrer">Global</a> called <code>siteSettings</code> and add a Lightswitch with a field handle of <code>disableIndexing</code> so you can reference throughout the site.</p>

<p>In the <code>&lt;head&gt;</code> section of your templates, it can be as simple as:</p>

<pre><code>{% if siteSettings.disableIndexing %}
    &lt;meta name="robots" content="noindex,follow"&gt;
{% endif %}
</code></pre>

<h1>robots.txt</h1>

<p>The second piece is some Craft magic. If you don't already have a <code>robots.txt</code> file, create one inside Craft's <code>templates</code> folder or remove it/move it if it exists in your public root folder.  </p>

<p>Craft is smart enough to parse your <code>robots.txt</code> file as Twig code and even output the proper mime type. This allows you to dynamically change it as it is requested.</p>

<p>When <code>robots.txt</code> is requested, instead of serving the file directly as your web server would, Craft will instead take the request and because there is a file that matches <code>robots.txt</code> inside templates folder, it'll use it.</p>

<p>Put this in your <code>robots.txt</code> file:</p>

<pre><code>User-agent: *
Disallow: {% if siteSettings.disableIndexing %}/{% endif %}
</code></pre>

<p>We're basically telling search engines not to index anything when the Lightswitch is ticked. That's a pretty basic robots.txt file but it gets the job done for this example. There's a ton more <a href="https://moz.com/learn/seo/robotstxt" rel="noreferrer">options</a>.</p>

<h1>Wrap up</h1>

<p>That's really it.  The only thing to remember is to UNTICK the box when your site goes live or when you want search engines to revisit. </p>

<p>Lots of people in WP land get burned because they forget to turn it back on so this isn't really a great thing to do in the first place but this answer goes into some basic Craft concepts.</p>

<p>If you want more control, instead of a Global lightswitch, you could setup one up on a per entry basis much like Yoast does. This is pretty common for landing pages used for PPC so they don't get indexed to begin with. Just remember not to include them in your sitemap as well.  Also consider the <a href="https://github.com/nystudio107/seomatic" rel="noreferrer">SEOMatic</a> plugin which can do all of this and more.</p>

CC BY-SA 3.0
2017-12-26T17:16:20.997Z
2017-12-26T17:22:24.300Z
2017-12-26T17:22:24.300Z
RitterKnight
<p>Craft 3.5.10 added a new <a href="https://craftcms.com/docs/3.x/config/config-settings.html#disallowrobots" rel="nofollow noreferrer"><code>disallowRobots</code></a> config setting. If set to <code>true</code>, Craft will add a <code>X-Robots-Tag: none</code> HTTP header to frontend requests, indicating to search engines that the page should not be indexed or crawled.</p>

CC BY-SA 4.0
2022-03-17T22:52:54.017Z
Beware, on a recent build, I found this tag/header to be enabled by default, and it took some time to track down why Google was not indexing the site.
4midori
CC BY-SA 4.0
2022-03-18T07:18:32.140Z
The default value for the `disallowRobots` setting is (and has always been) `false`. [It *is* enabled (i.e. set to `true`) in the official Composer starter project, though](https://github.com/craftcms/craft/blob/main/config/general.php#L36) – unless the current environment does not have an environment variable `ENVIRONMENT` set to `'production'`. Probably that is what you ran into?
Mats Mikkel Rummelhoff
CC BY-SA 4.0
2022-03-19T18:28:57.477Z
Not sure. On this build, I don't think I made any changes in general.php or .env. I used an unfamiliar web host, so this setting may have been incorrect from the start.
4midori
CC BY-SA 4.0
2022-03-14T09:08:54.903Z
2022-03-14T09:08:54.903Z
Mats Mikkel Rummelhoff
<p>You could also use this code in <code>robots.txt</code> file placed inside Craft's template folder.</p>
<pre><code>{# Disallows #}
{% if craft.app.config.env != 'production' %}

{# Disallow access to everything when NOT in production #}
User-agent: *
Disallow: /

{% else %}

{# Disallow access to cpresources/ when live #}
User-agent: *
Disallow: /cpresources/

{% endif %}
</code></pre>

CC BY-SA 4.0
2022-03-10T09:33:10.540Z
2022-03-10T09:33:10.540Z
Samuel
<p>Crafts does absolutely nothing in your frontend by itself unless you use certain twig functions (that mostly render css/js files). So it's totally up to you what seo options you provide. It's note like other cms that render your headers or footers or create  non editable links thus you have the freedom to define everything by yourself. </p>

<p>So if you don't want to appear in search engines just don't include seo optimisations. If your question is about how to do that you are in the wrong board here because that is off topic since it's not related to craft </p>

CC BY-SA 3.0
2017-12-26T13:41:10.693Z
2017-12-26T13:49:10.693Z
2017-12-26T13:49:10.693Z
Robin Schambach
<p>Can you let me know what the process would be? I heard there was a no track tag we can put in the site header so the sites won't show up in search engines.</p>

<search>
Need no search engine tracking option in Craft websites
# Need no search engine tracking option in Craft websites

(Asked by: username_0 on 2017-12-26)

Can you let me know what the process would be? I heard there was a no track tag we can put in the site header so the sites won't show up in search engines.

**Tags:** search


---

## 4 answers

### Answer by username_1 on 2017-12-26. Score: 6

If you're used to another CMS such as WordPress, you're probably used to using a Discourage search engines from indexing this site option. As Robin's answer mentions, the training wheels are off. 
However, that doesn't mean you can't replicate WP's functionality.
When you tick that box, there's a few things that happen behind the scenes:

If robots.txt file is requested, WP auto generates a version that tells google to skip indexing the site.
A  is added to each page.
XML-RPC pinging is disabled (so WP won't "ping" services like ping-o-matic when you update pages on the site).

Craft doesn't automatically ping any services so you don't have to worry about the last option but the first 2 are pretty straightforward to setup.
Global Lightswitch / templates
Create a Global called siteSettings and add a Lightswitch with a field handle of disableIndexing so you can reference throughout the site.
In the  section of your templates, it can be as simple as:
{% if siteSettings.disableIndexing %}
    
{% endif %}

robots.txt
The second piece is some Craft magic. If you don't already have a robots.txt file, create one inside Craft's templates folder or remove it/move it if it exists in your public root folder.  
Craft is smart enough to parse your robots.txt file as Twig code and even output the proper mime type. This allows you to dynamically change it as it is requested.
When robots.txt is requested, instead of serving the file directly as your web server would, Craft will instead take the request and because there is a file that matches robots.txt inside templates folder, it'll use it.
Put this in your robots.txt file:
User-agent: *
Disallow: {% if siteSettings.disableIndexing %}/{% endif %}

We're basically telling search engines not to index anything when the Lightswitch is ticked. That's a pretty basic robots.txt file but it gets the job done for this example. There's a ton more options.
Wrap up
That's really it.  The only thing to remember is to UNTICK the box when your site goes live or when you want search engines to revisit. 
Lots of people in WP land get burned because they forget to turn it back on so this isn't really a great thing to do in the first place but this answer goes into some basic Craft concepts.
If you want more control, instead of a Global lightswitch, you could setup one up on a per entry basis much like Yoast does. This is pretty common for landing pages used for PPC so they don't get indexed to begin with. Just remember not to include them in your sitemap as well.  Also consider the SEOMatic plugin which can do all of this and more.


---

### Answer by username_2 on 2022-03-14. Score: 3

Craft 3.5.10 added a new [`disallowRobots`](https://craftcms.com/docs/3.x/config/config-settings.html#disallowrobots) config setting. If set to `true`, Craft will add a `X-Robots-Tag: none` HTTP header to frontend requests, indicating to search engines that the page should not be indexed or crawled.

> **Comments:**
> **username_3**: Beware, on a recent build, I found this tag/header to be enabled by default, and it took some time to track down why Google was not indexing the site.  
> **username_2**: The default value for the `disallowRobots` setting is (and has always been) `false`. [It *is* enabled (i.e. set to `true`) in the official Composer starter project, though](https://github.com/craftcms/craft/blob/main/config/general.php#L36) – unless the current environment does not have an environment variable `ENVIRONMENT` set to `'production'`. Probably that is what you ran into?  
> **username_3**: Not sure. On this build, I don't think I made any changes in general.php or .env. I used an unfamiliar web host, so this setting may have been incorrect from the start.  

---

### Answer by username_4 on 2022-03-10. Score: 1

You could also use this code in robots.txt file placed inside Craft's template folder.
{# Disallows #}
{% if craft.app.config.env != 'production' %}

{# Disallow access to everything when NOT in production #}
User-agent: *
Disallow: /

{% else %}

{# Disallow access to cpresources/ when live #}
User-agent: *
Disallow: /cpresources/

{% endif %}


---

### Answer by username_5 on 2017-12-26. Score: 1

Crafts does absolutely nothing in your frontend by itself unless you use certain twig functions (that mostly render css/js files). So it's totally up to you what seo options you provide. It's note like other cms that render your headers or footers or create non editable links thus you have the freedom to define everything by yourself.

So if you don't want to appear in search engines just don't include seo optimisations. If your question is about how to do that you are in the wrong board here because that is off topic since it's not related to craft


---

