<p>Craft will set a <code>X-Robots-Tag: none</code> header for front end requests, if any of the following are true:</p>
<ol>
<li><p>The <a href="https://craftcms.com/docs/3.x/config/config-settings.html#disallowrobots" rel="nofollow noreferrer"><code>disallowRobots</code></a> general config setting has been set to <code>true</code> in your <code>config/general.php</code> file</p>
</li>
<li><p>The request URL includes a valid token (as per the <a href="https://craftcms.com/docs/3.x/config/config-settings.html#tokenparam" rel="nofollow noreferrer"><code>tokenParam</code></a> config setting (default is <code>'token'</code>).</p>
</li>
<li><p>The request URL includes a valid preview param <code>x-craft-preview</code> or <code>x-craft-live-preview</code> (note: this only applies for Craft 4.12+ and 5.4+, which are upcoming releases at the time of writing this).</p>
</li>
</ol>
<p>If neither of these are the case, you might have a plugin installed that is setting the <code>X-Robots-Tag: none</code> header. For example, I believe SEOMatic will set it if Craft's <code>devMode</code> config setting is set to <code>true</code> (which should not be the case for sites in production, for a <a href="https://craftcms.com/knowledge-base/what-dev-mode-does" rel="nofollow noreferrer">plethora of reasons</a>).</p>

CC BY-SA 4.0
2024-08-17T22:24:52.270Z
Thanks very much for your reply. It helped very much. disallowRobots wasn't set to true and I didn't have any plugins that interfered with devMode. In general.php I had 'devMode' => false when maybe it should have been $isProd (not sure if that made a difference) but what I think was the problem was in .env (which I didn't realise) I had ENVIRONMENT=dev when it should be ENVIRONMENT=production. It is now being indexed. Your pointers set me on the right path.
cannon303
CC BY-SA 4.0
2024-08-17T23:39:07.700Z
@cannon303 Ah, happy to hear that you sorted it out! Getting those environments mixed up is a common gotcha for sure.
Mats Mikkel Rummelhoff
CC BY-SA 4.0
2024-08-21T15:00:24.190Z
I like to `->disallowRobots(App::env('CRAFT_DISALLOW_ROBOTS') ?? false)` on `config/general.php` then add `CRAFT_DISALLOW_ROBOTS=true` to .env
Leo Leoncio
CC BY-SA 4.0
2024-08-21T21:49:09.650Z
@LeoLeoncio Env vars prefixed with `CRAFT_` will actually override whatever is in your `general.php` file, so in practice the above is completely redundant – i.e. if you removed `disallowRobots` from your config file altogether (the default is `false`) and had `CRAFT_DISALLOW_ROBOTS=true` in your `.env` file in applicable environments, you'd have the exact same result. See https://craftcms.com/docs/5.x/configure.html#environment-overrides
Mats Mikkel Rummelhoff
CC BY-SA 4.0
2024-08-15T21:37:10.417Z
2024-09-02T21:17:22.550Z
2024-09-02T21:17:22.550Z
Mats Mikkel Rummelhoff
<p>Here's how I approach this (without seomatic)</p>
<p>.env</p>
<pre><code>CRAFT_ENVIRONMENT=dev

CRAFT_DEV_MODE=true
CRAFT_DISALLOW_ROBOTS=true
</code></pre>
<p>config/custom.php</p>
<pre><code>&lt;?php

use craft\helpers\App;

$isProd = in_array(App::env('CRAFT_ENVIRONMENT'), ['prod', 'production', 'live']);

return [
    'isProd' =&gt; $isProd
];
</code></pre>
<p>_layout.twig</p>
<p><code>{% set isprod = craft.app.config.custom.isProd ?? false %}</code></p>
<p>inside <code>&lt;head&gt;&lt;/head&gt;</code></p>
<pre><code>{{ not isprod ? tag('meta', {
    name: 'robots',
    content: 'noindex'
}) }}
{{ not isprod ? tag('meta', {
    name: 'googlebot',
    content: 'noindex'
}) }}
</code></pre>
<p>config/routes.php</p>
<pre><code>&lt;?php

return [
    'robots.txt' =&gt; ['template' =&gt; 'routes/robots.txt']
];
</code></pre>
<p>templates/routes/robots.txt.twig</p>
<pre><code>{% set isprod = craft.app.config.custom.isProd ?? false %}
{% header &quot;Content-Type: text/plain; charset=utf-8&quot; %}
{% if isprod %}
# live
User-agent: *
Disallow: /admin/
Disallow: /cpresources/
Disallow: /cache/
sitemap: https://my-site.com/sitemap.xml
{% else %}
# lower env
User-agent: *
Disallow: /
{% endif %}
</code></pre>

CC BY-SA 4.0
2024-08-21T15:17:49.120Z
2024-08-21T15:17:49.120Z
Leo Leoncio
<p>Hello my website is not being indexed by Google. In Google Lighthouse it says:</p>
<pre><code>Page is blocked from indexing

Blocking Directive Source X-Robots-Tag: none
</code></pre>
<p>However I cannot find anything that could be causing the blocking. There is nothing in the .htaccess file that would point to anything and I don't have a robots.txt file and in the head of the pages I have:</p>
<pre><code>&lt;meta name=&quot;robots&quot; content=&quot;index,follow&quot; /&gt;
&lt;meta name=&quot;GOOGLEBOT&quot; content=&quot;INDEX, FOLLOW&quot; /&gt;
</code></pre>
<p>If you want to take a look the website is the-ropewalk.co.uk and is built using Craft CMS 3x. It's not a new site at all. It's about 4 years old and it was indexed at some point so something has changed. I did a full search on the site and indeed some of the templates used for the shop had robots set to no-follow but those were only used on the shop and not form the rest of the site or home page. I hyave changed those but still no joy.</p>
<p>Is there any settings that CraftCMS has that I might have accidentally set that might mean the site is not visible to Google?</p>
<p>What else should I check please?</p>
<p>Any pointers would be helpful.</p>
<p>Thank you.</p>

<craft3>
Website not getting indexed with Google
# Website not getting indexed with Google

(Asked by: username_0 on 2024-08-15)

Hello my website is not being indexed by Google. In Google Lighthouse it says:
Page is blocked from indexing

Blocking Directive Source X-Robots-Tag: none

However I cannot find anything that could be causing the blocking. There is nothing in the .htaccess file that would point to anything and I don't have a robots.txt file and in the head of the pages I have:



If you want to take a look the website is the-ropewalk.co.uk and is built using Craft CMS 3x. It's not a new site at all. It's about 4 years old and it was indexed at some point so something has changed. I did a full search on the site and indeed some of the templates used for the shop had robots set to no-follow but those were only used on the shop and not form the rest of the site or home page. I hyave changed those but still no joy.
Is there any settings that CraftCMS has that I might have accidentally set that might mean the site is not visible to Google?
What else should I check please?
Any pointers would be helpful.
Thank you.

**Tags:** craft3

---

## 2 answers

### Answer by username_1 on 2024-08-15. Score: 3 | Accepted answer

Craft will set a `X-Robots-Tag: none` header for front end requests, if any of the following are true:

1. The [`disallowRobots`](https://craftcms.com/docs/3.x/config/config-settings.html#disallowrobots) general config setting has been set to `true` in your `config/general.php` file
2. The request URL includes a valid token (as per the [`tokenParam`](https://craftcms.com/docs/3.x/config/config-settings.html#tokenparam) config setting (default is `'token'`).
3. The request URL includes a valid preview param `x-craft-preview` or `x-craft-live-preview` (note: this only applies for Craft 4.12+ and 5.4+, which are upcoming releases at the time of writing this).

If neither of these are the case, you might have a plugin installed that is setting the `X-Robots-Tag: none` header. For example, I believe SEOMatic will set it if Craft's `devMode` config setting is set to `true` (which should not be the case for sites in production, for a [plethora of reasons](https://craftcms.com/knowledge-base/what-dev-mode-does)).

> **Comments:**
> **username_0**: Thanks very much for your reply. It helped very much. disallowRobots wasn't set to true and I didn't have any plugins that interfered with devMode. In general.php I had 'devMode' => false when maybe it should have been $isProd (not sure if that made a difference) but what I think was the problem was in .env (which I didn't realise) I had ENVIRONMENT=dev when it should be ENVIRONMENT=production. It is now being indexed. Your pointers set me on the right path.  
> **username_1**: @cannon303 Ah, happy to hear that you sorted it out! Getting those environments mixed up is a common gotcha for sure.  
> **username_2**: I like to `->disallowRobots(App::env('CRAFT_DISALLOW_ROBOTS') ?? false)` on `config/general.php` then add `CRAFT_DISALLOW_ROBOTS=true` to .env  
> **username_1**: @LeoLeoncio Env vars prefixed with `CRAFT_` will actually override whatever is in your `general.php` file, so in practice the above is completely redundant – i.e. if you removed `disallowRobots` from your config file altogether (the default is `false`) and had `CRAFT_DISALLOW_ROBOTS=true` in your `.env` file in applicable environments, you'd have the exact same result. See https://craftcms.com/docs/5.x/configure.html#environment-overrides  

---

### Answer by username_2 on 2024-08-21. Score: 1

Here's how I approach this (without seomatic)
.env
CRAFT_ENVIRONMENT=dev

CRAFT_DEV_MODE=true
CRAFT_DISALLOW_ROBOTS=true

config/custom.php
 $isProd
];

_layout.twig
{% set isprod = craft.app.config.custom.isProd ?? false %}
inside 
{{ not isprod ? tag('meta', {
    name: 'robots',
    content: 'noindex'
}) }}
{{ not isprod ? tag('meta', {
    name: 'googlebot',
    content: 'noindex'
}) }}

config/routes.php
 ['template' => 'routes/robots.txt']
];

templates/routes/robots.txt.twig
{% set isprod = craft.app.config.custom.isProd ?? false %}
{% header "Content-Type: text/plain; charset=utf-8" %}
{% if isprod %}
# live
User-agent: *
Disallow: /admin/
Disallow: /cpresources/
Disallow: /cache/
sitemap: https://my-site.com/sitemap.xml
{% else %}
# lower env
User-agent: *
Disallow: /
{% endif %}


---

