<p>You can simply create a 'robots.txt' file in your templates folder and populate it using twig (templates do not need to end in <code>.twig</code>). Not sure what you're trying to control exactly, but something like this should technically work.</p>

<pre><code>User-agent: *
{% set entries = craft.entries.limit(null) %} 
{% for entry in entries %}
    Disallow: {{ entry.uri }}
{% endfor %}
</code></pre>

<p>Terrible example but you get the idea. Then just delete the robots.txt file that is in your public folder.</p>

CC BY-SA 3.0
2016-01-28T18:30:52.407Z
Also doesn't work. The problem turns out to be nginx configuration. But definitely the best solution once I have nginx tamed.
Marion Newlevant
CC BY-SA 3.0
2016-01-29T19:30:39.530Z
Glad you solved this — I thought perhaps that something else was going on with your setup, because (knowing you) I'm sure you tried this (or an equivalent). But doesn't hurt to start with the simplest answer :)
Douglas McDonald
CC BY-SA 3.0
2016-01-28T09:33:19.393Z
2016-01-28T09:33:19.393Z
Douglas McDonald
<p>Some good answers here.  This is not necessarily Craft related but wanted to quickly point out what's going on since it can involve Craft's routing.</p>

<p>When nginx—or any other web server for that matter—gets fed a request, it needs to figure out <em>what</em> to do with it. Is it a static file, a fastcgi/wsgi or proxy request, etc? </p>

<p>In general, the more specific you are with your config, the easier time nginx has to figure out what you want it to do. And requests like this (the <code>=</code> sign in this case) nginx will process first. After these location specific requests, nginx moves onto less and less specific requests. </p>

<p>How does this factor into the request getting to Craft? It's important to note that when you request an entry from Craft, the entry doesn't actually <em>exist</em> on the file system so, by default, nginx will throw up a 404. </p>

<p>Craft and other CMS that use a single-file controller pattern (index.php) are basically rewriting "would be" 404 errors back to index.php (<code>rewrite</code> or <code>RewriteRule</code> in Apache) to process them instead. nginx calls these content handlers. </p>

<p>I like to refer back to <a href="https://www.digitalocean.com/community/tutorials/understanding-nginx-server-and-location-block-selection-algorithms" rel="nofollow">this article</a> on how nginx processes a request.  It's good reading, much better than the official documentation.</p>

CC BY-SA 3.0
2016-02-01T01:13:41.427Z
2016-02-01T01:13:41.427Z
RitterKnight
<p>It turns out this is an nginx problem. <a href="https://trepmal.com/2014/03/28/nginx-robots-txt-and-copy-pasta/" rel="nofollow">This post</a> describes it:</p>

<blockquote>
  <p>The troubling part of the nginx configuration looks like this</p>
  
  <p><code>location = /robots.txt {
     access_log off;
     log_not_found off;
  }</code></p>
  
  <p>What’s inside isn’t the problem. It’s pretty basic, whether found (access_log) or not (log_not_found) don’t log it ( off;).</p>
  
  <p>The problem is with this: location = /robots.txt</p>
  
  <p>That equals sign means that when the request matches, perform only these rules and nothing else. When you have a robots file, then no problem – it just gets served up, plain and simple. But if you don’t, there’s nothing in that block that lets WordPress handle the request, so you get a 404.</p>
</blockquote>

<p>So if I comment out that part of the config file, everything works as it should.</p>

CC BY-SA 3.0
2016-01-31T23:55:04.713Z
As mentioned in the article you pointed to, you could also throw in a ``try_files $uri /index.php?$args`` at the end of that block to have nginx process that specific request via PHP. Since the there's no content handler and the file doesn't exist, nginx throws a 404.
RitterKnight
CC BY-SA 3.0
2016-01-28T18:35:29.790Z
2016-01-28T18:35:29.790Z
Marion Newlevant
<p>I would like my <code>robots.txt</code> file to be generated by Craft. Things that I have tried that don't work:</p>

<ul>
<li>a <code>craft/templates/robots.txt</code> template</li>
<li>a <code>craft/templates/robots.txt.twig</code> template</li>
<li>a route for <code>robots.txt</code> that points to a <code>robots.twig</code> template</li>
</ul>

<p>Both of these work fine for a <code>foo.txt</code> file. The error I get is <code>404</code>,  not the Craft 404, but the system one - nginx.</p>

<404><nginx>
Unable to manage robots.txt in Craft
# Unable to manage robots.txt in Craft

(Asked by: username_0 on 2016-01-28)

I would like my `robots.txt` file to be generated by Craft. Things that I have tried that don't work:

* a `craft/templates/robots.txt` template
* a `craft/templates/robots.txt.twig` template
* a route for `robots.txt` that points to a `robots.twig` template

Both of these work fine for a `foo.txt` file. The error I get is `404`, not the Craft 404, but the system one - nginx.

**Tags:** 404, nginx


---

## 3 answers

### Answer by username_0 on 2016-01-28. Score: 4 | Accepted answer

It turns out this is an nginx problem. [This post](https://trepmal.com/2014/03/28/nginx-robots-txt-and-copy-pasta/) describes it:

> The troubling part of the nginx configuration looks like this
>
> `location = /robots.txt {
> access_log off;
> log_not_found off;
> }`
>
> What’s inside isn’t the problem. It’s pretty basic, whether found (access\_log) or not (log\_not\_found) don’t log it ( off;).
>
> The problem is with this: location = /robots.txt
>
> That equals sign means that when the request matches, perform only these rules and nothing else. When you have a robots file, then no problem – it just gets served up, plain and simple. But if you don’t, there’s nothing in that block that lets WordPress handle the request, so you get a 404.

So if I comment out that part of the config file, everything works as it should.

> **Comments:**
> **username_2**: As mentioned in the article you pointed to, you could also throw in a ``try_files $uri /index.php?$args`` at the end of that block to have nginx process that specific request via PHP. Since the there's no content handler and the file doesn't exist, nginx throws a 404.  

---

### Answer by username_1 on 2016-01-28. Score: 5

You can simply create a 'robots.txt' file in your templates folder and populate it using twig (templates do not need to end in .twig). Not sure what you're trying to control exactly, but something like this should technically work.
User-agent: *
{% set entries = craft.entries.limit(null) %} 
{% for entry in entries %}
    Disallow: {{ entry.uri }}
{% endfor %}

Terrible example but you get the idea. Then just delete the robots.txt file that is in your public folder.

> **Comments:**
> **username_0**: Also doesn't work. The problem turns out to be nginx configuration. But definitely the best solution once I have nginx tamed.  
> **username_1**: Glad you solved this — I thought perhaps that something else was going on with your setup, because (knowing you) I'm sure you tried this (or an equivalent). But doesn't hurt to start with the simplest answer :)  

---

### Answer by username_2 on 2016-02-01. Score: 1

Some good answers here. This is not necessarily Craft related but wanted to quickly point out what's going on since it can involve Craft's routing.

When nginx—or any other web server for that matter—gets fed a request, it needs to figure out *what* to do with it. Is it a static file, a fastcgi/wsgi or proxy request, etc?

In general, the more specific you are with your config, the easier time nginx has to figure out what you want it to do. And requests like this (the `=` sign in this case) nginx will process first. After these location specific requests, nginx moves onto less and less specific requests.

How does this factor into the request getting to Craft? It's important to note that when you request an entry from Craft, the entry doesn't actually *exist* on the file system so, by default, nginx will throw up a 404.

Craft and other CMS that use a single-file controller pattern (index.php) are basically rewriting "would be" 404 errors back to index.php (`rewrite` or `RewriteRule` in Apache) to process them instead. nginx calls these content handlers.

I like to refer back to [this article](https://www.digitalocean.com/community/tutorials/understanding-nginx-server-and-location-block-selection-algorithms) on how nginx processes a request. It's good reading, much better than the official documentation.


---

