<p>I think it's there to protect those who can't have craft/ below web root. In your case it can't be found by search engines, so you <em>can</em> delete it.</p>

CC BY-SA 3.0
2015-08-12T16:55:01.117Z
2015-08-12T16:55:01.117Z
carlcs
<p>The specific directive to disallow the <code>/craft/</code> directory isn't necessary if your craft folder is below the webroot.</p>

<p>However, it's good practice to have a <code>robots.txt</code> for each site, so I don't recommend deleting it altogether. You can change the directives to fit whatever robots criteria you need for your site. (For example, if you launch a beta site, you'd want to Disallow the whole thing to avoid duplicate content penalties.)</p>

CC BY-SA 3.0
2015-08-12T16:59:58.013Z
2015-08-12T16:59:58.013Z
Michael Rog
<p>I dont understand why we need the <code>robots.txt</code>file in the Craft CMS package. I think it is confusing. (I had a problem with Google indexing one of my BETA hidden website because of that.)</p>

<p>Reason : When I initiate my domain name on the hosting it creates automatically a <strong>public folder</strong> inside the <strong>virtual host main folder</strong>.  </p>

<p>So for example my hosting server make the domain name point to <code>/lamp0/web/vhosts/public/</code>. So I just have to upload <code>craft</code> and <code>public</code> folders below in <code>/lamp0/web/vhosts/</code>.<br>
Then I just have to rename <code>public</code> in <code>htdocs</code> and no need of <code>robots.txt</code> file because <code>craft</code> folder is below domain name root.</p>

<p><a href="https://i.sstatic.net/KEwFe.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/KEwFe.jpg" alt="enter image description here"></a></p>

<pre><code># Don't allow web crawlers to index Craft
User-agent: *
Disallow: /craft/
</code></pre>

<p>=========================<br>
<strong>UPDATE</strong><br>
OK I see now in the Craft install documentation (see below). I missed it sorry.<br>
Perhaps would it be good to insert a <code>README.txt</code> file in the package next to robots.txt file to explain that again :)
<a href="https://i.sstatic.net/JKDJ6.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/JKDJ6.jpg" alt="enter image description here"></a></p>

CC BY-SA 3.0
2015-08-12T17:00:05.953Z
Most probably don't. However, not all hosting providers allow users to edit files above the webroot (or some users just prefer both `/craft` and public files in the same directory), in which case providing a default robots.txt file preconfigured to disallow `/craft/` folder indexing is probably a good security measure (especially for those that don't know such a thing even exists). But feel free to add it as a suggestion to [http://feedback.buildwithcraft.com/](http://feedback.buildwithcraft.com/)
Douglas McDonald
<config><installation><seo>
File robots.txt needed in Craft package?
# File robots.txt needed in Craft package?

(Asked by: username_0 on 2015-08-12)

I dont understand why we need the robots.txtfile in the Craft CMS package. I think it is confusing. (I had a problem with Google indexing one of my BETA hidden website because of that.)
Reason : When I initiate my domain name on the hosting it creates automatically a public folder inside the virtual host main folder.  
So for example my hosting server make the domain name point to /lamp0/web/vhosts/public/. So I just have to upload craft and public folders below in /lamp0/web/vhosts/.
Then I just have to rename public in htdocs and no need of robots.txt file because craft folder is below domain name root.

# Don't allow web crawlers to index Craft
User-agent: *
Disallow: /craft/

=========================
UPDATE
OK I see now in the Craft install documentation (see below). I missed it sorry.
Perhaps would it be good to insert a README.txt file in the package next to robots.txt file to explain that again :)

**Tags:** config, installation, seo

> **Comments:**
> **username_1**: Most probably don't. However, not all hosting providers allow users to edit files above the webroot (or some users just prefer both `/craft` and public files in the same directory), in which case providing a default robots.txt file preconfigured to disallow `/craft/` folder indexing is probably a good security measure (especially for those that don't know such a thing even exists). But feel free to add it as a suggestion to [http://feedback.buildwithcraft.com/](http://feedback.buildwithcraft.com/)  

---

## 2 answers

### Answer by username_2 on 2015-08-12. Score: 3 | Accepted answer

I think it's there to protect those who can't have craft/ below web root. In your case it can't be found by search engines, so you *can* delete it.


---

### Answer by username_3 on 2015-08-12. Score: 4

The specific directive to disallow the `/craft/` directory isn't necessary if your craft folder is below the webroot.

However, it's good practice to have a `robots.txt` for each site, so I don't recommend deleting it altogether. You can change the directives to fit whatever robots criteria you need for your site. (For example, if you launch a beta site, you'd want to Disallow the whole thing to avoid duplicate content penalties.)


---

