Spaces:
Running
Running
ai: Include Sitemap into REP.
Browse files* Okay, so i forgot this -_
uhh...
- src/crawlers/robots.txt +3 -1
src/crawlers/robots.txt
CHANGED
|
@@ -1,2 +1,4 @@
|
|
| 1 |
User-agent: *
|
| 2 |
-
Allow: /
|
|
|
|
|
|
|
|
|
| 1 |
User-agent: *
|
| 2 |
+
Allow: /
|
| 3 |
+
|
| 4 |
+
Sitemap: https://umint-ai.hf.space/sitemap.xml
|