Question about installation step 5
Dec 18, 2011 5:53 pm by cyberkeyhan
Dec 18, 2011 5:53 pm by cyberkeyhan
Hi,
I followed installation steps and installed the pro version, it works fine.
I have created sitemap.txt with the languages I want to be indexed, but I didn't understand the step 5 in the readme file, "modify robots.txt....
1- I don't find any robots.txt file on the root or anywhere else on my site, should I create one myself?
2- Except the sitemap: yourdomain.com/sitemap.txt what else should be inside?
I don't know for example what should be DISALLOWED and what allowed from security aspect.
Should I put a(*) for User-agent, allowing all user agents or limit it for example to only Googlebot?
Is it correct to put the following in the file:
User-agent: *
Disallow:
Sitemap: http://yourdomain.com/sitemap.txt
Best Regards
Re: Question about installation step 5
Dec 19, 2011 10:53 am by Edvard
Dec 19, 2011 10:53 am by Edvard
You can just create a blank robots.txt file and put this:
Sitemap: http://yourdomain.com/sitemap.txt
SIMILAR TOPICS
| Pre Sales Question ? | Oct 18, 2017 6:37 am | Replies: 1 | Post by: jackB2010 |
| Question about SSL | Jul 9, 2017 8:30 am | Replies: 3 | Post by: Travelaroundtheworld |
| Multiple question (login, authorization to edit...) | Feb 20, 2017 12:47 pm | Replies: 9 | Post by: José Soares |
| SSL wildcard installation on your server | Feb 3, 2017 7:50 am | Replies: 1 | Post by: partychemicals |
| PHP strict standards errors / PHP 7 Compatability question | Oct 6, 2016 10:53 am | Replies: 1 | Post by: furnivat |