Configuring robot.txt for drupal

[sadashiv]

Hi,

I have my site made in drupal 7 and I am using gtranslate pro on the website. I found that robot.txt don't restrict bots when they are prefixed with language code.

I went to webmaster tools and tried testing /admin/config is crawl-able by the bot or not and it showed that it is not. I tried hi/admin/config and it shows that the url is allowed. I have a bunch of urls as shipped with default robot.txt in drupal which should not be crawled. Is there a way that these urls are not crawled by the bots even if they get visited with language code or I have to set the robot.txt with all language code i.e. /hi/admin/config /de/admin/config and so on?

Thanks,
Sadashiv.

[Yana]

Hi,

In the robots.txt file you should add your site maps . For example if you have created sitemap.txt you need to add
Sitemap: http://yourdomain.com/sitemap.txt

In the sitemap.txt you should put the languages which you want to be indexed by search engines.
If search engines indexed the languages which you do not want to use you can add redirection rules in your .htaccess file after RewriteEngine On and redirect to the homepage. Here is an example of the redirection rule
RewriteEngine On
RewriteRule ^(af|ar|az)/(.*)$ /$2 [R=301,L]
RewriteRule ^(be|bg|ca)/(.*)$ /$2 [R=301,L]
RewriteRule ^(cs|cy|da)/(.*)$ /$2 [R=301,L]

[sadashiv]

Hi,

Would like to know whether we can restrict pages under translations i.e. If I have marked /admin as disallow in robot.txt then /de/admin should not be crawled, or do I need to add disallow /de/admin separately in the robots.txt?

Thanks,
Sadashiv.

[Yana]

Dear Sadashiv,

You can find more info here http://www.robotstxt.org/orig.html