Forum Archive - Configuring robot.txt for drupal

Configuring robot.txt for drupal
Jun 30, 2015 9:00 am by sadashiv
Hi, I have my site made in drupal 7 and I am using gtranslate pro on the website. I found that robot.txt don't restrict bots when they are prefixed with language code. I went to webmaster tools and tried testing /admin/config is crawl-able by the bot or not and it showed that it is not. I tried hi/admin/config and it shows that the url is allowed. I have a bunch of urls as shipped with default robot.txt in drupal which should not be crawled. Is there a way that these urls are not crawled by the bots even if they get visited with language code or I have to set the robot.txt with all language code i.e. /hi/admin/config /de/admin/config and so on? Thanks, Sadashiv.
Re: Configuring robot.txt for drupal
Jul 1, 2015 8:03 am by Yana
Hi, In the robots.txt file you should add your site maps . For example if you have created sitemap.txt you need to add Sitemap: In the sitemap.txt you should put the languages which you want to be indexed by search engines. If search engines indexed the languages which you do not want to use you can add redirection rules in your .htaccess file after RewriteEngine On and redirect to the homepage. Here is an example of the redirection rule RewriteEngine On RewriteRule ^(af|ar|az)/(.*)$ /$2 [R=301,L] RewriteRule ^(be|bg|ca)/(.*)$ /$2 [R=301,L] RewriteRule ^(cs|cy|da)/(.*)$ /$2 [R=301,L]
Re: Configuring robot.txt for drupal
Jul 1, 2015 2:02 pm by sadashiv
Hi, Would like to know whether we can restrict pages under translations i.e. If I have marked /admin as disallow in robot.txt then /de/admin should not be crawled, or do I need to add disallow /de/admin separately in the robots.txt? Thanks, Sadashiv.
Re: Configuring robot.txt for drupal
Jul 1, 2015 3:25 pm by Yana
Dear Sadashiv, You can find more info here


How do I add Indian languages in Drupal GTranslate?Sep 18, 2011 6:10 amReplies: 26Post by: rag_gupta
Drupal site: doGTranslate is not definedFeb 17, 2017 5:06 pmReplies: 10Post by: logicp
Configuring it properly with Cloudflare SSL proxy enabledMar 8, 2017 4:38 amReplies: 3Post by: brunocosta
Drupal 7 modulo errorJan 13, 2017 5:44 pmReplies: 3Post by: gretel
Configuring sub-directories to point to subdomainsAug 15, 2016 1:30 pmReplies: 17Post by:

Try GTranslate with a free 15 day trial