Configuring robot.txt for drupal

Popular automatic website translation tool

Configuring robot.txt for drupal

Postby sadashiv » Tue Jun 30, 2015 1:00 pm

Hi,

I have my site made in drupal 7 and I am using gtranslate pro on the website. I found that robot.txt don't restrict bots when they are prefixed with language code.

I went to webmaster tools and tried testing /admin/config is crawl-able by the bot or not and it showed that it is not. I tried hi/admin/config and it shows that the url is allowed. I have a bunch of urls as shipped with default robot.txt in drupal which should not be crawled. Is there a way that these urls are not crawled by the bots even if they get visited with language code or I have to set the robot.txt with all language code i.e. /hi/admin/config /de/admin/config and so on?

Thanks,
Sadashiv.
sadashiv
 
Posts: 64
Joined: Thu Oct 09, 2014 2:02 pm

Re: Configuring robot.txt for drupal

Postby Yana » Wed Jul 01, 2015 12:03 pm

Hi,

In the robots.txt file you should add your site maps . For example if you have created sitemap.txt you need to add
Sitemap: http://yourdomain.com/sitemap.txt

In the sitemap.txt you should put the languages which you want to be indexed by search engines.
If search engines indexed the languages which you do not want to use you can add redirection rules in your .htaccess file after RewriteEngine On and redirect to the homepage. Here is an example of the redirection rule
RewriteEngine On
RewriteRule ^(af|ar|az)/(.*)$ /$2 [R=301,L]
RewriteRule ^(be|bg|ca)/(.*)$ /$2 [R=301,L]
RewriteRule ^(cs|cy|da)/(.*)$ /$2 [R=301,L]
Regards,

Yana Ghahramanyan - GTranslate Team

Please leave your feedback on your CMS plugin directory. It is very important for us!
Google Translate Joomla
Google Translate WordPress
Google Translate Drupal
Yana
 
Posts: 4134
Joined: Thu Jan 12, 2012 6:21 pm

Re: Configuring robot.txt for drupal

Postby sadashiv » Wed Jul 01, 2015 6:02 pm

Hi,

Would like to know whether we can restrict pages under translations i.e. If I have marked /admin as disallow in robot.txt then /de/admin should not be crawled, or do I need to add disallow /de/admin separately in the robots.txt?

Thanks,
Sadashiv.
sadashiv
 
Posts: 64
Joined: Thu Oct 09, 2014 2:02 pm

Re: Configuring robot.txt for drupal

Postby Yana » Wed Jul 01, 2015 7:25 pm

Dear Sadashiv,

You can find more info here http://www.robotstxt.org/orig.html
Regards,

Yana Ghahramanyan - GTranslate Team

Please leave your feedback on your CMS plugin directory. It is very important for us!
Google Translate Joomla
Google Translate WordPress
Google Translate Drupal
Yana
 
Posts: 4134
Joined: Thu Jan 12, 2012 6:21 pm


  • Related Topics
    Replies
    Views
    Last post

Who is online

Users browsing this forum: No registered users and 0 guests

2GLux