Welcome our webmaster and SEO forum
Please enjoy the forum, contribute what you can, and wind up the Moderators!
Page 1 of 2 12 LastLast
Results 1 to 10 of 11

Thread: hide your robots.txt from visitors and show it only for validated robots

  1. #1
    Bagi Zoltán's Avatar
    Bagi Zoltán is offline Boss Cart consultant
    Join Date
    Feb 2007
    Location
    Veszprém, Hungary
    Posts
    1,225

    Default hide your robots.txt from visitors and show it only for validated robots

    After some hours searching and hacking i have finally found everything to build a solution which makes possible to hide the content of your robots.txt file from visitors but make it display ONLY for validated user agents such as googlebot, Yahoo Slurp and msnbot.

    You may find the whole thing very strange why would somebody hide that content? My answer for this question is the following:
    That content (folder structure of the core script files) is a private information, and don't want to share it with every script kiddies to make the possible to hurt my site.

    How to execute this hack? I will guide through..

    1. As first step you need to add these lines to your .htaccess file, or if you don't have create one and upload it to the root domain folder.


    Code:
    RewriteEngine On
    RewriteCond %{http_user_agent} !(googlebot|Msnbot|Slurp) [NC]
    RewriteRule ^robots\.txt$ http://seo.i-connector.com/  [R,NE,L]
    AddHandler application/x-httpd-php .txt
    I think i don't have to explain the first row, the second and the thirs says that if you are not one of the three big search engines and want to reach the robots.txt file you will be redirected to the main domain. It is very handy since a lot of people set their homepage as the landing page of 404 errors, so the cloacking won't be recognised. (will talk about the cloacking a bit later as well)
    The fourth row make possible that your robots.txt file behave as a php script.

    Now you are ready with the first step, lets see what else you need to do.

    2. Open a text editor or your favourite web editor application and insert the code below into a new file save as reversedns.php and upload it to your root folder.

    PHP Code:
    <?php
    $ua 
    $_SERVER['HTTP_USER_AGENT'];
    if(
    stristr($ua'msnbot') || stristr($ua'Googlebot') || stristr($ua'Yahoo Slurp')){
    $ip $_SERVER['REMOTE_ADDR'];
    $hostname gethostbyaddr($ip);
    if(!
    preg_match("/\.googlebot\.com$/"$hostname) &&!preg_match("/search\.live\.com$/"$hostname) &&!preg_match("/crawl\.yahoo\.net$/"$hostname)) {
    $block TRUE;
    $URL="/";
    header ("Location: $URL");
    exit;
    } else {
    $real_ip gethostbyname($hostname);
    if(
    $ip!= $real_ip){
    $block TRUE;
    $URL="/";
    header ("Location: $URL");
    exit;
    } else {
    $block FALSE;
    }
    }
    }
    ?>
    This script can be famaliar for many of you. This is a hacked version of the reversedns.php file which was presented some months ago. According to the hack if the robot can not be validated the script will redirect it to your main domain. So i return back for a min to the cloaking or not cloaking issue. I had to recognise that google are not capable to protect my rankings from exploits, so i have to defend myself, hence i belice it is not a bad cloacking only a protection solution. If somebody mask him/herself as googlebot he/she will fail during this robot valadiation so will be redirected to the main domain via php. No way to recognise the cloacking!

    3. And as the last step
    Open the robots.txt file you would like to protect and insert the code below to the first line.
    PHP Code:
    <?php include("reversedns.php"); ?>
    You are done, and your robots.txt file is in safe!

    Thanks!
    Last edited by Bagi Zoltán; 08-28-2008 at 07:30 PM.

  2. #2
    Midlandi is offline Senior Member
    Join Date
    Feb 2007
    Posts
    159

  3. #3
    Bagi Zoltán's Avatar
    Bagi Zoltán is offline Boss Cart consultant
    Join Date
    Feb 2007
    Location
    Veszprém, Hungary
    Posts
    1,225

    Default

    Thank you Midi, it took my afternoon.

  4. #4
    temi's Avatar
    temi is offline Facilitator
    Join Date
    Jun 2003
    Location
    London, England.
    Posts
    10,303

    Default

    This is brilliant Bagi, thanks for sharing, I know what you created this for originally

    Fellow UK WW members, please digg this post

    * Comprehensive UK Web Directory List . eCommerce software UK
    * BossCart.com can build you a.
    Register your domain names at Velnet
    ::
    Add Eco sites to The Green Directory free of charge.
    Use LBS Free PHP Directory Script . Web Hosting Blog

  5. #5
    pow-wow is offline Senior Member
    Join Date
    Jan 2007
    Posts
    245

  6. #6
    gkd_uk is offline Super Moderator
    Join Date
    Mar 2007
    Location
    UK
    Posts
    2,551

  7. #7
    Bagi Zoltán's Avatar
    Bagi Zoltán is offline Boss Cart consultant
    Join Date
    Feb 2007
    Location
    Veszprém, Hungary
    Posts
    1,225

    Default

    Thank you guys

  8. #8
    temi's Avatar
    temi is offline Facilitator
    Join Date
    Jun 2003
    Location
    London, England.
    Posts
    10,303

    Default

    Quote Originally Posted by pow-wow View Post
    Nice post! this is great
    Biodun, did you digg the article?

    * Comprehensive UK Web Directory List . eCommerce software UK
    * BossCart.com can build you a.
    Register your domain names at Velnet
    ::
    Add Eco sites to The Green Directory free of charge.
    Use LBS Free PHP Directory Script . Web Hosting Blog

  9. #9
    SkinnerW's Avatar
    SkinnerW is offline Senior Member
    Join Date
    Jun 2007
    Location
    Moscow - New York
    Posts
    992

    Default

    Bagi,

    Do you mind if we refer to your article in UKWW blog?

    Digged, stumbled and rep added
    Advanced CMS software with directory, blog, classifieds and portal features

  10. #10
    Bagi Zoltán's Avatar
    Bagi Zoltán is offline Boss Cart consultant
    Join Date
    Feb 2007
    Location
    Veszprém, Hungary
    Posts
    1,225

    Default

    No Skinner, that is absolutely no problem. Thanks for the digg the rep and the stumble

Page 1 of 2 12 LastLast

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Similar Threads

  1. Robots Exclusion Protocol: now with even more flexibility
    By temi in forum General Webmaster Talk
    Replies: 1
    Last Post: 07-29-2007, 02:26 PM
  2. Move to create less clumsy robots
    By gkd_uk in forum General Webmaster Talk
    Replies: 1
    Last Post: 05-30-2007, 10:55 AM
  3. images and search engine robots
    By mrcrowley in forum General Webmaster Talk
    Replies: 4
    Last Post: 05-25-2007, 09:24 AM
  4. create Simple robots.txt file
    By temi in forum General Webmaster Talk
    Replies: 2
    Last Post: 04-20-2007, 07:46 PM
  5. Robots.txt
    By temi in forum General Webmaster Talk
    Replies: 0
    Last Post: 03-24-2006, 01:07 PM

Bookmarks

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124