Welcome our webmaster and SEO forum
Please enjoy the forum, contribute what you can, and wind up the Moderators!
Page 1 of 2 12 LastLast
Results 1 to 10 of 11

Thread: Search Engine Crawler

  1. #1
    SeoDezin is offline Junior Member
    Join Date
    Mar 2011
    Posts
    3

    Default Search Engine Crawler

    When search engine crawler visit a site. Where does it visit first?

  2. #2
    jhonmark555 is offline Senior Member
    Join Date
    Mar 2011
    Posts
    109

  3. #3
    nullifygirls is offline Member
    Join Date
    Mar 2011
    Posts
    32

    Default

    search engine crawler target is first page

  4. #4
    SDeveloper is offline Member
    Join Date
    Oct 2010
    Posts
    36

    Default

    First they gets a list of URL's to visit and store. How? A file called "robots.txt." tells the search crawler which files/pages it can request, and which files/page or directories it's not allowed to visit.

  5. #5
    seoservices7890 is offline Junior Member
    Join Date
    Apr 2011
    Posts
    18

    Default

    Of course it visits the first page of a web site first.

  6. #6
    powwow is offline Junior Member
    Join Date
    May 2011
    Posts
    12

    Default

    This depends on how it got to your site.

    If you submitted a site map than it will go to the main page first.
    If you only put links on some sites and the crawler got to your site through those links it will go to the page where the link points.

  7. #7
    paddy12 is offline Member
    Join Date
    Oct 2009
    Location
    ireland
    Posts
    73

    Default

    well always when we start linking we target main page or index page first so it is so daffiness that crawler go to index page first as every one said and it tool complete data of your site. What will be the time period of crawling of your site no one can tell you the exact time.

  8. #8
    jamhassan is offline Member
    Join Date
    Mar 2011
    Posts
    63

    Default

    Quote Originally Posted by nullifygirls View Post
    search engine crawler target is first page
    dear Crawler does not crawl only First page . its try to get all the links from Sitemap.xml & looks for any changes or any new pages added to your website & update their log if their is any change

  9. #9
    Markwebuk is offline Member
    Join Date
    Jun 2007
    Posts
    94

    Default

    In my opinion, it first scans the main page of the website and then it goes on to the hyperlinks.

  10. #10
    Belisma Scott is offline Member
    Join Date
    Dec 2010
    Posts
    82

    Default

    It first target to crawl is homepage of a website.

Page 1 of 2 12 LastLast

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Tags for this Thread

Bookmarks

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124