Results 1 to 5 of 5
  1. #1
    Member
    Join Date
    January 18th, 2005
    Posts
    51
    Hi

    I know the basics of robots and how they work but I have gotten some mixed reviews about them. Here are some questions I have which any input would be helpful:

    1. Do I need to have a robots.txt file on my server?
    2. If YES, what should it look like?
    3. Should I be frequently checking my log files to see what robots visited me?
    4. What should/could I be doing with robots to improve my site rankings?

    Google once indexed me and I had loads of traffic very quickly but then about 2 weeks ago I was dropped but my pages still seem indexed however no traffic. I'm not sure if robots could play any part in improving my site's web postioning.

    Thanks in advance.

    Damn it's hot out today

    Regards,

    Dylan

    http://www.collegesanduniversities.org

  2. #2
    Member
    Join Date
    January 18th, 2005
    Posts
    110
    1) No, and it applies on a per domain basis, not per server (but if you do, it MUST reside in the root folder of any given domain, ie the URL MUST be www.example.com/robots.txt). I think you are thinking of .htaccess as well here
    2) There is a nice site out there that contains the definition document (robotstxt.org, I think)
    <added> Google search for "robots.txt" (no quotes), brings up the ref site, and the SEW tutorial. Bonus
    3) Hell, yes
    4) Robots.txt isn't the right tool to attract spiders, its specifically used to exclude bandwidth suckers

    Typically its used to exclude known "bad bots" (some email harvesters, and genuine bots that feed SEs that have no worthwile traffic for instance, or bots that just go crazy and request thousands of documents a second, threatening to bring your server down)

  3. #3
    Member
    Join Date
    January 18th, 2005
    Posts
    51
    Thanks for your response, makes sense. However, if robots are used only for keeping out bandwith suckers then what exactly am I looking at them in my logs for? Would I want to pay attention to the bots that visit me because it gives me an idea what ses are coming to my site?

    What benefit would I get by looking at the bots visting my site? I mean it might be nice to see when google comes by but does it really matter if I know they came there or not?

    Regards,

    Dylan

  4. #4
    Member
    Join Date
    January 18th, 2005
    Posts
    110
    >> if robots are used only for keeping out bandwith suckers

    The search engine spiders (also known as robots) are good to have on your site... most of them. The file robots.txt is used to exclude those that you DON'T want using your precious bandwidth

    >> but does it really matter if I know they came there or not?

    It does if the answer is "not". Knowing when they appear, and what they ask for can give you clues as to what they like about your site. If Gbot regularly appears and asks for index.html and the same half dozen pages of a 200 page site every week, you *know* which are the pages it likes. If you have new content you want spidered quickly, guess where the links should go....

    Sometimes you can watch them "choke" on a new page, go and check it, and oops, killer HTML error there, best fix that etc

  5. #5
    Member
    Join Date
    January 18th, 2005
    Posts
    51
    Got it, I understand now and will begin analyzing my web logs but exactly what am I supposed to be looking at?

    I run WebTrendsLive on my site and I notice the following web logs on my server: access_log, access_log.processed, and error_log. If you were me (fortunately your not), what would you be looking for?

    Thanks for any help.

    Regards,

    Dylan

  6. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Looking for Some Insight
    By nate3445 in forum Midnight Cafe'
    Replies: 6
    Last Post: May 13th, 2011, 03:23 PM
  2. Restricted by robots.txt without robots.txt?
    By mayfly in forum Search Engine Optimization
    Replies: 10
    Last Post: August 26th, 2009, 05:13 PM
  3. Insight from the Outside
    By deepestblue in forum Virtual Family and Off-Topic
    Replies: 3
    Last Post: August 26th, 2005, 08:10 PM
  4. Some Google Insight
    By Trust in forum Search Engine Optimization
    Replies: 40
    Last Post: April 11th, 2005, 05:34 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •