Results 1 to 12 of 12
  1. #1
    ABW Ambassador
    Join Date
    June 30th, 2007
    Location
    Syracuse, NY
    Posts
    677
    Robots.txt question.
    I didn't see this when trying to search.. but thinking it might be a really dumb question, lol.

    Do I need one? I currently don't have one, but in my log file I see googlebot visiting it..? Which makes no sense.. and I'm wondering if having a robot.txt file would help speed the SE's to index me? They visit, no index yet. Unless that's normal?

    Thanks.

  2. #2
    notary sojac Herb ԿԬ's Avatar
    Join Date
    January 18th, 2005
    Location
    Central/Western NY State
    Posts
    7,741
    "good" search engine 'bots use it, if they find one on your site, to tell where they can or can't go.

  3. #3
    ABW Ambassador
    Join Date
    June 30th, 2007
    Location
    Syracuse, NY
    Posts
    677
    Yeah, I know it can tell good bots where not to go, but would it be to my benefit then to have one that tells them where to go? I figured they'd just spider through, since I didn't have one telling them they couldn't, lol.

  4. #4
    Beachy Bill's Avatar
    Join Date
    November 20th, 2005
    Posts
    8,266
    My top producing sites do NOT have a robots.txt file. The have been on line so long and have such good rankings in the SERPS that I am not about to put one in place. Nor am I going to add a Google site map. Y'know, "If is ain't broke - don't fix it."

    With other sites I have built over the years I have some with and some without robots.txt and/or site maps. I have not seen a difference either way.
    Bill / Marketing Blog @ 12PM - Current project: Resurrecting my "baby" at South Baltimore..
    Cute Personal Checks and Business Checks
    If you are too busy to laugh you are too busy.

  5. #5
    ABW Ambassador
    Join Date
    June 30th, 2007
    Location
    Syracuse, NY
    Posts
    677
    Okay, thanks.

    Did do the sitemap thing.

    Still a bit puzzled why I see googlebot visiting robot.txt when I don't have one, lol.

  6. #6
    notary sojac Herb ԿԬ's Avatar
    Join Date
    January 18th, 2005
    Location
    Central/Western NY State
    Posts
    7,741
    Lightbulb
    if, on that line you see 404, it means it wasn't found

  7. #7
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    Los Angeles
    Posts
    4,053
    If nothing else, I put up a blank robots.txt so the sites won't get hit with a bunch of 404's.

  8. #8
    Moderator MichaelColey's Avatar
    Join Date
    January 18th, 2005
    Location
    Mansfield, TX
    Posts
    16,232
    I just wanted to clarify that it's "robots.txt". I've renamed the thread.
    Michael Coley
    Amazing-Bargains.com
     Affiliate Tips | Merchant Best Practices | Affiliate Friendly? | Couponing | CPA Networks? | ABW Tips | Activating Affiliates
    "Education is the most powerful weapon which you can use to change the world." Nelson Mandela

  9. #9
    ABW Ambassador
    Join Date
    June 30th, 2007
    Location
    Syracuse, NY
    Posts
    677
    Thanks. Maybe that's why I wasn't finding much searching, LOL.

    Ah, yeah.. thanks. There is 404 showing up so I added in a blank one.

    Not to be off topic, but somewhat mentioned this.. since I've been seeing the bots (yahoo, google, and msn) in my log, shouldn't I be in the index soon? For Digg and yahoo answers, I got indexed quick. So wondering if I should be concerned or not yet?

    Thanks.

  10. #10
    Resident Genius and Staunch Capitalist Leader's Avatar
    Join Date
    January 18th, 2005
    Location
    Florida
    Posts
    12,817
    Quote Originally Posted by Hardaka
    Okay, thanks.

    Did do the sitemap thing.

    Still a bit puzzled why I see googlebot visiting robot.txt when I don't have one, lol.
    It visits it (or tries to) because the only way for it to know if there is a robots.txt (and if so, what it says) is to try to access it.
    Not to be off topic, but somewhat mentioned this.. since I've been seeing the bots (yahoo, google, and msn) in my log, shouldn't I be in the index soon?
    That depends on what they think of the inlinks you have going to your site. Sometimes they spider away for what seems like ages before even forking over a listing under "yoursite.com." On the other hand, if you get a link from a site that gets spidered often, and that the engines like, you can be in within a few days (under "yoursite.com").

    Also, some engines are faster at it than others.

    Of course, getting *meaningful* rank, under keywords people actually look for, is a lot harder than just getting "indexed." But that's getting into SEO, which is a topic all its own.
    There is no knowledge that is not power. ~Hemingway

  11. #11
    Lite On The Do, Heavy On The Nuts Donuts's Avatar
    Join Date
    January 18th, 2005
    Location
    Winter Park, FL
    Posts
    6,930
    Quote Originally Posted by Hardaka
    Still a bit puzzled why I see googlebot visiting robots.txt when I don't have one, lol.
    Crawling and indexing, by search engines, begins with checking that file to see what you've said isn't available for crawling. So they check it often.

    Since this file lists things you don't want crawled... without one, the search engines default is to assume that anything on your site is available to be crawled. So not having a robots.txt file won't prevent you from being crawled and indexed and even well ranked.

    That said, the robots.txt file is one main element available to you in sculpting your site's PageRank flow. Choosing not to use one is fine for many, many websites, so don't fret over it. But if you decide to focus on your site's SEO and want to tune your website for optimum crawling, indexing and ranking, you'll want to circle back and learn more about the best use of the robots.txt file and the other tools within this skillset.

  12. #12
    ABW Ambassador
    Join Date
    June 30th, 2007
    Location
    Syracuse, NY
    Posts
    677
    That depends on what they think of the inlinks you have going to your site. Sometimes they spider away for what seems like ages before even forking over a listing under "yoursite.com." On the other hand, if you get a link from a site that gets spidered often, and that the engines like, you can be in within a few days (under "yoursite.com").
    Ah okay, I had thought if I submit it, then it'll show up. Google must like digg then, I posted one of my pages on digg and it was indexed within 36 hours. None of my other pages have been yet, but I guess it'd be spammy to submit each page to digg, huh? LOL Thanks.

    Donuts: Oh yeah, I want to do those things. Definitely need to go and research that, google does visit a couple times a day it seems, but just to the robots.txt so maybe if I do what you mentioned, it might be more inclined to crawl everything. Hoping so!

    And I thought most of my work was done when the website was done. I keep learning new things.

  13. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Restricted by robots.txt without robots.txt?
    By mayfly in forum Search Engine Optimization
    Replies: 10
    Last Post: August 26th, 2009, 05:13 PM
  2. Robots.txt
    By Rhia7 in forum Midnight Cafe'
    Replies: 0
    Last Post: April 18th, 2009, 12:34 AM
  3. Do you use a robots.txt?
    By Mr. Sal in forum Voting Booth
    Replies: 11
    Last Post: November 12th, 2003, 07:29 PM
  4. robots txt
    By reflections in forum Programming / Datafeeds / Tools
    Replies: 5
    Last Post: December 26th, 2002, 06:22 PM
  5. robots.txt file question
    By Doc Sawyer in forum Search Engine Optimization
    Replies: 4
    Last Post: December 7th, 2002, 07:10 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •