Results 1 to 2 of 2
  1. #1
    Join Date
    June 3rd, 2005
    DySE and Bots
    I am beginning to get hits from googlebot but it isn't going any deeper than the / or "home" page. Should I add a robots.txt file for specific instructions? Or will it just take time to index deeper? I'm not sure how "mod_rewrite" works with a googlebot or any other search bot.


  2. #2
    ABW Ambassador cusimano's Avatar
    Join Date
    January 18th, 2005
    Toronto, Canada
    Give the Google spider some time to come back. Spiders don't necessarily spider everything in one visit.

    You would use a robots.txt file to tell spiders what not to spider at your site. You do not have to do in anything in robots.txt to allow the Google spider into your website.

    mod_rewrite is transparent. To the outside world (users and spiders), the virtual HTML webpages at your website look like real .html files.

    Yours truly,
    Cusimano.Com Corporation
    per: David Cusimano

  3. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Curious about bots
    By LearnForce Partners in forum Midnight Cafe'
    Replies: 7
    Last Post: January 17th, 2012, 10:38 AM
  2. Adsense & Bots
    By oranges in forum Search Engine Optimization
    Replies: 1
    Last Post: March 16th, 2004, 10:59 AM
  3. Bots & Clicks
    By Javi in forum BettyMills
    Replies: 3
    Last Post: January 14th, 2004, 11:39 AM
  4. Bots and Spiders
    By drumminlogan in forum Search Engine Optimization
    Replies: 1
    Last Post: September 14th, 2002, 04:02 PM
  5. What is the google bots name?
    By Tami in forum Search Engine Optimization
    Replies: 3
    Last Post: April 4th, 2002, 05:19 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts