Results 1 to 2 of 2
June 8th, 2005, 01:24 PM #1
DySE and Bots
- Join Date
- June 3rd, 2005
I am beginning to get hits from googlebot but it isn't going any deeper than the / or "home" page. Should I add a robots.txt file for specific instructions? Or will it just take time to index deeper? I'm not sure how "mod_rewrite" works with a googlebot or any other search bot.
June 8th, 2005, 05:22 PM #2
Give the Google spider some time to come back. Spiders don't necessarily spider everything in one visit.
You would use a robots.txt file to tell spiders what not to spider at your site. You do not have to do in anything in robots.txt to allow the Google spider into your website.
mod_rewrite is transparent. To the outside world (users and spiders), the virtual HTML webpages at your website look like real .html files.
per: David Cusimano
By LearnForce Partners in forum Midnight Cafe'Replies: 7Last Post: January 17th, 2012, 10:38 AM
By oranges in forum Search Engine OptimizationReplies: 1Last Post: March 16th, 2004, 10:59 AM
By Javi in forum BettyMillsReplies: 3Last Post: January 14th, 2004, 11:39 AM
By drumminlogan in forum Search Engine OptimizationReplies: 1Last Post: September 14th, 2002, 04:02 PM
By Tami in forum Search Engine OptimizationReplies: 3Last Post: April 4th, 2002, 05:19 AM