Results 1 to 5 of 5
June 28th, 2006, 01:08 PM #1Slurp Making Strange Requests
One of my sites is getting some weird requests from the slurp bot. It's requesting pages like:
These pages don't exist and have never existed. And I've certainly never had a directory called SlurpConfirm404. The ip address points to Inktomi Corporation, so it's not a rouge bot masquerading. Anybody else seeing this or have any ideas?But are you still master of your domain?
June 28th, 2006, 01:13 PM #2
googled the first page name and looked at the cache of the result there...
maybe someone there (and elsewhere) scraped / linked to your site or something and now slurp bot is visiting to confirm it's a 404 error page still or not or whatever...
June 28th, 2006, 01:18 PM #3
Thanks Donuts, I just did the same thing, and came back to edit my post. That'll teach me to post first and search second.
D'oh I really feel like and idiot now! I just glanced at Donuts' post and thought we searched for the same thing. I searched for "SlurpConfirm404/" and found my way to a page on Yahoo with the explanation. If anybody else is seeing this, here's what they say:
Some web servers send a site navigation page or other response page with a "HTTP 200 OK" response instead of a "HTTP 404 Not Found" result for page-not-found conditions. To check on web server handling of page-not-found conditions, Slurp will occasionally send deliberately odd URLs built from random words to sites from which no 404 results have been seen. These URLs are built intentionally to not match any actual content at the site. We save information on the web server response to requests for non-existent pages so we can correctly recognize and remove obsolete URLs in our search database.
A Slurp check for 404 results from a web server consists of requests for up to 10 such URLs. The check for 404 behavior is not a normal part of Slurp site refresh, so such requests will be rare.
Last edited by NewcastleB; June 28th, 2006 at 01:25 PM. Reason: Me Posting without reading the whole postBut are you still master of your domain?
June 28th, 2006, 02:21 PM #4
Ah, cool, so they check to see how you would handle a 404 - then compare that to other responses, as needed - so they can know that your typical 404's, looking like what they know are real 404, become more meaningful.
Anticipating poor webmaster skills - good.
Obviously, you're not in that category. Not only do you likely have proper 404 responses NewcastleB, but you comb through your logs and figure out things you don't understand. Kudos for you, webMASTER.
June 28th, 2006, 08:10 PM #5
By Trust in forum Search Engine OptimizationReplies: 10Last Post: February 17th, 2004, 12:27 PM
By frank3iii in forum Search Engine OptimizationReplies: 1Last Post: September 6th, 2003, 09:20 AM
By ellen-s4y in forum Search Engine OptimizationReplies: 1Last Post: September 5th, 2003, 09:12 AM
By Electropulse in forum Search Engine OptimizationReplies: 5Last Post: July 31st, 2003, 01:44 PM
By AffiliateBuddha in forum Search Engine OptimizationReplies: 2Last Post: April 17th, 2003, 01:29 PM