Results 1 to 8 of 8
  1. #1
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    Winterpeg, the Mosquito Capital of Canada
    Posts
    2,299
    WTF are these ppl at Ineed hits doing now?
    I sent a very angry e mail voicing displeasure on some things also my inclusion since Oct now on hold because

    8006 ERROR_SITE: Failed to retieve robots.txt
    Problem: The robots.txt file for your web site, could not be retrieved.
    Solution: Please ensure that the robots.txt file returns either a 200 or 404 status code.
    HUH??????? Haven't a clue how to make them return certain codes.

    I can find info on making a robots txt page, but not with info as they say, Anyone more familiar with what they are actually needing?
    Based on my last e mail to them and the slow reponse I was anticipating possibly a quicker reply from someone in here

    ...............
    WW

    Make a difference! Support your local Cancer Care providers.

  2. #2
    ABW Ambassador DesignerWiz's Avatar
    Join Date
    January 18th, 2005
    Location
    U.S.A
    Posts
    2,777
    Hello WW,

    Try something like this:

    < !-- These are folder disallows

    Disallow: /folder or file type name here/
    -->

    Name = robots.txt (file in ROOT)
    Copy Example Type Below:

    User-agent: *

    Disallow: /cgi-bin/

    Disallow: /images/

    Disallow: /paid_customers/

    Disallow: /download/

    Disallow: /jasc/

    Disallow: /webstats/

    Disallow: /private/

    Disallow: /scripts/

    Disallow: /cute/

    Disallow: /*.gif$

    Disallow: /*.jpg$

    Disallow: /*.jpeg$

    Disallow: /*.doc$

    User-Agent: Googlebot-Image

    Disallow: /

    Ray Thomas
    DesignerWiz.com CEO
    Development Resource & Javascript Public Archive Center
    http://DesignerWiz.com
    ABW Board: Category: Programming / Coding

  3. #3
    ABW Ambassador DesignerWiz's Avatar
    Join Date
    January 18th, 2005
    Location
    U.S.A
    Posts
    2,777
    BTW:
    If you like .. we have a free Robots.txt Validation Testing Tool located in our "Free URL Tests" area of our service when you add your robots.txt file.

    Ray Thomas
    DesignerWiz.com CEO
    Development Resource & Javascript Public Archive Center
    http://DesignerWiz.com
    ABW Board: Category: Programming / Coding

  4. #4
    ABW Ambassador DesignerWiz's Avatar
    Join Date
    January 18th, 2005
    Location
    U.S.A
    Posts
    2,777
    Hello WW, I had a chance to investigate what exactly I think Ineedhits wanted.

    Are there custom 404 pages on your web site?
    If you do have a custom 404 page, does it return the 404 page status that is supposed to occur, or does it redirect and return a 200.

    If the custom 404 page returns a 200, then google will only TRY to interpret the page returned and won't be able to. Google (others) will then decide that you do not want to be crawled and will exit the server.

    Ray Thomas
    DesignerWiz.com CEO
    Development Resource & Javascript Public Archive Center
    http://DesignerWiz.com
    ABW Board: Category: Programming / Coding

  5. #5
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    United Kingdom
    Posts
    1,797
    You might be right, Ray. Walleye - this is how you check. Go here: http://www.svendsen-net.dk/?ref=/web...hkheader.phtml .

    Type in http://www.yourdomain.com/robots.txt (you don't need to worry about the other fields) to see what status code it gives you back.

    If you don't have a robots.txt, you should get a 404 error. If you do, you should get a 200OK status.

    If you get one or other of these status codes returned, then the problem is with Inktomi's spider and not with your site. Really you should have a robots.txt file anyway, but that's a different issue.

    Search Engine Marketing and Positioning - 1 Design 4 Life

  6. #6
    "An Englishman In New York" TJ's Avatar
    Join Date
    January 18th, 2005
    Posts
    3,282
    I'm getting reports from them stating Error 4000... "Unknown Error" it was on 3 pages first, and now 5! They say it should get better after the next refresh

  7. #7
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    Winterpeg, the Mosquito Capital of Canada
    Posts
    2,299
    Thing is I have no spiders text anywhere on any of my sites

    These people have got me thoroughly annoyed since I signed on in October with things like this and not being able to easily update pages as they were so proud to claim at that time

    ...............
    WW

    Make a difference! Support your local Cancer Care providers.

  8. #8
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    Winterpeg, the Mosquito Capital of Canada
    Posts
    2,299
    I went and added the thing to my site
    I hope someone from there reads this board and this post because I certainly won't be this quick when I need to renew
    it's just not gonna happen
    I think they now have me up to 6 hits in the past 3 weeks on a page that I do fairly well on in the search engines... what a waste

    ...............
    WW

    Make a difference! Support your local Cancer Care providers.

  9. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Restricted by robots.txt without robots.txt?
    By mayfly in forum Search Engine Optimization
    Replies: 10
    Last Post: August 26th, 2009, 05:13 PM
  2. Robots.txt
    By Rhia7 in forum Midnight Cafe'
    Replies: 0
    Last Post: April 18th, 2009, 12:34 AM
  3. Google wants nothing but robots.txt!
    By login in forum Search Engine Optimization
    Replies: 2
    Last Post: November 19th, 2004, 09:05 AM
  4. Do you use a robots.txt?
    By Mr. Sal in forum Voting Booth
    Replies: 11
    Last Post: November 12th, 2003, 07:29 PM
  5. robots txt
    By reflections in forum Programming / Datafeeds / Tools
    Replies: 5
    Last Post: December 26th, 2002, 06:22 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •