Results 1 to 11 of 11
  1. #1
    Member
    Join Date
    January 18th, 2005
    Posts
    103
    hi,
    i have some new websites, and slurp is only tring to crawl the robots.txt file, that I don't have,

    is there a way to make it crawl the rest of my site?

    thanks

  2. #2
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    1,658
    http://www.positiontech.com

    AS SEEN ON SE!!!

  3. #3
    Member
    Join Date
    January 18th, 2005
    Posts
    161
    Only incoming links from other sites already in the BOW

    Just hang in there, it takes some time [img]/infopop/emoticons/icon_wink.gif[/img]

    Nick

    Tioka Internet // Web design & SEO

  4. #4
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    Canada
    Posts
    1,650
    It would probably be a good idea to create a robots.txt file for your site, even a very, very basic one. If nothing else, it will reduce the number of errors showing up in your logs.

    Here's a tutorial I found useful:

    http://www.searchengineworld.com/rob...s_tutorial.htm

  5. #5
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    1,916
    I'm of the opposite opinion... since I feel that any possible mistake in the file could cause a robot not to crawl.. I prefer to just run without one unless there is something I need to exclude.. [img]/infopop/emoticons/icon_smile.gif[/img]

    2c

  6. #6
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    United Kingdom
    Posts
    1,797
    I disagree - if a spider is looking for a robots.txt and being polite, then it does no harm to provide one; particularly as - in this instance - Slurp is being particularly considerate and doesn't seem to want to come in unless invited.

    Search Engine Positioning - 1 Design 4 Life

  7. #7
    Member
    Join Date
    January 18th, 2005
    Posts
    161
    Yes. The only problem is that when you do put one in, the little sod will probably still just pick only that file up for a few months [img]/infopop/emoticons/icon_wink.gif[/img]

    I've never really understood what makes Slurp tick. I just give it time and eventually I get in. It's always take a long time though....

    Nick

    Tioka Internet // Web design & SEO

  8. #8
    Member
    Join Date
    January 18th, 2005
    Posts
    103
    but is it possible to tell slurp, through the robots.txt, that he's trying to get so many times,
    to start crawling my pages?

  9. #9
    ABW Ambassador Andy's Avatar
    Join Date
    January 18th, 2005
    Posts
    4,178
    After I added my robots.txt file, slurp started visiting quite regularly. I can't say it's because of the robots.txt file, but I'm getting a lot of action out of him now that I didn't before I uploaded that file!

    Try it and see if it helps, as long as it's formatted correctly, it certainly can't hurt!

    Andy

  10. #10
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    1,658
    <BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR> but is it possible to tell slurp <HR></BLOCKQUOTE>

    Pay the $39 bucks.

    Best investment you'll ever make.

    Time is money. Don't waste time waiting.

    AS SEEN ON SE!!!

  11. #11
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    1,916
    $39 gets you one page right? each other is $25?

    But INK is now crawling everything they can find (free) as of a month or so ago aren't they?

    INK deep crawls my site all day long it grabs a page here and there..kind of weird

  12. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Blocking Certain Forum Pages with robots.txt
    By Rob_BAM in forum Search Engine Optimization
    Replies: 3
    Last Post: May 29th, 2012, 08:58 AM
  2. No change to Robots.txt but google says pages blocked
    By JI7009 in forum Search Engine Optimization
    Replies: 7
    Last Post: November 7th, 2009, 11:34 AM
  3. Restricted by robots.txt without robots.txt?
    By mayfly in forum Search Engine Optimization
    Replies: 10
    Last Post: August 26th, 2009, 05:13 PM
  4. robots txt
    By reflections in forum Programming / Datafeeds / Tools
    Replies: 5
    Last Post: December 26th, 2002, 06:22 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •