Results 1 to 22 of 22
  1. #1
    Grandma broke her coccyx! Uncle Rico's Avatar
    Join Date
    May 8th, 2007
    Location
    North Carolina
    Posts
    2,238
    GSiteCrawler Problem: Not Crawling
    I installed GSiteCrawler and added a new site, but when I tell it to Crawl this porject, it only crawls the main page. Is there some setting that I am missing?
    Last edited by Uncle Rico; June 22nd, 2009 at 08:56 AM.

  2. #2
    Grandma broke her coccyx! Uncle Rico's Avatar
    Join Date
    May 8th, 2007
    Location
    North Carolina
    Posts
    2,238
    Actually, it's not crawling any pages at all. Wierd.

  3. #3
    The Seal of Aproval rematt's Avatar
    Join Date
    November 19th, 2006
    Location
    The Windy City
    Posts
    4,140
    Seymour, did you except the default settings?

    -rematt
    "I know that you believe you understand what you think I said, but I'm not sure you realize that what you heard is not what I meant." - Richard Nixon

  4. #4
    Grandma broke her coccyx! Uncle Rico's Avatar
    Join Date
    May 8th, 2007
    Location
    North Carolina
    Posts
    2,238
    Let's see..

    URL's are case sensitive <checked>
    Remove trailing slashes <un-checked>
    Remove HTML comments before parsing <checked>

    File Extensions to Follow:
    asp,aspx,cfm,cgi,do,htm,html,jsp,mv,mvc,php,php5,phtml,pl,py,shtml

    <blank>

    <blank>

    <blank>

    <blank>

    <blank>

    Action on Error 404 <No Nothing>

    Priority for this Project <100>

    Google site-map options:
    <checked>
    <checked>
    <checked>

  5. #5
    The Seal of Aproval rematt's Avatar
    Join Date
    November 19th, 2006
    Location
    The Windy City
    Posts
    4,140
    Everything looks fine. Did you use the Wizard or overwrite an existing settings file?

    -rematt
    "I know that you believe you understand what you think I said, but I'm not sure you realize that what you heard is not what I meant." - Richard Nixon

  6. #6
    Grandma broke her coccyx! Uncle Rico's Avatar
    Join Date
    May 8th, 2007
    Location
    North Carolina
    Posts
    2,238
    Used the Wizard twice now. I even re-installed the tool again and got the same results. Nothing.

  7. #7
    The Seal of Aproval rematt's Avatar
    Join Date
    November 19th, 2006
    Location
    The Windy City
    Posts
    4,140
    Seymour, check your robots.txt an make sure you haven't accidentally disallowed your main directory.

    -rematt
    "I know that you believe you understand what you think I said, but I'm not sure you realize that what you heard is not what I meant." - Richard Nixon

  8. #8
    Grandma broke her coccyx! Uncle Rico's Avatar
    Join Date
    May 8th, 2007
    Location
    North Carolina
    Posts
    2,238
    Also, I have used GSiteCrawler in the past on the last PC I had and it worked fine, so I am somewhat familiar with the tool.

  9. #9
    Grandma broke her coccyx! Uncle Rico's Avatar
    Join Date
    May 8th, 2007
    Location
    North Carolina
    Posts
    2,238
    Quote Originally Posted by rematt
    Seymour, check your robots.txt an make sure you haven't accidentally disallowed your main directory.

    -rematt
    Checked. The only items I list are ..

    User-agent: *
    Disallow: /some-directory/
    Disallow: /some-file.php
    ...
    ...

  10. #10
    Grandma broke her coccyx! Uncle Rico's Avatar
    Join Date
    May 8th, 2007
    Location
    North Carolina
    Posts
    2,238
    It may be a long shot, but I wonder is this is related to Google de-indexing my site pages. In the last 4-6 months, the number of indexed pages in Google has gone from around 1000 down to 198 as of this morning.

  11. #11
    The Seal of Aproval rematt's Avatar
    Join Date
    November 19th, 2006
    Location
    The Windy City
    Posts
    4,140
    Seymour, try going to the URL tab and manually entering your main site URL. Make sure that you check; Manual, Include and Crawl, change the priority to 1 and the frequency to daily. Once you've changed these settings select Recrawl and see what happens.

    -rematt
    "I know that you believe you understand what you think I said, but I'm not sure you realize that what you heard is not what I meant." - Richard Nixon

  12. #12
    The Seal of Aproval rematt's Avatar
    Join Date
    November 19th, 2006
    Location
    The Windy City
    Posts
    4,140
    Quote Originally Posted by SeymourButts
    It may be a long shot, but I wonder is this is related to Google de-indexing my site pages. In the last 4-6 months, the number of indexed pages in Google has gone from around 1000 down to 198 as of this morning.
    No. GSite should crawl your pages regardless of what Google has indexed. You may want to make sure the option to load existing pages from Google is off just for the heck of it, but it shouldn't make a difference.

    ]-rematt
    "I know that you believe you understand what you think I said, but I'm not sure you realize that what you heard is not what I meant." - Richard Nixon

  13. #13
    Grandma broke her coccyx! Uncle Rico's Avatar
    Join Date
    May 8th, 2007
    Location
    North Carolina
    Posts
    2,238
    Quote Originally Posted by rematt
    Seymour, try going to the URL tab and manually entering your main site URL. Make sure that you check; Manual, Include and Crawl, change the priority to 1 and the frequency to daily. Once you've changed these settings select Recrawl and see what happens.

    -rematt
    Under the "Project" tab, you can enter the "Project Name" and "Main URL", but that's it. This version of GSiteCrawler is v1.23.

  14. #14
    Grandma broke her coccyx! Uncle Rico's Avatar
    Join Date
    May 8th, 2007
    Location
    North Carolina
    Posts
    2,238
    I think this is a bad version. I tried to crawl another site and get the same problem.

  15. #15
    The Seal of Aproval rematt's Avatar
    Join Date
    November 19th, 2006
    Location
    The Windy City
    Posts
    4,140
    Same version I'm using. Three tabs to the right of the project tab is the URL List tab. Select that and you will see what URLs have been crawled (if any). Once there, follow the instructions above to manually enter a URL.

    -rematt

    BTW, I'm going to have to run for about 90 minutes. Maybe we should continue this in chat later so we don't drive everyone else crazy with this thread.
    "I know that you believe you understand what you think I said, but I'm not sure you realize that what you heard is not what I meant." - Richard Nixon

  16. #16
    Grandma broke her coccyx! Uncle Rico's Avatar
    Join Date
    May 8th, 2007
    Location
    North Carolina
    Posts
    2,238
    Quote Originally Posted by rematt
    Same version I'm using. Three tabs to the right of the project tab is the URL List tab. Select that and you will see what URLs have been crawled (if any). Once there, follow the instructions above to manually enter a URL.

    -rematt

    BTW, I'm going to have to run for about 90 minutes. Maybe we should continue this in chat later so we don't drive everyone else crazy with this thread.
    Found it. All 3 checked, priority is 1, and frequency is 1, but still no crawl.

    Sure, maybe later is fine. I will keep playing around.

  17. #17
    Grandma broke her coccyx! Uncle Rico's Avatar
    Join Date
    May 8th, 2007
    Location
    North Carolina
    Posts
    2,238
    I also edited the MS Firewall to allow GSiteCrawler, but that had no affect.

  18. #18
    ABW Ambassador
    Join Date
    January 4th, 2006
    Location
    USA
    Posts
    2,477
    I'm not sure if it is related to your problem, Seymour. I just went to check my google webmaster tools. All my sites with sitemaps that created with GSiteCrawler are showing

    "Sitemap errors and warnings".

    The status: "We encountered an error while trying to access your Sitemap. Please ensure your Sitemap follows our guidelines and can be accessed at the location you provided and then resubmit."

    They were fine last time I checked a few days ago and I have changed anything on my sites since then.

    The weird thing is they don't tell you what the errors are. I read through google webmaster tools' help pages, still have no idea where should I start to find the error .

    Could that be GSiteCrawler is having some problems?

  19. #19
    ABW Ambassador Georgie Peri's Avatar
    Join Date
    January 18th, 2005
    Location
    Norwalk, CT
    Posts
    846
    something else to try .. not sure what effect .. but make sure your running the exe file as Administrator ..Right Click / Run As Admin ..
    OpA! Giasou Ti kanies!

  20. #20
    Full Member
    Join Date
    January 18th, 2005
    Posts
    396
    I know it is a looong shot but (since I did this mistake awhile ago) --- is the case of what you are trying to scan ... php rather than PHP ...

  21. #21
    Grandma broke her coccyx! Uncle Rico's Avatar
    Join Date
    May 8th, 2007
    Location
    North Carolina
    Posts
    2,238
    Quote Originally Posted by Magi
    something else to try .. not sure what effect .. but make sure your running the exe file as Administrator ..Right Click / Run As Admin ..
    Magi gets the big cookie award. Ran as administrator and all worked. Thanks for your help and everyone else who responded.

  22. #22
    ABW Ambassador Georgie Peri's Avatar
    Join Date
    January 18th, 2005
    Location
    Norwalk, CT
    Posts
    846
    Lightbulb
    Quote Originally Posted by SeymourButts
    Magi gets the big cookie award. Ran as administrator and all worked. Thanks for your help and everyone else who responded.

    I have the best award anyone can ask for .. its the FREE EBOOK(http://forum.abestweb.com/showthread.php?t=118724) by YOU!!!

    OpA! Giasou Ti kanies!

  23. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. GSiteCrawler Question
    By Uncle Rico in forum Midnight Cafe'
    Replies: 2
    Last Post: June 11th, 2008, 10:08 PM
  2. Ask Jeeves crawling ?
    By Cosmo in forum Search Engine Optimization
    Replies: 1
    Last Post: May 11th, 2004, 03:25 PM
  3. Is the (google)bot crawling?
    By AffiliateBuddha in forum Search Engine Optimization
    Replies: 11
    Last Post: February 8th, 2003, 12:49 PM
  4. Dynamic Page Crawling
    By Sam Bay in forum Search Engine Optimization
    Replies: 12
    Last Post: December 20th, 2002, 08:43 AM
  5. Google crawling...
    By MsMarySunshine in forum Search Engine Optimization
    Replies: 1
    Last Post: January 27th, 2002, 04:14 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •