Page 1 of 2 12 LastLast
Results 1 to 25 of 39
  1. #1
    Newbie
    Join Date
    January 18th, 2005
    Posts
    37
    Hello,
    Quick question for the google experts. My site is listed in google and has been for the last 6 months but the only page that ever gets spidered is the index page. I have over 3000 pages on my site, and I would really like to see more than the index page in the google index. Here is my question, how many pages can I submit to google per day without them thinking I'm spamming the engine.

    Thanks for the help in advance. [img]/infopop/emoticons/icon_biggrin.gif[/img]

  2. #2
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    ÄúsTrálíĺ
    Posts
    1,372
    try putting a
    <pre class="ip-ubbcode-code-pre">
    meta name="robots" content="index,follow"
    </pre>
    tag to each of your pages.

    As long as your other pages are linked from your index page you should be right. I don't mean all 3000, but if it they broken down into departments etc, you should be right.

    Also, make sure your robots.txt file is correct, and not disallowing google inside.

    [edit to add the code tag]

    [ 12-13-2001: Message edited by: Pete ]

    [ 12-13-2001: Message edited by: Pete ]

  3. #3
    Newbie
    Join Date
    January 18th, 2005
    Posts
    37
    Thanks for the reply.
    I have a page that is my sitemap, that has a link to every page on my site. There is also a link to my sitemap from my index page. I don't use a robot txt file, since all my pages have unique content on them. Should I use a robot file and if so what should I put in it to allow googlebot to find my sitemap?

  4. #4
    ABW Founder Haiko de Poel, Jr.'s Avatar
    Join Date
    January 18th, 2005
    Location
    New York
    Posts
    21,609
    runrun,

    Welcome! <IMG src=http://www.abestweb.com/smilies/welcome.gif>

    The robots.txt file should look like

    <BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>
    # http://www.yoursite.com
    # Robot Exclusion File -- robots.txt
    # Last Updated: 12/13/01 21:20:10

    User-agent: *

    Disallow: /cgi-bin/dirthatyoudon'twantspidered
    Disallow: /logs/
    <HR></BLOCKQUOTE>

    Go to http://robogen.com/ for an easy program to help you!

    Haiko

  5. #5
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    ÄúsTrálíĺ
    Posts
    1,372
    A robots.txt file is what you dont want the spiders to index

    I've read that it is often bad to create an empty robots.txt file.

    mine is simply
    --------------------
    User-agent: *
    Disallow:
    --------------------

    This means that any spider is allowed to travel anywhere.
    When I do get spidered, google, altavista, inktomi & fastsearch all hit every page.

  6. #6
    Newbie
    Join Date
    January 18th, 2005
    Posts
    591
    Just to add to the above suggestions...

    I would follow the above suggestions and NOT submit any pages to google manually. Let google find them after you made the changes above. Google will probably not include any pages you hand submit anyways because google likes to include pages it finds.

    One other thing, google needs to detect changes in your page(s) that it does have indexed or it won't crawl the page. Google just comes by, takes a quick look and says "nothing changed" and moves on. I'd make the changes suggested above and make sure you change your main page a little so google will take the time to crawl it.

  7. #7
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    2,279
    <BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR> Google will probably not include any pages you hand submit anyways because google likes to include pages it finds.<HR></BLOCKQUOTE>

    Sorry, CK, but hogwash. I've never had a manually submitted page NOT included.

  8. #8
    Guest
    helpful.
    Thank you.

  9. #9
    Full Member
    Join Date
    January 18th, 2005
    Posts
    260
    if you do want to sub multiple pages try www.selfpromotion.com if you join ($10) there is a multi-sub tool.. you can give it lists of 25 pages and it will submit 1 a day

  10. #10
    ABW Founder Haiko de Poel, Jr.'s Avatar
    Join Date
    January 18th, 2005
    Location
    New York
    Posts
    21,609
    And ....
    http://www.getsubmitted.com <IMG src=http://www.abestweb.com/ubb/icons/icon14.gif>

    Haiko

  11. #11
    Full Member
    Join Date
    January 18th, 2005
    Posts
    260
    Does getsubmitted submit 1 url a day?

  12. #12
    ABW Founder Haiko de Poel, Jr.'s Avatar
    Join Date
    January 18th, 2005
    Location
    New York
    Posts
    21,609
    Spiderman,

    No ... but they have gotten many, many, many of my sites into the top ten of Google! (after they were optimized)

    See Also: http://www.abestweb.com/tools/semarketing.htm


    Haiko

    [ added caveat & link ]

    [ 12-16-2001: Message edited by: Haiko ]

  13. #13
    Full Member
    Join Date
    January 18th, 2005
    Posts
    260
    ok,
    I like selfpromotion.com though, lots of info there.. helped me a lot when I started and it's free ($10 for 'extras' as the actress said to the bishop)

  14. #14
    Newbie
    Join Date
    January 18th, 2005
    Posts
    591
    Cedric Says: <BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>Sorry, CK, but hogwash. I've never had a manually submitted page NOT included. <HR></BLOCKQUOTE>

    The reason google included pages you hand submitted was because google found it from another link, not because you hand submitted it.

  15. #15
    Newbie
    Join Date
    January 18th, 2005
    Posts
    591
    And to Followup a little with some quotes from Google's Site:

    <BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>...it is NOT necessary to submit your site to Google in order to be included in our index. In fact, the vast majority of sites listed are not manually submitted for inclusion.
    ...If you like, you may submit your URL as well. Submission is not necessary and does not guarantee inclusion in our index. Given the large number of sites submitting URLs, it's likely your pages will be found in an automatic crawl before they make it into our index through the URL submission form. We DO NOT add all submitted URLs to our index...

    Google's robots jump from page to page on the Web via hyperlinks, so the more sites that link to you, the more likely it is that we'll find you quickly.
    <HR></BLOCKQUOTE>

    Here's the link to google with the above wisdom: http://www.google.com/webmasters/1.html

  16. #16
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    2,279
    Of course it's not necessary if your page is linked from somewhere else. But what you said earlier, and what I was commenting on, was "Google will probably not include any pages you hand submit anyways" -- that's still hogwash.

    As to whether the pages were included because they were submitted or because they were linked from somewhere else? Only the Googlegods know for sure.

    But you don't get penalized for submitting.

  17. #17
    Newbie
    Join Date
    January 18th, 2005
    Posts
    591
    There really isn't a much of a point in hand submitting links to google for these reasons:
    1) Google says "it's likely your pages will be found in an automatic crawl before they make it into our index through the URL submission form."
    2) Google says "Submission is not necessary and does not guarantee inclusion in our index."
    3) Even IF google does accept a hand submitted listing without finding it itself, then your page rank will probably be zero because google knows of no other links pointing to it. A page rank of zero will put you at the bottom of the search results. Is this your desired result from hand submitting?
    4) Low page rank can cause google to drop you from google's index. Here is a quote from google on why your page may not be in their index:
    <BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>...the links pointing to your page changed significantly and you no longer have a sufficiently high PageRank, or your page had low PageRank to begin with and a small change caused you to be dropped from the Google index.
    <A HREF="http://www.google.com/webmasters/2.html" TARGET=_blank>http://www.google.com/webmasters/2.html
    </A>
    <HR></BLOCKQUOTE>


    5) Google says hand submitting is a waste of your time. <BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>You are free to submit as often as you wish. However, given the nature of our inclusion process, your time is better spent improving the content and links of your site. http://www.google.com/webmasters/facts.html
    <HR></BLOCKQUOTE>

    Although I don't think you are likely to get a page into google by hand-submitting it, it may be possible, but since google uses Page Rank technology and your ranking in google is going to be based on the links pointing to it, it is a waste of time unless you want to be consistently listed at the bottom of the search results.

    You can probably figure out if google accepted your hand submitted pages. If your google page rank for those pages are ZERO, then you succeeded in getting a hand submitted page into google. Congrats!

    Given the title of this forum, I was under the impression the question was about getting listed in google in a fashion that pages submitted might actually be found by someone using google.


    post edited to fix links to info sources

    [ 12-16-2001: Message edited by: Commission King ]

    [ 12-16-2001: Message edited by: Commission King ]

  18. #18
    Newbie
    Join Date
    January 18th, 2005
    Posts
    37
    thanks for all the great info.
    I asked this question since I designed several pages to do well for the keywords that I noticed didn't have much competition and fit in with my theme. I get the googlebot to my index page everyday but it never travels any further, even after the robots.txt and changes made to my index page. I was able to get several pages listed a few moths ago by doing the submissions by through the addurl link. I was just wondering how many pages would be acceptable to submit to google per day?

    Again thanks for all the info.

  19. #19
    Newbie
    Join Date
    January 18th, 2005
    Posts
    37
    I forgot to mention that I have a pagerank of six on the index page and a 4 or 5 on the rest of my 8 pages listed in google. The 8 other pages all do very well for their respected keywords. That is the reason I would like to get the otheres listed. 1 more quick question, how often will google update their directory since I have been listed in dmoz for the past 2 months but still nothing in the google directory.

    Thanks again.

  20. #20
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    2,279
    Okay, one more word on manual submits to Google and then I'm done.

    My personal experience is that pages linked and submitted manually get listed faster than those just linked -- particularly if Google has recently indexed the site doing the linking.

    Freak accident? Fluke of nature? Perversion of all that is holy? I dunno.

    I just know what I see happening. And my *lowest* page ranking (of the ones I have checked, that is) is a four.

  21. #21
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    2,279
    Oops, have to take that back -- did find a few threes (a couple of layers deep in the site and not linked from anywhere other than natural navigation links) -- not manually submitted pages, though.

  22. #22
    Newbie
    Join Date
    January 18th, 2005
    Posts
    591
    If you have a Page Rank of 3 or 4, then google found it from other links. :cool:

  23. #23
    Newbie
    Join Date
    January 18th, 2005
    Posts
    591
    runrun,

    Google updates about once per month. Google spiders pages during the month (I see google visitng my sites mostly between the 8th and 15th). Google updates their directory later in the month about 2 weeks after I see google come through my site. So probably just after Christmas (Jadakiss is the ultimate predictor on this).

    Have you checked out google's reasons why google might not spider your entire site?
    If not, here is a link that lists some reasons why you might have problems: http://www.google.com/webmasters/2.html

    ps. If your site is a competitor of mine, I would reccommend spending atleast 4 hours every day hand submitting pages to google. [img]/infopop/emoticons/icon_biggrin.gif[/img] [img]/infopop/emoticons/icon_biggrin.gif[/img] [img]/infopop/emoticons/icon_biggrin.gif[/img]

  24. #24
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    2,279
    <BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>If you have a Page Rank of 3 or 4, then google found it from other links<HR></BLOCKQUOTE>

    But not until after the manual submit. [img]/infopop/emoticons/icon_wink.gif[/img]

    I'm not saying pages don't get found by links, I'm saying my experience is that, for whatever reason, manual submits get the pages included faster.

    And that, back to the beginning of this conversation, to suggest that there is a penalty for manually submitting is hogwash.

    Now I'm really really done.

    No really.

    Unless I'm not. [img]tongue.gif[/img]

  25. #25
    Newbie
    Join Date
    January 18th, 2005
    Posts
    591
    <BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>I'm not saying pages don't get found by links, I'm saying my experience is that, for whatever reason, manual submits get the pages included faster.
    <HR></BLOCKQUOTE>

    I don't seem to have that problem. Google seems to find my new links virtually everytime I had new pages and includes them in the next update. But then again, I don't waste my time hand submitting and maybe google knows that. [img]/infopop/emoticons/icon_smile.gif[/img]

+ Reply to Thread
Page 1 of 2 12 LastLast

Similar Threads

  1. Hi my site crawls pages in google are day to day decreasing
    By istham in forum Search Engine Optimization
    Replies: 0
    Last Post: April 30th, 2008, 01:48 AM
  2. FREE Google Site Creator, Submit, Updater.
    By SonjaHelga in forum Search Engine Optimization
    Replies: 13
    Last Post: July 13th, 2005, 06:35 AM
  3. Site Title and other pages dropped from Google
    By tonyfelice in forum Search Engine Optimization
    Replies: 6
    Last Post: January 3rd, 2003, 06:23 AM
  4. Submit new pages to Google, or wait?
    By Andy in forum Search Engine Optimization
    Replies: 6
    Last Post: August 3rd, 2002, 09:35 PM
  5. Google and site pages
    By john dundas in forum Search Engine Optimization
    Replies: 1
    Last Post: May 6th, 2002, 07:53 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •