Results 1 to 19 of 19
  1. #1
    pph Expert! Gordon's Avatar
    Join Date
    January 18th, 2005
    Location
    Edmonton Canada
    Posts
    5,781
    Which is the best meta tag to use?

    <*META name="robots" content="all"*>
    <*META name="robots" content="index,follow"*>
    Thanks in advance

  2. #2
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    808
    I use the <*META name="robots" content="index,follow"*>

    It works the best for me [img]/infopop/emoticons/icon_smile.gif[/img]

    Regards
    Amit

    [ 07-26-2002: Message edited by: ToolsToUse.com ]

  3. #3
    pph Expert! Gordon's Avatar
    Join Date
    January 18th, 2005
    Location
    Edmonton Canada
    Posts
    5,781
    Thanks very much Mr. Muscles with the great tat. [img]/infopop/emoticons/icon_wink.gif[/img]

    [ 07-26-2002: Message edited by: Gordon ]

  4. #4
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    2,341
    I don't use either of these tags. Is this type of tag really essential. I thought SEs usually crawled most links from a page anyway.

  5. #5
    pph Expert! Gordon's Avatar
    Join Date
    January 18th, 2005
    Location
    Edmonton Canada
    Posts
    5,781
    @ perfectG
    From what I have gathered it is an instruction to the search engines to index all the links. (2, 3, 4, 5 or whatever deep) I can only asume (I do not know for sure maybe markymark will put me right) that if you do not put this in some search engines do not do this they will only index 1 or 2 deep.

  6. #6
    Newbie
    Join Date
    January 18th, 2005
    Posts
    34
    <BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>Originally posted by Gordon:
    From what I have gathered it is an instruction to the search engines to index all the links. (2, 3, 4, 5 or whatever deep) I can only asume (I do not know for sure maybe markymark will put me right) that if you do not put this in some search engines do not do this they will only index 1 or 2 deep.<HR></BLOCKQUOTE>

    No, that's not how it works.

    Engines crawl what they want, when they want, unless told not to. The tag values you're asking about are useless because they're redundant: They tell a robot to do what it would do anyway. Those values only exist to satisfy completists.

    Read the spec: http://www.robotstxt.org/wc/norobots.html

  7. #7
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    2,341
    Islander
    Thanks, that is what I thought but it was only a hunch. I also heard that some search engine ignore instructions in the robot.txt file too? Seems pointless for things like log directories but that could easily be filtered out by search engines, as could cgi-bin directories.

    If I was a search engine, I would want to trawl all directories - catch the spammers.

  8. #8
    Newbie
    Join Date
    January 18th, 2005
    Posts
    2,694
    log files are not automatically filtered out by SE's. One of mine has a PR of 3 on Google. In fact, neither are robots.txt files.

    If you want to see what CNN's robots.txt or IBM's robots.txt looks like, do a search on Google [img]/infopop/emoticons/icon_wink.gif[/img]

    Soooooo, it never hurts to have a robots.txt and be specific.

  9. #9
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    2,341
    Thanks for that eaglefire. May start using them then. I noticed in the Google dance, several of my log analysis pages are indexed.

  10. #10
    ABW Ambassador Andy's Avatar
    Join Date
    January 18th, 2005
    Posts
    4,178
    How in the world does Google find log analysis pages if there's no link going to them?

    Do you need to exclude everything in the robots.txt file, even if there's no direct links to them? I was told to not include logs and other folders unless you have a connection to them that would be spiderable from your pages.

    Is this wrong? :confused:

    Andy

  11. #11
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    2,341
    Obviously Google looks to see what directories are present. That is the only way it could find my log files as they are in a directory I named as advlogger. No links there, but Google can obviously see the directory.

  12. #12
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    2,341
    Just a thought of how Google may have indexed my log files even though no links pointing to them. I have Google Toolbar installed and obviously check my stats. Google could have gotten the URL from that and visited the pages.

  13. #13
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    United Kingdom
    Posts
    1,797
    Islander is right re: robots meta tags.

    As for those using AdvLogger (fine, fine script, can I just recommend it again - www.perlonline.com ) , were you using SSI or Javascript ? Actually, Googlebot shouldn't be able to get in anyway - it's password protected.

    Best practise is to disallow all of your cgi-bin except any portion that you want the spiders to index.

    I have to say that I look out for people's stats in Google when I am conducting keyword research. People who don't password protect that horrendous Webalizer thing are my favourites [img]/infopop/emoticons/icon_wink.gif[/img] That includes some major CJ merchants.

    Interestingly, I would have thought that the Webalizer stats for CJ merchants would show a whole bunch of qksvr hits, but they don't. I could see exactly how much traffic each affiliate site sent, including in one instance, my own.

    Think about what this means in terms of search engines, then do a search for the main keywords for your site at Google. See a lot of your merchants there ? Draw your own conclusions from this - I know what mine are.

  14. #14
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    2,341
    <BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>Originally posted by markymark:


    As for those using AdvLogger (fine, fine script, can I just recommend it again - www.perlonline.com ) , were you using SSI or Javascript ? Actually, Googlebot shouldn't be able to get in anyway - it's password protected.
    <HR></BLOCKQUOTE>


    I am using the javascript.

    Thanks Marky, it was you who originally recommended them to me. I am soooo happy you did.


    Just checked, the type of thing that is indexed is this:
    www.xxxxxxx.com/advlogger/ admin.cgi?funct=analysis
    also www.xxxxxxx.com/advlogger/ admin.cgi?funct=hourstats www.xxxxxxx.com/advlogger/ admin.cgi?funct=entrypages www.xxxxxxx.com/advlogger/ admin.cgi?funct=topreferers

    etc

    21 in all for one domain

  15. #15
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    United Kingdom
    Posts
    1,797
    Weird. I must admit to being out of my depth with odd stuff like this. That script should really be in the cgi-bin though. Just create a robots txt file like the one below and that will sort it:

    # Robots.txt file for http://www.thatdomain.com
    #
    # Bans from cgi-bin and images
    #
    # Disallow /cgi-bin/

    User-agent: *
    Disallow: /cgi-bin/

    User-agent: *
    Disallow: /images/

    User-agent: *
    Disallow: /advlogger/

    This will exclude all your image files from being indexed, your entire cgi-bin and - if you haven't put AdvLogger in the cgi-bin, advlogger as well.

  16. #16
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    2,341
    Thanks Marky.
    Will putting advlogger inside a cgi-bin work on its own, or do I have to do the robot.txt file too?

  17. #17
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    2,341
    Just tried the indexed URLs and they take you to the password login screen, so I should be alright. Need to get them out of Google though to make more room for more $$$ pages [img]/infopop/emoticons/icon_wink.gif[/img]

  18. #18
    ABW Ambassador Andy's Avatar
    Join Date
    January 18th, 2005
    Posts
    4,178
    <BLOCKQUOTE class="ip-ubbcode-quote"><font size="-1">quote:</font><HR>Just a thought of how Google may have indexed my log files even though no links pointing to them. I have Google Toolbar installed and obviously check my stats. Google could have gotten the URL from that and visited the pages.<HR></BLOCKQUOTE>

    That makes sense! There are so many details to keep up with - ugh! I just added a robots.txt file to my site, I got tired of seeing requests for it in the error logs. I also use the "robots" meta tag on some of my pages, especially the ones I do or do not want indexed, just for insurance.

    Andy

  19. #19
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    551
    I just want to give a <IMG src=http://www.abestweb.com/ubb/icons/icon14.gif> to advlogger.

    Installed quickly and simply in cgi bin, security checks worked fine and the results are good.

    I will evaluate for a few weeks then move up to the full feature version. I also like the fact that they are so confident that they tell you NOT to buy without using the free version first.

    Thanks for the link Marky :cool:


    [speeling mistook]

    [ 07-28-2002: Message edited by: Wayder ]

  20. Newsletter Signup

+ Reply to Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •