Results 1 to 25 of 25
  1. #1
    ABW Ambassador flamingoworld's Avatar
    Join Date
    January 18th, 2005
    Posts
    5,208
    Why is the CJ robot visiting several of my domains that are not even developed yet or have any links, much less CJ links?
    These domains are not linked to from anywhere, so how is the robot finding them and why is it visiting them?

    Connie Berg
    http://www.flamingoworld.com


  2. #2
    ABW Founder Haiko de Poel, Jr.'s Avatar
    Join Date
    January 18th, 2005
    Location
    New York
    Posts
    21,609
    Connie,

    Do your domains have anything to do with the 2392... id? Easy script to run to find the other domains.

    Haiko


    The secret of success is constancy of purpose. ~ Disraeli


  3. #3
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    1,916
    Did you ever click through a CJ link from any of these sites (testing, etc).... that will do it...

    If not, no clue.

    ----
    -JM

  4. #4
    Member
    Join Date
    January 18th, 2005
    Posts
    118
    Connie,

    You already know my theory on this spybot.

  5. #5
    ABW Ambassador Sam Bay's Avatar
    Join Date
    January 18th, 2005
    Posts
    1,603
    Is it bad if CJ Bot visits a site, no matter what the site's condition is?

  6. #6
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    2,419
    According to one of Todd's posts, I'm pretty sure he said their bot would only visit pages that had generated sales in the last x days.

    That series of posts concerning the bot was at best - "a major blunder" (sorry Todd but thats the way I see it).

    If it's visiting domains not developed with no links on it, that seems to clearly throw out the purposes of the bot being as Todd explained it.

  7. #7
    ABW Ambassador flamingoworld's Avatar
    Join Date
    January 18th, 2005
    Posts
    5,208
    There are NO links on these domains. They are not developed! They are just parked domains.

    They are not linked to or from anywhere. Not a link in site!

    Haiko, you said "Do your domains have anything to do with the 2392... id? "

    I am confused???? What is the 2392 id? My domains don't have anything to do with anything!

    Connie Berg
    http://www.flamingoworld.com


  8. #8
    ABW Founder Haiko de Poel, Jr.'s Avatar
    Join Date
    January 18th, 2005
    Location
    New York
    Posts
    21,609
    PM explaining 2392 sent.

    Haiko


    The secret of success is constancy of purpose. ~ Disraeli


  9. #9
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    Nunya, Business
    Posts
    23,684
    I know some registrars like dotster have dotster affiliate links(cj links) on parked domains. Don't know if your parked domains are like this?

    "Nothing focuses the mind better than the constant sight of a competitor who wants to wipe you off the map."
    --Wayne Calloway

  10. #10
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    1,916
    If that were the case she wouldn't have access to the logs to see the hits from the spider.. Unless they were parked at the registrar (or expired domains) then once moved to Connie's servers the robot continued to visit.....

    Good supposition TN1... Very interested to know whats going on with this.

    Are they just doing GET / or DEEP (non existant?) URL's? Either way I guess they are getting 404's?

    Weird.

    ----
    -JM

  11. #11
    Member
    Join Date
    January 18th, 2005
    Posts
    118
    Joe,

    This robot probes deep, and is relentless, one webmaster was quoted as saying he had literally hundreds of pages, and it stayed the whole day.

  12. #12
    ABW Ambassador flamingoworld's Avatar
    Join Date
    January 18th, 2005
    Posts
    5,208
    Nope, no expired domains, NO links at all, just a sign saying under construction.
    I host all my domains at the same place, not a dotster place or anything.

    Has to be that they just got it from checking into what domains I own and are spidering them or something.

    Connie Berg
    http://www.flamingoworld.com


  13. #13
    Ad Network Rep ToddCrawford's Avatar
    Join Date
    January 18th, 2005
    Location
    Santa Barbara, CA
    Posts
    2,162
    quote:
    Has to be that they just got it from checking into what domains I own and are spidering them or something.
    This is not the case.

    We only spider referring URLs and registered URLs. I have no idea why any other domains would get spidered. I am sure there is a logical explanation. We have no interest in spidering non-referring pages. All we are looking at are pages that are sending traffic. The system gets it instructions from our referring URL table. We do not add URLs.

    I'll double check just to make certain that there are no issues.

    Connie, if you could send me the domains, I'll have someone look up how we got them in the database and maybe we can uncover this mystery.

    Todd Crawford
    Commission Junction

  14. #14
    Member
    Join Date
    January 18th, 2005
    Posts
    118
    Todd,

    Why didnt you say this in the first place? Please unregister all of my urls from my deactivated account.

    Thanks

    Glenn

  15. #15
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    843
    "We only spider referring URLs and registered URLs."

    Connie, I'm going to read alittle more into this and suspect that if you have remote files (graphic or otherwise on your site coming from other domains) the robot will be following the referring URLs of those files.

    Have you got any of your graphic files coming from other locations? I'm sure you must have already suspected this?

    You're a big site. I'm sure you must have had possibly even your main logo coming from a remote location for bandwidth reasons at one time.

    I would also talk to PCDudes (where you host) and ask them about how this might all be...if you already havn't.

    One thing I am sure of... is that the CJ robot is collecting information for reasons beyond simply publisher compliance.

  16. #16
    ABW Ambassador flamingoworld's Avatar
    Join Date
    January 18th, 2005
    Posts
    5,208
    I do have another site that I sometimes use for images. That one shows no visits from the CJ Robot.

    The site that was spidered has no files or anything on it at all. Just the placeholder page, with no links or images on it.

    This is a new domain. Nothing has been done with it yet whatsoever.

    The CJ Robot was the only visitor there besides myself.

    Todd, where do I send the domain name to?
    I don't care to post it here.

    Connie Berg
    http://www.flamingoworld.com


  17. #17
    ABW Ambassador ShoreMark's Avatar
    Join Date
    January 18th, 2005
    Location
    NJ, USA
    Posts
    912
    quote:
    Originally posted by kelly626:
    One thing I am sure of... is that the CJ robot is collecting information for reasons beyond simply publisher compliance.
    Of that, I don't think anyone has any doubt. It could be a positive thing though, they could want a big database of statistics to help their marketing team attract more merchants for affiliates to represent...

  18. #18
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    843
    The next question would be if this parked domain is a virtual host domain tied with your main site?

    If so, the parked domain would share the same IP number as your main site. Even though it could have its own separate log file, the spider may be following the IP number to find it.

    I am not seeing that PCDudes offers virtual hosting. Again, I would talk to them for possible clues as to how a spider could find your otherwise unknown domain.

    Please share with us all CJ's explanation if they respond. I'm sort of doubtful they will give a proper explanation that could give such an insight into how or why spybot does what it does.

  19. #19
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    843
    "It could be a positive thing though, they could want a big database of statistics to help their marketing team attract more merchants for affiliates to represent"

    Tod's statement today in another thread saying how they are so busy buying up the rest of the internet is ringing in my ears.

  20. #20
    ABW Ambassador flamingoworld's Avatar
    Join Date
    January 18th, 2005
    Posts
    5,208
    No virtual hosting. Not parked, has it's own ip. Waiting to be developed in the next few weeks or month. Still in idea development.

    Completely independant of other sites.

    Connie Berg
    http://www.flamingoworld.com


  21. #21
    Member
    Join Date
    January 18th, 2005
    Posts
    118
    Connie,

    You said the domain name is new, as in you just made it up or it became available? My thought is if it became available, it might have once belonged to another unfortunate Commission Junction affiliate.

    If you will note Todd stated something to the effect of registered urls.

  22. #22
    ABW Ambassador flamingoworld's Avatar
    Join Date
    January 18th, 2005
    Posts
    5,208
    Nope, it was new, I made it up, it is flamingo related. Never been owned before.

    I kinda got a thing for flamingos ya know!

    Connie Berg
    http://www.flamingoworld.com


  23. #23
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    843
    Just a few more questions.

    Why did you feel a need to check the logs files for a site that there could be no traffic for other than yourself of course?
    Were you just checking to see if the logging was at work?

    You're saying the site has not been submitted to any search engine, you have never linked to its location through any source, and are completely sure that PCDudes server applications are not to blame for spybot finding your parked domain.

    Do you deem it possible that CJ checked a whois database of some sort to find domains owned by you?

  24. #24
    Member
    Join Date
    January 18th, 2005
    Posts
    118
    quote:
    "It could be a positive thing though, they could want a big database of statistics to help their marketing team attract more merchants for affiliates to represent"

    Tod's statement today in another thread saying how they are so busy buying up the rest of the internet is ringing in my ears.


    huh, just checked out the idealab website, oddly enough, there seems to be an added logo there.

  25. #25
    ABW Ambassador flamingoworld's Avatar
    Join Date
    January 18th, 2005
    Posts
    5,208
    I was actually in my admin getting ready to set up some things for the new site, getting email set up and stuff and happened to notice that there had been some traffic and decided to check it out.

    I don't know of any server applications that could cause the spybot to find it. I have only had the domian 2 months.

    I haven't set up any affiliate accounts for this domain, and only recently starting thinking of what I was going to do with it.

    Maybe once the full disclosure is out on the robot is, it will be no big deal. I was just curious, not really upset.

    Connie Berg
    http://www.flamingoworld.com


  26. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Google Robot Searching for robot.txt
    By ahmar in forum Search Engine Optimization
    Replies: 4
    Last Post: December 26th, 2004, 01:26 PM
  2. Ecomworks Inc. robot?
    By waytogo in forum Rakuten LinkShare - LS
    Replies: 5
    Last Post: August 29th, 2004, 11:40 PM
  3. Robot.txt example...
    By eggerda in forum Search Engine Optimization
    Replies: 7
    Last Post: September 18th, 2003, 11:56 AM
  4. Robot txt
    By mousejockey in forum Programming / Datafeeds / Tools
    Replies: 12
    Last Post: January 14th, 2002, 05:05 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •