Results 1 to 17 of 17
  1. #1
    Ad Network Rep ToddCrawford's Avatar
    Join Date
    January 18th, 2005
    Location
    Santa Barbara, CA
    Posts
    2,162
    I want to clarify this hot topic with a concise overview of what occurred.

    CJ has always reviewed (aka spidered, read, downloaded) publisher webpages. We are contractually obligated to monitor the network for quality and content. Much of this has been done manually in the past.

    During a recent upgrade to a spidering tool, a bug caused it to request pages at too high of a frequency for some servers to handle. This event may have affected up to 5% of our publisher network. This event only occurred once.

    Additionally, we requested some page URLs in lower case that may have caused 404 errors on some webservers. This event may have affected a larger percentage of our network than the first incident above.

    We apologize if your web logs or servers were adversely affected. The problems have since been remedied. We are currenlty evaluating implementation of this tool to ensure that none of these issues occur again.

    Commission Junction is committed to providing a quality network of advertisers and publishers who are equally committed to driving quality pay-for-performance results.

    Sincerely,

    Todd Crawford
    Commission Junction

    Todd Crawford
    Commission Junction

  2. #2
    Member
    Join Date
    January 18th, 2005
    Posts
    129
    Tom,

    As one of the first people to point out this spider, I appreciate this more cadid reply.

    I would though point out that there are other technical problems with the spider. Someone pointed out that it seems to have a problem with underscores.

    This caused 404 errors on my site and increased labor since we track and examine all 404s.

    Example:

    GET /pics/fn-12575700-kutcher

    I'm not sure what it was trying to get here but there's some bits missing. Case isn't the issue here.


    There must have been hundreds of these on the 24th alone.

  3. #3
    ABW Ambassador ShoreMark's Avatar
    Join Date
    January 18th, 2005
    Location
    NJ, USA
    Posts
    912
    quote:
    Originally posted by ToddCrawford, CJ.com:
    I want to clarify this hot topic with a concise overview of what occurred.

    Why the mysterious "maybe in July" statement then?

  4. #4
    Ad Network Rep ToddCrawford's Avatar
    Join Date
    January 18th, 2005
    Location
    Santa Barbara, CA
    Posts
    2,162
    You are right. The 404 errors were also caused by requests to URLs that have underscrores (_) in them. I was trying to keep the post more general and not confuse everyone with too many details.

    Todd Crawford
    Commission Junction

  5. #5
    ABW Ambassador Ron Bechdolt's Avatar
    Join Date
    January 18th, 2005
    Location
    Affiliateville, USA
    Posts
    7,927
    Thanks for the information Todd. Always good to hear it from the top.

    Hey, Todd, would it possible to get a section at CJ with publisher updates, like important information like this. Right now the only area we get is the top left of the page or our main publisher screen. Often these messages are left up there for days and weeks and rarely updated. It would be wonderful if, perhaps somewhere in this box, there was a link to a page that gave a history of recent publisher messages or alerts. And don't delete them once they are over, too. That way we can look back and see if an issue we are experiencing may already have been addressed.

    This would take very little time on someones part at CJ and provide a valuable resource for all publishers.

    Thanks.

    Ron - 7 Days A Week Marketing
    Every day is a chance to learn something new and an opportunity to teach others.

  6. #6
    Ad Network Rep ToddCrawford's Avatar
    Join Date
    January 18th, 2005
    Location
    Santa Barbara, CA
    Posts
    2,162
    Great idea Ron. I'll pass it along. I am going to spearhead an initiative to ensure that we are more communicative with our publishers on "what's up at CJ'.

    Todd Crawford
    Commission Junction

  7. #7
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    1,608
    Todd

    I'd rather have stuff that can increase sales be put back at the publisher screen, rather than clutter it with this itsy bitsy spider that does nothing but bang its head into the walls of our websites.

    Bring back the reports, easy accessibility, and the (improved) reversal rate.

    And as you can see, that dumb spider issue can easily be addressed and communicated clearly here at ABW, getting it to the right people, without using the precious bandwidth and space of CJ, which CJ appears to be trying to conserve.

    [This message was edited by waytogo on April 29, 2003 at 01:00 AM.]

  8. #8
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    1,608
    quote:
    I am going to spearhead an initiative to ensure that we are more communicative with our publishers on "what's up at CJ'.

    Todd Crawford
    Commission Junction
    Great idea Todd

    How about enlightening us as to what happened Monday Apr 28 when we cannot log-in to CJ.

  9. #9
    ABW Ambassador Andy's Avatar
    Join Date
    January 18th, 2005
    Posts
    4,178
    quote:
    Why the mysterious "maybe in July" statement then?

    I guess Todd must have not noticed this question, since he ignored it...

    Andy

    "If you were born to be shot, you'll never be hung." -Unknown

  10. #10
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    796
    Todd, please clue us in on this spider. Won't it obey the robots.txt file? If it ignores the robots.txt files it's already a rogue spider.

    I have nothing to hide and am willing to make the CJ spider welcome if I know what it's specific purpose is and that it will behave. Otherwise, it's going to hit a few firewalls that are already in place for rogue or unwelcome spiders.

    Also, if CJ is monitoring our sites for quality and content, it's only proper to let both publishers and advertisers know what CJ's criteria for quality and content are. Will I get dunned for using cheap graphics? Will my competitor get dunned for squatting on someone's trademarks? Will CJ agressively identify parasites or pirates and eject them from the network?

    What are CJ's criteria for quality and contents in regard to their network's advertisers? Will advertisers get dunned for reversals getting out of control? Will they get dunned for extremely poor conversions compared to their industry (now that's a real quality issue, isn't it?)

  11. #11
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    1,916
    BTW, it is my belief that this "spider" is using referrer data to seed its "crawling".

    just FYI.

    Not hard to figure out....and..could be a lot of fun to play with. *evil laugh*


    Not very happy about this thing... time to deny it.

    ----
    -J
    The neighbors complain about the noises above ...

  12. #12
    ABW Founder Haiko de Poel, Jr.'s Avatar
    Join Date
    January 18th, 2005
    Location
    New York
    Posts
    21,609
    From what I understand, the Spider is a "compliance tool" it will search sites that have a cj link on them to ensure that specific words aren't on the site or url. Many merchants want to ensure that their offerings aren't on non-compliant {with the PSA} sites. The newest version of the spider had some extra in it this time and that is why it crashed some servers.

    Haiko


    The secret of success is constancy of purpose. ~ Disraeli


  13. #13
    ABW Ambassador
    Join Date
    January 18th, 2005
    Location
    Nunya, Business
    Posts
    23,684
    The new spider from what i've read also spidered sites with no cj links on them, didn't conform to robot standards. It was a test spider that should have been tested inhouse before unleashing it on affiliate sites knocking some out, same time the Google bot was around.
    The info they get from spidering our sites is used to get or retain clients or they might charge them for the info.
    "If an advertiser has specific questions about specific publishers, we are always willing to answer their questions and educate them on the publishers' business models and marketing tactics. "

    "Nothing focuses the mind better than the constant sight of a competitor who wants to wipe you off the map."
    --Wayne Calloway

  14. #14
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    796
    Haiko, with some search engines believed to be running a witch hunt for affiliate sites (fact or fiction doesn't matter, some people think they are) and with some people not wanting their link ranking to bleed off through affiliate links, I think you'll find some affiliate links out there that the CJ spider will never find.

    I'm sure you know some people take their search engine listings very, very seriously, and that can include making affiliate links invisible to spiders.

    I hope CJ knows what they're doing if their spider ransacks an affiliate site and comes home empty handed. I hope they're smart enough to manually browse those sites for affiliate links that render to human eyes but not to the spider's.

  15. #15
    ABW Ambassador cditty's Avatar
    Join Date
    January 18th, 2005
    Location
    Memphis TN
    Posts
    1,434
    I was the site that was spidered that had no CJ links on it. Todd found it in the CJ database. It turns out that the person that owned the domain before me, used CJ to promote it. Todd told me the domain was removed.

    Chris

    quote:
    Originally posted by TrustNo1®:
    The new spider from what i've read also spidered sites with no cj links on them, didn't conform to robot standards. It was a test spider that should have been tested inhouse before unleashing it on affiliate sites knocking some out, same time the Google bot was around.
    The info they get from spidering our sites is used to get or retain clients or they might charge them for the info.
    "If an advertiser has specific questions about specific publishers, we are always willing to answer their questions and educate them on the publishers' business models and marketing tactics. "

    "Nothing focuses the mind better than the constant sight of a competitor who wants to wipe you off the map."
    --Wayne Calloway


    ----------------------------
    Scriptsforyoursite.com - Featuring datafeed import scripts for Art.com, Backcountry Store, Coldwater Creek, Mondera.com and many more.....

  16. #16
    ABW Founder Haiko de Poel, Jr.'s Avatar
    Join Date
    January 18th, 2005
    Location
    New York
    Posts
    21,609
    quote:
    Originally posted by Whiznot:
    Haiko, with some search engines believed to be running a witch hunt for affiliate sites (fact or fiction doesn't matter, some people think they are) and with some people not wanting their link ranking to bleed off through affiliate links, I think you'll find some affiliate links out there that the CJ spider will never find.


    I agree!

    quote:

    I'm sure you know some people take their search engine listings very, very seriously, and that can include making affiliate links invisible to spiders.


    Yepsters!

    quote:
    I hope CJ knows what they're doing if their spider ransacks an affiliate site and comes home empty handed. I hope they're smart enough to manually browse those sites for affiliate links that render to human eyes but not to the spider's.


    Me2!

    Haiko


    The secret of success is constancy of purpose. ~ Disraeli


  17. #17
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    796

  18. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. good spider or bad spider?
    By Gordon in forum Newbie Affiliate FAQs & Helpful Articles
    Replies: 3
    Last Post: September 3rd, 2004, 09:38 AM
  2. New spider?
    By womanht in forum Midnight Cafe'
    Replies: 1
    Last Post: August 8th, 2004, 06:00 AM
  3. CJ's spider bs
    By Linda in forum Commission Junction - CJ
    Replies: 41
    Last Post: June 3rd, 2003, 03:50 PM
  4. CJ Spider
    By Dirk Gardner in forum Commission Junction - CJ
    Replies: 63
    Last Post: May 6th, 2003, 02:23 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •