Results 1 to 10 of 10
July 11th, 2006, 11:34 AM #1CJ DNS Services Completely OFF CJ's Network
Monumental day as CJ is no longer providing DNS services for its domains and DNS is now mostly provided by VC.
Recapping recent events, a few weeks ago I had started this thread in regards to CJ's primary name server (ns2.cj.com) being unavailable. Within a few days, CJ had switched primary dns to a VC server (fs1.vclk.net), but for only the cj.com domain.
Last Sunday afternoon/eve there was an outage at CJ, affecting all domains except for cj.com.
Throughout yesterday, CJ/VC had switched the primary dns for all CJ domains (in addition to cj.com) to the VC name server at fs2.vclk.net. Hence, CJ is no longer using any of its server for DNS services and the work has mostly been transferred to VC's DNS servers.
nslookup -q=soa qksrv.net ns1.mediaplex.com
primary name server = fs1.vclk.net
responsible mail addr = systems.valueclick.com
serial = 2006071002
refresh = 7200 (2 hours)
retry = 900 (15 mins)
expire = 604800 (7 days)
default TTL = 600 (10 mins)
qksrv.net nameserver = ns1.mediaplex.com
qksrv.net nameserver = ns2.mediaplex.com
ns1.mediaplex.com internet address = 126.96.36.199
ns2.mediaplex.com internet address = 188.8.131.52
Thanks to ripe's Akamai reminder, There's a double lookup issue for all of CJ's tracking domains.
In summary, CJ's tracking domains are using Akamai's DNS servers for name resolution. Unlike the standard setup where visitors are directed directly to Akamai for DNS resolution, CJ's setup has visitors going first to the VC/Mediaplex servers, which in turns point visitors to the Akamai's name servers.
With this setup, Akamai's robustness is negated by any issue/outage from the VC DNS servers. From a performance/ speed issue, especially within the context of tracking services, the extra complete trip to lookup the Akamai host is very troubling.
Searching for www.qksrv.net A record at g.root-servers.net [184.108.40.206]: Got referral to F.GTLD-SERVERS.net. [took 39 ms]
Searching for www.qksrv.net A record at F.GTLD-SERVERS.net. [220.127.116.11]: Got referral to ns1.mediaplex.com. [took 72 ms]
Searching for www.qksrv.net A record at ns1.mediaplex.com. [18.104.22.168]: Got CNAME of track.cj.akadns.net. and referral to e.root-servers.net [took 86 ms]
Searching for track.cj.akadns.net A record at e.root-servers.net [22.214.171.124]: Got referral to H.GTLD-SERVERS.net. [took 78 ms]
Searching for track.cj.akadns.net A record at H.GTLD-SERVERS.net. [126.96.36.199]: Got referral to asia4.akadns.net. [took 116 ms]
Searching for track.cj.akadns.net A record at asia4.akadns.net. [188.8.131.52]: Reports track.cj.akadns.net. [took 225 ms]
Enter any CJ domains and look for "fs1.vclk.net" in the INFO>SOA>Primary nameserver: box.
DNS Lookup - A Record. Enter www.qksrv.net, or any of the other cj tracking domains (w/ the www).[url="http://www.wholesaler.com"]Wholesaler.com[/url]
July 11th, 2006, 11:37 AM #2
Thanks for your regular updates on this.
I'm catching a bit of what you are saying (too techie for me) but think I am missing the point. Can you speculate on why this is all happening?
July 11th, 2006, 11:46 AM #3
- Join Date
- January 18th, 2005
They have been down quite often lately. I'm using several ISPs and they had rotating DNS problems these last few days. Everything was fine with some ISPs and down with others.
Anyway, WHY do they have to use VC for their DNS services?
Is it an other LMI story?
I wonder if they are telling their merchants about the lost sales.
July 11th, 2006, 11:48 AM #4
it seems pretty in line with the whole LMI thing....drop the data off at the front door and then continue on with what started the process (requesting a publishers page / link)
If it does not go to a VC server first, how would they ever be able to determine the most appropriate content to display....when and IF it ever gets to that.
July 11th, 2006, 11:53 AM #5Originally Posted by Zeus
Off topic but you should look into www.opendns.com. I've been beta testing it for a few months and it provides excellent name resolution service, instead of using your ISP's dns servers. I've known David for a few years and have been using his everydns service since 00.
July 11th, 2006, 11:56 AM #6Originally Posted by mrbshouse
July 11th, 2006, 12:12 PM #7
if the search string is included in this, it would be a good quanity of data.
July 11th, 2006, 12:22 PM #8
- Join Date
- January 18th, 2005
"OpenDNS makes the Internet experience safer, faster and smarter for you and everyone using your network."
I'm changing my routers setting.
July 12th, 2006, 03:21 PM #9Originally Posted by 7-days
IMHO, CJ's management and oversight. of its technical operations can be improved upon.
CJ's strategy seems to be that of deploying best-of-breed products, in lieu of a complete understanding of its own operations. Similar to what I was referring to here, in regards to doing whatever is necessary from a marketing standpoint to keep the accounts coming in.
Pursuing this strategy has led CJ to the current difficult issues.
In summary, I do not believe anyone at CJ has "ownership/management" of CJ's DNS setups. Instead, there have been various "administrators" who have made modifications to CJ's DNS records, per specific requests, but no manager who monitors and ensure the deliberate and effective delivery of DNS services.
For example, the 12/04 outage was due to CJ effectively running on one name server, for months and without CJ's awareness.
When CJ was aware of an issue and took actions, the actions are sometime unfortunately incomplete. After my posting in regards to primary name server ns2 being down, CJ initiated a primary server change, but only for only cj.com and not the tracking domains. Consequently, Sunday's outage affected those tracking domains which were not switched over to VC's nameservers, as was done for cj.com.
These are straightforward outage issues, unlike what is IMHO the smoking gun as to no one is managing DNS.
Since being a CJ affiliate, i've always wondered about the "tracking is on a separate system", posting it here as a question to Todd.
With the recent outage, and a reminder to review Akamai's dns services for CJ, I now understand how CJ can assert tracking is on a separate system. Unfortunately though, this issue communicate a lack of management and oversight in CJ technical operations, for years ongoing.
As a crude example, suppose you operate a retail storefront and business is booming, customers are knocking down your door. So you go out and purchase a new and much larger door and a have it installed. You get the invoice for the installation as well as periodic maintenance invoices, so you assume the door is installed and you're providing a larger doorway for customers to enter your store. There's even a counter on the new door and you can see how many people are going through it. So one day, you decide to check out your new door and see that it was installed directly behind the old door, thereby providing your customers with no larger of an entry as your old door.
It breaks my heart every time I see this:
Searching for www.qksrv.net A record at j.root-servers.net [184.108.40.206]: Got referral to F.GTLD-SERVERS.net. [took 365 ms]
Searching for www.qksrv.net A record at F.GTLD-SERVERS.net. [220.127.116.11]: Got referral to ns1.mediaplex.com. [took 78 ms]
Searching for www.qksrv.net A record at ns1.mediaplex.com. [18.104.22.168]: Got CNAME of track.cj.akadns.net. and referral to g.root-servers.net [took 83 ms]
Searching for track.cj.akadns.net A record at g.root-servers.net [22.214.171.124]: Got referral to A.GTLD-SERVERS.net. [took 51 ms]
Searching for track.cj.akadns.net A record at A.GTLD-SERVERS.net. [126.96.36.199]: Got referral to zb.akadns.org. [took 67 ms]
Searching for track.cj.akadns.net A record at zb.akadns.org. [188.8.131.52]: Reports track.cj.akadns.net. [took 129 ms]
In summary, mediaplex.com (CJ) is analogous to the old door and akadns.net (Akamai) is the new door and apparently you have to go through the old door before you can use the new door.
I'm reminded of the Steve Jobs plea to the fist MAC team to cut one additional second of the boot time, hence giving the cumulative effect of saving the world 5 years of time. CJ receives millions of dns requests per day and assuming (conservatively) 100ms additional for each request, that works out to 1.16 day of wasted time for every million dns requests.
I remember first seeing this a couple of years ago, and filing it away as temporary configuration. Revisiting the Akamai/dns issue this week, i'm horrified that this the de facto standard.
July 12th, 2006, 03:34 PM #10
Thanks, that really does help.
Sounds like CJ needs to hire a dedicated contractor to help them with some doorway remodeling!
By thomsenhk in forum Starting an Affiliate Program & Merchant Q&AReplies: 5Last Post: February 27th, 2012, 12:29 PM
By ~Michelle in forum Midnight Cafe'Replies: 6Last Post: September 11th, 2004, 02:21 PM