Results 1 to 15 of 15
  1. #1
    Member
    Join Date
    January 18th, 2005
    Posts
    59
    Hi folks,

    I've had my website's host enable server logs for my merchant site, as one more way to make sure that all affiliate sales get tracked. Frankly, I don't fully trust PayPal, I don't fully trust cookies, and I definitely don't trust referrer info gathered by javascript.

    The problem is, this is a huge pain in the tush. Server logs are not designed for people to read easily, they are just recorded server transactions.

    Can anyone recommend a good server log analyzer, or formatter, for this purpose? I don't need graphical displays of the top 10 referrers (already have that, not useful for this), I just need something to help me organize this info for tracking affiliate sales.

    I'm on an Apache server, Unix I think, and the logs are compressed. I'd like to be able to convert to my local time, and have custom grouping or something. I really don't know what I need, but if I can customize it I can get it.

    I'd use crystal reports if I owned it (love that program!), but I don't.

    Free is always better, but I'm willing to pay for a program that does what I need. It would be worth it with the time saved going through server logs every time I make a sale with no cookie or referrer found.

    Thanks folks!

    Toxey

  2. #2
    Member
    Join Date
    January 18th, 2005
    Posts
    122
    Funny you posted this..

    We had an affiliate ask us to see the stats for his subdomain today that we are hosting. Never thought about it I just figured everyone had access to counter scripts.. We have stats but they are for the whole domain and we have no way to separate his stats from the rest.

    Anyway I was looking and I found 2 I liked. I like them because they can be hosted from one site/server and you can use the counter code on any site. It can be on any server anyplace.

    The free one was PowerPhlogger and the one I liked the most $100.00 was LoggerX. I am not sure if I can put a link in here?? So you will have to email me if you need that. If you know how to use a search engine you can find them.

    They may even be good scripts for the smarter affiliate to start a free counter site with. Just think about all the counter banners you could have all over the internet!! I thought about it and we have WAY TO MANY irons in the fire now so we will just stick to getting one for the people using our subdomains.

    So you guys know the people using our free hosting with their own domain do have stats.. This is just a problem for people using the subdomains.

  3. #3
    http and a telephoto
    Join Date
    January 18th, 2005
    Location
    NYC
    Posts
    17,708
    My other hosts use stuff like AWStats or Webalizer, which I think are free to install server wide and easy to configure for the sites that exist on the server. Plesk doesn't have something built in?
    Deborah Carney
    TeamLoxly.com BookGoodies.com ABCsPlus.com

  4. #4
    Member
    Join Date
    January 18th, 2005
    Posts
    122
    quote:
    Originally posted by loxly:
    My other hosts use stuff like AWStats or Webalizer, which I think are free to install server wide and easy to configure for the sites that exist on the server. Plesk doesn't have something built in?


    Yes plesk has Webalizer and that will track the whole domain. It cant break out the subdomains. Or if it can I have not figured it out??

    Webalizer has pretty crummy stats anyway. IMO

    I am just a stat freak.

  5. #5
    http and a telephoto
    Join Date
    January 18th, 2005
    Location
    NYC
    Posts
    17,708
    The plesk servers I have been on had webalizer stats by subdomain. Yes, they are crappy, but they are better than nothing
    Deborah Carney
    TeamLoxly.com BookGoodies.com ABCsPlus.com

  6. #6
    Member
    Join Date
    January 18th, 2005
    Posts
    59
    I've actually got webalyzer, as well as about 9 other programs I've tried and rejected.

    The problem is those are all statistics based. Who sent the most traffic, which is the most popular exit and entry page, what are my top ten yada-yada's.

    This is great for me as webmaster of my site, and it helps me to know where to focus efforts at improving customer experience and all that.

    But they're all detail poor, since their focus is basically on percentages, not individual transactions.

    I need to be able to open up my server logs, and be able to quickly find the time that a sale occurred (without translating timezone differences) and be able to determine (backwards from when sale occurred) the path that person followed to get to the sale. If it leads back to an affiliate (but the buyer had cookies totally disabled) I'll be able to give credit where credit is due...

    Unless they found it via an affiliate site, then a week later did a Google search to find the product. Then the referrer info will not show the affiliate at all. The ultimate would be if I could search and match the visitors IP with server transactions throughout my log history, and KNOW that they visited an affiliate, clicked on a specific product link, and then two weeks later bought the product.

    Just because they've turned off cookies doesn't mean the sale didn't come from that affiliate, and they should be given credit manually.

    I just don't want any affiliate sales to not be properly credited to the affiliate. I think that's bad for my program, and might discourage an affiliate from "hanging in there" until their pages really start producing sales. Plus, I'm an honest guy and wouldn't feel right doing it any other way.

    Again, progs like webalyzer are cool, but they're not transaction based. They're aggregate totals and percentages and graphs. I need to simplify the process, but I still need to deal with the individual transactions.

    Thanks, hopefully somebody's got the perfect proggy for that. I'll check out the two mentioned ones, perhaps I need to reshape my thinking about how this needs to be done.

    Best regards,

    Toxey

    [Edited to fix typos]

  7. #7
    ABW Ambassador
    Join Date
    January 18th, 2005
    Posts
    4,423
    We use and offer urchin, pretty much the king of the mid-range stat programs. But on top of that, people should be tracking things on their site independently, there is so much data you can track that is helpful - for this keyword, what does someone click on? What are the big sellers on each category page?

    This is where I really feel sorry for people trapped with webmerge and other similar products. There is so much data being given to you, you should have your sites dynamically using that data to serve your customers better.

    Chet
    --edit my post took a bit to save and missed toxey's followup, and he is correct, that is outside of normal log based stats, that is process/backend stat reporting, which is loads more important.

  8. #8
    ABW Ambassador FFoc's Avatar
    Join Date
    January 18th, 2005
    Posts
    1,015
    The tool you are looking for is 'grep'.

    No graphical gewgaws, and you will find the data you need quickly and easily.

    Though, you should really build in the tracking in the ecommerce site code. It will make you lots happier and be tons easier on you in the long run.
    “An adventure is an inconvenience rightly considered. An inconvenience is an adventure wrongly considered.” - G.K. Chesterton

  9. #9
    Member
    Join Date
    January 18th, 2005
    Posts
    59
    Thanks, I'll check out grep. It looks a little simple for my purpose, but I might be able to bend it to my will. :P

    As for building it into my ecommerce site code, I'm sure you're right that I'd be much happier but I'm not sure exactly how to do that. Although my site has been designed to appear very much like an ecommerce site, it's actually hand coded. Unless I can be sure that my products won't get ignored by search engines or I'll get penalized for parameter filled links, I'm going to keep doing this by hand. (Slow, yes. Do I have absolute control as a result? Yep.)

    Thanks much,

    Toxey McDavid

  10. #10
    Full Member
    Join Date
    January 18th, 2005
    Location
    UK
    Posts
    273
    Analog is a free log analysis program but is fiddly to setup.

    Personally I use Faststats because it does as its name suggests and is very fast as I have found other stats programs are a bit slow, which is ok for regular reports but a pain if one wants a quick ad-hoc one.

    I wrote the following page:
    http://www.wynsoft.com/logs/
    ages ago about processing webserver logs.
    It is is still basically valid and has links to the above products and a few more free and non-free ones.

    Les

  11. #11
    ABW Ambassador FFoc's Avatar
    Join Date
    January 18th, 2005
    Posts
    1,015
    Ok, how about this -- whenever you set your affiliate cookie, also note the IP addy in a log file with the AFID and the date.

    Check to see that the IP isn't in the file already with another's AFID (if so, replace), and also run a simple cronjob daily to remove those entries from the file that are beyond your cookies' normal expiration date.

    Then upon checkout, if there's no AFID cookie, compare the customer's IP vs your IP/AF chart.
    “An adventure is an inconvenience rightly considered. An inconvenience is an adventure wrongly considered.” - G.K. Chesterton

  12. #12
    http and a telephoto
    Join Date
    January 18th, 2005
    Location
    NYC
    Posts
    17,708
    IP's aren't static, especially for aol users. They can change minute to minute.
    Deborah Carney
    TeamLoxly.com BookGoodies.com ABCsPlus.com

  13. #13
    ABW Ambassador FFoc's Avatar
    Join Date
    January 18th, 2005
    Posts
    1,015
    That's why you check the cookie first and foremost. Toxey was looking for a reasonable backup.
    “An adventure is an inconvenience rightly considered. An inconvenience is an adventure wrongly considered.” - G.K. Chesterton

  14. #14
    Member
    Join Date
    January 18th, 2005
    Posts
    59
    quote:
    Originally posted by FFoc:
    That's why you check the cookie first and foremost. Toxey was looking for a reasonable backup.


    That's exactly right. Some people refuse all cookies, some browsers don't send referrer information, and that makes affiliate tracking impossible. But the server logs do a much better job of tracking referrer information because it's "pre-browser" (AFAIK).

    The problem is, I'm looking at 1000's of lines that look like this:

    quote:
    cache1-2.ruh.isu.XXX.sa - - [11/Jul/2004:02:01:22 -0700] "GET /PI-Supplies/paul-vunak-street-safe.html HTTP/1.0" 200 9373 "http://www.google.com/search?q=paul+vunak+video&hl=en&lr=&ie=UTF-8" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Hotbar 4.5.0.0)"



    Basically this says (in this order) who: 1) Asked for the page, 2) two blank fields I don't use shown as "- -", 3) When they asked for it 4)What page they asked for (and some codes), 5) What page they actually asked for, 6) and what browser they were using when they asked for it.

    Frankly, going through 1000's of lines interpreting this kind of stuff to make sure a $40 sale might or might not be an affiliate sale (i.e. they refused cookies, or shareasale didn't get the referrer info because their browser didn't send it) is a huge pain, and will get worse as sales increase. It's even worse when you have more than one visitor on the site (trust me, it's a mess).

    I'm just looking for something that will let me filter through this stuff. I start off with the following info: product sold, time and date sold. If we're lucky, I get a cookie I serve from my site that says how they found me (this time only, not if they found me from an affiliate weeks ago).

    If I don't get this cookie, or if I want to make sure I don't miss any affiliate sales that were referred weeks ago to a no-cookie browser, then I'd have to comb line-by-line through server logs for 60 days prior to the sale (length of my ShareaSale cookie). Frankly, this is too much to do for every sale. I'm currently willing to check backwards from sale to most recent entry point, but I'd like to do more.

    So what I need is, armed with the information of Product Sold and Time/Date Sold, I'd like to find the entry in the server log (easily!) and be able to track backwards and see the path that led to the sale. I'd also like to look back even further (60 days) to see if an affiliate site was used to access that product page before the sale.

    So, I don't need graphs and charts, but simplification of information. Actually, graphically would be okay if it separated visitors and showed me their paths.

    Sorry for the ridiculously long post, but I wanted to be clear about exactly what I needed so that hopefully someone will say "Hey! I know exactly what he needs!".

    I have programming experience, so it's possible I could write what I need myself, but it would involve learning a great deal more than I have time for. I'd much prefer to find a program that already has solved this problem, rather than make it myself.

    Thanks for the info so far, and if somebody has an alternate solution that sounds far less crazy I'd love to hear it.

    P.S. FFoc, I'm a little lost on the whole AFID Crontab thing, but it sounds interesting. I'll let my brain ruminate on that a while, thanks!

    [edited later for spelling errors, hopefully all of them]

  15. #15
    ABW Ambassador FFoc's Avatar
    Join Date
    January 18th, 2005
    Posts
    1,015
    If you build a referrer-to-X list into a separate file, you'd need to cull it on a regular basis, or it would quickly become unwieldy.

    Cron is a facility (as I'm sure you already know) to automate running a specific task on a set regular schedule. I use it to update my datafeed sites on the same day that the merchant updates their feed, as long as I know their schedule in advance.

    Basically, you run a script (be it shell, perl or php) from your crontab once per day (week, month) that removes the old entries (you define old, probably the same length of time as your AFID cookie duration) from your Referrer-to-X file.

    And to lessen the possibility of having dynamic-IP-related mis-referrals, you can use a DNSBL provider to mark the dynamic IPs in your list of IPs for manual tracking.
    “An adventure is an inconvenience rightly considered. An inconvenience is an adventure wrongly considered.” - G.K. Chesterton

  16. Newsletter Signup

+ Reply to Thread

Similar Threads

  1. Odd finding in server log
    By knight01 in forum Midnight Cafe'
    Replies: 2
    Last Post: July 12th, 2007, 12:18 AM
  2. Impressions vs. Server Log
    By John Powell in forum Commission Junction - CJ
    Replies: 6
    Last Post: October 23rd, 2005, 05:54 PM
  3. Replacement for the Free WebTrends Log Analyzer
    By stevethegreek in forum Midnight Cafe'
    Replies: 4
    Last Post: September 12th, 2002, 01:22 AM
  4. Which web log analyzer?
    By icepenguin in forum Midnight Cafe'
    Replies: 25
    Last Post: August 1st, 2002, 09:18 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •