What happened to my statistics?

Once upon a time, when websites were just HTML, stats on a site came from counting how many lines were in the server logs. Things evolved and businesses blossomed and became too expensive for mere mortals (remember webtrends?) but those big packages could filter out the bots and handle big page views, they could even make a good stab at the differnce between pageviews, visitors and on the big expensive packages unique views (even if it was only a guess!)

Then along came Google... they had to have a pretty good analytics solution to make their advertising solutions work, they got even more information if you used it as well and all you had to do was add a little bit of code to your site... Great! almost, umm well pretty good... except it doesn't work for people who have javascript disabled, it won't measure image or asset views, some proxies seem to block it and we see that the script fails some of the time from our monitoring.

More importantly Google doesn't measure how many search engines spider your site, which is interesting considering Google is best known as a, um, search engine. I have seen some sections of sites where bot views are 40x the number of real human views! Sites that get over a million bot searches a day to 25,000 real views, so proper sites with circa 10,000,000 pages.

Here is the rub, to measure bot activity or asset usage you need log based analysis, but Google doing analytics for free seems to have put all the serious ones out of business! I spoke to webtrends, they told me they had stopped doing log based stats; they have gone to tag based tracking like DC Storm. These packages are great for tracking specific actions and really good for measuring marketing/advertising effectiveness and attributing sales/conversions.. But rubbish for finding out how much load your server is getting or if people are stealing images off you!

Interestingly one of the last good log based analytics packages is Urchin, now owned by Google it's also really hard to get hold of, only being available through authorised re-sellers.

So we had a demo of IBM Unica Netinsight. This is log based analytics on steroids. It reads your logs onto a database (which you can query with ODBC if you like) combines your logs with tags for even better event tracking, combines the logs with your own CRM database or even GIS address data and presents it all in a web based data analysis package that seems to be actually usable... unlike many packages that present you with a uselessly documented complex blank screen from which you need to pry your data kicking and screaming (are you listening DC Storm!) We are learning a lot just by playing with the interface... useful things that are changing the way we view users... which is what analytics is supposed to be!