Knock! Knock! Who’s There?

Well, you will excuse my posting about comScore again. Since what their study implies is no trivial matter to our field, I would like today to go around some of the top blogs and summarize what their positions are:

Eric Peterson is probably the one who attracted our attention to it. He’s been doing a great job at asking comScore more details about the methodology. Eric knows how the Spring Cookie Deletion Fever does to WA: he started the first one (well that was Winter, but Spring is cuter). You got to follow what is going to be published on that blog to get the best state of the affair.

Anil Batra is rather cool with it. I mean not that he doesn’t appreciate the impact of the study, but he reminds us that, in an imperfect data world, and with all things being equal, trends are still what is actionable. He also reminds us of the actual relativity of the WA technology, where the same dataset will give out totally different results depending on which application you use. I tested it some years ago: I ran 5 different applications on the exact same (I mean the redundancy here) on the same set of logs, and never came with 2 identical numbers. I mean, not 2 applications shared similar numbers on any of the metrics, even the most basic ones. Well, I guess our world is one of quantum mechanics.

Gary Angel is right to point out, IMHO, that Anil’s position, although the most prevalent one in our field, could be a perilous one, since it excuse the accuracy problem maybe too lightly. Gary offers some could critique of that position, and puts the study into context: the deletion phenomenon must differ greatly depending on what type of site you run.

Stéphane Hamel offers a good summary of the study. He seems to be on Anil’s side of trend analysis, and not trying to provide hard numbers. Again, this is what most people in our field believe. Me included. However, I am growing less comfortable about the inaccuracy problem, especially in light of the integration with other customer data system who do a much cleaner job at counting “1”. Stéphane reminds us that comScore has also a lot of interest in positioning their methodology as the best one, especially with the upcoming IPO.

Ian Thomas (you got to scroll down one post) offers a position similar to Stéphane. I agree with him that there are hard numbers we wouldn’t like to pretend offering anyway. I always tell my clients that if they want to know how much they sold this month, they will be much better off looking at their bank account! I look forward to seeing how Gatineau will address the sessionization problem.

Craig Danuloff offers a quieting statement of the situation. Yes, comScore is probably right, and so what? Like Gary, he points out that the phenomenon won,t affect you the same way depending on what you do online. He also offers some good ideas that could help us “control” the deletion impact a littler better.

Ian Houston is actually getting his hands dirty with it, and doing some very interesting experiments. He,s got access to analyses from a WA application and comScore for the same site. Go read it: he’s got some troubling results. And, please, try to help him if you happen to be using comScore and a WA application.

And as I discussed in my previous post, Avinash Kaushik is calming us all down with a reminder of what an imperfect world we live in. But then again…

You got to read IAB’s letter to the panelists. They ask good and hard questions.

In another world, people like Ron Shevlin are also asking questions to comScore about their methodology. It seems that another of their recent studies on online banking is raising some hair. I commented on his post that I don’t think comScore is evil, but you got to be quite solid on your feet when you rock other people’s world.

On a final note, this situation has been an opportunity to be reminded Jim Sterne’s famous quote about Web Analytics being true but not accurate. I think it would be justified to organize a debate with comScore people at the upcoming EMetrics.

And you? Have you lost sleep?