Recently, my friend Gary Angel (certainly *the* one blogger you should follow in Web Analytics if you had to keep only one) wrote about something I thought had profound consequences. In a few words, Gary says that an online marketing agency could not fundamentally be the same one who reports on whether strategies and campaigns were successful or not. I must say that, in general, I certainly agree with him.
I was attending a small Web Analytics conference here in Montreal this week, and that question came up. There were several people from agencies (here, by agencies, I mean online Marketing ones, not measurement consultancies), and their stance, not surprisingly, was that, “absolutely, we are objective, and everybody is on board with what we do.” I don’t think they’re lying, not even bullshitting a little, and those analysts really believe it. However, I’m not convinced that analysts in agencies can be totally free to tell the absolute truth. Pressure must be very high to present multi-million dollar projects as successful. Gary uses even stronger language:
In a world where there are lies, damn lies, and statistics, why would you let your Agencies measure their own performance? If you’re agencies are siloed, they have every incentive (and ability) to make their channel look maximally successful. If you’ve concentrated everything in a single agency, that agency has every incentive (and ability) to make their entire program look successful and not delve too deeply into any single piece.
This is a complex question that has deep root in agency cultures, agency-client relationships, and how much those relationships leave room for errors.
Everybody knows that creativity is at the heart of what agencies do; gee! it’s the reason why clients hire them. I think too many people believe that creativity is antithesis to measurement, or at least that in the face of rather poor results, a very original concept must have the larger share of value. I have discussed that elsewhere, and all I wanted to touch here is the creativity for creativity’s sake moto that is often the main dynamics at many, many agencies (you know, “I didn’t make an extra buck, but my agency won prizes”).
The other point is I think the main factor why it is very hard to build strong analytical cultures at agencies. I very much doubt that clients are yet ready to accept candid and transparent admissions of failure. And this is especially true if their own culture leaves little room for mistakes. Since basically, to many clients’ mind, one hires an agency for its expertise, “they sure know what they’re doing”, it is easy to imagine that expertise means being right, always. Anyway, why would you pay someone to mess up, which you can easily do yourself for far less money?
So, I am not sure analytics have a viable proposition within agencies. Yes, you can call my own and Gary’s bias on this, since both our firms are in the measurement business.
The choice of bias is yours.
[August 29th Update]
Gary Angel in a new post reviewing Marshal Sponder’s new book Social Media Analytics, quotes from Marshall’s book about the role of agencies in measuring what they do for their clients in Social Media (NOTE: I am reproducing the book quote here without permission. Since this is a very small excerpt, I believe it will not pose a problem):
…I have come to believe marketing and communications agencies are not the most appropriate entities to measure marketing or PR campaigns run on behalf of their clients, especially within social media. Too often there is an inherent conflict of interest, as MarCom firms measure their client’s online buzz, and data can be skewed, often unintentionally, to show the successful completion of agreed-upon campaign goals.
I’m going a bit cross-cultural here, but the same problems crop up when interactive or traditional agencies also offer UX testing services in-house. I’ve always believed that evaluation activities (whether it be campaign performance, usability, customer satisfaction, or other) should ideally be done by a third party, neither client nor agency (Peter Merholz, of Adaptive Path, wrote a *scathing* blog entry a few months ago about this very subject http://bit.ly/kRUztd). Of course, since we live in a world where user research is still considered a luxury or an afterthought at best, this all might seem like wishful thinking.
Salut Jean-François
Yes, I guess this post would apply to that type of expertise as well. There is often a *lot* involved in UI (politics, investment choices, etc), and with so much at stake, one can easily imagine there could be a lot of temptation to “torture the data” as Jim says below!
This is essentially the same argument I had with Christopher Berry about web analysts reporting to Finance instead of Marketing:
http://christopher-berry.blogspot.com/2011/06/analytics-at-marketingtechnology.html
When analysts report to the people whose work they are analyzing, you get pressure to torture the data, whether implicit or explicit. That is just the way it works.
Hi Jim
Certainly; you were one of the very first to bring that up some time ago. I thought Gary’s post was another good occasion to have the discussion. Thanks for commenting.
So, you are presuming that clients want objective assessments of advertising success? That they are willing to pay for sound measurement? That they are willing to make changes based on that information? How charming.
It would, of course, make sense to do all that, wouldn’t it?
Now, take a look at those same clients. To what degree do they make an effort to obtain objective assessments of success elsewhere in the business? How much do they invest in sound measurement? Do they routinely make business decisions based on such information? Maybe not so much as they oughta, right? (Some do, many don’t.)
I hear what you’re saying, yet I doubt that real measurement failure is in the agencies. Many clients are not sufficiently motivated to measure and make data-driven decisions. As long as that is true, it really doesn’t matter who is assessing advertising programs.
@Meta Oh boy! You ARE right!! And I believe I say that in part in my post. I do think that most clients are not willing to let their agencies make mistakes, admit them, and improve on them. And this is, I think, very often linked to their own measurement culture, or lack thereof.
I have been educating (bah! I should just say “preaching”!) for the last 10 years about the importance of measuring, and I still come across everyday organizations that don’t seem to care at all. And many believe that because they plugged Google Analytics on their site they are doing analytics.
I must confess that this still depresses me.
Thanks for commenting
“The other point is I think the main factor why it is very hard to build strong analytical cultures at agencies.”
Couldn’t agree more.
The agency I worked at, in theory, wanted a strong analytics department. Everything they did went against that though and when I realized they didn’t want to grow it I left.
An agency we work with now has a forgettable analytics team mostly because I don’t think they (the leaders) take it seriously and they’re only hiring entry-level kids who barely know what they’re doing. They’re certainly not paying money to people who have actually done web analytics before. And that’s too bad, I’m sure they’d do a better job if they were able to treat it like a real competency.
I think it really does boil down to your point of their just trying to spin the results. Why would they go out of their way to showcase the failures of the campaign?
Hi Bryan
Thanks for sharing that hands-on experience. I sure think it is a hard proposition, maybe not impossible, and here I am willing to admit that there could be exceptions, but still hard, because in part of the nature of the client-agency relationship. See what Meta says to that effect, and I can’t disagree.
Since humans tend to behave the way they’re paid to, maybe client should start rethinking what behaviors they want to reinforce. Or plainly ask themselves if they are able to handle the truth…
Hi,
I do agree with the idea that auditing your own work doesn’t deliver too accurate results (regardless of an industry). On the other hand, I strongly believe that authors should be the first ones to check, measure and verify their own work.
I run a digital shop, we’ve been doing some crazy stuff, mainly using Flash. Some years ago, when even Google Analytics couldn’t track Flash, we’ve been investing our own time and money to create a technology (which later became http://www.useitbetter.com) that would help us find out how exectly people interact with our websites, so we could optimize them, bring more benefit to our clients and more fun to users.
I know that my agency is not the only one that tests and measures its own work. Many agencies want to make great and effective stuff because they love their job and strong portfolio brings new clients.
Using an objective auditor on every stage of a project that requires testing/measurement is usually impossible because of time and budget limitations. Even it would be possible, it would probably kill the creative process in many cases.
Don’t get me wrong, I have nothing against hiring a third-party, specialized analytic vendors. If I were a client and had money, I would do that because in-house teams are often less skilled. But first of all, I would set up a win-win situation, in which every side would benefit from honest reporting. If an agency would be paid for improving its work why would it want to juggle numbers? If the client was engaged in a project and co-responsible for its outcome why would he want to blame and drop the agency?
best,
Lukasz
Hi Lukasz
Thank you for your long comment. I do not disagree with you, and as someone who offers specialized professional online analytics services, I very well know that only very few clients seriously invest in it. So, sure, I think that an agency that offers analytics within its projects is better than one that doesn’t, although one can imagine how pernicious it could be to hide an agenda behind the appearance of pseudo-factual results.
I still think that it is a tough proposition, i.e. being totally freed from internal pressure to give the clients the story they want to hear, tougj, but not impossible…
In an ideal world, analytics would be a separate function. As other people have noted, there is no buy-in from either the client side or agency side to be 100% transparent…clients don’t want to hear the bad news, and agencies don’t want to deliver the bad news, each for their own reasons. I work at an agency and have definitely felt the pressure from on high to leave certain things out of reports or to make a bigger deal about other things.
I think the push for analytics to be done outside of agencies sets up a few issues, which, of course, are informed from my own personal experiences/biases:
1) The person who put together the media schedule and overall strategy is just down the hall from me. Now place that person in an entirely different company, each of which have their own software and ways of creating schedules, and it will be that much more difficult to put together the pieces of the puzzle.
2) Our agency works in a niche market, and truth be told, sometimes the recommendations we get from outside vendors (SEO, PPC, etc) suggest a profound misunderstanding of this market. Unless analytics vendors (auditors/consultants/etc) could become niche-specific, this throws a real learning curve their way.
3) I think local advertising is on the verge of exploding; that being said, a significant segment of businesses moving online will be from the SMB segment. Getting a small business to pay for advertising, let alone paying to analyze that advertising from *another* vendor, would be a large battle. These are clients that need Google explained to them…so imagine teaching just Google Analytics! That takes a much deeper and longer-lasting relationship than I think consulting could bring…but I could be wrong on that since, to be honest, I have not yet worked with an analytics consultant or witnessed that type of consultation.
Now, I do understand where the sentiment for third-party auditing comes from. I’ve seen too much pulling-the-wool-over-the-eyes to disagree with you or Gary Angel. That being said, there are a lot of details that would need to be ironed out for this move to work.
PS, I am “one of those”–someone from an agency who signed up for the newsletter 🙂
Hello Anonymous (I usually don’t allow anonymous comments, but yours was insightful)
You bring very interesting arguments to the conversation.
1) I can see how process easiness can be a good argument. Certainly, the more ingredients one controls, the better the chances that the cake will be good. However, this does not guarantee that analytics would enjoy the freedom it needs.
2) Good point here. Not being very knowledgeable of a niche market can certainly impair one’s capacity to make recommendations. However, I still believe that one doesn’t need that level of knowledge in order to identify what doesn’t work. But I can see your point, and it is certainly a strong one.
3) Well, yeah, you’re touching something very sensitive. Obviously, SMBs, and especially SBs, prefer one-stop shops, and can’t really afford highly priced services. You got a point here. They (the SMBs) will have to live with what they can pay.
I would have been interested to see what you think of clients’ responsibility in this situation.
And thank you for subscribing to the upcoming newsletter!! Agencies are most welcome!
Finally, why don’t you come to San Diego in September to the X Change Conference? That’d be great to debate it together with Gary.
Thank you for contributing.
Hi Jacques,
Thanks for accepting my comment. I typically don’t post anonymously, but this time I had my reasons.
I must note that our agency has learned a lot from analyzing data. I think any agency that takes analytics seriously can add a lot of value for their clients. For instance, when we started our AdWords campaigns, we had CTRs with a monthly average of 1.7% and “goal conversion rate” was unheard of–we weren’t even tracking goals on our clients’ websites! Fast forward a few years and we now have campaigns with CTRs at a monthly average *over* 6% and conversion rates for those campaigns hovering at 4%.
Coming from the agency perspective, it’s predictable that my cynicism is (mostly) directed towards the clients. I’ve literally had clients tell me “I don’t care how many graphs or Google Analytics charts you show me, you’re wrong!” We’ve presented in-depth case studies about the fact that if our client just changed X, Y and Z they could more than double their goal conversions, only to be told “no.” And even when a client has taken interest in the data, it’s more or less because they’ve hired someone to be the full-time “gotcha!” person, where the smallest error can be cause for dismissal as an agency.
Overall, I think clients are uneducated about the possibilities of analytics. Whether the agency teaches them, the client hires an analyst, or the client reach out for consultation, someone needs to educate clients about the possibilities. When a client really gets this stuff, it’s like magic! Everything just seems that much better in the world. Moreover, I think this education also needs to come with a culture change…one that allows for mistakes (within reason). I’ve never had a client thank me for catching something that went wrong and moving strategy in a different direction; mostly I just hope we don’t get yelled at in a meeting where that kind of thing has to be discussed.
Anyways, after mulling over what I wrote last night, I think my point-of-view really comes from a frustration of dealing with SMBs exclusively while watching the web analytics world engage in an almost strictly enterprise-level discussion…so, conversations like this, where there is a movement for advocating that the analytics function live outside the agency, are unrealistic for the types of clients I deal with. Not that I disagree with you: I’ve watched vendors present numbers and metrics to clients about how great a job their doing, only to be suspicious that they’re either incompetent or lying through their teeth. We’ve taken some of that business directly from those vendors and done things like reduce CPC by 25%, increased visits by 100%, etc. In other words, even though I come from the agency side, I see the need for analytics living outside of the agency…I just have my doubts about whether or not it’s viable in most cases.
Great addition to your previous comment! I mentioned in my post that clients definitely had some share of the responsibility. This “culture of analytics” that T. Davenport talks so much about, the “learning culture” that so many writers push forward, is certainly NOT that easy to come by! Rare are clients who accept some margin of error (“within reason” as you well put), and maybe this is asking a lot. I don’t know.
At the end of the day, analytics must bring more value. Why do it if it doesn’t? And I have asked analysts when was the last time they found at least $50,000 in their data, and not many hands were raised. The same has to be said of creativity in a business context: it brings value or it doesn’t, and the more that value can be measured beyond “It’s good for the brand”, the better.
The problem with asking analysts about the dollar value of their discoveries is that most analysts can’t relate what they do to dollar revenue or costs, even when the value is there. This is not a problem unique to data analysts, it’s true of most people.
Hi again Meta,
I agree. However, I believe analysts should try as much as possible to follow the value chain, and connect what they discover with financial value. Whenever financial value is created, someone takes credit for it. I don’t say analysts will be able to get their share all the time, but demonstrating the connection between their work and gains, even once in a while, will do them some good!
You’ll get no argument about that from me! It’s just that there’s a difference between the information lacking value and the analyst lacking the ability to recognize and explain the value. I’m sure much of your audience was staring that $50,000 in the face many times over and not knowing it.