(First published in the WAO/FACTOR newsletter in May 2013)
Analysis Governance – By Analysis Governance I basically mean the framework within which analyses will be performed. In other words, of all the possible metrics we can collect data about, which ones should we focus on so that we know we are executing the digital strategy flawlessly? That’s right, KPIs, among other things.
You just can’t say “Go into the analytics application(s), and tell us what you see”. Well, I say you can’t, but you would be surprised (and saddened, believe me) to know that many, still many companies do just that. They believe everything there is to know, what is important to analyze, is what the product(s) they acquired will tell them. A sign that this is happening? You still look, and look only at the default reports. Yes, customizing reports is a first and good sign that you want your applications to adapt better to what you need to know.
Key Performance Indicators
For years I have identified focused, deliberate definition of one’s own KPIs as the first indicator that a business was now engaged in the right path of making analytics meaningful, beyond the reporting stage. An organization can stay indefinitely at the reporting stage, however. I have known organizations that are still there after ten years of implementing technology, and upgrading it, regardless of my calls for pushing the analysis framework further (they’re almost all in government though, which means I shouldn’t mind so much I guess).
Determining your KPIs, and the other operational metrics that influence them most, will constitute your analysis framework, and it will give the overall direction to the mysteries you will need to tackle, the questions you will ask, and eventually the recommendations you will come up with.
So, in good governance, those performance indicators and related metrics would all be defined and documented through an official process that should include as many stakeholders as possible.
I have talked about this process extensively, and will not bother you any further with it. Here’s a 1-hour webinar I recently gave:
What is the Mystery?
A deeper question than determining KPIs (yes, there are those questions), is the one of what we really try to uncover, to dig out, to bring to light. What is the deep mystery we need to solve? KPIs will naturally be within the realm of one’s strategy, and analytics will be an activity aimed at reporting on that strategy, with an obsession on small incremental improvements. This is a good thing; at the end of the year the sum of all those improvements will bring gains to the organization. However, analysis will then always stay within the horizon of the predictable, of what the past helps us envision using complex regressions. This is analytics whose main role is to reassure.
Are we in business to be content with a 5% annual increase? Should this year be the same as all past years, plus a little more? What if we could do things really differently? What if there were whole new types of clients we could serve, and never knew about? Could those outliers in the data be signs of opportunities? Besides using our KPI framework, we need to leave room for bold questions, detached from the safety of our own performance evaluation metrics. This is analytics whose main role is now to inspire.
A lot more on this in the coming months.
Questions and Hypotheses
Day in day out, analysts have to answer hundreds of questions about campaigns, customer segments, the best place to insert a call-to-action. “What works?” is the main question. It is essential to keep asking it, and search in the data for past proofs that allow us to make reliable predictions about what to do. This is how we can keep the farm.
However, we need to ask a lot more “What if” hypotheses and search in the data for facts that allow us to make valid predictions about what could happen is we tried. This is how you bet one of the geese.
Write down all your hypotheses, and make one or two officially the mystery to solve for the year. Let yourselves draw wild conclusions from even little blips in the data, even if there are only too few data points to even draw a trend line.
Look for signs, not proofs.
Governance means to be explicit about what we do, and consistent in doing it. It doesn’t mean to be left with only the past to repeat.