Analytics On The Bleeding Edge: Transforming Data's Influence

Analytics teams are named for the silos and limitations within which they trap themselves.

Paid Media. Owned Media. SEO. BI. Customer Service. Data Warehousing. Email. And, a thousand other silos (depending on your company size).

One outcome of this reality is that while every team works hard to do their very best work, it is rare that they earn strategic influence from their work. That’s not really surprising, if your view of your scope is narrow… Your impact will be narrow as well.

The other dimension to consider is most Analtyics teams kick into gear after the campaign is concluded, after the customer interaction has taken place in the call center, and after the funds budgeted have already been spent. When you only look backwards, it limits your ability to have an impact.

Finally, few analytics teams obsess about predictive analytics in a way that allows them to dictate future action. This is a huge miss… Left to their own accord, how many companies will make the same decisions data would recommend? Astonishingly few.

Transforming Data’s Strategic Influence.

The above-observed realities were on my mind as I took on a new role to lead Global Strategic Analytics. This time around, my goal was for the analytics team to chart a very different path… To solve for expansive influence, before, during, after, money is spent by the organization.

A key part of how this manifested in our work was doing truly super-advanced machine-learning powered analysis to answer hard questions that few can successfully. This is of course exciting and very cool.

But the difference in the team’s impact comes from the combination of an audacious vision and putting together the people-process-structure that powers our desire for data to have an expansive influence across the company’s decision-making needs.

I lovingly call our strategy analytics on the bleeding edge. It is powered by the union of:

1. intelligent analytics initiatives
2. application in data to new and unexplored areas
3. upgrades to processes to create deeper integration with Finance & Strategy teams
4. power real-time actions and future decision using automation and algorithms

While our team’s journey had many, many, miles to go, I want to share the broad strokes of our vision and execution in the hope that it’ll help spark your imagination – perhaps you can use our core philosophy to reflect on your organization’s current status and future path.

I’ve boiled our approach into four smart clusters. It is most directly applicable to larger companies, but some components do apply to companies of all sizes.

Let’s look at them one at a time… Ready?

Smart Lessons | Analytics Cluster.

Like every good analytics team, we started doing work that you’ll recognize.

Executive scorecards, post-campaign analysis, some limited data puking (only when we absolutely can’t get away with it because someone who influences our existence is asking!), using Google Analytics in a smart way, setting good behavior standards like always having Targets (pre-set, always pre-set) and using methodologies that don’t suck and 90% or higher significance, etc., etc., etc.

What may or may not be as common, but is an integral part of our analytics strategy is the extensive use of controlled experiments to answer life’s hardest questions.

Is campaign strategy x better than campaign strategy y? Because they are both different, both have different executive sponsors and it is insanely hard to know based on data we have which one is better.

Does advertising really have a long-term business impact? Surely you’ve been asked that one before, and there is a short-term answer but the long-term one needs a sophisticated controlled experiment.

What does the diminishing returns curve look like for TV GRPs for our company? More shouting is not really better – and it is expensive!

This is very hard to do, we now have a proven seven-step experimentation process, with one of the coolest algorithms to pick matched-markets (normally the kiss of death of any large-scale geo experiment).

Underpinning our Smart Lessons work is the very basic – incredibly complex – art of picking the right Key Performance Indicator. It underpins every dimension of success.

Hence, if your assessment is that you are messing up Smart Lessons, it is because of this simple reason: Wrong/bad KPIs.

The choices our team helps make are powered by another awesome innovation: The Impact Matrix.

Smart Start | Analytics Cluster.

It has always gnawed at my soul that most companies only turn to data after all the money has already been spent. After the campaign is done. Long after your CRM-powered emails have been sent. Long, long,  after your TV ads have stopped running.

On those non-normal occasions when the campaign did not quite work, it pained me that we were learning after the money had already gone poof!

Hence, in this role, in particular, I've been deeply obsessed with using data before any money was spent. For data to provide a degree (high!) of confidence that, if we spend the money, it would most likely deliver a positive impact on the business.

This cluster, a part of our awesome win before you spend strategy, is called Smart Start.The first component is a gloriously scaled global creative pre-testing program.

Creative is the thing you see in the ad. The text. The goats. The slow music. The repeated mention of the product (hopefully). The use of a celebrity (or not!). Yada, yada, yada.  It turns out, hold on to your seats, the creative has approximately 60% influence on the ultimate success of your campaign!

Not the audience, not your ad stack, not your targeting strategies, not your obsession with other little bits that are currently sucking up 98% of your attention.

The creative is what matters, and, unfortunately, few people who do modern analytics focus on the creative.

We pre-test pretty much everything in an online labish environment, and predict whether a piece of a TV or Billboard or Radio or YouTube or Facebook creative will be successful. With the support of our progressive CMO, we spend money on creative that passes pre-test.

In a unique feature of our analytics strategy, we practice a trust, but verify approach to lab testing. We routinely put failed in-lab creative in the market and use both passed in-lab and failed in-lab to see how they do in the real world. This helps us understand the quality of signal we get in-lab (it is around 67% for Yes and 89% for No).

Now, our Marketing teams know before they spend money if the campaign’s creative will deliver success.

The other gloriously scaled global practice is our pre-flight check, which we call the Minerva Check – named for the Roman goddess of wisdom. :)

The Minerva Check is a collection of media plan minimums required for success. Reach. Frequency. Duration. GRPs. Passed creative rates. Tactical strategies. Ad unit types. Etc.

We identified these minimums from the massive amount of data we have for our past campaigns. Meta-analysis. Matched market tests. Product/regions/channels. Throw everything in there, and out comes a list of things that every media plan has to meet to be greenlit.

Now, our Marketing teams know before they spend money if the campaign’s creative will deliver success.

It is not perfect (I like perfect). It is not enough (I want more). But, with incredible certainty, we can now say, before any money is spent, that the chance that Media Plan X will deliver success is 3% and the chance that Media Plan X-modified (with Minerva incorporated) will deliver success is 97%.

Intelligently applied data proving its value when it really, really matters – before the company budget is spent.

I’ll give you one guess as to how much our VP of Finance loves this capability. :)

Smart Execution | Analytics Cluster.

For the last year, the thing that I have been obsessed with, along with our small team, is the next cluster: Smart Execution.

Real-time data in the hands of humans is a colossal waste of technical, financial and human resources. It takes too long for humans to process the data through themselves, their team, the bureaucracy, the agencies, and the technology stack to convert real-time data into action.

(From 2006: Is Real-Time Analytics Really Relevant?)

Yet, there are, often literally, tons of signals coming off your ad and analytics stack that you can use to identify if things are going right or wrong and take quicker action – by eliminating humans from the process!

My deep love for this cluster comes from the competitive advantage you can build for your company…

In our in-flight optimization journey thus far, we have worked to identify signals that are believable, and identifying at which point they become believable (ex: statistically significant).

The next step was to create a collection of decision trees. It sounds complex, it is not.
Here’s an example. The benchmark for the beautiful metric AVOC is 15.3%. The decision tree is: If it is about 20% for our campaigns, then sing happy birthday.  If it is between 10 – 20%, then raise a flag. If it is below 10%, then stop.

Then, automate the execution of this decision.

Now repeat this across many, many metrics, for many dimensions, in the three clusters you see above.

Boom!

You have the start of a fabulous in-flight optimization engine.

We’ve now made data influential and useful while we are spending money!

You can imagine that excellence in the orange (Smart Start) and blue (Smart Execution) clusters means there is more green coming out of our green cluster (Smart Lessons) than red (bad news).

Another secret agenda: By being excellent at Smart Start (win before you spend) and Smart Execution (win while you spend), you end up making Smart Lessons (the thing that occupies so much of your present life) utterly boring!  By the time you see the results, you pretty much already know what they are going to say. :)

Smart Future | Analytics Cluster.

I hope you got the sense that I am very, very passionate about each cluster above and that they are my favorite children.

That would be a misleading conclusion.

Yes, it is all fantastic work that is obsessive about making data more useful in more novel ways for our company than is common in other companies.

But my favorite child is cluster four: Smart Future.

It answers the hardest questions a CMO asks:

What is the true incrementality of all my marketing spend?

What is the bottom line (ex., sales) impact of my brand marketing?

How does the portfolio of all our activities – owned, earned, paid media, promotions etc. – work together, and how do the channels complement each other?

How effective are our efforts in the context of all the actions our competitors are taking to impact our company?

I see you nodding your head. You are being asked these questions, and you know the depth of analytical difficulty.

Our innovative approach…

Due to non-public IP, it is a cluster I can’t tell you much about.

Except to say that most companies, when they attempt to answer the aforementioned questions, take the approach of using existing statistical approaches that require explicit programming. We’ve chosen to use machine learning algorithms that learn from the underlying structures inside massive amounts of our datasets without explicit programming. That’s the magic.

As the second box indicates we not only use this approach to look backwards. Rather, delightfully, we also look into the future. We have the ability to model scenarios, budgets, channel allocations to maximize effectiveness and efficiency for our future campaigns.

It is very unique and difficult work.

Analytics on the Edge.

So what does a strongly proactive and truly influential data strategy at the bleeding edge look like?

Machine learning algorithms help to create the optimal budget and channel allocations that flow into Smart Start programs, ensuring core elements of marketing are pre-wired to deliver success. The data then flows through automated decision trees, making decisions in close to real-time maximizing success in the real world. The end result is data stories and scorecards that help our leaders get a unique view into marketing’s impact.

Data not as a side-show. Data as an influential core.

Cool, eh?

We are not perfect, we are not complete, we have miles to go, but I am so proud of the work the team has done and the business impact delivered.

There is so much packed in each box, I could write a 98-page book for each. :)

A reflective assignment for you.

How expansive is your analytics strategy? How much influence does it have on the marketing (or product or customer service or HR or whatever else) that your company undertakes?

Print out the image below.

Sit down with a pen for when you have a calm 20 mins to think.

Write down across each cluster how and where your analytics efforts are being applied today.

If you see that you have something filled in each of the Smart Start, Smart Execution, Smart Lessons, and Smart Future cluster, raise your hand up because I’m high-fiving you at that very moment!

Well done.

If you found that not all the boxes are filled, no biggie. Start to sketch proposals to take to your leadership.

If you see that your efforts are mostly centered in the green Smart Lessons cluster, be happy. That is where most companies are. But, also be hungry because you want more.

I hope that in this note you’ve found enough very specific starting points, informing you about what you need to do in order to get to the bleeding edge or of analytics and, ultimately, implement a strongly proactive and truly influential data strategy.

If you want my recommendation for the best next cluster to focus on, it is Smart Start. Influence the company with data before any money is spent, and untold love will shower on you (along with promotions and extra headcount for your team!).

Bottom line.

It is very hard in a modern corporation, which, still, primarily runs on opinions, to make data influential. (Not the be-all and end-all, influential.)

Yet, we have so much data, we have such vast opportunities as Analysts to have a material impact on the company’s profitability and even the direction of the business. We just need to unlock our imagination.

It is not easy. But, nothing worth having is.

Carpe diem.

As always, it is your turn now.

Please share your feedback on my ideas and lessons from your anatlyics journey, via comments below. Thank you.