If only John Wanamaker knew about experimental design.*
Wanamaker’s old saying about not knowing which half of the advertising budget is wasted may be a cliche, but for many marketers the sentiment holds true. Thoughtful campaigns can fail to gain traction, and soft launches can soar without clear insights on why. Fortunately, today’s marketers have techniques available to them that reveal which elements of a campaign resonate with customers. Among the most promising is experimental design.
What is experimental design?
Experimental design is a research, testing and optimization approach used to organize data and run statistical tests on it to identify cause-and-effect relationships between inputs and outcomes.
In the context of a marketing campaign, experimental design allows brands to separately tag different elements and test variations of each element to see which combination performs the best. More than just identifying the best version of each element, experimental design allows brands to assess which elements in combination have the highest impact.
For example, a brand that is trying to identify the best design for a web page could separately tag different variations of the headline, tagline, image, and the call-to-action button and then test four or five variations of each to see which combination most effectively induces customers to take action. This allows marketers to test many different variations of different page elements quickly and at scale.
How does experimental design differ from A/B testing?
One of the most common methods marketers use to test different messages is the A/B test. As the name implies, A/B tests allow brands to randomly divide an audience into two groups and deliver one version of a message to one group and deliver a different version to the second group, and assess which performs better. A/B testing can be thought of as testing the champion version against the challenger version.
A/B tests are very effective at showing which of a small handful of options performs the best, as Ron Kohavi and Stefan Thomke point out in their 2017 article in Harvard Business Review. For brands that want to test only two or three variations of one element, an A/B test can produce a usable result with a modest sample size. That ease has made A/B testing very popular.
Yet A/B tests are also quite limited in today’s climate of massive scale and real-time interactivity. They cannot, for example, efficiently test multiple versions of multiple page elements against one another or show why a particular message performs better than another. Nor can they show why a failed message produced no or a negative response.
And that’s a problem for modern marketers.
Knowing why some messages work and some fail is critical to improving marketing results over time. It’s the only way marketing will move from being data-driven to insight-driven.
Knowing why some messages work and some fail is critical to improving marketing results over time.
Likewise, brands can’t use A/B tests to efficiently assess multiple elements of the same message. They would have to run a separate A/B test for each variation, which would very quickly multiply to an unmanageable number of tests. Four different versions of five separate elements on a web page would require more than 500,000 separate A/B tests, for example, each of which would need to be seen by a large number of people to amass a statistically valid result.
With experimental design, in contrast, marketers can test a number of variations and then use predictive analysis to identify the best observed and best predicted combination of elements for the brand.
With experimental design, in contrast, marketers can test a number of variations and then use predictive analysis to identify the best observed and best predicted combination of elements for the brand.
Experimental design in action
To illustrate what that looks like in practice, consider an experiment run by the Persado data science team at a recent offsite designed to determine the best design for a paper airplane. The basic elements that need to be tested are the size of the paper (letter sized or legal sized), the weight of the paper (light or heavy), and two different design options (the Bulldog or the Harrier). Three elements and two variations for each produces eight versions of the airplane.
An A/B testing approach would require the plane designers to pit every plane against every other plane and test how well specific variables perform against another specific variable. Experimental design, in contrast, only requires four tests, because it arranges the findings on each element in such a way that it can predict how they perform in different combinations. One interesting additional element of how Persado conducts experimental design is the notion of best observed outcomes and best predicted outcomes.
The best observed option is the one that is actually observed to perform the best. But testing even paper airplanes with a handful of elements and variables can take all day to run through all the test conditions and you may have missed the best predicted outcome based on all the data available. Humans simply can’t observe or predict the best possible outcome based on such large volumes of data as those used by marketing.
The bank’s marketing leaders understood why people engaged and converted on the site and used the insights to inform other marketing creative and campaigns.
A/B testing requires a hypothesis that you want to prove or disprove. What if there is a combination of variables that far outperformed anything the human tester could hypothesize?
The best predicted outcome could be headline A along with body copy B, image C and call-to-action D. The best predicted outcome is one that the machine learning algorithm predicts will perform the best based on experimental design. These options are then put into real world tests to test actual performance.
In the case of paper airplanes and the data science team, the team members didn’t even get around to testing the best predicted version which beat the best observed option (for the curious, it was a Bulldog design built with legal size paper and heavyweight stock).
In this way, experimental design provides more insights to marketers in less time with less work than would be required using A/B testing.
How can experimental design help businesses grow revenue?
It can take a person multiple contacts with a brand before they decide to become a customer, yet every opportunity to communicate and engage with people counts. If a brand can increase the number of people who respond to their marketing messages, they are going to increase conversions. Every interaction counts in today’s marketplace so the ability to know what works to engage quickly and effectively is paramount.
Knowing which messages resonate, which do not and why gets to the heart of competing on customer experience. A banking customer of Persado’s offers a case in point: the bank engaged Persado to improve the impact of a website campaign, but the experiment found no difference in uptake across the various messages compared to the control.
That was a very unusual result and difficult to understand, until the Persado team discovered that customers were responding differently depending on the channel they used to reach the web site. People who clicked through from an email had one engagement profile and people who clicked through from a social media ad had another, and so forth. The differing responses effectively canceled each other out when viewed in aggregate. But when separated by channel as part of experimental design, Persado could identify differential messages that produced better results—and ultimately better conversions. Most importantly, the bank’s marketing leaders understood why people engaged and converted on the site and used the insights to inform other marketing creative and campaigns.
Such is the promise of experimental design.
Mark Young is a Senior Data Analyst at Persado, where he provides statistical support to other Persado teams and ensures proper statistical rigor in Persado’s tests and results.
*John Wanamaker was a pioneer in American retailing, famously known for the phrase “Half the money I spend on advertising is wasted; the trouble is I don’t know which half.” Lesser known is the fact that Wanamaker was one of the first marketers to take out half page and full page ads and hired the world’s first copywriter, nearly doubling revenue as a result. Nearly 140 years later brands such as JPMorgan Chase and Dell Technologies work with Persado to know with mathematical certainty which part of their creative drives the business forward and, most importantly by using AI, why.
Share
