I. Don’t. Know. Those may be the scariest three words in marketing. No one wants to admit to their boss that they aren’t sure whether a campaign will perform with a particular customer segment on a given channel. Yet even the most experienced marketers can’t know for sure if a campaign will perform. Customer needs simply change too quickly. A marketer’s judgment may also be clouded by their own personal experiences, preferences, and biases. That is why A/B testing has been embraced by digital marketers. Yet we can do even better. The advent of AI and data analytics allows marketers to upgrade their testing approach to use experimental design in marketing. Let’s explore why you should and how to do it.
What is experimental design?
Experimental design is a research, testing, and optimization approach used to organize data and run statistical experiments with it. With experimental design, you can identify cause-and-effect relationships. In marketing, that means you can see how your campaigns (cause) produce customer engagement (effect).
How can we use experimental design in marketing?
Using experimental design in marketing lets brands identify which elements of their marketing messages connect for customers and which ones fall flat. It does that with an approach that separately tags each element of a campaign to isolate data related to it. For example, you can tag the different words or phrases that make up an email subject line or the copy on a web banner ad.
After tagging each element separately, the brand can develop a hypothesis about what appeals to their customers. Then run an experiment in which they present different versions of the same campaign to gauge real-world reactions. The results allow marketers to see how customers respond to each version overall and to each of its separate components. In practice, that means you only need to run one experiment to get a slew of insights into what makes your customers take action.
Using experimental design in this way lets a brand efficiently test different versions of a tagline, image, description, and call-to-action button—all at once. Digital marketers can test multiple ideas and then adjust campaigns so that the version they put into market has the highest potential for customer engagement. Only by using experimental design in marketing can brands say with absolute certainty, “I know this will work.”
What are the limitations of experimental design?
Experimental design in marketing brings deep insights with extreme efficiency. Yet that experiment still needs a large enough sample size to reach significance. The exact size depends on the number of elements and variants you are including in the experiment. If there are a lot of them and your organization has limited digital traffic reaching significance can take…awhile. Some marketers would rather just go with their instincts rather than wait for statistically valid results.
Experimental design also requires specific technology designed for the purpose.
How does experimental design differ from A/B testing?
As the name implies, A/B tests allow brands to test an A version and a B version of a message. Some marketers refer to an A/B test as a champion versus challenger experiment. These typically require marketers to randomly assign members of their audience into two groups. Each sees one or the other message. The engagement results tell the marketers which version performed better than the other in its entirety.
Note, marketers can run these kinds of tests with more than two variants. They just divide the audience randomly by the number of variants you want to compare.
For brands that only want to test two, three, or four variations of a single element, an A/B test can produce a usable result with a modest sample size. That ease has made A/B testing popular.
What are the limitations of A/B testing?
A/B tests have limitations in today’s climate of massive scale and real-time interactivity. Chief among them is that A/B tests cannot efficiently test multiple versions of multiple page elements against one another. A marketer would have to run a separate A/B test for each element and each version they want to assess. That would quickly produce an unmanageable number of tests. Each test would need a large enough audience to produce a statistically valid result.
With a combination of experimental design and data analytics, in contrast, a marketer could gather data on multiple versions and multiple elements with just one experiment.
There’s another limitation to A/B testing that may be even more important. It’s that A/B tests can’t tell you why. Because they only show the results of one full message against another full message, they don’t reveal the impact that subtle changes can have.
That’s a problem for modern marketers. Knowing why some messages work and some fail is critical to improving marketing results over time. This is especially true when trying to optimize results with segment marketing.
Despite those limitations, A/B testing is still a valuable and valid form of testing. This is especially true for marketing teams that want test results in just a few hours. Or for those with too little data to employ experimental design.
Experimental design in action
To illustrate experimental design in marketing, consider how Persado does it.
When a Persado customer plans a marketing campaign and wants to run an experiment, the brand’s human creators craft the message and then give it to us. We run it through the Persado Motivation AI Platform, which analyzes what the message is trying to motivate people to do.
Persado is an enterprise Generative AI solution built on a large language model and knowledge base containing more than ten years of high-performing campaign language from Fortune 500 firms.
Once the Persado Motivation AI has analyzed the purpose of our customer’s campaign, it then generates alternative options. Many or most will be predicted to out-perform our customer’s human-generated original. By “out-perform” we mean the message will result in more clicks, more click-throughs, more conversions, fewer abandoned carts, etc.
Sometimes, our clients simply choose one of those predictive alternatives and use it. But if they want to be 100% sure they have the best performer, they run a language experiment using experimental design. The point is to measure how real consumers respond to the different messages. An experiment can include as few as four and as many as 16 versions of the message.
The data that come out of those experiments show which messages perform the best with which consumers—and why. The Persado AI can see how each version performed as a whole. It also shows the impact of each element of the message.
For example, we can see how much impact the subject line or CTA had on the overall message lift. Think of those elements as sources of motivation.
Experimental design in marketing allows marketers to compare multiple messages and elements of messages at the same time.
Using an A/B testing approach would require Persado to create a different campaign to test every message element against every other message element. It would require hundreds of tests for one campaign. Experimental design, in contrast, only requires one test. The findings on each element are arranged in a way that allows the AI to quickly predict how they perform in different combinations, so you can get the best campaign in front of your customers quickly.
How luxury retail company Tapestry leverages experimental design with Persado
Luxury fashion house Tapestry works with Persado to run digital experiments for its brands. These include Coach, Kate Spade, and Stuart Weitzman. E-commerce leaders at Tapestry sat down with Persado to talk about how they leverage experimental design, Generative AI and brand language to optimize customer engagement.
Listen to their conversation on-demand here.
Best observed vs. best predicted outcomes in experimental design
One interesting additional element of how Persado conducts experimental design is through “best observed” and “best predicted” outcomes. The best observed option is the one that Persado put into the market. But even with experimental design, testing all the elements and variables to get an observed outcome could take time.
The best predicted outcome, in contrast, could be headline A along with body copy B, image C and call-to-action D. That one may not have appeared in the experiment. The machine learning algorithm can predict it will perform the best based on the experimental results. If the brand agrees, Persado can then create the best predicted version and broadcast it to the customer’s audience.
In this way, experimental design provides more insights to marketers in less time with less work than would be required using A/B testing.
How can experimental design help businesses grow revenue?
Potential customers often interact with a brand multiple times before they decide to make a purchase. That means that every opportunity to communicate and engage with people counts. If a brand can increase the number of people who respond to their messages, they increase conversions.
Knowing which messages resonate, which do not, and why, gets to the heart of competing on customer experience.
A Persado banking customer offers a case in point. The bank engaged Persado to improve the impact of a website campaign. Yet the experiment found no difference in click-through rates across the various messages compared to the control.
That was an unusual result and difficult to understand. That is, until the Persado team discovered the impact that the customer journey could have. People who clicked to the website from an email engaged at one rate. People who clicked through from social media engaged at another.
The differing responses canceled each other out when viewed in aggregate. But, when Persado designed an experiment to test different messages for different audiences on different channels, the performance improved, leading to better conversions. Most importantly, the bank’s marketing leaders understood why people engaged and converted on the site and used those insights to inform other campaigns.
Listen to leaders from Chase and Persado talk about these findings.
Such is the promise and the result of experimental design.
Find out more about how Persado Motivation AI merges our capabilities with experimental design and Generative AI to help brands deliver marketing impact. Reach out about our risk-free trial options today.
Share
