If people-based A/B testing is that much better than past methods of ad testing, why isn’t everyone doing it already? Anything new requires change. If you’re prepared to invest in new data-driven resources, you may be ready to make that change and start seeing some big results.
Here’s what you need to put in place for people-based A/B testing and measurement:
1) Identity Graph
To conduct people-based A/B testing, you need to have a way to identify people across devices so that you can properly divide people—not devices—into test and control groups. A deterministic identity graph such as LiveRamp is the most accurate way to link multiple devices to the same person ID in a privacy-conscious manner. This foundational layer will inform the rest of the setup.
2) Ad Server
Ad servers decide which ad to show a user. Historically, ad servers used only cookies or device IDs to decide which ad to serve, but they failed to take into account that people may have multiple associated cookies or devices. Adding an identity layer makes it possible to factor cross-device usage when split-testing ads. People-based ad servers such as Thunder utilize identity graphs to conduct A/B and multivariate tests by randomly assigning users they see into different groups.
3) Minimum Inputs
Now that you have the tools, you need to decide the inputs for the test. How are you measuring success—is it conversions, clicks, or something else? Once you know what you want to test for, you need to find out how you’re currently performing based on that metric (example: you currently achieve a 1% conversion rate with your ads). This input is your baseline effect.
Next, for your organization, what is the minimum desired effect (MDE) you wish to detect to determine if one ad strategy is more worthwhile than another? Some organizations may consider a 5% lift in performance to be immense—this is particularly true for very mature sectors such as CPG and insurance, where brand choice is very sticky. Other organizations may want to see a 20% lift in performance before choosing the best strategy and further investing media dollars on that winning strategy.
Sample size calculators, which are readily available online, can help determine the minimum sample size you need to achieve statistical significance and be confident the results will hold true through repeated experiments. Once you know the needed sample size, you can plan how much media you’ll need to purchase accordingly.
4) Analytics Team
One final resource is key to implementing people-based A/B testing—people. Whether it’s in-house or a managed service by the agency or tool provider, you’ll need a team to validate the test setup, read the returned data, and decide on a future ad strategy based on the results. For some tools, this decision can be automated and the ad campaign can auto-optimize.
However, for all advertisers, to build sustained learnings and future insights, a dedicated person or team to look at the qualitative aspects of the ad campaign to inform future strategy is a must. If you simply test ads without a hypothesis and without studying what may have led one to win, you’ll be spending media dollars without improving future ad strategy. Advanced people-based testers have analytics managers who can synthesize learnings across all tests for creative and media teams to use in the next set of ads.