How to Create Holdback Groups in an Experience Targeting Activity in Adobe Target

This article is aimed at experimentation developers

Wait a minute - holdback groups in Adobe Target Experience Targeting…?
We’re talking about personalisation, not A/B testing.
Aren't you confusing things?

When setting up a personalisation activity, the fact that personalised content will outperform the generic experience is kind of a given, right? But wouldn’t it be great to be able to track just how strong that performance boost is?

While other experimentation tools offer a default holdback for personalisation campaigns out of the box, this option isn’t available for Adobe Target Experience Targeting activities. Sure, you can set up an A/B test for a specific audience - but what if you’re targeting multiple audiences? 

Content

Why You Should Have Holdback Groups in Experience Targeting

Let’s say you own a company that offers bookings for cruise vacations on its website. While all of your cruises offer entertainment for every type of traveller, people come to your website with a specific idea of their ideal cruise experience. You want to reflect their persona both in the imagery used on the cruise description page, and in the focus of the amenities offered on board. 

Let’s think about the types of travellers you might need to customise your content for: Couples, families with small children, families with teenagers, singles, seniors, single seniors, groups of friends, groups of female friends, groups of male friends, honeymooners

 ... the list goes on.

So if you wanted to measure the lift of your personalised content for each of these audiences out of the box, we’re looking at 10 A/B tests. For just one personalisation campaign. Phew..!

And we haven’t even talked about comparing the effectiveness of personalisation between these audiences. For Example: How do single seniors respond to personalised content, compared to families with teenagers?

Setting this up as multiple A/B tests doesn’t sound feasible, does it?

Creating Holdback Groups for Multiple Audiences in a Single Adobe Target Experience Targeting Activity

So how do we address this requirement in Adobe Target? Enter profile scripts. 

If you’re new to profile scripts and user profiles, I recommend you to read the Adobe Target documentation here:

https://experienceleague.adobe.com/docs/target/using/audiences/visitor-profiles/profile-parameters.html?lang=en

With the help of a short and simple script, you can divide your visitors into holdback and experience groups - a profile attribute that you can then use to enhance your experience targeting audience conditions.

Here’s an example of a script that does just that: 

Let's see what the script does:

  1. Check if the visitor already has the attribute “campaign123_control’ stored in their user profile. If so, the script will not proceed, as there is already a value to target the visitor with.
  2. Generate a random number from 0 - 99. Think of this range as your 100% of the users that visit your website.
  3. If the generated number is less than or equal to 4, return a value of ‘holdback’ for the user profile attribute ‘campain123_control’ for this visitor.

Otherwise, return a value of ‘experience’ for this attribute.

Since your range of numbers starts with 0, you only need to include numbers up to 4 to get a 5% holdback (0, 1, 2, 3, 4).

So what happens is that 5% of all visitors to your website will be marked as ‘holdback’ for ‘campaign123_control’ and 95% will be marked as ‘experience’. The distribution of this attribute is random (random generation), just like the holdback allocation would be for an A/B activity.

Holdbacks are persistent. Once a visitor has been assigned the ‘holdback’ value, this particular script will not be run for them again (that first “if” condition). If the attribute exists, its value cannot be overwritten.

Wait a minute - are you saying that the user will be held in holdback limbo forever? 😮

Yes and no: the decision is yours.

There’s a reason I named the script ‘campaign123_control’ and not ‘experimentation_control’ or something similarly universal.

If you were to use only this profile attribute to create holdback audiences in all of your activities, users assigned to the holdback group would never see personalised content - ever. From any campaign. 

Global Holdback - What’s the Point of a “hold out” Group?!

Excluding a fixed fraction of users from any experimentation at all can be a perfectly desirable effect. There are certainly arguments in favour of establishing such a global holdback, or rather hold-out group. 

At one level you want to measure the effects of your individual tests, but you may also want to know the value of your entire experimentation programme, beyond a relative improvement.

To put it bluntly: What is your testing programme, as a whole, actually worth? 🧐

Once a Winner - Always a Winner?

In addition, a global hold-out group can give you an insight into the long-term success of individual optimisation efforts. For example, improvements you made some time ago that were based on winning experiences from previous experimentation campaigns.

Suppose variation B of a test was a winner way back in spring, and you rolled it out to 100% of your visitors. But is it still a winning experience half a year later? Could that lift have changed (due to seasonal effects, due to other improvements you made later)? 

Interesting stuff there for sure, and if you have the option, definitely worth exploring.

 😎Option being the key word here: the “luxury” of keeping a global holdback (or hold-out) group really only makes sense if you have the traffic to support it. Which is… a lot.

Activity Based Holdback Groups

So stepping back to a more modest campaign level, you’ll want to keep your holdback as heterogeneous as possible. Just because a user is excluded from this particular test doesn’t mean they shouldn’t be exposed to other personalisation or testing experiences. The pool of visitors you’re separating should have the same level of optimisation. Let them start with roughly the same level of bias, you might say.

How do you achieve this? Simply put, you create a profile script for each personalisation campaign for which you want to create a holdback group.

The script above provides you with a holdback group for campaign 123

For campaign 456, you’ll need to create a new script to set this new attribute:  

				
					if (!user.get('campaign456_control')) {
var ran_number = Math.floor(Math.random() * 99);
wenn (ran_number <= 4) {
            return 'holdback';
 } else {
return 'Erfahrung';
}
}
				
			

If you want a split other than 5/95, simply change the second if condition to your desired value. For a split of 10/90 you would write: 

if (ran_number <= 9) …

You could set up multiple scripts for multiple activities. A visitor could be attributed different values for all of them. For example: One user could end up in the holdback for campaign 123, but see the variation for campaigns 456 and 789. Another user might see the personalised experience for campaign 456, but fall into the holdback group for campaigns 123 and 789.

Either way, once a user has been assigned one of the two values for any of the campaign attributes, that value will stick. A user who falls into the holdback group for campaign 123 will continue to do so in every following session.

How to Use the campaign control Attribute in an Audience

Ok, so your user has been placed in a random group by a script, but how does this actually work when setting up an experience targeting activity with multiple audiences? First, create 2 separate audiences - one for each of the attribute values:  

You can then combine these two audiences with any of the base personas you are targeting in your activity. For example, your family travellers. 

Add it as a condition to any of the audiences you have set up in your experience targeting activity. 

By using either of the two values, you split a group of users with a shared persona (your family travellers) into two random groups to be used as separate audiences to be served either default or personalised content. 

By comparing the results for these two audiences, you can see just how effective the personalised content is for this particular audience.

Ok… this seems like I could have just set up an A/B test. 🤔

If family travellers were the only group you were targeting, I’d agree. But remember, you wanted to personalise the cruise detail page for couples, seniors, honeymooners, female group travellers, …🤯

With this solution, you can set up unique experiences for all of them - and leave a default version for their respective holdback group - in a single action! Just follow the steps described above for each of them. You’ll surely appreciate the added layer of information it brings up to your result report. 👌

❤️I encourage you to give this custom solution a try in your next experience targeting activity. Happy insights!

How to Get Started with Adobe Target and AI-Powered UX Testing and Personalisation

Denise Paché

Denise Paché is a Senior Experimentation Developer at Up Reply, specialising in data-driven personalisation and experimentation with Adobe Target. She has supported clients in translating their CRO strategies and audience requirements into meaningful campaigns. For the blog, she will share insights on working with Adobe Target and hands-on solutions for personalisation use cases.

Let’s Take Your E-commerce to the Next Level

Unlock new opportunities and redefine the customer experience through personalised, data-driven strategies with Up Reply.