A/B testing: your ticket to defining high intent customers for funnel acceleration

Stephanie Menz, Senior Manager of Customer Marketing at Toast, made this presentation at the Customer Marketing Summit in September 2021. Watch presentations OnDemand now.

My name's Stephanie Menz and I'm on the customer marketing team at Toast.

In this article, I'll be talking about using A/B testing to help define your target audience, focusing specifically on:

• Customer marketing upsell
• What the upsell funnel looks like
• Creative testing
• A/B testing 101
• Testing a call to action (CTA)
• Messaging testing
• Testing to refine your high-intent audience
• An example A/B testing segmentation campaign
• Testing personas

Toast

Toast provides a powerful online platform that's built for restaurants by restaurateurs. It's better for restaurant owners, their guests, and their staff.

We help restaurants work more efficiently, manage a productive team, grow their business online, attract and retain guests, and more.

Customer marketing upsell

Different upsells serve different purposes and different ways to sell these solutions into your customer base:

For example, I've laid out:

• New product launch
• Product-led growth as a channel, which makes finding add on products discoverable within your business's product
• Customer success, which is a channel that can be used for upsells that you may need to support as a customer marketer

And so much more...

Here, I'm going to dive into a use case around a new product launch that applies to all customers and discuss how I use A/B testing to help define that target audience.

This is going to be focused on the process of getting from "Okay, we're launching a new product" to "How can we refine our target audience to increase funnel acceleration and ultimately conversion?"

Our main goal is to generate higher quality leads from our customer base that’ll have higher conversion throughout our upsell funnel. But to get started, we first need to define that upsell funnel.

What does an upsell funnel look like?

TAM

At the top of your funnel, you have your total addressable market, which is your customer base that is fit for the product.

Hand-raisers

You have your hand-raisers which could be customers that show they are interested in the product.

Demo held

Maybe the next stage is a demo. Maybe customers need a demo with their customer success manager or a sales rep before purchasing.

Win

Maybe they don't and they go straight from hand-raisers to the opportunity to win. It may look different depending on your organization.

Creative testing

Next, I want to dive into creative testing. Now we have defined our funnel, it's time to get started, but where?

Begin testing the customer base

When I went through this exercise, I wanted to start with the basics and make sure I had the positioning nailed down for this product. Then I wanted to move on to the best audience segmentation that I could evolve to.

My first step was to start testing messaging into our customer base.

Using the messaging framework from product marketing, you can come up with a base email you can send to your customer base and we can begin with testing the subject line. I like to think of refining the target audience exercise as a funnel as well.

So you're going to want to start broad and refine to the point where you end up with customer personas for this solution. Getting to the persona set may take additional cross-functional collaboration with teams in your company.

For example, you may need to collaborate with your business intelligence team to analyze customers attached to the product to help understand the different attributes as to why and I'll talk more about this later on.

Firstly, we need to start testing messaging into our customer base and A/B testing the subject line. Emails are a great channel to test the messaging as it's scalable, and the results are usually easy to digest.

I'm starting with the subject line because I want customers to open the email and I want to know how I can get them to open it, so ultimately, they can take that action of buying the product and becoming a hand raiser.

A/B testing 101

When creating your subject line for A/B tests, you want to make sure you have your subject line versus a control.

For example, if I want to test personalization in this subject line and this will be our test one, subject line A will have the customer’s name, and subject line B will not have the customer's name but they're still the same subject line.

This way, we're able to get an easy read on which subject line is a winner. Maybe, for example, if we're hosting a webinar around the product, we can start testing the example of variation A being "Stephanie's Bakery, are you free next week?", versus variation B, which is "Hi there, are you free next week?".

The winning subject line in this example is variation A and then that will become the control, the winner for test two. So for test two, we can maybe do a date and time test with the personalization token.

So we have variation A being "Stephanie's Bakery, are you free on 10/5?" versus variation B, which is "Stephanie's Bakery, are you free next week?"

From here, we can see which one's the winner and keep testing with our customer base.

Examples of subject lines to test

Here are some of my favorite subject line tests that I like to do. This is by no means an exhaustive list.

There are so many different tests you can do with subject lines but I just wanted to highlight a couple:

Personalization tokens

One is personalized, so using terms like 'my', or 'your'. Some examples of this could be 'schedule my demo', or 'chat with my customer success manager'.

Value proposition

Thinking through value propositions and testing different value propositions within the subject line, making sure that the subject line is short and actionable.

Seasonal

Testing into seasonality, maybe your solution helps with some customer pain points around back to school or the holidays, you could test into messaging there versus generic messaging to see if one influences open rates.

Date/time

Also, think through the date and time as well.

Testing a call to action (CTA)

After doing some testing and identifying which subject line drives customers to open their email, I think it's a good opportunity to start testing which calls to action drives customers to take the behavior we want them to take, i.e. becoming a hand raiser and purchasing the solution.

Examples of CTA tests

Let's think through some of the different CTA tests you can try. Here are a couple of CTA tests you may want to try within your organization - a couple of favorites of mine:

Personalized vs. control

One is personalized so using terms like 'my', 'your', some examples are 'schedule my demo', 'chat with my customer success manager'.

Urgency vs. control

Another is to create a sense of urgency from the call to actions such as 'apply now' or 'get started now'.

Promotion vs. control

Special promotions are another CTA you can test into to see if that helps drive customer behavior.

Placement of CTA in an email

Outside of the copy of the CTA, you can also test different creative treatments of the CTA within the email by having it images or as a button.

Every company is different and everyone's audience has different needs with different messaging and positioning that resonates best with them. Therefore, identifying the best CTA is great because you can then test it across different channels that you use within your company.

Messaging testing

We now have a good sense of what subject line will drive someone to open our emails and which call to action will drive someone to click a link in our email, so now's a good opportunity to test different messaging in our email.

At this point, we're still at the top of the funnel with a broad target audience, our messaging is still generic and we have the same goal of wanting the customer to open the email and take an action.

Examples of messaging to test

Here are some examples of different messaging you can test:

Bulleted list with product benefits

One is a bulleted list of product benefits. It can be as simple as explaining the product in paragraph format.

Bullet list of the acquisition process

You could do a bullet list of the acquisition process to set customer expectations.

Overview video of product experience

An overview video of the product experience can be used to highlight product benefits and you can A/B test that against the bulleted list of the product benefits.

Seasonal examples

Maybe there's a seasonal example as well. For example, if the holidays are coming up, include a gift card module.

Testing to refine your high-intent audience

Now that we’ve established what we want our messaging and positioning to look like, we can move into audience testing and start digging into the data to refine our target audience.

How to create audience A/B test

As I went through this exercise at Toast, I thought through the different attributes that I could test into. One of the things I love about Toast is how there's data readily available for marketers to use.

To begin, I went through the current attached data set for our product to understand which segments were converting in which regions.

I was then able to collaborate with our business intelligence team and they did an exercise that went deep into our customer data set to help me understand if additional products, time of year, and more influenced customer attach for this particular product.

Together, we were able to come up with hypotheses around target audiences that we could test them on.

Dig into the data

I wanted to lay out some things to think about as you go into your audience testing.

First, dig into the data you have readily available to you:

• Are you able to see if segments are converting at a higher rate than others?
• Maybe you're a B2B company and you sell across verticals so maybe you can look into the healthcare, IT, or whatnot.
• Are there different regions?
• How about time as a customer?
• Time in business?
• Are there customers attached that have a positive or negative customer NPS?

Define hypothesis

From looking at the data, you can then define your hypothesis.

For example, if you decide to focus on New York City doing a regional test, your hypothesis could be "If we send your customers content related to a region such as case studies, blogs, maybe something related to a big event happening in New York City that your product is the solution for, we can expect a higher-touch from this cohort versus a control group".

You can then send New York-related content to your NYC cohort, leaving a subset out as the control group.

Create the content with the tools from your toolbox

As you go through this exercise think of all the different tools in your toolbox you can use:

• Are there relevant case studies?
• How about other blog posts?
• Is there any messaging related to the segment that you can test in the email

The list goes on and on.

Audience tests

From the data analysis to looking at all these different attributes and segments, we were able to identify an opportunity to uncover a high intent segment for our testing needs.

The first was restaurant types as well as different regions that we could test them on. If you're a B2B company, maybe your restaurant types look different and that's a vertical such as healthcare or IT that you want to test them on.

Now that we've decided which customer segments we want to hone in on and test into, let's begin the testing process. What does that look like?

Example A/B testing segment-specific campaign

Here's an example of an A/B testing schedule for a segment-specific campaign. We're trying to establish the best messaging, creative, and channel mix for the segment that will ultimately drive customers from the segment to convert.

While we're testing, we want to hold out a randomized control group from this segment to measure the lift of the segment receiving marketing campaigns versus not, to understand conversion.

For example:

• Maybe we have cohort A which is the segment we're testing into and they're getting segment-specific messaging.
• Maybe then we have cohort B, which is the segment we're testing into that will receive general messaging.
• Then maybe we have cohort C which is just a randomized group of customers that we're also going to test into with the generic messaging.

We want to measure what the attached looks like across all those cohorts to understand if our hypothesis of if we send segment-specific content, then we will see a higher conversion rate is correct.

In this example, let's say this is our quarterly testing schedule. For month one, we want to test messaging, positioning, and creativity via email. After we have our messaging/positioning nailed down, we then can start testing maybe into different channels.

In month two, maybe we test an in-product message channel and maybe we're hosting a webinar around the new solution.

During month three, maybe we want to start testing a sales motion with the customer success team, and we're going to partner with them and enable them to help with this different sales motion.

For email messaging, we have some different A/B tests laid out here that we want to try and test, and then move on to our different channel tests once we know which messaging and positioning resonates with this customer base.

Refining target audience based on data

If we go back to our audience test funnel you can see that our audience test is refined, our messaging is refined, and we're continuing to move forward with our hypotheses of testing into these segments.

Maybe we'll start expanding channels and doing these segmentation tests in these channels, as an example I gave previously.

Refining audience tests

As we further refine our audience tests, here are some items we can think of testing.

Segmentation tests

We can think about different creative treatments, including segment-related photos, and segment-related messaging, including use cases in the emails, or whichever channel we're using. Additionally, we can also support content related to other segment-specific needs, such as case studies and blog posts.

Regional tests

Maybe there's an event that ties in with the solution that helps customers with that. Test it out, and see if you can use this in your comms. This could take the form of a case study from that region and other supporting content related to that region's needs.

Intersections

Depending on how large your audience is within these two segments, maybe there's an intersection between the test that you're doing and you can get even more specific and continue to create hypotheses between these high-intent, high-converting audiences.

As you continue to tailor your messaging, your cycle then starts again with creative testing as well.

Testing personas

Once you hit that persona stage, you just keep testing which subject line will help them convert and which messaging and positioning works best for this audience.

The cycle just continues and continues, which I think is one of the exciting things about being a marketer; there's so much you can test to gain a refined insight into your customer base.

Key takeaways

Following this article, I hope you’ve established how to create a simple A/B test, how to create segment-specific campaigns, apply different email tests of the messaging and positioning, and methods for testing in different channels.

I also hope you now understand how data can help you refine your target audience. Consider which segments, reasons, and attributes are most likely to resonate with your customers, as this'll help you create hypotheses. These insights and data will inform your upsell targeting strategy.

Additionally, think about how you can apply different content within segment-related tests to help increase engagement amongst your customer base.

Finally, I think one of the biggest things I learned from setting up a new funnel is not to over-engineer the marketing. A/B tests are a simple and great tool to help refine your messaging and positioning and a great tool to help define your target audience.

I hope you found this framework helpful.

Thank you.