On Product Marketing Life this week we got together with Global Product Marketing Manager at Uber Eats, Nadia Niky, to discuss all things experiments. Nadia explains why she thinks running experiments is such a vital element of the PMM role, how she prepares for hers at Uber Eats, measures and analyzes their success, and the times it’s gone wrong, plus, key learnings, top tips for others, and more.

Full transcript:

Emma Bilardi - PMA 0:00

Hi, everyone, and welcome to the Product Marketing Life podcast brought to you by Product Marketing Alliance. My name's Emma Bilardi and I'm a content marketer here at PMA. Today we're joined by Nadia Niky, Global Product Marketing Manager at Uber Eats. Thanks for joining us today, Nadia.

Nadia Niky 0:53

Thank you so much, Emma. It's so great to be here.

Emma Bilardi - PMA 0:55

Excellent. So I'm going to dive right in with the first question. Can you tell us a little bit about yourself and your role at Uber Eats?

Nadia Niky 1:02

Absolutely. Yeah. So I'm a product marketing manager on the marketplace and consumer experience team on Uber Eats. Being in a three-sided marketplace, with eaters, delivery partners, and restaurants can get quite complex and our team, the marketplace team, we run a lot of new product experiments.

Basically what we do is test new products in different regions before rolling the feature out globally. So some of the products that I've gotten to work on include Uber Eats promotions, our $0 delivery fee products, and priority delivery. So if you've ever received a promotion from us as an Uber Eats customer, that was probably our team.

Emma Bilardi - PMA 1:47

Excellent. Today we're going to be talking about marketing experiments. Can you tell us why they're important?

Nadia Niky 1:56

Totally, yes. So I believe marketing experiments are essential, especially in our role as PMMs. I find a lot of marketing can be intuition but what experiments really allow is for us as marketers to get a signal before launching a campaign out globally. So it saves the business, time, and money but it also provides us PMMs lessons and learnings that we can apply to the future.

Ultimately, to think about it, it helps us gain confidence and get relative certainty on the success of future campaigns. Some examples of experiments are I've worked with production teams on storyboarding ideas before we go into production. This is an example of really testing out a concept in a focus group, before going into production and investing so many assets and money into producing the video. Other tests include acquisition tests, so testing which channels work out best, this can also save the business money.

Emma Bilardi - PMA 3:38

So how do you get to the point of knowing what experiments to run and which elements to test? What goes into the preparation for that?

Nadia Niky 3:48

Totally. So experiments can definitely be complex. I personally categorize them into product forward and marketing forward experiments. For the first bucket for product, these experiments typically require a lot of engineering resources, and other high cost or high build time requirements. These are often super collaborative at Uber, we'll conduct what we call jam sessions.

We'll get a cross-functional team together, and brainstorm and ideate then prioritize different features that we want to test and launch. That's the first bucket but I'll focus on the second bucket, which is the marketing experiments. This is really where, as PMM, we bring insights to the table and speak on behalf of our consumers and represent them. For marketing experiments, as a PMM, I usually lead the strategy for this.

I loop in our creative team, brand, designers and get them all in a room or a virtual room and the tricky part is to design. I put together a matrix generally of what is the highest impact versus the complexity of the experiment. It's always a trade-off, right? Sometimes you can run really complex experiments that have high impact but that may take a lot of time versus an easy experiment, but it has a low impact. What we do is we discuss what are some high impact, but relatively low lift experiments that we can run. It depends on ultimately your goal and what you're trying to learn.

I'll take an example of we've run a lot of CRM campaign experiments, and the standard subject line, image, click-through rate, or I should say the button, the call to action, are typically some of the first elements in CRM to be tested just because they could have a high impact. So it depends on the product obviously, I'm speaking on behalf of our Uber Eats products, but you need to change up the test to reflect what you're trying to learn and what is the path that your customers are typically going through?

Emma Bilardi - PMA 6:06

So how do you choose which metrics to measure?

Nadia Niky 6:10

Yeah, so in this case, data science and marketing analytics are our best friends here. Typically how this works is I'll put together a one to two-page experiment brief after the jam or brainstorm that I've conducted. First, we'll try to think through classic growth marketing, build a funnel of what are the metrics that are relevant to each stage of the funnel, and if conversion is your goal, for example, so I'll prepare this one-pager, and then I'll definitely make sure to get data science or marketing analytic's perspective.

Because I think a key trick here is... sometimes it surprises us completely that it may be hard to pull a certain metric or measure something. Like brand campaigns, street billboards are sometimes more difficult to measure than performance or paid marketing. Definitely targeting the metrics to the channels and the learnings is really, really key.

Emma Bilardi - PMA 7:19

Okay, so what are the indicators of success once you've analyzed the results?

Nadia Niky 7:24

I think it's that we've ultimately learned something. Experiments don't necessarily have to be 'success'. So in my experience, the ones where we failed or we've learned something completely surprising can sometimes be the ones that benefit us in the long term. As long as experiments are set up methodically, and we've taken the time to really plan out the design, then really, that's what you're here for. The learnings that come out of it are gold, and there's a lot of money and time that go into these experiments so sharing out the results is really important. I think that's really what I consider a success after we've gone through analysis.

Emma Bilardi - PMA 8:17

Absolutely. We've talked a little bit about the right way to run an experiment or the successful way. So is there a wrong way to run the experiments?

Nadia Niky 8:29

Good question. I would say having too many variables, so not really being clear on what you want to learn, I think that's what the root cause is. It's really easy to get carried away and want to add more learning, more questions, you can go down many different paths because the setup of the experiment is ultimately up to you. So, it could go many different directions, and having too many variables can, to use data science terms, you could lose signal, or it won't be statistically significant if you have way too many cohorts.

I think that's something to be cautious of, so keeping them simple. And then always including a control group. I've definitely run experiments where we lose sight of that, and a control group, for those that may not know is a baseline ultimately. It's comparing back to was this experiment or these variables effective or were they not effective? And then keeping that control group. Any scientist that's listening will obviously be nodding and say that's crucial.

Emma Bilardi - PMA 9:48

So from a personal perspective, have you ever conducted an experiment that spectacularly failed? And if so, what did you learn from it?

Nadia Niky 9:59

Yes, I would say my example for this one would probably be not a past experiment but just a general campaign. Being on the Uber Eats promotions team, we do run a lot of promotions. I think what was happening at the time is we were launching a bit too quickly so our Q&A, the process that checks our comms are looking great, ticking all the boxes, that process really couldn't keep up with the speed at which we were launching these experiments.

What ultimately happened was, we sent out an email for an Uber Eats promotion to our existing customers that had no percentage off. We sent out this comms and we were like, "Thank you so much for being a customer, enjoy percentage off", but it was negative, it was a cluster of different keys, and we were like, "Oh, gosh, this looks like a big fail", it was really bad. And because we were launching these experiments across the US it went out to hundreds of 1000s of users.

So we issued a second follow up email saying, "Sometimes we mess up too". I think looking back on that, what would have helped, and what we need to focus on is sometimes just slowing down for that little moment right before launching an experiment, just to make sure everything's tightened up and you've checked all the comms and the messaging is right, is really important. I think that's where we could have saved the campaign. But ultimately, we learned and we fixed that bug after the fact.

Emma Bilardi - PMA 12:08

And you held your hands up and admitted you were wrong, which always helps. I like that, thank you for being a valued customer here's nothing. We like to end the show on a few words of wisdom. So what's the best piece of career advice you've ever been given?

Nadia Niky 12:29

I love that you ask this question, Emma because I think different people listening to the podcast, bring different pieces of advice. This is obviously curated by different people they've worked with, I think it's a really valuable piece in the podcast. For me, I would say, 'be 1% better every day' has been the best piece of advice that I've been given. I personally apply this in my career, but also in my personal life.

What it means to me is, we're humans, we can make mistakes, we learn from our careers and our journeys. But what it's about is just trying our best to make incremental improvements, small changes every day, it could be something that scares you, it could be something that you actively improve. By doing that, I find it has ultimately, it compounds over time, and it has the opportunity to drastically change our paths, both our career paths, but also personal lives.

Emma Bilardi - PMA 13:41

I love that. Thank you so much. Thanks for joining us, Nadia.

Nadia Niky 13:45

Thank you so much. This was a lot of fun.

Emma Bilardi - PMA 13:48

Yeah, I had a lovely time. It was great to meet you. Take care.

Nadia Niky 13:51

Thank you so much. Bye.