I want to start this article today by inviting anyone reading to partake in a little bit of quiet. Before we get stuck in, take yourself back to the first time you ever heard the word experiment, or scientific method, or maybe the first time you were in a lab.

Okay, have you got it? Keep yourself there throughout this article.

For me, I remember being about 11 years old in fifth grade, Mrs Kane was my science teacher and I thought she knew everything. She had these fancy poster boards and big whiteboards and words like hypothesis and independent variable and dependent variable and observations and data and drawing conclusions.

As I reflected on that I thought, this is actually what we do as product marketers, but we might not be thinking about that sort of framework or through that lens. So I invite you today to come on a journey with me and think differently about all of the things we do, whether you're at that seed-stage company, or you're a billion-dollar entity with 6,000 employees and 27 global offices and publicly traded.

The art and the science of learning something new

I have two boys, five and a half and four years old, so while I was preparing this piece I asked them “What does experiment mean to you?” And my eldest son raised his hand and said, “Mama. I know. It's when you start to think about something in a way that you've never thought about it before, but then you take the potions, and you mix the potions together and see what happens on the other end”.

I said, “Oh, this is great. I'm gonna write this in my notes, I gotta get it down”. Next, I turned to my four-year-old because of course, he doesn't want to be outdone and he's raising his hand, “Mama. I know what it is. It's like when I have this idea in my head of a picture that I want to draw. And then I put it down on paper with my crayons and my markers and maybe throw some Playdoh in there too. And then it's like, something brand new that I've never seen before”.

And so from the eyes of a five and a half and a four-year-old, experimentation is the blend of the art and the science of learning something new that you didn't know before. So just think about experimentation as the blend of that art and the science to learn something new that you didn't know before.

Where do we start?

I always say empathy is currency. I am a recovering English major and I always go back to some of those examples, and one of my favourite books and now a fantastic play, To Kill a Mockingbird. I think about how do we embrace and embody the insatiable curiosity of Scout with the tenacity and the fortitude of Atticus? And we start first getting to that empathy, of really just getting out there, getting curious and talking to people.


Now I've worked for everything from startups to Disney, and obviously you're saying, especially if you're at seed stage, or you've got fewer than 50 people in the room, "I don't have money to go spend $20,000 or hire some fancy agency, or vendor, to go out and do these longitudinal qualitative and quantitative studies. What am I going to do?"

The birth of a hypothesis

Honestly, you all by nature of reading this article, are most likely extroverted and have insane networks. So if you start thinking about it from the beginning, you have this problem, or you have this thing that you're trying to solve. Chances are you more than likely know 10 to 15 people across your entire life that might have a similar situation going on, and it's much easier to pick up the phone and ask them for 15 or 30 minutes of their time because you already have that relationship with them, than trying to go out and hire this firm and spend all this money and this time trying to recruit.

I'm not saying that's a bad situation, and if any of you reading are from agencies that do that, God bless you, it's good work. But at the same time, use what you have and be a little bit scrappy. So go talk to those 10 to 15 people. This is the genesis of your hypothesis, bringing us back to that scientific method. I want to observe and listen to the market. In my experience, once you get through those 10-15 people, you hit the point of diminishing returns, you start to hear the same kind of trends over and over again of, “I had this problem. I tried to use Excel, it didn't work”.

And then you're saying, “Okay, I've got some really good ideas. I've kind of climbed into their skin and walked around a little bit”. But now you really need to get to scale. You need to start doing those surveys. You need to substantiate some of the initial learnings that you got out of those first points of curiosity.

Getting curious


Any Family Feud, Friends, Skit fans reading? So what do they do? They go out and they survey 100 people to gain relative statistical significance. The objective is to start narrowing down some of those pieces of your hypothesis. How do you get much more specific about:

  • The people whom you're serving,
  • The problems that they have,
  • How they're incentivized,
  • What their goals are,
  • Any sort of maybe seasonality in their business,
  • Who their clients are,

...and all those types of things. Now again, you can go the full scrappy route and use things that are relatively easy to find like a Google survey or SurveyMonkey or other things, or you can go out to the route of hiring out a firm to do the sort of quantitative research for you, and it doesn't need to be this protracted activity that’ll take six months or a year. Nowadays, you can spin it up in a few weeks.

You would be amazed at what people do and tell you about for a $5 Starbucks gift card or a T-shirt, it's amazing to me, I've worked in all sorts of different industries, all different types of companies serving all different types of markets, and universally the humanity of coffee and T-shirts works.

The who and the why

Okay, so now we've gotten curious, we've talked to people we have the kernels of a hypothesis, but we really want to break that into the who and the why.

If you don't have that, and I know many of you reading have probably seen a lot of this research out there, 85-90% of start-ups fail. Why do they fail? Because they didn't have product-market fit.

Instead, there's some cool people in a room who are like "I have this problem!". Or maybe there are some great engineers in the room who are like, "Wouldn't it be awesome if we built this super shiny, cool thing?". And then they sort of toss it over the fence to us in marketing and say, "Go find a value prop, go find my market, let's go sell this thing!", instead of getting product marketers in at the ground level to define the who and the why before anything gets built.

The what, the where, the when

So that's the first section, and I'm really on a crusade to drive this research and importance. Now let's take the next three. The what, the where, the when, again, thinking about "What am I building out for my test plan?" Ultimately, my sort of experimentation or your go-to-market plan is wrapped up in these questions:

  • What are we offering?
  • Is it a product?
  • Is it a service?
  • Is it a combination of those things?
  • Where?
  • How are we going to distribute this?
  • How are people actually going to buy it?
  • How are people going to experience it?

But then also the when. There's so much seasonality that goes into how we go-to-market and how we sell. How many of you reading this are in budget season right now? Putting together a strategic plan and all that, well, so are our clients and other businesses. And what are they trying to do in Q4? And why does that look like a lot of B2B companies where our revenue is so heavily backloaded, the back of the year?

Because they want to spend their money so they can convince their higher-ups they need the money next year, right? You've gotten to that point in the year, and so ultimately, from a sales perspective, and from your market’s perspective, there's a lot of seasonality.

So you really need to understand 'when' not just in the time period of the year but also in the construct of the competitive landscape. Ask these questions:

  • What's going on in the market?
  • Is there M&A activity going on?
  • Are there divestitures going on?
  • Is there some other sort of market trends that are driving the timing that will be really, really important?

Value, value, value

And then the last two questions I'm so passionate about, you've got to build packaging and pricing that conveys the value that those people are going to ascertain from using your product and service. Because price tends to be the number one thing in your go-to-market strategy that pegs you to something else.

I'm in financial services, we all have so much baggage when it comes to money. Whatever your background, what is or was your perception of money, they don't just stop when you walk into the front door of your particular audience. So as you're pricing something, you have to go back to the who and the why. What is their perception of having three different packages and pricing, the good, better, best model that many of us are aware of in SaaS, or just all you can eat model or usage-based pricing or all these different things that will immediately put into their heads, sort of engender in their minds, what value am I going to take out of that product or service?

Similarly, from an experimentation standpoint, going back to that scientific method, pricing and packaging are really, really profound levers on the outcome of your experiment. Again, thinking about the competitive landscape:

  • Are you a new entrant?
  • Are you creating a new category?
  • If you're a new entrant into an established category are you going to undercut price to steal share?

So there are all of these different levels, levers and variables that you need to be aware of as you're putting together this test plan. And then the last thing, and this is a little bit controversial, but I really think that product marketing should have a quota, and what I mean by that is an incentive structure is so profound in terms of what behaviours it drives in your particular organisation.

Align your quad

Just think about it, what's the one thing that will connect you to sales more than anything else? Having the exact same quota, signing up for business goals, actual closed business, ACV, bookings, revenue, etc. And so I think a lot of times we're so focused on what that immediate output would be, or if people have come from the demand-gen side or lead-gen side, you're so focused on how many leads am I driving? Or sometimes even what I call vanity metrics within a particular tactic, such as:

  • How many downloads did I get?
  • How many people went to my website?
  • How many likes?
  • How many clicks? Etc.

All of those are fine metrics directionally, but ultimately, we need to be focused on acquisition, revenue, how much money did we close at the end of the day? Because that is a true measure of product-market fit, how you're positioning, how your distribution channels are performing, how you’re packaging and pricing all of the things that I just went through, that's the ultimate measure of your success.

So I'm encouraging you all, if you've got what I call 'the quad' all aligned to that - so product, marketing, sales and service, all focus on that same objective. It's really hard to start having conflicts within that if you're all on the same team, you're all trying to score the goal. You might hold different positions, obviously across the field, but ultimately we're trying to win the game at the end of the day within the constructs of whatever those rules are.

Okay, so let's recap. So we got curious, we talked to people, we created our hypothesis, we got that at scale, we have our testing plan together, all of those crucial fundamental elements. So let's get to it, now the fun begins, now the potions come together. I'm going to mix a little bit of this and a little bit of that and put it together and see what happens.

Experiment and collect data

The first deliverable in more practical terms that I focus on with my team is the map. The messaging map, bringing all of those questions and the answers to those questions together in one place, such that you have a really good idea of:

  • Whom are we targeting?
  • What do they care about?
  • Why?
  • What's your independent variable?
  • How are you differentiated in that particular scenario?
  • What are you offering them?
  • When?
  • Where are you offering it to them?
  • How much?
  • And how? Etc.

That should all be on one map. If we're all trying to get to the same destination, a map is super helpful. Now, the interesting thing about this map is it serves a foundation, a lot of people are like, 'Really? You want me to put this into a brief? You want me to put it on a piece of paper?'. But this is what happens when we start working with a lot of those cross-functional teams, and suddenly everybody's asking you for things like:

  • "I gotta get a Twitter post",
  • "I need copy for the website",
  • "I have to have this one sheet for whatever", or
  • "I have a pitch deck over here".

If you don't have that one central place where all of your messaging positioning resides, and that can be distributed, it gets confusing.

Shared objectives

So then it gets interesting. We have demand-gen in my group, too. So we've got product marketing, demand-gen, and events marketing all in one umbrella, we still partner and have lots of different cross-functional teams that we're working with but ultimately, we have full control over the budget and the spend, how we're prioritising, which marketing channels we're going after, and then we also have the accountability to see how we're progressing against that budget, what's our return on investment, and so on.

So we have a little bit more control of the lab environment, as it were, then maybe some other organisations do. But that's where when you're collecting data, and you're running this sort of mini-experiment within your ultimately much bigger go-to-market strategy experiment, where you start looking at a lot of those directional variables and those are important, but again, you really need that product marketing team, or maybe there's some other cross-functional lead that is responsible for the ultimate goal:

How much business are we winning at the end of the day?

And it's not just about those marketing channels, if you're in a B2B environment, it's certainly about sales. And if you have the same objective, I know I'm repeating myself a little bit but this is point is key, if you have the same objective as your sales team, there's less reticence for them to let you into their calls, to let you listen in, to do ride alongs, to actually go in on those pitches, so you can hear firsthand.

When I was at a company called Jive, I was on the road 80% of my time, and that was invaluable to understand and sit with, whether it was an account executive, inside salesperson, or otherwise, and just understand, are they even using any of the content we were putting out? Are they speaking in the same messages? To see across every single industry vertical, how do we show up? And that feedback is so important.

Some people call it the voice of the customer. That's fine. It's just another channel that you should be collecting data and bringing that back in.

The blend of art and science

And then something happens, we all get to launch day, we’ve got our go-to-market strategy, we know who we're targeting, we’ve got our messaging, we have our product, we've got our pricing, we've got packaging, all of it. And lightning hits the clock tower, and all of a sudden, you go to the place where roads are not needed anymore.

But in the process of that launch, you generally leave a wake of fire all behind it. I think that's job security. Some people may disagree, but it really is earning all those little fires that cropped up along the way or things that you didn't anticipate. It's very similar to art, going back to that art and science and the blend.

When Picasso or Van Gogh or U2 are creating something in their basement, in their home, in their studio, whatever, it just exists in their minds, but when you put it out into the market, then it takes on a whole other life of its own and it gives you the opportunity to develop those sequels.

Draw conclusions, take action


So think about that now, if I'm in this mindset of scientific experimentation and exploration, I'm giving myself the opportunity to go and test, learn and iterate and take all of that data, drawing different conclusions and re-run the experiment. I'm going to change this variable, this time, I'm going to see what happens.

Now, you have to be working in an organisation where you have some little tests where you've proven out the model so you have a little bit more leeway to continue in your world of experimentation. A lot of organisations, Morningstar included, are very data-driven.

So, isolate it, is there a particular market, segment, product, maybe, that's a little bit less risky, a little bit smaller, that allows you to sort of prove the model? Have that first experiment and then ultimately scale to these sequels. Take the observations that you had and the data about that particular market. What are other local markets, different segments, different geographies, etc. where I can really scale this type of behaviour in these plans?

A lot of people have these great research organisations, when I worked at Disney for example we had a whole army of PhDs looking at market research all the time.

And sometimes that data would just exist in a really pretty deck and people would go to the presentation, sit in the meeting and say, "Wow, that's really fascinating". And then just go on their merry way and do their thing, get back to their daily jobs.

So really, I highly encourage you to take those pieces into action and do something about it. That's where the value comes in.

Never stop learning - an anecdote

One of my favourite anecdotes is about the opening day of Disneyland which was in July of 1955. And Walt Disney, coming off of the high of Snow White which was a huge box office success - really the revenues from that movie fueled a lot of the innovation of the company, and he said, "You know what, I have this idea. Instead of just taking my daughters to this carousel in a park in LA, I'm going to go build this fully immersive storytelling experience called Disneyland".

He went on this insane PR spree, trying to raise money, sponsorships, he had a whole dedicated show for it. And so he's gearing up for the launch, and on the first opening day, the premise was like what we would see as embargoed press engagement. He said, "Okay, well, I'm going to invite a couple of hundred international press people to come see and then tell the world about the greatest place on Earth".

What he didn't realise was that some of those international press people actually passed out some tickets to their friends and over 20,000 counterfeit tickets were printed. Hold that in your head for a second.

So Walt's walking through Disneyland Park talking to some Imagineers, and one says, "We have some bad news. We actually weren't able to get cement into all of the areas of the park so that there are sidewalks everywhere for everybody to walk on".

And he said, "Oh no worries, I can direct press people where they need to go and just avoid those areas." Remember how I mentioned there were 20,000 counterfeit tickets that were printed?

Yeah, so in 1955, most people were a little fancier than we are today. Most women wore heels even to an amusement park. Most men had on very nice shoes and children were sort of dressed appropriately to go to these amusement parks. So Southern California in July with wet cement and women in heels and 20,000 people you weren't anticipating means your MVP launch is people sinking into the cement and losing their shoes in front of 100s of international press people.

Let that sink in for a moment.

So he could have done a couple of things here, right? He could have said, "Oh, man, throw in the towel. We're going to get all this terrible press", etc. But he said, "You know what? No, we are going to take this as a learning", and in Disney parlance, it's all about the guest, "Whatever that feedback is from our guests, we're going to go out and find them shoes".

So everyone backstage basically went and found shoes and got them to these people whose shoes were either literally in the cement or broken or non-functioning so that they can still experience Disneyland. And then they sat and strategically marked off places where the cement wasn't quite dry.

It could have been the end of one of the most iconic brands as we know it, but because he thought differently, he took feedback, he acted very quickly on the data and the feedback and came up with a more interesting solution, people still had a fantastic time. And now Disney Parks is a nearly $15 billion entity at this point. I would imagine nearly everyone reading has experienced Disney Parks in some way, shape, or form.

And so what I want you to take away from my article is that Disneyland, or any products or services or things that we're creating, will never be completed as long as there's imagination left in the world. So I highly encourage you to embrace the spirit of the scientific method of experimentation and the notion that Disney also proffered, that it's kind of fun to do the impossible.