How to Use Smoke Tests to Validate Your Product or Feature Ideas

How to Use Smoke Tests to Validate Your Product or Feature Ideas

Think you’re sitting on the next big idea for a product or a product feature? Before you spend dozens of hours and tens of thousands of dollars on the idea, you need to validate it.

In other words, you need to make sure your audience is just as excited about the idea as you are.

So, how can you validate the ideas sitting in your startup notebook or your product feature backlog without wasting resources? By running a smoke test.

What is a smoke test?

The term smoke test was most recently borrowed from the computer programming world. There, a smoke test is the process of running test cases involving the “important functionality of a component or system, to ascertain if crucial functions of the software work correctly”.

Using this definition, smoke tests address questions like…

  • Does the program run?
  • Does the UI open?
  • Does clicking the key functionality do anything?

Essentially, does everything work as it’s intended to? Smoke tests are specifically designed to be run quickly and frequently during the software development stages.

In the marketing world, smoke test has a similar meaning. However, instead of being focused on “does it work properly” like programmers, marketers want to know “will it make me money”.

Trevor Lohrbeer of Teikametrics and Lab Escape defines a smoke test as…

“TrevorTrevor Lohrbeer, Teikametrics & Lab Escape:

“A way to test market acceptance before creating a product.” (via SlideShare)


So, if you put the time and money into creating the product, will it actually sell or will it fall flat? A smoke test is designed to help you answer that question.

Dominic Coryell of GIMME GROWTH explains further…

“DominicDominic Coryell, GIMME GROWTH:

“It’s helping to shape your product roadmap by using growth experiments in a pretty fast way. What we’re trying to do basically is just prove that your product backlog is actually worth moving forward.

Validate, learn, iterate.” (via YouTube)

Why do marketers use smoke tests?

Consider these two situations…

  1. You have a new business idea that you’re excited about and want to develop.
  2. You have a new product feature idea that you’re excited about and want to develop.

In both cases, you’re about to spend a lot of time and money. You will also likely need help along the way, whether it comes in the form of a co-founder or a product team.

What if that idea, after months of development and tens of thousands of dollars invested, falls flat on its face? Crickets.

Marketers use smoke tests to avoid this exact scenario. Dan Martell, a 5x founder who now helps entrepreneurs build their tech products, explains…

“DanDan Martell, Entrepreneur:

“People get excited about an idea and then build the product and then find out 6 months later that nobody else was excited.

1. Idea: Problem, Pain, Passion
2. Validated Research
3. Clickable Prototype
4. Pre-Sales

I’ve used this approach to build ALL my companies and it doesn’t matter if it’s for a consumer application, or a solution for big Fortune 500 companies – it can be done.” (via

Dan’s four step process might look something like this, when applied…

  1. How do you come up with or choose the right idea? Choose a problem you’ve experienced, choose something that’s a must-have (not a “nice to have”), choose something you’re passionate about.
  2. Find customers using your competitor’s product and conduct validation research. Your friends will tell you your idea is great; people who are already paying to solve the problem you want to solve are the best choice.
  3. Design 3 core features of the idea, not the entire product. What can you build in a 6-week product sprint vs. a 6-month development period?
  4. Get your target audience (see step two) to pre-buy the product. Work with them to develop the full product.

Now, this process doesn’t just apply to startups and entrepreneurs. It’s certainly the way that Dan positions it above, but it can easily be applied to bigger companies looking to test new product ideas, feature ideas, service ideas, etc.

This type of agile thinking is what matters.

Dan’s process proposes something very interesting: that there’s something wrong with the current minimum viable product (MVP) concept. If you’re a marketer who is interested in rapid experimentation and product efficiency, then there is.

Here’s how the product team at Spotify explains it…

MVP Problem
Image Source

The most common objection to this pre-sale (or smoke test) process is that customers will be upset that full functionality is not available. It’s never smart to fall short of customer expectation, right?

Eric Ries, CEO at Long-Term Stock Exchange and author of Lean Startup, explains why this isn’t as big of a deal as it seems for agile teams…

“EricEric Ries, Long-Term Stock Exchange:

“If you’re worried about disappointing some potential customers – don’t be. Most of the time, the experiments you run will have a zero percent conversion rate – meaning no customers were harmed during the making of this experiment.

And if you do get a handful of people taking you up on the offer, you’ll be able to send them a nice personal apology.

And if you get tons of people trying to take you up on your offer – congratulations. You probably have a business.

Hopefully that will take some of the sting out of the fact that you had to engage in a little trickery.” (via Startup Lessons Learned)

Example: Pijon Box

During a presentation on smoke testing, Dominic shared a case study from Pijon Box, a care package service for the parents of college kids. Pijon Box wanted to test add an “Add to Box” feature so that parents could add more products to their care packages, if they wanted to.

Here’s how rolling out that feature would look typically…

Typical Product Way
Image Source

Of course, if the Pijon Box team had taken this route, they would’ve spent a month on development and missed the end of April deadline for the last box.

So, instead, they tried this smoke test approach…

Smoke Test Way
Image Source

They found out which products they had extra of in the inventory, put them into an email and sent the emails out to customers. If the customers clicked the link, the Pijon Box team would have to manually invoice them and notify the warehouse.

The result? 9.55% of active subscribers clicked the link and converted. Monthly gross profit grew by 45% because of the “Add to Box” feature. Marginal gross profit per Pijon Box (on those boxes) grew over 357%.

Not too bad for a day’s work. Now Pijon Box had validated the idea and knew they wouldn’t be wasting a month of resources developing it.

Example: Flowtown

Another example comes from Dan, who used smoke testing frequently to valid Flowtown. Dan calls it a minimum economical viable offer (MEVO)…

“DanDan Martell, Entrepreneur:

“Create a MEVO.

It’s saying, ‘I’m gonna create an offer, a valuable offer, and the minimum amount of features that I feel will get somebody to part with their money and I’m gonna create that landing page and get them to sign up, even if the product doesn’t exist on the backend.’

It’s what I used in my company, Flowtown. And what you do is once they sign up, if you have no product, just say, ‘Hey, we’re still early. We’re releasing people little bit by little. Your credit card wasn’t charged.’ Even though, again, you accept the credit card and you use different tools for that, like Stripe. And say, ‘When we’re ready, we’re gonna let you know.’

That is true validation.” (via

Here’s what Flowtown looked like in February 2009…

Flowtown Version 1
Image Source

That’s right, a service company. After three months of working with 15+ customers, they decided to use all of the feedback they had gathered to double down on creating a product. (The plan all along.)

That resulted in…

Flowtown Version 2
Image Source

Which only resulted in one customer over three months. So, they went through many more iterations (smoke tests)…

Flowtown Iterations
Image Source

Finally, they arrived here…

Flowtown Version 4
Image Source

Within the first day, they had their first customer. Then, growth took a turn for Flowtown…

Flowtown Growth
Image Source

Two years later, they were acquired by Demandforce.

Something that they emphasize is what they call the “bullshit bar”. That is, having people actually pay for the idea…

Bullshit Bar
Image Source

Here’s another one…

Bullshit Bar
Image Source

It’s important to consider the transfer of funds when validating an idea for a product or feature. A lot of people will tell you the idea is great. “Oh, yeah, sure, I’d use it.” But would they really spend their money on it? Even something $5 or $9.99 a month can add an extra layer of validation.

What should you smoke test first?

Let’s assume you’re in one of these two scenarios…

  1. You have a dozen business ideas you’d like to smoke test.
  2. You have a dozen product features you’d like to smoke test.

How do you decide which to smoke test first?

Derek Sivers suggests you “test the riskiest assumptions first”.

Otherwise, consider the following…

  1. High impact ideas. If you’re prioritizing your experimentation, you likely have some measure for “anticipated impact”.
  2. Ideas that have been backlogged for too long. Smoke test it and move on.

The 6-Step Smoke Test Process

In his presentation, Dominic lays out the six steps of his smoke test process, which is arguably the most definitive and widely used.

Step 1: Define the necessary proof.

As with most tests, smoke tests begin with a hypothesis. If I do X, I expect Y.

For example…

  • If we improve monthly gross profit by X% with the “Add to Box” upsell email, I expect it’s worth spending the resources to automate the feature.

  • If we capture info from more than X% of visitors to the landing page in the first X weeks, I expect it’s worth building the actual product.

  • If we can pre-sell $X in X weeks, I expect it’s worth building the actual product.

Step 2: Design an experiment without engineers.

The idea of a smoke test is to limit the amount of time and money invested before validation. So, your smoke test should be able to run without the help of an engineer.

For example…

  • Manually invoicing customers and notifying the warehouse vs. having engineers automate the process.

  • Creating a landing page to capture zip code and email for an eventual release vs. having engineers create the entire product.

Step 3: Drive the right traffic to your experiment.

Once you have the smoke test set up, you’ll need to drive some traffic to the test. This shouldn’t be a spray and pray effort. You want to make sure you’re recruiting the right kind of traffic for your test.

Here’s how Dominic knows if he’s recruiting the right audience for his smoke tests…

“DominicDominic Coryell, GIMME GROWTH:

“I think that there is no better way to find the right audience than to see if people share what you’re showing them because if people do share, that means that they’re willing to show their networks about what it is that you’re doing, that they believe in what you’re doing.

Sometimes that can get skewed if you incentivize this.

It helps guide the audience you’re creating. So, on Facebook, if people aren’t sharing after they come in from an ad, then I know it’s probably the wrong audience whereas from a search ad, if they are sharing, maybe the intent is higher.

So it helps guide the decision of where to put money for your experiments.” (via YouTube)

Eric turns to AdWords when he’s looking to generate traffic for his smoke tests…

“EricEric Ries, Long-Term Stock Exchange:

“If you want to build an online service, and you don’t test it with a fake AdWords campaign ahead of time, you’re crazy. That’s the conclusion I’ve come to after watching tons of online products fail for a complete lack of customers.

Just make sure you choose a maximum daily budget that is affordable for you to run the campaign for a few weeks with. I would aim to get no more than 100 clicks per day – over the course of a week or two, you’ll get pretty good conversion data.

Here are the stats you want to pay particular attention to: the overall conversion % of customers from landing page to completed registration, the click-through-rate for your ads on different keywords, and the bounce rate of your landing page for different keywords.” (via Startup Lessons Learned)

Now, if you’re testing a feature, you can simply send existing users to the smoke test. You may want to consider using segmentation, however. For example, perhaps you will only test the feature among active users who have signed in at least 3 times in the last 30 days.

Whatever route you go, be sure of two things…

  1. You have tapped into the right audience. The best product or feature in the world will fail miserably if it’s presented to the wrong audience.
  2. You have generated enough traffic to make a decision. If you’re a large enough company to be A/B testing, use a sample size calculator upfront as you normally would. If you’re a small company (or not yet a company at all), aim for those 100 clicks / day for two weeks that Eric mentions to start.

Step 4: Define your metrics and measure them.

This (almost) goes without saying, but be sure you have tracking in place and are measuring KPIs before starting your smoke test. You should know where people are coming from, how they’re behaving at each stage of the funnel, what device they’re using, etc.

Choose KPIs that relate back to the proof you established in step one.

Step 5: Analyze the results and iterate / optimize.

If you fall into the “testing a product idea from scratch” camp, you likely do not have the traffic (and cannot afford the traffic) to make your test statistically significant.

That doesn’t mean you can’t run a smoke test. It just means that your smoke test will not be perfect and there is a lot of room for error, so take results with a grain of salt. (Think of it the way you think of heuristic analysis, for example.)

Even if you can get a statistically significant result, chances are that result will be “maybe”.

That’s where A/B testing and informal iteration comes into play. Eric explains his iterative process…

“EricEric Ries, Long-Term Stock Exchange:

“Use Google Optimizer to try different landing pages (even radically different landing pages) to see if any particular way of talking about your product makes a difference to your conversion rates.

And if you’re getting conversion rates that you feel good about, try asking for a credit card or other method of payment.

If that takes your conversion rate to zero, that doesn’t necessarily mean you don’t have a business, but I’d give it some serious thought. Products that truly solve a severe pain for early adopters can usually find some visionary customers who will pre-order, just based on the vision that you’re selling.

If you can’t find any, maybe that means you haven’t figured out who your customer is yet.” (via Startup Lessons Learned)

Example: Suprizr

Another example Dominic gave during his presentation was Suprizr, which promised to deliver a surprise meal based on your preferences within an hour.

Here’s what the landing page looked like before…

Suprizr Version 1
Image Source

It had a <3% email capture rate. Here’s where the importance of testing comes into play again. This low conversion rate might, to some, indicate that the idea is not valuable.

Instead of abandoning it altogether, Suprizr changed their approach by first asking for zip code…

Suprizr Version 2 (A)
Image Source

And then email, when they were told that Suprizr was not yet available in their area…

Suprizr Version 2 (B)
Image Source

The result? 30% of visitors entered their zip code, 25% entered their email and 10% of them shared the site.

Sometimes, your first attempt at a smoke test will not pan out the way you thought. Continue iterating before giving up on the idea completely.

Step 6: Call it.

Some of you will have the luxury of being bigger businesses with statistically significant data to analyze while some of you will not. No matter your situation, eventually you will have to make a decision.

It can be a good idea to set a designated timeframe for step five as analysis and iteration can go on for ages, which would defeat the purpose of running a smoke test in the first place.

Work with the data you have, make a decision and move on to the next.

Insights > Winning

As with regular A/B testing, strive to create an insights > wins smoke test culture. After all, as Dominic explains, even if your idea is proven to be a loser, you still win…

“DominicDominic Coryell, GIMME GROWTH:

“Now, if we hadn’t have won, that would have been fine too because we would have still won.

We would’ve felt good as a team that we didn’t waste time, we didn’t waste money. And we would’ve learned more about our customers.

It’s this type of process and approach that I find really, really valuable because it’s so easy for me, especially on the product side, to get so focused on the perfection and what the product should feel like and make all of these assumptions that people are going to use it without actually testing to see if people are going to use it.” (via YouTube)

Always consider what other insights you can draw out…

  • Why might this idea have failed?
  • Was it an execution error?
  • Did some iterations work better than others?


Before you launch your next product or feature idea, run a smoke test to validate it. You’ll save yourself time, money and (potentially) embarrassment.

Just follow these simple smoke test steps…

  1. Define the necessary proof.
  2. Design an experiment without engineers.
  3. Drive the right traffic to your experiment.
  4. Define your metrics and measure them.
  5. Analyze the results and iterate / optimize.
  6. Call it.

Remember, smoke tests are often thought of strictly in the startup sense, but the process is easily applicable to big companies looking to launch product features or new products / services as well.

The post How to Use Smoke Tests to Validate Your Product or Feature Ideas appeared first on CXL.

via CXL Read More…

How will Google’s new ‘Ad’ label impact marketers?

How will Google’s new ‘Ad’ label impact marketers?

Google started testing a new ‘Ad’ label in January this year, and late last week it was confirmed that this will now be rolled out globally.

This white label with green text and a green outline will replace the green label that was launched in June 2016.

The instant reaction to this is that the new labels fit in quite seamlessly with the rest of the paid placement, perhaps creating less of a contrast between them and their organic counterparts.

So why has Google made the change now, what impact will it have have, and will users even notice the change?

The official line on this update is that Google wants to streamline the number of colors on its results pages, particularly on mobile devices. A Google spokesperson revealed:

“After experimenting with a new search ad label with a green outline, we’ve decided to roll it out. The new ad label is more legible and continues to make our results page easier to read for our users with clear indication of our ad labeling.”

Additionally, they claimed that “the color change had no bearing on consumers’ ability to distinguish ads from organic listings on the page.”

So why make the change at all?

First of all, these changes never occur in a vacuum. This is just an indication of a wider trend and should be viewed in the context of the removal of right-hand side ads, expanded text ads, and the consistent drive towards a ‘mobile-first’ approach.

Add in the growth of ad blockers, intensifying competition in the search industry (with both Facebook and Pinterest upping their efforts), and the constant pressure on Google to grow its revenues, and the reasons for moving to a less noticeable ‘Ad’ label become apparent.

We should also beware the source of this information. Google may say it has had no impact in testing, but that seems a convenient line for a company that is close to obsessive in its desire to attract more paid clicks through attention to the minutiae.

Google is famed – sometimes ridiculed – for this constant tinkering, but it does work.

Their highly-publicized ‘50 shades of blue’ experiment was seen by some as a step too far, but Marissa Meyer made sure to state that it drove an extra $200m in ad revenue. Even at a company of Google’s size, those figures talk.

It is also worth remembering where we have come from with these ‘Ad’ labels. People can have short memories – a fact that such frequent adjustments take advantage of – and this latest change makes sense when viewed at a higher level.

Google’s ‘Ad’ labels have gone from garishly overbearing to their latest camouflage iteration in the course of just two years:

The change from yellow to green in mid-2016 was reported to have a positive impact for paid search CTR, and few will doubt that last week’s move was led by exactly the same motive.

But is this just a myopic attempt to gain clicks (and the accompanying revenue) in the short term? Or is there more at play here?

For many in the organic search industry, this will just be another step in the inexorable march towards paid search domination of results pages.

One assumption at the heart of Google’s latest update is that users simply want to get to the result that answers their query, whether a brand has paid for their click or not. Giving more space to paid placements and a never-ending stream of new products to make these ads more attractive undoubtedly gives prominence to sponsored listings.

But, the counter-argument goes, people prefer organic listings. They know an ad when they see it and will go out of their way to avoid it.


However, one of the reasons this has held sway in the past is that paid search landing pages have at times been of lower quality or of lesser relevance to the query than organic listings. Brands are willing to pay their way to the top, while that right has to be earned in SEO. The quality of the search results in each camp reflected this.

Which brings us to the growing impact of content marketing and user experience signals in SEO. These factors are essential for any successful SEO strategy and they touch all aspects of a brand’s digital footprint – including paid search.

All that effort site owners have put into creating ‘great content’ to improve their SEO rankings plays directly into the hands of AdWords. If Google can convince brands that the best way to get this new content in front of people is to pay for that right, they will do so. The same great content ends up in front of consumers, so everyone wins. Brands still get the traffic (at a higher price), users get the result they want, and Google makes more money.

Someone has to lose, though, and SEO traffic seems most likely to assume this position.

A diminished SEO landscape would be to the detriment of user experience, though, and no monopoly (even one as seemingly immovable as Google) has a divine right to market ownership. Higher CTR for paid listings will have to go hand-in-hand with a better user experience if this pitfall is to be avoided. If the quality of results starts to dip, alternative search engines do exist.

Another argument is that perhaps the role of paid search is starting to change. The AdWords business model is beautifully crafted for a direct response strategy, but it has its limits when it comes to brand marketing. As brand budgets start to move into the digital space, it would make sense to have a less obvious ‘Ad’ label if Google wants to encourage advertisers to spend this budget on AdWords.

As always, there is much room for speculation, even if the central thrust behind this move seems to be an intended increase in paid search revenues.

One thing is for sure, though: we will be keeping a very close eye on CTR for both paid and organic listings over the upcoming days and weeks to see how this plays out.

via Search Engine Watch – Category: seo Read More…

5 Steps To An Influencer Marketing Strategy

5 Steps To An Influencer Marketing Strategy

Influencer marketing is vital for any consumer-facing brand in 2017. The concept is certainly not new. Many brands have used influencers as marketing assets in some form or another for decades. So, how has influencer marketing evolved? The rise of […]

Post from: Search Engine People SEO Blog

5 Steps To An Influencer Marketing Strategy

Written by Nick Andrew Rojas,

The post 5 Steps To An Influencer Marketing Strategy appeared first on Search Engine People Blog.

via Search Engine People Blog Read More…

New Hub Feature: Activate, Manage and Delete ANY WordPress Plugin or Theme

New Hub Feature: Activate, Manage and Delete ANY WordPress Plugin or Theme
So you asked, and we listened! Today, we’d like to officially announce the ability for you to upgrade, activate, deactivate and manage any WordPress plugin and theme straight from The Hub. And, of course, that means you can now upgrade all of your plugins and themes across as many sites as you like with one “Update Everything” […]

via The WordPress Experts – Read More…