Startup Validation — Dos and Don’ts

What should you be validating?

What is the difference between the idea, the solution, and the problem?

The all-encompassing magical survey

Photo by Alex Green from Pexels

The key problems with surveys.

  1. Filling a survey is not the same as making a purchase.
    It’s not a real-life buying scenario. Customers in reality vote with their money. How they spend it, where they spend it. Either for your solution or an alternate one. People are happy to tell any kind of bullshit when filling a survey but when it comes to real-life transactions, they lose faith at the counter. They tell you anything you want to hear in the survey then spend exactly zero dollars at the end.
  2. Helpful people.
    Anyone filling the survey is trying to help and support you. People who will end up buying your stuff are not supportive of you. They do not and should not really care about you as a person. They don’t want you to feel good. They are looking for stuff that makes THEM feel good. Therefore people with personal supportive motives in the surveys have absolutely no value when you evaluate the market.
  3. Wrong audience.
    Those who are filling the surveys are often not your real target audience. This is most prominent in a B2B setting but we still see the “400 people were supporting the idea by filling the survey and giving us positive feedback! This means there is a vacuum on the market!” It’s like a drug addiction for founders. How do you know who filled it? Or that they have money? Or that they make decisions about money?
  4. I personally love a lot of stuff, would love to spend money on lots of stuff, have plenty of problems that need to be solved, BUT if it isn’t me calling the shots over our family budget, I am a no one. This is the so-called non-decision-maker error/bias. Another personal example of this is asking me about dairy products: happy to answer, but I am on a lactose-free diet — relevancy issue.
  5. Bad survey design.
    Market research is a profession. There is a reason for that. If you are not a professional, chances are high that your survey design will be bad. Full of unintended influences, prone to errors. A great survey uses multiple, similar questions from different angles to gauge the same topic. This is used to filter false positive or false negative answers and to check the coherency of the surveyed.
  6. Trolls. Survey Trolls.
    Take me as an example again. I am the biggest NPS (net promoter score) troll. Whenever I am asked a net promoter question (and I didn’t have a 10/10 experience) I just pick a random answer around the threshold values. Or if a survey starts to bore me, I just pick answers fully random. I know that if I always pick the first answer, it will be easily sorted out, so I don’t do that. Just a random pattern. Sort it out if you can.
  7. A friend’s story in the matter: After a survey, they realized one of the biggest demographic group turned out to be 90+ year old female miners. Why? In the local language, those three characteristics were first the possible options in their respective drop-down lists. So people trolling or not liking the share of details just clicked the first available.
  8. The good survey is long. And short.
    You read that right. It’s not a grammar mistake or an oxymoron. The shorter the survey, the more people will fill it, the more data you will have. The longer survey can be well designed and might provide useful information. The short, professionally designed survey doesn’t exist. Therefore a well designed and at the same time useful survey (for startup problem validation) doesn’t exist. Sorry.
  9. No free text entry field.
    I can say with 100% certainty that not all answer options will come to your mind during the survey design phase. Trust me on that one. And you will just omit the free text option. Why would you since you have no hope of doing anything with that data if you get 400 or even more responses? Still, you are adamant that the answers or better, the preset options include the real answers of potential customers.
  10. Behavioral Economics.
    This is where I should have started probably. During the last decades, it became apparent that we make decisions highly influenced by external stimuli. Or better put, even if unintentionally; all choices and all context have unintended, decision-bending behavioral influences. Some by design, some by fallacy. A primer on this topic is Dan Ariely’s Predictably Irrational. Quick and entertaining read on the most important research in the field. Your survey will contain lots of biases. You can try to correct these, but think about all the biases that will still be in effect and you have no control over them.

Situations where survey is a great solution

Photo by Nick Hillier on Unsplash
  1. Quantitative research on numbers: Age, experience, factual data which are easy to recall and are not opinionated. Demographics on your existing customer base.
  2. Surveying existing customers: Their response rate will be adequate even if the survey is well-designed, therefore long and detailed.
  3. Conjoint analysis: When you are trying to figure out the right package to be offered or gauge the set of features that will be the most desired. Another profession within the profession. You might see it as the knowledge that can be absorbed in a weekend. Your call.

How to do validation properly

Guiding principles for validation interviews

  1. No preparation for the interviewee. Never tell the interviewees what exactly the discussion is about. Just tell them you are looking for stories on a specific problem space.
  2. Never tell the exact goal in advance. The interview always consists of two parts. Before and after revealing the exact idea or problem you are trying to validate. People are always trying to help you and pinpointing too many details will influence their answers. And that’s bad for you in that phase. Don’t let it happen.
  3. Don’t have preset questions. Preset questions are about you and not the person you are talking to. The validation interview is a conversation. A free-flow conversation between a (mostly) questioner and the person questioned. Be curious!
  4. Look for stories. If you wanted one secret sauce from our vast library of innovation secrets, this would be it. Most of the decision-making happens on an emotional level and stories are the best representation of this emotional map. Not to mention that when done properly, stories are the best representation of truth. The truth of the customer. It doesn’t matter if it is a factual truth or just a subjective memory because this is the playing field your service or product will compete in.
  5. Create an emotional baseline. Evaluating the interview is based on emotional spikes against the baseline. If you jump right into it, you will have no time and no chance to establish this baseline. You will not know what was important to the interviewee and what they only said is important.
  6. Always give them a chance to figure out the actual goal. Be prepared to lead them into a dead end. People love puzzles. Your research is a puzzle to them. They should believe that they are in the right domain and got you figured out. For this to work effectively, prepare a research goal that you can use as a distraction. By guiding the interviewee to believe in this direction you help create the baseline and steer the conversation later to the domain of your interest.
  7. This is not a UX interview. We don’t care how current solutions are used. We are measuring if there will be a buying intent, and how strong the problem perception is. E.g. how easy it will be to create a buying trigger.
  8. Make it short. The best validation interviews are 10–15 minutes long. Anything after that is not necessary. Remember, you are trying to figure out what the problem perception is. How important the targeted problem is. If you have to dig for 25–50 minutes, then they don’t have a burning problem — you would need to heavily invest into door opening during sales to face that depth.
  9. Stay as long as you can. Although from the buying perspective you will not learn much more from the 15-minute mark on, you can still learn a lot about what your future solution should entail. Objections and context. And respect the time of your counterpart. If they are spending time with you and enjoy the conversation, don’t be rude and never tell them, ok, we can stop here, I had enough. Be respectful even if the interview is just another dead end.
  10. Don’t pay or reward the interview directly. It is an influence, a significant one. People will try to help you if they get rewarded. It can become extremely hard to remove this anchor and get honest feedback. In our experience, most people are willing to help you if it takes 10 minutes max without any additional reward. Just for the joy of being part of a bigger story or in the end, learning about new innovation and drawing inspiration from it.
  11. The interview is the reward and a great place to get serendipity working. Respect the people spending time with you by telling about the innovative idea. They might connect you with the next person to be interviewed or a potential lead, investor. Or they might become one of these.

What’s next?

--

--

Customer experience strategy resulting in new or improved products and services. We measure customer experience and create strategies.

Love podcasts or audiobooks? Learn on the go with our new app.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
AbilityMatrix

AbilityMatrix

914 Followers

Customer experience strategy resulting in new or improved products and services. We measure customer experience and create strategies.