Sample size is the total number of responses your survey receives.
You may think you can craft the perfect survey questions, throw some addresses in the “To” line of your email, hit send, then sitback to await the results that will come pouring in. But sending a survey without carefully considering your sample size can return inaccurate data, leading to potentially expensive mistakes in brand strategy.
According to one Price Waterhouse Cooper (PwC) research study, customers are 32% likely to stop engaging with a brand after a single bad experience. If you change your customer experience based on data from a flawed sample size, you may actually hurt the CX for a large portion of your customers. There’s a good chance you’ll lose more customers than you gain.
The biggest determinant of a survey’s accuracy is its sample size. Carefully designing your sample sizes will ensure the most accurate data, which in turn, leads to the best customer experience decisions for your brand. This can be difficult to do for even the most experienced survey creators.
Jump down to our visual below on tips for designing a well-run survey.
What Is Sample Size?
Sample size is the total number of responses your survey receives. Contrary to popular belief, this is not the number of surveys you send out for your audience to take.
In the perfect world, you’d want to hear from the entire population you want to reach to ensure you have perfect results. Unfortunately, this isn’t possible, which is why surveys rely on sample sizes.
For example, if you want to survey your customers about their experience ordering from your online store following an update to the ordering system, you should send the survey out to a random selection of your customers who have ordered online after you implemented the update. If you send it to 1,000 people who ordered from you, but only 200 respond, 200 is your sample size.
Before finding your ideal sample size, you’ll need to carefully consider a few significant variables.
5 Sample Size Variables To Know
In order to get an accurate sample size, you’ll need to consider margin of error, confidence level, population size, standard deviation, and Z-score.
1. Margin Of Error
When analyzing your survey data, margin of error is the percentage of potential error in how the survey results reflect the views of the entire population. This is normal in every survey, but the smaller the percentage, the more accurate your results are. To get a smaller margin of error, the larger your sample size needs to be. You want to aim for a margin of error of 4-8%.
Margin of error is often reported using “+/-,” illustrating that each response can be higher or lower than the survey found. For example, a survey with a margin of error of +/-2% means a result of 20% may actually be 22% (+2%) or 18% (-2%).
2. Confidence Level
Confidence level is a percentage informing you of how confident you can be in your results. If your confidence level is 95%, you are confident that you could replicate the results of the survey 95% of the time. The most common confidence levels for a survey are 90%, 95%, and 99%.
A larger sample size typically yields a higher confidence level, so the higher the confidence level you want, the larger the sample size you’ll need.
3. Population Size
Population size refers to the size of the entire group whose opinions you want to know. If 1,000 people ordered from your online shop in the last six months, the population size for your survey would be 1,000.
4. Standard Deviation
Standard deviation is a measure of the variance between your sample values. A low standard deviation means your survey results are closer together in value; a larger standard deviation indicates the results to your survey are more spread out from the mean reported.
For example, if you have only two survey responses of 10 (very satisfied) and 1 (very unsatisfied), the mean of the two results is 5.5. On the other hand, if you have two different survey responses of 5 and 6, the mean is also 5.5. However, the 5 and 6 are closer in value than 1 and 10. The variation between 5 and 6 is lower than the variation between 1 and 10. So while the mean is the same, the standard deviation is higher for the sample with the scores of 1 and 10.
This example is an oversimplification of standard deviation. In reality, standard deviation is calculated using an equation, but all spreadsheet programs such as Excel do this automatically. This is rarely — if ever — factored by hand in the industry.
5. Z-Score
A Z-score measures how many standard deviations a particular survey response is from the mean. In order to have a normally distributed curve to indicate you’ve surveyed enough people, you want a Z-score from -1 to 1 to represent 68% of your sample population, from -2 to 2 to be 95% of your sample population, and from -3 to 3 to be 99.7% of your sample population.
The Z-scores for the most common confidence levels are:
- 90% confidence: 1.65
- 95% confidence: 1.96
- 99% confidence: 2.58
How To Calculate Sample Size
Once you’ve considered all your variables, you can determine your sample size. You can find out your sample size by hand using the formula below: You can also use Chattermill’s sample size calculator to easily figure out your sample size.
How To Minimize Sample Size Errors
Sample size is so important to get right because it can have a profound effect on the results of your survey. Overall, the larger your sample size, the more accurate your results. However, a large sample size can also make your survey more expensive to run and more complicated and time-consuming to analyze, so having a large sample size for the sole purpose of having a large sample isn’t effective by default.
To avoid sample size errors that may negatively affect the credibility of the results:
- Carefully design your samples so they are representative of the larger population you wish to learn from to avoid population specification error, which occurs when you target the wrong population to sample.
- Randomly select people for the survey to avoid selection error, or only having people respond who volunteered.
- Actively seek your samples out through multiple modes of communication and employ adequate follow-up to encourage the most people possible to participate; this helps avoid non-response error, which occurs when the subsection that responds to your survey is different from the subsection that doesn’t.
- Check your subpopulation for any potential biases to prevent sample frame error, an error caused by pulling from a biased subpopulation that skews your results.
Improving customer satisfaction is a must for any brand that wants customers to remain loyal to them. If you want to keep your customers coming back, you need to have an accurate idea of how customers feel about their experience with you, whether it’s in person or online.
Surveys with a strong sample size are an easy way to get feedback from your customers so you can make the improvements you need to see customer experience — and your bottom line — increase over time.
Chattermill takes the guesswork out of designing surveys and analyzing survey data. Book a demo to see how Chattermill can help you scale your business with customer experience data.
