What is the effect of increasing the standard deviation on the margin of error?

Click here to jump to our Margin of Error Calculator

When essential business decisions are being made based on survey data from market research, you need to know the data you’re using is accurate and reliable.

But can any market research — or any research in general — be 100% accurate? How confident are you that your sample accurately reflects the population size as a whole?

While it’s impossible for survey results to be completely representative all of the time, you can figure out how close you are based on the margin of error built into your survey.

But what exactly is a margin of error, how can you calculate it and how much can or should it impact the level of confidence you have in your survey results?

Keep reading to find out.

What is margin of error?

Margin of error is information that’s provided alongside the results of a piece of research, such as a poll, survey, or a scientific study. You’ll recognize it because it’s expressed with a plus and minus sign together, e.g. + or -1%

Researchers use this to provide additional information that helps you interpret their results and to understand how the study was carried out. The figure tells you that the true result may differ from the percentage figure provided, and how much more or less than the stated percentage the reality might be.

Margin of error provides a clearer understanding of what a survey’s estimate of a population characteristic means. A plus or minus 2 percentage points means that if we ask this question using a simple random sample 100 times, 95 of those times it would come out at the estimated value plus or minus 2 points.

The larger your random sample (the more responses you get), the smaller your margin of error will be and the more confidence you’ll have that your sample results are reliable.

Find out how big your sample size should be with our free calculator

When is margin of error used?

Margin of error is used when you have a random or probability sample.

That means the survey respondents have been selected at random from your population as a whole and every population member has a known, non-zero probability of being included.

It’s not appropriate if the sample has been selected in a non-random way, for example when you use an opt-in research panel.

A research panel sample is typically a quota sample, where participants are selected because they have particular characteristics. Additionally, the respondents volunteer for the panel in return for benefits, so they are not randomly selected from the population size at large.

So although it is a popularly known term, it has a specific application in survey research and it won’t always be relevant to your market research data.

Here are a couple of scenarios:

A sports team has a complete list of everyone who has purchased tickets to their games in the past year. If they randomly select a sample of that population for a survey, they can calculate the margin of error on the percent of people who reported being a fan of the team.
An organization has a complete list of employees. They poll a simple random sample of these employees on whether they preferred an additional day of leave or a small bonus payment. They can report the margin of error on the percentage preferring each option.

Other kinds of error

Margin of error accounts for the level of confidence you have in your results, and the amount of sampling error you expect based on the size of the sample. But there are other kinds of survey errors that may influence your results too.

These include:

Coverage error

Where your sampling frame doesn’t cover the population you are interested in.

Non-response error

This happens when certain respondents don’t take part in your survey.

Measurement error

This can arise from problems with the questionnaire.

To learn about other sources of error, check out our guide to random and non-random sampling error.

What is a confidence interval?

A confidence interval (CI) is a range of values that include a population value within a degree of confidence.

The values are typically shown in the results as a % value when a population mean sits between a lower and upper limit. Researchers use a CI to measure how accurately the survey sample resembles the overall population.

That’s because it’s almost impossible to find a sample that 100% matches the characteristics of the total population.

Researchers can choose a CI of any level, but a 95% CI is the most common.

This means researchers can be 95% certain that the results contain the actual mean value of the total population. This can be demonstrated using a normal distribution:

Using the graph above, you can see that if the probability of population mean value is between 1.96 and -1.96 standard deviations (sometimes called the z score), the sample mean is 95% (hence the 95% CI).

This is the most common industry formula for calculating the CI. However, when the costs of an error are extremely high, e.g. a multi-million dollar decision is at stake, the confidence interval should be kept small. This can be done by increasing the sample size.

How do you calculate margin of error?

Margin of error is calculated using a formula:

Z * √((p * (1 – p)) / n)

Where

Z* is the Z*-value for your selected confidence level, which you’ll look up in a table of Z scores:

p is the sample proportion

n is the sample size

The sample proportion is the number within the sample that has the characteristic you’re interested in. It’s a decimal number representing a percentage, so while you’re doing the calculation it’s expressed in hundredths. For a 5% sample proportion, it would be 0.05.

The most commonly used confidence level is 95%, so we’ll use that for an example calculation. The Z*-value for a 95% confidence level is 1.96.

We’ll set our sample size at 1000. Next, we’ll follow the process to calculate the margin of error.

How to calculate margin of error with your survey data

Subtract p from 1. If p is 0.05, then 1-p = 0.95.
Multiply 1-p by p. So that’s 0.05 x 0.95 – which gives you 0.0475.
Divide the result (0.0475) by the sample size n. So 0.0475 divided by 1000 = 0.0000475.
Now we need the square root of that value, which is 0.0068920. This is the standard error.
Finally, we multiply that number by the Z*-value for our CI, which is 1.96. So 0.0068920 x 1.96 = 0.0134395. That’s a margin of error of just over 1%.

Let’s try it with a real-world example.

Imagine you are a business surveying your current customers. You’ve run a study with a randomly selected sample of 1,000 people from your CRM list. The results tell you that of these 1,000 customers, 52% (520 people) are happy with their latest purchase, but 48% (480 people) are not – yikes. You want to add a margin of error to these results when you report them to your shareholders.

We’ll assume you want a 95% level of confidence, so the z*-value you’re working with is once again 1.96.

The number of customers who are happy with their latest purchase was 520, so that’s the number you’ll use to work out the sample proportion. 520 (p) / 1,000 (n) = 0.52

1-p is 0.48
0.52 (p) x 0.48 (1-p) = 0.2496
0.2496 / 1,000 = 0.0002496
The square root of 0.0002496 = 0.0157987
0.0157987 x 1.96 (the z*-value) = 0.0309654, or in other words, 3.1% (when you round it up).

You can now report with 95% confidence that 52% of your customers were happy with their latest purchase, + or – 3.1%

There are a couple of conditions for using this formula. They are:

n x p must equal 10 or more
n x (1-p) must equal 10 or more

Usually, survey research involves quite high numbers of people in a sample, so unless you have a very small sample, or the sample proportion within your sample is very small, there won’t be a problem. If you’re getting numbers below 10 for either of these checks, you may need to increase your sample size.

Free eBook: Learn how to increase response rates to your survey

Determining your sample size

Choosing the right sized sample is a critical part of creating a survey that will give you reliable results, especially when you’re using the data to make critical business decisions.

Determining sample size is based on the number of people you will need to represent the full population. For example, using 2,000 people as a representative sample of the UK workforce.

If you choose a sample that’s too small, you won’t get reliable results. The larger the sample, the smaller the margin of error. Of course, larger sample sizes mean costlier research, so ultimately there’s a trade off.

Determining the size of your sample is a relatively straightforward process:

Calculate margin of error

Your margin for error (also referred to as confidence interval) essentially determines how confident you can be that your sample will represent the same results you’d get if you were able to survey the total population.

If you used the margin of error formula mentioned above — Z * √((p * (1 – p)) / n) — and the associated z score table, you have your own margin of error calculator.

Understanding your confidence level

Your confidence level determines how certain you would be that the actual mean falls within your margin of error — and that if you were to repeat your survey multiple times, you’d still get the same results.

The lower your sample size, the lower your confidence level will be. The higher the sample size, the higher your confidence level.

What’s the standard deviation likely to be?

In a population, standard deviation helps you to understand how the completed responses will differ from each other, as well as how they’ll deviate from the mean number.

A low standard deviation means that the values are clustered around the mean number, whereas a high standard deviation means they are spread across a range, with some small and large outlying figures.

What’s the total population size?

Population size is the main factor in determining the sample size you need to get the desired confidence level in your results.

The ‘population’ represents the whole group that you want to understand through your research, while the sample size is the number of people you need to survey to get a representative sample of that whole.

Still unsure how to determine your sample size? Check out our Sample Size eBook.

How to reduce the margin of error in your survey results

Reducing your margin of error is an essential part of increasing the reliability of your surveys and ensuring you’re generating better, more accurate results.

There are several ways you can reduce the margin error, some are more practical and will depend on a few variables around your project.

Reduce the variables

It stands to reason the more variables you have in your study the higher the chances that errors will creep into your study.

Reducing the variables in the way you gather your sample or complete your survey can also help to reduce the standard variation, meaning the margin of error will be lower.

Increase the total number in your sample

Often the easiest way to reduce the margin for error is to increase the size of the sample. Having a larger group complete your surveys statistically means you’ll be more likely to generate a representative response for your study and reduce the margin of error.

Use a lower confidence level

A lower confidence level provides you with a more precise margin of error – however you should be wary about this as lowering the confidence level reduces the confidence that your survey will accurately represent the overall population.

Use a single-sided confidence interval

Unlike a two-sided CI, a single-sided interval has a smaller margin of error, because it only shows if one of your parameters is above or below the cut-off value on one side i.e. it will only tell you if the parameter is above or below (it won’t show both).

You can use this method to reduce your margin for error if you’re confident the parameter will fall on one side of the cut-off value.

How Qualtrics can help

No matter the survey, research project or poll, Qualtrics CoreXM has the tools and capabilities you need to derive insight from data — and go from insight to action.

Used by more than 13,000 customers, Qualtrics CoreXM brings your survey, research, and polls into one platform, rather than multiple tools, enabling you to scale your activities, improve data quality, and uncover new opportunities.

With CoreXM, get insights faster through automation, reduce project costs and admin by consolidating your tools, and increase project impact by leveraging best-in-class survey methodologies and tools.

From simple polling and internal feedback surveys for strategy to gleaning customer, product, brand, and market insights, Qualtrics CoreXM empowers every department to carry out and benefit from research.

And if you want to get even more from your research, take advantage of Qualtrics’ Research Services and a network of experienced partners. Through trainer experts, unlock cost and time savings, while benefiting from flexible service options that let you choose how much or how little support you need.

Want to supercharge your research? Find out more about Qualtrics CoreXM.