FACT CHECKER - Is A Survey On The Budget Statistically Valid?
Is 212 people responding to a non-random on-line survey out of 22,000 voters statistically valid? And if so, is the on-line survey a good indicator of what Moose Jaw residents are thinking?
These were questions Council faced when the results of the 2019 Budget and City Services Survey were presented at their September 23rd meeting.
“When 212 respondents out of a voting population of 22,000 (respond that) is less than one percent it is not statistically valid,” Councillor Swanson said at the September 23rd Council meeting.
The question to be asked is Councillor Swanson correct or not?
Councillor Brian Swanson said a survey about the 2019 Budget holds no statistical value due to the number who responded
To get the answers MJ Independent did a Fact Checker and asked an expert on the subject of statistics.
Dr. Andrei Volodin PhD is a Professor of Statistics at University of Regina and also Co-Editor of the Journal of Statistical Calculations and Simulations who agreed to answer some questions on the survey as it relates to the science of statistics.
According to Dr Volodin the minimal sample size - assuming 22,000 voters in Moose Jaw - to have an acceptable margin of error (five percent) and confidence level of 95 percent and have a statistical validity is 378 respondents.
For the non-random on-line 2019 Budget and City Services Survey there was only 212 respondents just under 60 percent of the minimal respondents to hold statistical validity.
It should also be noted in 2018 and 2017 the survey never reached the acceptable margin of error.
Additionally bias can enter into the survey rendering the results statistically meaningless. The bias is that only those most affected, interested or engaged are likely to participate in the survey.
So according to an expert in the field of statistics Councillor Swanson's statement is VALID.
In response City of Moose Jaw communications manager Craig Hemingway said the City never claimed the survey to be scientific but just another means to get feedback from residents to Council.
“The City of Moose Jaw has conducted resident engagement on the Budget process since 2015, when 195 respondents took part in the Citizen Budget survey. Public engagement is one of many actions we undertake in an effort to provide City Council with as much pertinent information as possible before Budget deliberations begin. We do consider all feedback important and continue to thank those 212 people that took the time to complete the survey,” Hemingway responded.
“We noted in our discussion that of the people that responded, this is what they told us,” he responded, adding “We never claimed that these surveys are scientific or spoke for the majority of the people in Moose Jaw..”
“We will continue to search for meaningful ways to engage as many residents as possible on important issues, including Budget.”
Below is the full interview with Dr Volodin and his responses:
1. Statistically speaking given that only 212 people out of a potential pool of 22,000 respondents replied to the survey is the survey valid? Does the survey hold a statistical value?
Answer: The problem I see there is that usually we consider 5% margin of error and 95% confidence level. These numbers should be written in some City Hall Moose Jaw documents for surveys, but I do not know where exactly.
“If we use 5% margin of error and 95% confidence level, taking into consideration that the population size (all citizens of Moose Jaw) is 22,000, then the MINIMAL sample size should be 378, which is strictly MORE than the actual sample size of 212. ”
We CANNOT infer that with 5% margin of error and 95% confidence level the survey provides statistically significant estimation of the proportion.
If the survey does hold any statistical value can you say what value and why?
Answer: As I mentioned above, the requirement of the minimal sample size is not met, so the results do not hold a statistical value with 5% margin of error and 95% confidence level.
2. Given that approximately 10 percent of residents do not have Internet access is the sample size a valid representation of the majority of Moose Jaw residents?
If so can you explain why? If not can you explain why not?
Answer: Even if we consider theoretically that all residents have Internet access, the sample size is still not large enough.
The fact that approximately 10 percent of residents do not have Internet access makes the situation even worse from statistical point of view.
These people were automatically excluded from a possibility to participate in the survey, so the sample is obviously NOT representative.
3. I do not know if you can speak directly regarding the questions asked in the survey but as a statistician do you have any concerns about using on-line platforms such as Survey Monkey (as was used here) to gauge public opinion?
Answer: The procedure of contacting statistically significant surveys is quite complicated.
One of the MAIN requirements for it is that ALL people that we chose for a sample SHOULD reply.
Because of that, we must have a possibility to contact the person that was not available at the first attempt again and ask the survey questions. If the second time there was no success, then we should contact the person again and again until we will get an answer..
“Such things are not possible to achieve by using on-line platforms such as Survey Monkey.
Only interested people will reply. ”
Such type of research is called an observational study, it is NOT real sampling or correct design of experiment. Usually, when we perform an observational study, we are getting biased results.
Dr Volodin provided an analogy to explain how bias occurs in surveys.
The example I can give here is a new medicine that a doctor decides to give to his patients to cure a disease. In the observational study the doctor simply asks a patient: Do you want to try a new drug?
Now, if the patients has a severe stage of the disease and already tried all previously available medicine, he/she will be happy to use the new medicine and will go for it.
If the patient does not suffer too much, then probably he/she would prefer to try previous known medications first because of unknown side effects of the new medicine.
So patients with the severe stage of the disease will be represented much more than others, which creates a bias
Statisticians try their best to avoid any bias by obvious reasons. Results with a bias are NOT statistically significant.
What are the odds of these types of surveys being statistically accurate?
Answer: Statisticians try their best to avoid any bias by obvious reasons. Results with a bias are NOT statistically accurate or significant. The experiment should be properly designed to avoid such things.
How can these types of on-line surveys be made more accurate and allow a wider demographic as well as allow more people to participate?
Answer: The only way I see to make these types of on-line surveys be more accurate is to make them the CENSUS.
This means that ALL residents of Moose Jaw have to reply.
But this is impossible because as you mentioned, 10% do not have even Internet access.
Usually such surveys are conducted by phone with important requirement: if a person did not answer the survey questions first time, we have to call him/her again and again until we get an answer.
4. Can the way the survey question is written skew the resulting statistical result? How important is it to write the questions to be neutral so as not to skew the results in one way or another?
Answer: Yes, of course! This is well know technique how to write a question in such a way that we sort of push a person to answer in the way we prefer.
This question is more to a professor in Psychology, but if you would like, I can dig more into examples of such tricky formulation of a survey question.
Additional Question: how could the survey be made better to reflect resident's opinions?
Answer: I think, some ideas are presented in my answer to Question 3: usually such surveys are conducted by phone with important requirement: if a person did not answer the survey questions first time, we have to call him/her again and again until we get an answer.