Posts
CBC Manitoba recently commissioned a survey of Winnipeggers’ attitudes about racial issues and race relations in the city. Over 1000 Winnipeg residents were telephoned last month and asked about a wide range of race-related topics, including immigration, justice, community, etc. The survey results indicate that Winnipeggers are, on the whole, tolerant and accepting people. CBC can hardly believe it.
The results of the telephone survey were surprising, and sometimes shocking. In general, the survey suggests Winnipeg is a tolerant city.
CBC thinks it’s “surprising” and “shocking” that a survey of Winnipeggers finds them to be a tolerant and even welcoming bunch of folks. Does CBC believe Winnipeg is full of close-minded bigots? What’s really “surprising” to me is that the president of the company that was hired to conduct the survey takes the same view.
"To tell you the truth, I was surprised to the degree in which Winnipeg society seems to be well integrated between racial minorities and the majority population," says Scott MacKay with Probe Research. "There are not any massive glaring issues when it comes to relations, at least sort of on a citizen-to-citizen level [period missing in original]"There are some more important issues … with institutions, but it seems that Winnipeg is a very tolerant and open society when it comes to dealing with racial minority groups. I was kind of surprised by that."
Mr MacKay seems to agree with CBC in thinking Winnipeggers biased and intolerant. That’s pretty bad, I think–but it gets worse. After a rudimentary description of the survey’s methodology, almost half of the story is given over to discussion of its shortcomings. Here Mr MacKay really gets out of hand.
While efforts were made to ensure the population surveyed reflected Winnipeg's population, MacKay notes that it's important to acknowledge that certain segments of the population are always underrepresented in telephone surveys.For example, MacKay said, such surveys can't reach the people who are most economically disadvantaged, because they don't have phones. In addition, Statistics Canada indicates Winnipeg residents in that category tend to be visible minorities and aboriginal people.
"[This] means that we probably have an under-expression of some of the people who have faced the most serious discrimination," MacKay said. "There's nothing really we can do about this, but it's important for us to acknowledge that there, in fact, may be more racism than we've been able to measure."
That is so wrong for so many reasons. The survey gathered no data from people who don’t own telephones, so, as a professional statistician, Mr MacKay knows nothing about them. His survey collected no information about their experience of discrimination, so any generalisations he makes about that are speculation. Mr MacKay has taken off his statistician’s hat and put on a political commentator’s hat. He has no special expertise in the latter area, unfortunately, so his opinion on discrimination experienced by Winnipeggers who do not have telephones is just as serviceable as that of the next guy.
He is also incorrect in saying “there’s nothing really we can do about this”. Of course there is: Go out and find the people who don’t own phones and ask them the survey questions. What he really means here is “CBC didn’t pay us enough money to track down Winnipeggers who don’t have telephones and interview them face to face”. Admittedly, that would have been far more expensive than hiring interviewers to sit in the Probe Research office for a week or so talking to Winnipeggers over the phone. Still, let’s be clear: The issue is money, not inability to gather the information. At Census time, Statistics Canada somehow manages to find and interview, not only Canadians who don’t have telephones, but even those who don’t have a home. It can be done.
MacKay also noted that an appropriate proportion of the survey's respondents were aboriginal – compared with the proportion of Winnipeggers who are aboriginal – but the aboriginals who responded to the survey tended to be more affluent and better educated that statistics indicate is true in the larger aboriginal population in Winnipeg.
"The fact that we probably did not represent the lowest, sort of bottom end of the socioeconomic strata of Winnipeg, especially in some of these minority groups, and the fact that we have sort of a higher level of respondents in the same groups as far as education and income go, are important to consider … because in some ways the experiences and attitudes we're seeing will be affected by that sampling," MacKay said.
Mr MacKay is raising doubts about the credibility of his own company’s survey. After this admission that the survey results are suspect (not to mention the earlier impromptu lapse into socio-political commentary), I wonder why anyone would hire Probe Research to conduct a statistical survey. They don’t believe their own findings. CBC agrees with Mr MacKay: The survey is highly questionable.
In other words, although the Race and Racism in Winnipeg poll showed relatively low levels of racial intolerance in the city, the poll may not have reached the people on the receiving end of the worst discrmination [sic] and racial intolerance.
If CBC had coughed up enough cash for a survey covering all Winnipeggers in the first place, extraneous speculation wouldn’t be necessary.
All surveys have limitations and shortcomings. The unprofessional way to relate them to users, as just seen, is to guess about what might have been found if the limitations did not exist. So, what is the professional way to explicate them? Let’s take an example. Statistics Canada does the Labour Force Survey (LFS) every month in all the provinces and territories, except Nunavut. This survey is the source of many of Canada's most salient economic statistics, most notably, the unemployment rate.
The LFS is done entirely by telephone, so it excludes all households that do not have phones. Does Statistics Canada ever speculate about what the unemployment rate might be if those without telephones had been included? Of course not. If it did that, it would devalue its own statistics and undermine the credibility of its own survey.
The latest Yukon LFS results are posted here (pdf document). Page 11 includes text describing some aspects of the LFS, including limitations. Here’s the bit pertinent to the present discussion:
In the Yukon the LFS sample is designed to represent approximately 83% of the working-age population (Yukoners 15 years of age and older). Yukoners living in unorganized areas, full-time members of the armed forces and people living in institutions are not represented in the sample.
That’s the end of discussion of the limitations of the Yukon LFS sample frame. It would be easy to speculate about the labour force experience of Yukoners who live in “unorganized areas” (i.e., very small communities or out in the bush) or institutions. However, Statistics Canada has no basis on which to conjecture because it gathered no information from those people.
The professional thing to do is simply to state what the limitations are and refrain entirely from supposition about possible effects on the findings.
via Chris’ blog. Chris, who lives somewhere in Manitoba, has posted some neat nature photos at his site.
