Comments by Jacob Ryten (*) on issues of credibility:

Dear Simon:

I read the article with great interest and was particularly keen on the discussion of credibility. I would expand it because I feel that sociological research can make an important contribution to answering the question "why do we believe in what we are told by a statistical office even though we cannot replicate anything it does?" "Moreover, not only can we not replicate it for reasons of logistics but additionally, the law forbids us to have access to the individual records required to replicate statistical compilation." These matters are covered in your paper but they can do with expansion and further analysis.

In particular, I feel the paper would gain if it included a somewhat more detailed examination of credibility. It makes no distinction between reactions to the compilation of those statistics that only challenge (if at all) a narrow set of interests and those that are perceived as a more generalized threat. Thus, the public does not question the reliability of statistics on the production of electricity because it is assumed that a statistical agency can add reliably and that the production of electricity (generally speaking) is all out in the open. There may be questions about the reliability of the statistic on the production of liquor (particularly if it is a state monopoly) not so much because the statistical agency cannot add reliably but because it is supposed that there are secret (illegal, underground) activities that remain undisclosed in official figures. The questioning attitude related to electricity could well increase if the statistics were used to fix arid change consumption tariffs.

There are always questions about socially reprehensible activities such as gambling even though they may not threaten anyone in particular. They are generally interesting in a gossipy sort of way. For example, not only is it believed that notwithstanding the credibility of the statistical agency, individual returns by casinos and gambling houses are falsified but also that coverage is incomplete and that the statistical agency cannot tell a false return from a true one. The same disbelief would apply to returns from nightclubs, escort agencies and so on. The criticism may be motivated by a sense of outraged equity or else by no more than curiosity or gossip.

There are very different questions about credibility when a statistical agency conducts a one-off survey - as distinct from the previous examples which were all of ongoing statistics - and its results are surprising. Statistics Canada conducted a survey on violence against women which resulted in an unexpectedly high rate of instances of maltreatment. The credibility of the agency was challenged on the grounds that: it sided with the feminist movement; its methods were inadequate; the questions it asked were biased because it was ideologically committed; or alternatively the questions were inadequate because the agency was not competent to take a survey of these matters, Extreme criticisms did not represent an overwhelming majority of the comments received but it meant that the debate round the survey’s results was not so much about violence against women, but rather about where Statistics Canada stood ideologically etc., all of which was not a desirable outcome.

There are instances where credibility is challenged strictly on the grounds of technical competence. This is particularly so where the statistic is used for purposes of income supplementing or income re-distribution. The CPI is a case in point. There is the challenge to the statistical agency’s competence at an elementary level: what do statisticians know about buying lettuce, fish or shoes, asks my cleaning lady. And she concludes that the cost of living is much higher than what Statistics Canada claims. This kind of criticism is usually a mixture two assumptions: technical incompetence and political motivation. The Government wants to save money and instructs its agents to lower the statistics. The proportions in which these two ingredients are mixed depends greatly on the rate of inflation. Generally, people overestimate inflation because they tend to select those instances in which price rise is fastest and forget those for which prices remain fixed or move comparatively slowly.

(*) Jacob Ryten was Assistant Chief Statistician of Canada.