Comments by Jacob Ryten (*) on issues
I read the article with great interest and was particularly keen on the discussion
of credibility. I would expand it because I feel that sociological research
can make an important contribution to answering the question "why do we
believe in what we are told by a statistical office even though we cannot replicate
anything it does?" "Moreover, not only can we not replicate it for
reasons of logistics but additionally, the law forbids us to have access to
the individual records required to replicate statistical compilation."
These matters are covered in your paper but they can do with expansion and further
In particular, I feel the paper would gain if it included a somewhat more detailed
examination of credibility. It makes no distinction between reactions to the
compilation of those statistics that only challenge (if at all) a narrow set
of interests and those that are perceived as a more generalized threat. Thus,
the public does not question the reliability of statistics on the production
of electricity because it is assumed that a statistical agency can add reliably
and that the production of electricity (generally speaking) is all out in the
open. There may be questions about the reliability of the statistic on the production
of liquor (particularly if it is a state monopoly) not so much because the statistical
agency cannot add reliably but because it is supposed that there are secret
(illegal, underground) activities that remain undisclosed in official figures.
The questioning attitude related to electricity could well increase if the statistics
were used to fix arid change consumption tariffs.
There are always questions about socially reprehensible activities such as
gambling even though they may not threaten anyone in particular. They are generally
interesting in a gossipy sort of way. For example, not only is it believed that
notwithstanding the credibility of the statistical agency, individual returns
by casinos and gambling houses are falsified but also that coverage is incomplete
and that the statistical agency cannot tell a false return from a true one.
The same disbelief would apply to returns from nightclubs, escort agencies and
so on. The criticism may be motivated by a sense of outraged equity or else
by no more than curiosity or gossip.
There are very different questions about credibility when a statistical agency
conducts a one-off survey - as distinct from the previous examples which were
all of ongoing statistics - and its results are surprising. Statistics Canada
conducted a survey on violence against women which resulted in an unexpectedly
high rate of instances of maltreatment. The credibility of the agency was challenged
on the grounds that: it sided with the feminist movement; its methods were inadequate;
the questions it asked were biased because it was ideologically committed; or
alternatively the questions were inadequate because the agency was not competent
to take a survey of these matters, Extreme criticisms did not represent an overwhelming
majority of the comments received but it meant that the debate round the survey’s
results was not so much about violence against women, but rather about where
Statistics Canada stood ideologically etc., all of which was not a desirable
There are instances where credibility is challenged strictly on the grounds
of technical competence. This is particularly so where the statistic is used
for purposes of income supplementing or income re-distribution. The CPI is a
case in point. There is the challenge to the statistical agency’s competence
at an elementary level: what do statisticians know about buying lettuce, fish
or shoes, asks my cleaning lady. And she concludes that the cost of living is
much higher than what Statistics Canada claims. This kind of criticism is usually
a mixture two assumptions: technical incompetence and political motivation.
The Government wants to save money and instructs its agents to lower the statistics.
The proportions in which these two ingredients are mixed depends greatly on
the rate of inflation. Generally, people overestimate inflation because they
tend to select those instances in which price rise is fastest and forget those
for which prices remain fixed or move comparatively slowly.
(*) Jacob Ryten was Assistant Chief Statistician of Canada.