Qualitative consistency masquerading as quantitative fit
pp. 387-394
in: Kees Doets, Daniele Mundici (eds), Structures and norms in science, Berlin, Springer, 1997Abstract
Much of physical science is based on the ability to control and negate the effects of variables not being studied in order to examine the quantitative effects of variables of interest. In contrast, "soft" psychologists and other social scientists who study people and institutions and societies can never negate the effects of variables they are not studying. The results are generalizations of a ceteris paribus nature, occasionally without even a grudging understanding that the effect of variables not studied — often incapable of being controlled — is critical in determining the "significance" and effect size of the variables that are studied in any particular context. The result is that the generalizations in most psychological and social science contexts are of a very simple, usually qualitative, nature, although ironically phrased in quantitative terms. (The ubiquitous linear model is an example; even though it is specified in quantitative terms involving slope and intercept, it will be "fit" whenever variables have a merely monotonic relationship, and we feel validated whenever we have predicted the right direction.)Psychologists and social scientists feel most comfortable when their results are consistent across qualitatively diverse situations; the results concerning the "clinical versus statistical prediction" controversy are a prime example. Our current formal theories of science and statistical generalization, however, hardly justify such comfort. For example, standard statistical models of hypothesis testing, effect estimation, or Bayesian inference all involve random sampling from a population, which is an absurdity in social science contexts other than those such as polling likely voters. Moreover, there is no precise theory about what constitutes a "conceptual replication," which is in fact preferred to an exact one.