Models of statistical distribution
For some time now I have tried to relate major innovations in science and mathematics to the movement of society in history. At the grandest level of generalization, there are observations such as Oswald Spengler’s when, in The Decline of the West, he contrasted ancient and modern ideas of number in terms of ‘magnitude’ and ‘function’ respectively and linked this to the money system. Ian Hacking in The Taming of Chance has shown how linear causality was replaced by probabilistic reason and statistics in the course of the nineteenth century; and this is undoubtedly related to the salience of crowds as opposed to unique effects. The homology between Darwinian evolutionism and Victorian capitalism was pointed out by Marx. It is plausible to posit a link between scientific/artistic modernism and the movement of world society in the decades leading up to the First World War. And the sciences of complexity that have emerged since the 1970s, with their language of chaos, fractals and phase transition, evoke the postmodern moment in social and cultural history.
If I have learned anything from these amateur inquiries, it is that the history of ideas and the history of society have at best a very loose chronological relationship. But that hasn’t stopped me from pursuing the connection. I have been sustained in this by a belief that social science is ideology and therefore in denial as far as social reality is concerned. This explains why the epistemology of economics remains trapped in the seventeenth century world of Galileo and Newton, caught between rationalism (microeconomic theory) and empiricism (econometrics); or why the methodological achievements of quantum mechanics – you can’t measure position and movement at the same time and if you observe something you change it – have had so little impact on the social sciences in the twentieth century. I have become convinced that the physicists and mathematicians, fondly assuming that their objects of study have nothing to do with human experience, are in fact a better guide than the social scientists to how ideas about the world are influenced by society. For this reason, I have avoided biological subjects since these lend themselves so readily to ideology, preferring rather to glean what I can from the study of stars, earthquakes, clouds, metals and elementary particles.
The history of science, technology and society has achieved a success in our time comparable to the ethnographic movement of the early twentieth century — using similar methods. Faced with their own inability to grasp Maxwell’s equations, these intrepid explorers have convinced us that we should not celebrate the inventions of great men, but rather should investigate how the laboratories were organized and the machines made to run. Since I do not aspire to pioneer a new segment of the academic division of labor, I have followed what I can of the scientific literature produced by its leading practitioners. Most of the mathematical reasoning passes me by. What I am looking for above all are the prose statements that reveal how these authors explain their discoveries in general terms. I hope to find here the implicit models of society held by the scientists. There is no way that such an amateur investigation could proceed other than by guesswork.
Nevertheless, I have some expertise in the field of statistics, which I have practiced and taught for four decades (not counting my early career as a scientific gambler on the horses); and I focus here on a remarkable shift in its dominant paradigm. When I was trained in the discipline, all the most powerful techniques were derived from the Gaussian or ‘normal’ curve with its parameters of mean and standard deviation. ‘Non-parametric’ statistics had made an appearance in fields such as social psychology, but they were mathematically weaker. At the same time, graph theory offered a more systematic approach to networks, which interested me as a student of migration and urbanization, but its limitations were all too apparent, as we will see. Only in the last decade have I become aware of the emergence of another statistical paradigm, based on the ‘power law’ distribution, which itself draws on ‘the new science of networks’. Both promote a much more dynamic and unequal understanding of the world than what went before. I have been led to speculate on why this might be so; and have turned to a historical idea, that humanity is currently caught between national and global versions of society.
I have not carried out an exhaustive survey of the intellectual antecedents for my approach. Instead, I have placed my main bet on an essay by Durkheim and Mauss, “The primitive forms of classification”, published originally in Année Sociologique and in English as Primitive Classification with an introduction by Rodney Needham. They sought to demonstrate that the classification of things in nature, the categories of understanding, replicate the classification of relations between men in society, whose forms ought to be considered to be prior. It was a daring example of Durkheim’s reductionist method for sociology and Mauss, who later pursued a non-reductionist anthropology in works such as The Gift, commented wryly that his contribution to this had been to collect the facts. The argument clearly prefigures Durkheim’s Elementary Forms of the Religious Life (1912) which to my mind is the most revolutionary work of modern social theory’s founding fathers.
As in Elementary Forms, the classification essay starts from the relationship between Australian totemism and clan organization. Variations are introduced by comparison with other Australian groups before the Zuni case is examined as an elaboration of the same principles, with the Sioux as an intermediate stage in what is taken to be a development of the system. Chinese astrology is introduced as one of many instances of more complex Asian societies to point out how a more abstract system, precursor of our own scientific rationalizations, shares some principles (such as hierarchy) with ‘simpler’ societies that are firmly rooted, they believe, in social morphology. There are lots of holes to be picked in this argument and Needham identifies most of them, for example that the direction of association between the classification of things and social classification is not established. Nor would we be as inclined to frame it in evolutionary terms. At least since Lévi-Strauss (1962), we have not drawn a firm line between intellectual abstraction and social complexity. But I still find the basic premise gripping and it inspired what follows.
Statistical patterns can be found in nature and society. Their distribution may conform to mathematical models. Take a large sample of adult human beings and measure their height. Most cases will fall between five and six feet with very few less than four or more than seven feet. Because this is a continuous variable, the results can be plotted on a graph to which a curve may be fitted. It too will have a single peak with fan tails on the high and low ends. We call this the ‘normal’ distribution or popularly the ‘bell-curve’. For more than a century statistical inference has largely been based on this curve with its parameters of mean and standard deviation.
More recently, another statistical pattern has been making the headlines. If you count the book sales on Amazon and plot them according to frequency, the curve hugs the vertical and horizontal axes, indicating a few very large numbers (the blockbusters) and many small ones (the ‘long tail’ of books like yours and mine). If the same data are plotted on a log-log scale, the result is a straight line sloping down from left to right. This is a typical manifestation of something called a ‘power-law’ distribution. A similar formula describes the frequency of words used in natural language; and the distribution of molecular reactions in cells reveals a few hubs linked to most reactions and many weakly connected molecules.
The ‘new science of networks’, growing out of the physics of complexity, has been announced by authors such as Albert-Laszlo Barabasi and Duncan Watts. Just as, in the late nineteenth century, the normal distribution seemed to lend unity to statistical patterns emerging in a number of apparently unrelated fields, such as criminology, astronomy and plant genetics, now the power-law distribution appears in fields as disparate as the worldwide web, stock markets, air transport, Hollywood actors’ networks, electric power grids, urban hierarchies and molecular biology.
Empirical phenomena lend weight to the mathematical models used by statisticians, but their relative prominence in our collective imagination reflects how we experience society in history. Modern statistics took off a century and a half ago as a way of regulating people through enumerating them. Soon the normal distribution became the basis for the development of ‘parametric’ or inferential statistics. The very word normal says it all — conformity to a standard revealed by a central tendency, meaning that a population can be described in terms of an average type. The key assumption is randomness. This means that every member of a group has an equal chance of being selected. The democratic premise is obvious. This is an egalitarian as well as an atomistic model. Moreover, the quantities have to be measured on an interval scale, so that size is a continuous variable, not broken up into the separate classes of nominal or ordinal scales.
Does the recent rise to prominence of the power-law distribution, with its premise of extreme inequality, tell us something about our collective experience of society today? Power-laws have been known for some time. In the nineteenth century, when urban economy was relatively free of national controls, they were discerned in the dramatically uneven growth of cities. Later both Pareto and Zipf proposed something similar in the form of rank-order distributions, the one for income distribution and the other for word frequencies. Pareto is credited with discovering the 20/80 rule — the idea that 20% of the people own 80% of the wealth. But the premise of inequality contained in this rule was not adapted to the ideology of twentieth century society.
The ‘normal’ image of the natural and social world was credible because it reflected the premises of national societies formed to regulate industrial capitalism. In the last century, anthropologists transposed the idea of the nation-state as the typical form of society to ethnographic descriptions of so-called primitive societies, thereby demonstrating that the shared model of cultural homogeneity was universal. The power-law distribution is characterized by a few very large quantities and many small ones. In network science, it is commonly observed that networks consist of a few hubs with many links and a large number of weakly-connected nodes. The discovery of power-laws is related to the physics of complexity, the attempt to study interconnectivity in a non-reductionist way (as opposed to the isolated atoms of the random universe). This science is mainly concerned with the edge between order and chaos and with critical moments of transition, as when chaotic water molecules assume the rigid pattern of ice. It is now thought that self-organization, including life, flourishes in this interstitial zone.
Specialized study of networks in social science arose in the 1950s as a result of the development of graph theory in mathematics. The assumptions of this theory are now revealed to be unrealistic. It described an inventory of nodes whose number is fixed and remains unchanged throughout the life of the network. All nodes are taken to be equivalent and are linked together randomly. These principles of randomness, stasis and equivalence were unquestioned for forty years. Territorial states lent some credibility to networks configured in their own image. Thus road maps do not diverge markedly from the model, each centre having roughly the same number of links as the others.
Stanley Milgram conducted an experiment to see how many personal links would be needed to connect any two individuals in the United States. He found the median number of links was 5.5, hence the popular idea of ‘six degrees of separation’, that all humanity is connected by six links on average. This ‘small world’ phenomenon does not sit well with the assumptions of a random universe. Then it was discovered that most Hollywood actors were linked by two or three degrees to Kevin Bacon through appearances in the same movies. Mark Granovetter established that some individuals convey information between clusters of job-seekers. And the typical clustering of networks was modeled by Watts and Strogatz in 1998. But the basic assumptions of original graph theory still held. The key shift emerged with the recognition that some network nodes are hubs and some persons are ‘connectors’ . People vary widely in their ability to make social connection and in this they resemble an air traffic grid, with a few O’Hares and many small airports. Networks were now seen as linking nodes of unequal size and depending on a few highly connected individuals. But what produces this effect?
Barabasi established a fit between patterns of website links and the power-law distribution. These networks are ‘scale-free’ and lack the parameters of the normal distribution. There is no characteristic node in the continuous curve described by the power-law which reflects the fact that networks grow over time. Skewed distribution of links may be accounted for by ‘preferential attachment’, so that growth with preferences accounts for the hub phenomenon (early-comers tend to attract more links) and undermines graph theory’s key assumptions of randomness, stasis and equivalence. There is an analogy with the market principle that ‘the rich get richer’. Indeed in the network economy ‘winner takes all’. The winner is often unpredictable until one node crosses a threshold and takes off. The trick is then to find the threshold. When hubs are weakened, the network as a whole may be visited by ‘cascading failure’.
The convergence of world markets and the internet has multiplied opportunities for scale-free networks. If corporate hierarchy was well-suited to the era of mass production for national markets, the rise of a web or network model of economy involves a shift from vertical integration to flat, virtual integration, as Castells has long insisted. I have shown in a recent book that, when the money circuit is detached from real production and trade, the market is revealed as a weighted and directed network, with the mass of ordinary stocks following a few market leaders. Already the power-law distribution has been harnessed to predictive models based on analysis of the movement of the eight or so main stocks in a given sector. But, as we now know, this process is also inherently unstable.
Barabasi claims that nature normally hates power-laws. Hitherto physicists have found them near the critical point of phase transitions, as when a metal is magnetized. Even if it can be shown to be regular, power-law growth is unpredictable. Statisticians can only say that sometimes a variable crosses a threshold and then it takes off. We have been led to believe for more than a century that the bell-curve is preponderant in the physical world and this has helped to make prevailing social ideologies ‘natural’. European societies still largely hold to these ideologies. But the Americans have long held that income inequality is inevitable and today even the radical democratic wing of internet society, the bloggers and the peer-to-peer activists, tend to accept the fact of power-law distributions, claiming that as long as choices can be made freely (equal opportunity), this inequality is acceptable, one might say ‘natural’ or even ‘normal’.
This whole paradigm shift in scientific and statistical models coincides with the breakdown of the nation-state’s monopoly of society and with it the corporatist premises of twentieth century economy, such as jobs for life and social planning. For three decades neo-conservative liberals subordinated national economy to global markets; and the digital revolution has given us a new emergent model of society in the internet. The norm of this new world market was stark inequality. The egalitarian premises of nation-states, seeking to curb capitalism’s polarizing tendencies, gave way to a world society where the winner takes all. All of this has been thrown into stark relief by the economic crisis of 2008-9. But for now the power-law is king. It’s a different model of statistics, for sure. Perhaps it captures society poised between national and world forms. Or maybe we reverted temporarily to the imbalance between market and state typical of the Gilded Age, before national regulation aspired to curb domestic capitalism. The pressing political question for humanity, now given a new urgency by the collapse of the credit boom, remains whether new forms of association will enable us to harness the polarities of the network economy for common ends.
There is an objective basis of sorts for these statistical models in nature and society. But the one that attracts most attention in a given period is likely to reflect underlying tendencies in social experience. Having been raised in the heyday of British social democracy, only to face the new liberalism and now global economic catastrophe, I have had to internalize a radical paradigm shift and then to unlearn it sharply. Shifts in statistical models, with some degree of lag perhaps, offer a window on social history.
When I carried out fieldwork in Ghana during the 1960s, I was amazed by how migrants found their relatives, after traveling 500 miles to an unknown city of a million people. They had no addresses or phone numbers written down. When they arrived in the central lorry park, they would look for someone wearing Northern dress and ask him where they could find people like themselves. Directed to a particular district, they would seek out a leading figure in the ethnic community. They might then be directed to someone else from their home village. By all means, within an hour or two, they would be sitting with their relative. These African migrants knew that we live in small worlds connected by fewer links than most of us imagine. They used contingent human encounters and network hubs like local big men, not street maps. Their method was news to me then, but it shouldn’t be now.
References
Barabasi, A.-L. 2002. Linked: the New Science of Networks. Cambridge MA: Perseus.
Castells, M. 2001. The Internet Galaxy. London: Oxford University Press.
Durkheim, E. 1965 (1912). The Elementary Forms of the Religious Life. Glencoe IL: Free Press.
Durkheim, E. and M. Mauss. 1963 (1903). Primitive Classification. Chicago: University Press.
Gerratana, V. 1973. Marx and Darwin. New Left Review 82: 60-82.
Gladwell, M. 2000. The Tipping Point: How Little Things Can Make a Big Difference. New York: Little, Brown.
Granovetter, M. 1973. The strength of weak ties. American Journal of Sociology 78: 1360-1380.
Hacking, I. 1990. The Taming of Chance. Cambridge: University Press.
Hart, K. 2000. The Memory Bank. London: Profile. Republished 2001 as Money in and Unequal World. New York and London: Texere.
Lévi-Strauss, C. 1966 (1962). The Savage Mind. Chicago: University Press.
Mauss, M. 1990 (1925). The Gift: form and reason of exchange in archaic societies. London: Routledge.
Milgram, S. 1967. “The small world problem.” Psychology Today 2: 60-67.
Pareto, V. 1972 (1906). Manual of Political Economy. London: Macmillan.
Spengler, O. 1962 [1918]. The Decline of the West (abridged edition). New York: Alfred Knopf.
Taleb. N. 2007. The Black Swan: The Impact of the Highly Improbable. New York: Penguin.
Watts, D. 2003. Six Degrees: the Science of a Connected Age. London: Heinemann.
Watts, D. and Strogatz. 1998. Collective dynamics of ‘small-world’ networks. Nature 393: 440-442.
Zipf, G. 1949 Human Behavior and the Principle of Least Effort. Cambridge MA: Addison-Wesley.
Comments |1|
You must be logged in to post a comment.
[…] men, not street maps. Their method was news to me then, but it shouldn’t be now. from “Models of statistical distribution,” by Keith Hart, The Memory Bank, 27 January 2009 :: via Koranteng’s […]