To Take the Measure of Lives: The Metrics and Morals of Quantifying Police Violence

“HARVARD ECONOMIST’S STUDY Finds No Racial Bias in Police Shootings,” the headline from Fox News Insider proclaimed. A chorus of conservative media sources peddled similar takes. Even mainstream outlets like The New York Times covered the controversial findings. As the nation reeled from a harrowing month of racially charged police shootings in Baton Rouge, Minneapolis, and Atlanta, an eminent social scientist, Roland Fryer, was challenging allegations that African Americans find themselves disproportionately in the crosshairs of law enforcement.

Fryer is a 39-year-old MacArthur grantee and John Bates Clark medalist with a penchant for applying quantitative methods to social problems. His previous research has analyzed topics ranging from educational policy to hate groups: the impacts of crack cocaine on neighborhoods, the persistence of gender and racial achievement gaps, the consequences of “acting white,” discrimination against job seekers with distinctively black names, and the funding of white supremacist groups. An interest in the black experience in the United States animates his work, but Fryer emphasizes empirical analysis and data-rich prescriptions over impassioned appeals for justice. As Stephen J. Dubner wrote in a profile of the economist, he “has a huge appetite for advocacy, but a far larger appetite for science.”

Fryer found the question of whether law enforcement officers are more violent with minority suspects than their white counterparts irresistible. As a black academic who grew up in challenging circumstances and experienced his share of unpleasant encounters with cops, he had a personal stake in the project. Crunching the numbers appealed to him more than grabbing a picket sign. As he compiled data, he asked searching questions. Was there a quantifiable pattern of racial bias in police violence? If so, how could this data inform public policy?

After spending some time observing cops on the beat in New Jersey and Texas, Fryer and his team collected and coded a staggering number of archived police reports. The data they trawled represented nearly five million entries from New York’s Stop, Question, and Frisk program, and another 1,300 records of police shootings from 10 cities, including Houston and Los Angeles. Fryer focused particular attention on 6,000 incident reports from Houston recording confrontations in which officers were justified in discharging their firearms but did not; these chronicles of restraint threw the split-second deliberations that lead to police shootings into sharp relief.

Using this trove of information, the study reached two interlocking conclusions. Police used more coercion with blacks and Latinos than whites, but were equally likely to fire on blacks and whites. The latter finding was a revelation, particularly given the publicity surrounding the deaths of African Americans at the hands of law enforcement officers over the past two years. In Houston, police were 23.8 percent less likely to discharge weapons when the suspect was black. Fryer had expected to uncover unmistakable evidence of bias against minorities in police shootings. What he found instead was “the most surprising discovery of [my] life” — that racial bias seemingly vanishes when police officers pull the trigger.

Unsurprisingly, this counterintuitive result has led skeptics to question the activism surrounding police shootings. Columnist Tom Ciccotta muses, “Perhaps the Black Lives Matter movement should be asked to reconcile these revelations with their loud response to the tragic shooting deaths of Alton Sterling and Philando Castille that occurred last week.” His colleague at The New American was even less charitable. Under the headline, “Study by Black Harvard Economist Refutes Black Lives Matter’s Claim,” Bob Adelmann charges that Fryer’s research “puts the whole ideology behind the Black Lives Matter movement into question.” Jonathan Tobin, over at Commentary, suggested the subversive implications of these findings for contemporary activism:

The notion that blacks are at risk from police fits in nicely with liberal myths about law enforcement […] [a] lie that has kept a group like Black Lives Matter going with its destructive agenda that has led to anti-police violence …

[That agenda] is actually costing African-Americans their lives.

There is more than one way to frame the study’s findings, of course, and the reporting of Fryer’s research in the media functions as an ideological litmus test. If perceptions about black lives at risk fit nicely with liberal myths about law enforcement, then intuitions about officers being impartial in their use of force fit nicely with conservative myths about law enforcement. Nothing about the unjustified killings of black men in Baton Rouge, Minneapolis, and Atlanta needs to be reconciled with this data; the numbers could never excuse these deaths or exculpate the responsible officers. The insinuation that, in light of these statistics, protestors need to justify their outrage at the shootings of innocent victims is a non sequitur.

Media fixation on lethal force is understandable, but this represents a single dimension of the panorama Fryer’s study presents. More dispositive, though less surprising, than his conclusions about police use of lethal force is his evidence that officers are quicker to resort to physical confrontation and to draw weapons when the suspect is nonwhite. Latinx and black subjects were more than 50 percent more likely to encounter violent escalation at the hands of law enforcement. Controlling for factors like resistance, Fryer found African Americans were still 17.3 percent more likely to elicit a show of force from cops. If his numbers are reliable, it is time to retire the canard that blacks have nothing to fear if they heed officer’s directions: police threatened or restrained compliant African Americans 21.1 percent more frequently than they did their white counterparts.

Moreover, this dimension of the study is a more damning indictment than the conclusion about shootings in one respect. As Benjamin Wallace-Wells points out, officers fire shots only when they have lost control of the situation; however, they have significantly more control over the situation when they apply lesser forms of coercion. An equally plausible headline for The New American could therefore declare, “Study by Black Harvard Economist Proves Black Lives Matter’s Claims.”

Suppressing some facets of this research while amplifying others games the results at the expense of seriously grappling with the implications of a pioneering study. Fryer and his team produced the most substantive survey to date on police violence. They painstakingly inventoried the most current available evidence to quantify and analyze a phenomenon that has raised concerns throughout the country. Their unsettling conclusions demand critical attention.

The reliability of a study like Fryer’s turns on the quality of its sources. Given that police forces exist to promote public safety, it surprises many that data collection on police shootings — particularly shootings that do not result in fatalities — remains surprisingly inconsistent, incomplete, and uncoordinated. There is no dedicated government entity with a mandate for standardizing this reporting. Nor are there uniform criteria for recording incidents in which law enforcement officers discharge weapons. What evidence can be gleaned is self-reported, produced for local municipalities rather than federal agencies.

Aside from variation, significant questions remain about the objectivity of the reporting. Although police reports furnish important data for research on police violence, they are not always as devoid of bias as their detached tone suggests. When violence escalates, the likelihood of elliptical reportage and coded justifications increases. “What’s there is crappy data,” David Klinger, a criminal justice scholar at the University of Missouri, lamented. So unreliable were statistics in the protean category of “legitimate homicides” that the Department of Justice stopped reporting them in 2009. Fryer imposes some regressive controls on these variables, but his findings still rely on the premise that police reports are fundamentally reliable and relatively exhaustive. He has maintained that the outcomes remain the same irrespective of whether one finds the police framing persuasive or obfuscatory. That said, much still depends on the accuracy of records provided by law enforcement. Omissions and selective detail can skew the data, raising concerns about its reliability. In this respect, Fryer’s study is hardly singular. He has to reckon with the same evidentiary vulnerabilities as all such studies based on self-reported data.

Some of the most pivotal evidence for this study comes from the incident reports from Houston. This reporting of confrontations in which officers are authorized to use deadly force but refrain from doing so provide an important statistical context for police shootings. By comparing decisions to open fire with decisions to hold fire, the researcher can more readily identify racial bias. Consider what happens when one removes this information from the calculations. One could only count the bodies and compare them to the number of incidents, using different filters to accommodate the variables. It would be impossible to know if officers fired in every situation where a threat warranted shooting or if they seldom resorted to force even in the most dangerous circumstances. Houston’s incident reports thus supply critical data, revealing a police force’s propensity to shoot in situations of danger and uncovering hidden patterns in those decisions.

Fryer’s close reading of Houston’s data provides valuable granular information but limits its global application. To generalize from his findings, one must assume that the rest of the United States looks a lot like Houston. That is a dubious assumption, to put it mildly. Police departments vary as significantly as the cities they exist to protect. A police force may, like Houston’s, mirror the community they serve demographically; alternatively, it may look as different from its citizens as Ferguson’s did at the time of the Michael Brown shooting. Training may emphasize preemptive fire or restraint in cases of open conflict. The department may follow the protocols of “Broken Windows Theory,” cracking down on even minor infractions, or may focus only on more serious crimes. Most relevantly, few departments collect and report data as methodically as Houston does. Taking Houston as a statistically representative sample, then, is risky business, a selection bias Fryer himself concedes.

Indeed, when anthropologist Cody Ross researched police shootings using a crowd-sourced database at Deadspin, he found significant regional disparities in racial bias. Departments in conurbations like Los Angeles, New Orleans, and Miami-Dade shot disproportionate numbers of black and Latinx suspects. Ross arrived at a strikingly different conclusion from Fryer: victims of police shootings were three times more likely to be black and unarmed than white and unarmed.

These apparently incongruous findings can be reconciled by considering the different ways the researchers go about measuring the role of race in police shootings. Ross measures racial bias differently than Fryer does because he is asking a different question. As Rosa Li summarizes his approach in Slate: “For people shot by the police, what is the relative likelihood that they are black versus white and armed versus unarmed?” Unlike Ross, Fryer has little interest in calculating rates of shooting by race. On that count, there can be no debate. In an astringent critique, Justin Feldman points out that, even in Houston, Latinx and black residents are two and five times more likely, respectively, to be shot by police than their white counterparts. But Fryer is after different quarry. He purports to identify “racial bias” by investigating whether these shootings exceed the limits of statistical discrimination based on race.

The term “statistical discrimination” requires explanation. Suppose that, in a given municipality, Latinx drivers are 20 percent more likely to drive without a license. To an economist, that means police officers have rational justification to pull over 20 percent more Latinx motorists than whites to demand identification. Anything eclipsing that benchmark invites charges of racial profiling. Using this metric, Fryer seeks to identify racial bias by determining whether police shootings of black and Latinx citizens occur at rates that transgress the bounds of statistical discrimination.

Employed properly, statistical discrimination can become a powerful tool for detecting unfair racial profiling. Perhaps the most prominent recent case involved a legal challenge to New York City’s “Stop, Question, and Frisk” program. A study co-authored by Andrew Gelman, Jeffrey Fagan, and Alex Kiss revealed that, using arrest rates by race from the previous years as the criterion for statistical discrimination, blacks were stopped 23 percent more and Latinx 39 percent more than was warranted. Their findings furnished evidence for a trial that ended this regime of stopping and frisking residents of minority communities.

Yet deploying the concept of statistical discrimination in this way remains controversial because it fails to account for systemic racism. Fryer concerns himself only with what happens once police have confronted a suspect. But numerous studies have documented how racial discrimination colors individual officers’ decisions to accost individuals in the first place. The more police target minorities for routine questioning, the higher the total number of minorities charged — even if the rates of arrest for these minorities are actually lower. Arrests from these stops can create their own momentum, propelling confirmation bias by raising the threshold for statistical discrimination. For this reason, critics such as Feldman have criticized the Harvard study’s methodology as inappropriate to the phenomenon under investigation.

More confrontations for lesser offenses also have the effect of lowering the perceived threshold for threat in the aggregate. Uri Simonsohn illustrates this principle with the video of a police officer subduing an African-American teenager at a pool in suburban Dallas that went viral a year ago. The officer’s aggression reflected an overestimation of threat: unarmed teenagers in bikinis pose little danger to public safety. On average, the threshold for perceived threat minorities pose diminishes. Such disproportionate responses lead to coercion and arrests of less dangerous people. But they also entail that, unless the researcher effectively controls for threat, more arrests for lesser offenses means fewer shootings as outcomes. The pool has become diluted — “with criminals and girls in bikinis” grouped together, as Simonsohn puts it. Perversely, then, the presence of bias creates the impression of a lack of bias in lethal confrontations. This diluting effect tallies with the results of Fryer’s research, and helps to explain the “surprising” findings of more violent encounters and fewer police shootings of racial and ethnic minorities.

So what difference does this study make for Latinx and black lives? Narrowing the focus to what happens once an officer stops a minority, as the Harvard study does, can obscure the larger context. This is true even if one accepts the validity of its findings. An analogy might help. Suppose a physician conducting a routine screening informed you that if you contracted the particular strain of cancer he was looking for, your treatment would involve more invasive therapies and destructive side effects, but that your chances of survival were similar to others in this condition. That might appear good news. But what if the physician then reminded you that you were 23 percent or 39 percent (to use the figures from the New York study) more likely to be diagnosed with this cancer than others? That would make you feel considerably more vulnerable.

Statistics are the currency of science, the regnant standard for human knowledge. Claim that lives are at stake, and this is an opinion; lard this assertion with numerical evidence, and your argument carries the signature of objectivity. Yet numbers, like words, require context for their meaning. Statistics are based on perspective, selection, and other limitations that demand careful interpretation. Different communities translate this data into narrative accounts and normative commitments, and power relations stamp the creation of policy from data. In many ways, as philosopher James Blachowicz points out, this “scientific” method of dealing with information looks more like poetry than positivism: we conscript the data to fit our task, not to discover immutable laws of nature or engineer human behavior. The reportage surrounding Fryer’s working paper exposes the plasticity of statistical texts in the hands of various interpretive communities — from activists taking these numbers as evidence of police brutality to their critics marshaling it as evidence of activist overreach. We need to handle these deployments of statistics with more critical awareness and sensitivity to context.

Fryer’s study makes an important contribution to the problems of police use of force against minorities, but it raises as many questions as it answers. New findings may complement or complicate his conclusions. In the meantime, partisans on the left and right will continue to enlist his research to support their own agendas, using statistics to lacquer their arguments with the veneer of respectability. These numbers have much to tell us, and they should be instrumental in formulating policy. But we should also be cautious in mistaking them for our only access to the real, should we lose sight of the personal experience behind the statistics. The visceral response that arises when we see faint specters of officers firing on innocent black and brown bodies on video, and the dread minorities feel when they see blue and red lights in their rearview mirrors, are present in a way that an actuarial calculus can never capture. Black and Latinx residents inhabit a world in which racial profiling and police harassment are palpable realities, not mere statistical abstractions. If black lives matter, they must matter more than numbers on a ledger sheet, or in a research abstract. As we deliberate strategies of policy and protest, we must value their voices, their rhetoric, and their experiences as part of a thick description that includes our best data.


Ryan T. Woods teaches at Georgia Gwinnett College and serves as associate editor at Marginalia Review of Books, a channel of the Los Angeles Review of Books.