FEBRUARY 7, 2013
THE NOTED SOCIOLOGIST and legal scholar Richard Lempert has warned, “Nothing is so helpful as good empirical research and nothing can be so bad as poor research that becomes influential.” Richard Sander (a UCLA law professor) and Stuart Taylor Jr. (a journalist/lawyer) describe their book Mismatch as “mostly an empirical book.” It should be evaluated as such. Sander and Taylor expend considerable effort to make Mismatch influential, with the book’s release timed the day before the US Supreme Court heard arguments in the affirmative action case Fisher v. University of Texas at Austin and countless op-eds and so on. The attention is more than unfortunate, for the book is highly misleading.
Sander and Taylor’s “mismatch” hypothesis asserts that affirmative action harms its beneficiaries by placing them in settings where they cannot compete academically and so learn less and perform less well than they would in less selective institutions. This thesis has been advanced over the years by a number of affirmative action critics, but none of these earlier efforts were as ambitious as Mismatch in attempting to claim the mantle of social science evidence.
Affirmative action in American law schools is a major theme in Mismatch, and Sander’s 2004 article “Systemic Analysis,” a study of black law students published without peer review in a student-edited journal, is the fountainhead for much of what Sander and Taylor say. Even after eight years and voluminous, empirically based criticism, Sander (with Taylor) sticks to his guns in Mismatch:
[A]ll the factual claims and the data presented in “Systemic Analysis” withstood all scrutiny. All of its tables, models, and analyses were replicated. […] [T]he debate (such as it was) concerned only the inferences I drew from the facts and models I presented.
To borrow from Mark Twain, there is evidence here of a “high talent for inaccurate observation.” Sander and Taylor use the concepts of “replicated” and “withstood all scrutiny” in ways that strain credibility and depart from social science norms. To give but one example, in “Systemic Analysis” Sander used Linda Wightman’s 2001 data to argue that ending affirmative action would only decrease African-American enrollment in law school by 14 percent, and this forms the basis for his climactic conclusion that ending affirmative action would actually increase the number of African Americans who become lawyers. But many scholars (including Wightman) have been careful to point out that this “grid model” paints an unrealistically rosy picture, because it is untethered to the schools where candidates actually apply (a logistic regression model taking this into account projected a 38 percent national decline). In fact, when my co-authors and I looked closely at Sander’s table, it was evident that he violated the very grid model methodology he espoused by deleting all students with the very lowest credentials, effectively puffing up his post–affirmative action performance estimates. Other scholars reject Sander’s blending of 2001 admissions data with bar passage data from students entering law school in 1991. Sander and Taylor repeat this flawed set of claims in their book.
Replication concerns in Mismatch are not, however, confined to “Systemic Analysis.” Sander and Taylor claim that the University of Michigan Law School’s bar passage rates for underrepresented minority graduates were “greatly exaggerated” in an acclaimed article by Lempert, Chambers, and Adams that was part of the record in the landmark Grutter v. Bollinger ruling. Their claim is advanced to support the dark suggestion that authors such as Lempert “would have a powerful motive for keeping this data secret.” (Note a performative motif in Mismatch that wears thin because of the misleading evidence: Sander and Taylor position themselves as nonpartisans reluctantly driven to conclusions by virtue of the overpowering evidence — a posture Sander repeats from earlier scholarly debate about “living wage” ordinances, where labor economists likewise failed to replicate his findings and found his methods unsound.) In another article, Sander claimed that only 76 percent of the University of Michigan’s black graduates had passed the bar, but Lempert, using a more complete and accurate data set, conservatively estimated the black student bar passage was 91 percent. He shared his results with Sander long before Mismatch was set in print. Throughout the book, Sander and Taylor refuse to acknowledge a failure to replicate when it challenges their thesis.
Dan Rubenfeld, a leading applied econometrician, instructs judges and lawyers in the Reference Manual on Scientific Evidence that when evaluating statistical evidence such as regression analyses they should ask: (1) “Has the expert provided sufficient information to replicate the multiple regression analysis?” and (2) “Are the expert’s methodological choices reasonable, or are they arbitrary and unjustified?” It is on these bedrock questions — regarding reasonable methodological choices, defensible assumptions, and the importance of having one’s findings validated by others — that Sander and Taylor’s mismatch argument completely fails. Putting aside my own critiques, co-authored with Chambers, Clydesdale, and Lempert (2005, 2006), here are the bottom line conclusions by methodologically sophisticated scholars who have carefully examined Sander’s data and analysis:
- Ayres and Brooks (2005): “Sander’s conclusion that these disparities [on the bar exam] are dominantly or solely caused by affirmative action does not withstand closer analysis. To the contrary, we have shown by looking at the actual achievement of blacks […] that affirmative action mitigated these racial disparities and that even more affirmative action would have been likely to produce more black lawyers.”
- Camilli and Jackson (2011): “[T]his study has shown that regression analyses of the kind conducted by Sander are incapable of producing credible estimates of causal effects.”
- Holzer and Neumark (2006): “Sander presents interesting data and a provocative argument. But the empirical case for the mismatch hypothesis in law schools has not been made.”
- Rothstein and Yoon (2008): “There is no plausible interpretation of the data under which the elimination of affirmative action would increase the number of black lawyers […] Rather, a shift to race-blind admissions would have reduced the number of blacks from the cohort studied here who became lawyers by over 50 percent.”
- Ho (2005): “In short, whichever way one cuts it, there is no evidence for the hypothesis that law school tier causes black students to fail the bar.”
If this isn’t enough to make a reader skeptical of Sander and Taylor’s claims, the upshot of their amicus brief in the Fisher case certainly should do so. It triggered a responding brief — from nearly a dozen of America’s leading social scientists and methodologists, including two members of the National Academy of Science and others without known prior positions on affirmative action — that represents an unflinching rebuke by social science heavyweights:
Sander’s research has major methodological flaws — misapplying basic principles of causal inference — that call into doubt his controversial conclusions about affirmative action. The Sander “mismatch” research — and its provocative claim that, on average, minority students admitted through affirmative action would be better off attending less selective colleges and universities — is not good social science. […] That research, which consists of weak empirical contentions that fail to meet the basic tenets of rigorous social-science research, provides no basis for the [Supreme] Court to revisit longstanding precedent supporting the individualized consideration of race in admissions.
Sander and Taylor are similarly unpersuasive when they discuss affirmative action and college graduation rates. They begin with a vague argument that Bowen and Bok’s pathbreaking affirmation of affirmative action in The Shape of the River “can be just as easily read as confirmation of mismatch,” and promise a detailed critique of Bowen and Bok on the Mismatch website. To date (three months after publication) no refutation has appeared. They then cite two broad-based studies, one by Light and Strayer and the other by Loury and Dratcher, that purportedly show that affirmative action harms African Americans’ overall prospects for college graduation. However, the former study is based on 1979 data and the latter on students entering college in 1972. Moreover, two years after the study Sander and Taylor cite, Light and Strayer reexamined their data in a more focused investigation and concluded their results are consistent with the conclusion that affirmative action works successfully in combination with retention programs.
As for Loury and Dratcher, Thomas Kane points out that their study shows college selectivity associated with higher black graduation rates in predominantly white institutions. These results are the opposite of what mismatch theory would predict. An overall negative association existed only because of high graduation rates at historically black institutions, rates plausibly due to campus climate, “critical mass,” or other variables having nothing to do with mismatch. The very same problem plagues Sander and Taylor’s effort to salvage evidence of law school mismatch by comparing black law students attending their “first choice” versus “second choice” law schools — they rely on an unpublished study by Sander’s “old friend” Doug Williams where the results are driven by a handful of historically black law schools that are starkly dissimilar from the other 95 percent of US law schools.
Sander & Taylor claim that most studies understate mismatch because they don’t account for an important fact: students at more elite institutions have been selected not only on the basis of routinely captured variables like test scores and grades but also on qualities like tenacity, qualities that are not captured in statistical models. In other words, unobserved characteristics can be a confounding factor when attempting to analyze the impact of affirmative action, especially when comparing the performance of students at elite institutions to students at other institutions. Sander was slow to embrace this point about unobserved characteristics, as it is inconsistent with positions he staked out in “Systemic Analysis.” In any event, in Mismatch he and Taylor argue that the evidence that affirmative action beneficiaries obtain benefits by attending elite institutions is “surprisingly weak.” In this vein, they applaud a statistical matching study by Dale & Krueger as a clever and effective way to overcome the selection bias associated with this problem of “unobserved characteristics.”
Not only is this a selective reading of Dale and Krueger’s original findings — they found that though there were too few African Americans to test separately, “disadvantaged” students benefited more by going to elite schools — Dale and Krueger followed up their original research by acquiring data on a larger sample in 2011. Repeating their analysis, they concluded that other things being equal, African Americans and Latinos enjoyed “large” benefits in the labor market from enrolling at highly selective colleges. Sander and Taylor’s decision not to mention these facts is significant both substantively and in what it says about the trustworthiness of the rest of their report.
The Dale and Kruger findings on earnings are consistent with a large body of other peer-reviewed research — including studies sensitive to selection issues — which show that attending a more selective college or university is associated with net gains in African Americans’ and Latinos’ college graduation rates. Unlike the claims made in Mismatch, this conclusion is strengthened by the convergence of findings from different investigators using multiple data sets and a range of analytical methods. Relevant studies include Alon and Tienda (2005); Small and Winship (2007); Fischer and Massey (2007); Melguizo (2008); Bowen, Chingos, and McPherson (2009); and Cortes (2010). No wonder then that in Fisher the American Education Research Association, the National Academy of Engineering, and other top research associations filed a brief concluding that:
[The] educational harms resulting from the so-called “mismatch” of minority students at selective institutions have not been established by the studies said to prove them and indeed are regularly contradicted by sounder and more widely accepted research.
Mismatch also aggressively attempts to trumpet the positive effects of affirmative action bans, such as Sander and Taylor’s claim that California’s Proposition 209 ushered in a “warming effect” that meant “strong black and Hispanic students accepted UCLA offers of admission at much higher rates.” This claim is based on methodologically questionable statistical adjustments that obfuscate this stubborn fact about freshmen admitted to UCLA: in the four years prior to Proposition 209, 24 percent of the African Americans in the top third of the admit pool chose to come to UCLA. In the first four years after Prop 209 the yield rate plummeted to 8 percent. There were less extreme drops in the middle-third of UCLA’s admit pool, and in the top third of black admits at the other UC campuses. In short, after Proposition 209 a larger share of top black students admitted to UC campuses chose to reject offers from UC in favor of selective private universities that utilized affirmative action (and likely maintained higher levels of “critical mass” in the student body).
In Mismatch Sander and Taylor delve into a fundamental, important issue where our values (individually and collectively) may be in conflict. Some of these differences cannot be resolved empirically, but other clashes can be powerfully illuminated by sound social science. Their thesis is not supported by the relevant body of peer-reviewed social science, and that “mismatch” does the debate about affirmative action — and the country — a great disservice.