Studies supporting mismatch theory are replete with 'demonstrably incorrect' assumptions, law prof says
Sherod Thaxton a professor at the University of California at Los Angeles School of Law. Photo by Todd Cheney/The University of California at Los Angeles School of Law.
Studies supporting an “academic mismatch” theory claim that students are harmed by racial preferences, but the data doesn’t support that assertion, according to a professor at the University of California at Los Angeles School of Law.
The professor, Sherod Thaxton, spoke with the ABA Journal in an email Q&A about the shortcomings that he sees in the research by his UCLA colleague Richard Sander, who is also a law professor.
Sander said in a recent blog post his study of about 6,500 law students at four law schools provides strong support for his academic mismatch theory—that law students with lower qualifications than their peers fall behind and have worse outcomes in a learning environment geared toward better-qualified students.
The study was conducted by Sander and Robert Steinbuch, a professor at the University of Arkansas at Little Rock William H. Bowen School of Law. According to Sander, the study found that “a student’s degree of mismatch in law school is by far the strongest predictor of whether he or she will pass a bar exam on a first attempt.”
Thaxton says the study has methodological flaws. His answers to the Journal’s questions have been shortened and lightly edited by the Journal.
Q. In an amicus brief submitted to the U.S. Supreme Court, you and other scholars assert that Sander’s mismatch theory is “bad science,” and his work has “serious methodological flaws.” Are you essentially saying the theory is bunk? Or that it makes assumptions that can’t be proven by the available data? Or both?
A. Studies claiming empirical support for mismatch theory are replete with both demonstrably incorrect and highly implausible assumptions, and there is no detectable effect of mismatch when these errors in assumptions are corrected. And this is not merely a matter of reasonable disagreements among experts about assumptions—rather, studies claiming support for mismatch theory routinely misapply basic principles of statistical inference that enjoy virtually universal acceptance.
Q. You have published a critique of law school research by Sander and Steinbuch in a 2022 paper at SSRN. In that paper, you mention findings that Black and Latinx students have lower law school grades, independent of undergraduate grade-point averages and LSAT scores. Why does that weaken the mismatch theory? Is racial discrimination affecting grades, which in turn affects bar passage? Are there other variables that might be at work?
Image from Shutterstock.
A. The core claim of mismatch theory is that Black and Latinx students learn less in law school than they would have if they attended a less competitive law school, in the absence of affirmative action, because “large admissions preferences create big credential gaps between students who receive preferences and their classmates.” This has been labeled “learning mismatch.” Mismatch theorists also posit that legal knowledge/training obtained in law school is primarily responsible for success on the bar exam. That is, mismatch meaningfully impacts learning in law school, and learning in law school meaningfully impacts the likelihood of success on the bar exam.
According to the theory, after accounting for a student’s degree of mismatch, differences in performance metrics that correlate with race—such as law school grades and the likelihood of bar passage—disappear. Yet nearly every credible study conducted both at specific law schools and on groupings of law schools have revealed one (or more) of the following: (1) Black and Latinx law students receive lower law school grades than similarly situated white law students, (2) entering credentials (i.e., undergraduate GPA and LSAT) account for less than a quarter of the variation in law school grades, (3) entering credentials and law school GPA only explain a modest proportion of the variation in graduation and bar passage rates (i.e., less than one-third), and (4) part of the effect of race on bar passage rates is “indirect”—that is, it operates through law school grades, independent of any credential gap. These empirical findings undermine this hypothesized causal chain of events articulated by mismatch theory along four important axes.
First, something other than a credential gap is causing Black and Latinx students to receive lower grades than their white student counterparts. Second, something other than entering credentials is predominantly responsible for differences in law school GPA among otherwise similarly situated students. Third, something other than entering credentials and legal knowledge/legal reasoning skills (proxied by law school GPA) is predominantly responsible for differences in graduation rates and bar passage rates. Fourth, a proper test of the impact of race on bar passage rates must disentangle the direct and indirect effects (through law grades) of race on bar passage rates.
There is evidence that race exerts a significant overall effect on the likelihood of bar passage that cannot be explained by mismatch theory. Perhaps the race effect is indicative of a “race tax” paid by nonwhite students (particularly Black and Latinx students, where the race effects are strongest), and this tax is the price nonwhite students pay in their encounters with white students (and white professors and white staff) because of racial stereotypes. And while it may not always be rooted in conscious racial animus, the cumulative impact of constant experiences of racial stereotypes and the energy expended dealing with them has been described by one analyst as “soul-destroying.”
Q. The Supreme Court amicus brief cites criticism that Sander makes “indefensibly strong assumptions” that aren’t supported by his data. What kind of study would be needed to show—or disprove—that race-conscious admissions policies cause Black students to fail the bar exam at higher rates?
A. Studies concluding that race-conscious admissions policies cause lower rates of Black and Latinx student bar passage suffer fundamental flaws because they do not adhere to the agreed-upon methods for making such causal inferences. Properly designed and executed randomized experiments are the “gold standard” for causal inference. Conducting a randomized experiment to test mismatch theory would require randomly assigning otherwise identical prospective students into differently ranked law schools and then compare these students’ performance on the bar exam in order to infer any causal effect of mismatch. But this type of experiment is infeasible because very few prospective law students, of any race, would forgo the opportunity to enroll in a higher ranked (i.e., more elite) law school, all else equal.
Consequently, scholars must conduct observational studies, and causal inference using observational data must be carefully designed to approximate a properly designed and executed randomized experiment. Observational studies are retrospective because analysts have access to the outcome data, potentially allowing them to tailor their studies to generate the preferred result, typically after examining many different statistical models that all vary with respect to how the variables are coded, what variables are included, and what subset of the data were analyzed.
Fortunately, properly conducted quality control facilitates the examination of the consequences of these various assumptions on the result. Of course, no social scientific study is perfect, but credible causal inference entails using the best available methods and data, and equally important, it entails awareness of, and transparency about, the limitations of the study. It is impractical, given the space limitations of our Q&A, to provide a more nuanced discussion of the conditions necessary for establishing causal claims using observational studies; however, I do so in a prior article, “How Not to Lie About Affirmative Action.”
Q. Is mismatch theory wrongly informing the debate over race-conscious admissions policies? Does the data actually support such policies? Or is the data inconclusive?
A. The available data indicates that affirmative action policies yield positive outcomes for their intended beneficiaries. Multiple analysts who examined the impact of affirmative action at both the undergraduate and graduate level discover some evidence of a “reverse mismatch effect” for Black and Latinx students—that is, affirmative action increases the likelihood of graduation, receiving a higher income, etc. There is also a vast literature concerning employment outcomes that contradict mismatch theory. Specifically, affirmative action does not adversely impact employee performance and compensation. These analysts have been careful to qualify their conclusions, given the limitations in available data, which is something that analysts who purportedly find support for mismatch theory have failed to do.
Q. Do you have anything else you would like to add?
A. At the center of the debate over mismatch theory are divergent views about the effects of race-conscious admissions policies on human behavior and not merely normative disagreements arising from different ideologies. In other words, this a matter of empirical claims and assumptions that are testable, data-driven, and that gain strength from the quality of the studies that adhere to accepted methods among experts seeking to advance knowledge in the face of uncertainty and solve problems about the world we live in. The consistent critique of the various studies reporting support for mismatch theory is their vast array of recurring conceptual and methodological shortcomings. The public debates over the appropriate methods to use, appropriate assumptions to make and appropriate inferences to draw are designed to identify errors, incentivize learning from those errors and improve knowledge. Given the U.S. Supreme Court’s pending decision in the Harvard and University of North Carolina affirmative action cases, and its potential impact on both legal and nonlegal education, it is vital that rigorous empirical research continues to inform the discussion about race-conscious admission policies. Thank you for the opportunity to contribute to this discussion.
