Last Name Selection in Audit Studies

In this article, we build on Gaddis (2017a) by illuminating a key variable plausibly related to racial perceptions of last names—geography. We show that the probability that any individual belongs to a race is conditional not only on their last name but also on surrounding racial demographics. Specifically, we demonstrate that the probability of a name denoting a race varies considerably across contexts, and this is more of a problem for some names than others. This result has two important implications for audit study research: it suggests important limitations for (1) the generalizability of audit study findings and (2) for the interpretation of geography-based conditional effects. This means that researchers should be careful to select names that consistently signal racial groups regardless of local demographics. We provide a slim R package that can help researchers do this.

S INCE Bertrand and Mullainathan (2004)'s pioneering study, hundreds of re- searchers have conducted audit studies to investigate the extent of racial (or ethnic) discrimination in America across myriad contexts (Crabtree 2017). 1 The results from these studies have done much to advance scholarly research on discrimination across the social sciences (Crabtree and Fariss 2016; Pager and Shepherd  2008).For this reason, articles that center on audit studies often garner tremendous attention within the research community and even in the broader public, where they have helped deepen public understanding about the serious barriers that racial (or ethnic) minorities face in nearly every aspect of common life.
There are reasons, however, to be at least somewhat concerned about the findings from these studies. 2 One of the largest concerns relates to the first and last names that researchers use to signal racial (or ethnic) identities in America (Butler and  Homola 2017).Researchers typically select these names based on either (a) those used in prior audit studies or (b) government-provided lists that contain populationlevel statistics of use across races (Gaddis 2017a).The potential problem here is that scholars often ignore the extent to which these choices accurately map onto how individuals perceive names.In his welcome and long-overdue study on that subject, Gaddis (2017a) demonstrates that current practices do not acknowledge or take into account the many different factors that shape public perceptions of given names.His findings have obvious implications for how scholars choose first names to represent racial identities.
Perhaps one of the most important findings from Gaddis (2017a) is that individual perceptions of first names can change considerably depending on the last names with which they are paired.Specifically, he shows that the last names researchers use in their audit studies can substantially strengthen (or weaken) the extent to which individuals perceive first names as belonging to specific racial (or ethnic) identities.In a separate, related article, Gaddis presents additional evidence that this is the case (Gaddis 2017c).This means that researchers should think carefully about the last names they pair with first names.
Unfortunately, as Gaddis (2017a) acknowledges, researchers often do not.He and others typically classify last names by race based on population-level usage statistics (Butler 2014; Butler and Crabtree 2017; Gaddis 2017a,c).This practice, however, relies on the assumption that individual racial perceptions are in line with the country-wide popularity of last names among racial groups.
In this article, we contend that this assumption ignores the crucial importance of geography and local demographics.We show that the probability that any individual belongs to a race is conditional not only on their last name but also on surrounding racial demographics.This result has two important implications for audit study research: it suggests important limitations for (1) the generalizability of audit study findings and (2) for the interpretation of geography-based conditional effects.This means that researchers should be careful to select names that consistently signal racial groups regardless of local demographics.
This article proceeds as follows.First, we describe how researchers typically select last names for audit studies, laying bare a key assumption behind these choices.Second, we explain why this assumption is too restrictive in many cases, as it requires individuals to ignore information about local demographics when inferring the race of an individual based on their name.Third, we present and justify an alternative assumption that allows individuals to combine information about name popularity and racial context.We contend that this assumption more closely mirrors how individuals infer the race of others.Fourth, we illustrate why these assumptions matter by showing that the probability of a name signaling a racial group varies across geographic contexts.Finally, we close by outlining several implications of this finding and introducing an open-source software solution that researchers can use to assess how the racial meaning of names varies across geographic locations.

Last Name Selection
Researchers generally select names for audit studies using one of two strategies.The first is to use names reported in other audit studies.This is a very common approach, made easier by evolving norms of transparency in social science research, which increasingly require that researchers make available details like this.The second strategy is to use population-level name lists, such as those provided by the United States Census Bureau.These lists contain a series of last names and report their frequency of use across racial groups. 3he problem with using population-level data, though, is that it requires researchers to assume that people evaluate names based on the national popularity of those names within racial groups.In other words, individuals assign the probability, P, that an individual belongs to a race, R, based on the frequency with which members of that race use a last name, S, in the population. 4Formally, this means that P(R|S). 5n some contexts, this might be a reasonable assumption.We can think, however, of many cases in which individuals likely make inferences about a person's race based not only on their surname but also on local demographics.This is because last names can signal different races in different places.Elliott et al. (2009) provide a couple of powerful examples for which this is the case.
Persons with the common surname 'Lee,' for example, are likely to be Korean or Chinese if they reside in a predominantly Asian neighborhood but not if they live in, say, Williamsburg, Virginia.Likewise, the Asian surname 'Ohara' could easily misidentify persons living in predominantly Irish neighborhoods.(P.4) The obvious implication of this is that individuals in a predominantly Asian neighborhood are more likely to think that someone with the last name "Lee" is Asian than white.Likewise, individuals who live in neighborhoods with a large proportion of Irish are more likely to think that a person with the name "Ohara" is Irish than Asian.
Although these are perhaps exaggerated examples, they highlight the fact that in some contexts, individual associations between names and race might be conditional on local racial demographics or location, which we denote as L. 6 This could happen when the subject of these studies might assume that the fictitious individual contacting them lives within their community, broadly defined.In light of this, we think that assuming P(R|S, L) is more reasonable than assuming P(R|S).
In the next section, we empirically illustrate what happens when we incorporate information on L when classifying last names.

Data and Results
To calculate P(R|S, L), we follow a well-developed health care literature and use Bayes' rule (Elliott et al. 2008, 2009; Fiscella and Fremont 2006; Imai and Khanna  2016). 7This approach provides us a "probabilistic prediction of individual [race or] ethnicity" (Imai and Khanna 2016: 265) for a given surname in a geographic area.In the context of this article, we interpret this quantity as representing the extent to which an individual believes a person belongs to a racial (or ethnic) group.
Before applying Bayes' rule, we need several inputs.First, we need a selection of last names.We take our list of 20 surnames from Gaddis (2017a).Second, we need a set of spatial units.For this application, we use the population of 3,142 U.S. counties. 8Data for this is supplied by the Census Data Application Programming Interface.Third, we need population-level data on surname use across racial groups as well as the racial demographics of counties.We use the R package wru to dynamically call these data.
We also use wru to apply Bayes' rule and generate the predicted probability of a name signaling a race (or ethnicity) (Imai and Khanna 2016) for all 3,142 U.S. counties.We generate these probabilities for three racial (or ethnic) groupsblacks, Hispanics, and whites.We then plot the results of this exercise in Figure 1.The vertical axis lists the 20 names we used.The horizontal axis displays the distribution of the probabilities generated using Bayes' rule across U.S. counties.The probabilities of a name belonging to a race sum to 1.The left panel plots P(white|S, L) for each name, the middle panel plots P(black|S, L), and the right panel plots P(Hispanic|S, L).
If a last name consistently signals a race (or ethnicity) across geographic contexts, we would expect the distribution of probabilities for that name to be closely grouped around 1. Similarly, if a name consistently does not signal a race (or ethnicity) across geographic contexts, we would expect the distribution of probabilities to tightly center on 0. We see, however, that both of these cases are rare, characterizing only about half the last names in our sample.Examples of this include Walsh, Nielsen, and McGrath.Regardless of where these names appear, individuals are likely to think that they belong to white individuals.
Importantly for audit study work, many names do not appear to consistently signal race across geographic contexts.Indeed, several of the names would seem to be particularly bad at signaling race no matter where they might be used.This can be seen in the low, flat distributions of probabilities seen for some of the names, such as Mosley, Jefferson, and Jackson.Last names like these appear not to provide any additional information about the race of their bearer.
One could argue, though, that individuals do not form their racial perceptions based on last names alone but on first names as well.This is objectively true.For instance, individuals are more likely to think that a man with the last name ''Washington" is black if his first name is ''Jamal" than if it is ''Chad."Yet the racial signal sent by first names is rarely so clear, as Gaddis (2017a) and Gaddis (2017c)  show.When first names do not clearly signal a race (or ethnicity), they are unlikely to help boost the signal sent through a last name, making it necessary for last names to convey a racial signal on their own.

Discussion
What are the implications of the empirical finding introduced above?We can think of at least two noteworthy consequences.The first has to do with generalizability.As a reminder, Gaddis (2017a) and Gaddis (2017c) show that first names alone often provide an insufficient signal about racial identity, causing people to rely on the information provided in last names.We show, though, that the racial (or ethnic) information provided by surnames varies across geographic context.Taken together, this suggests that the racial perception of some full names (first and last names) likely changes across space, leading to geographic variation in treatment intensity.If this true, then it means that the effect of racial cues based on these names likely varies geographically as well.This calls into question the extent to which previous audit study findings travel across spatial contexts, particularly when those studies are conducted at a local or regional level (e.g., Wallace et al.  2014).
The second implication has to do with treatment effect heterogeneity.Researchers often want to investigate the extent to which their impact estimates vary across spatial domains.For example, political scientists might want to know whether the effect of being black is different in the South.If individual perceptions of names are influenced by place, though, then it is not entirely clear how researchers should interpret statistically significant interactions between location and racial (or ethnic) treatments.The problem here is that scholars cannot know if these interactions are the result of some contextual-level process, the fact that the treatment varies across space, or both.

Conclusion
So what can scholars do about these issues?We recommend that prior to using a surname in an audit study, researchers should first examine the extent to which the probability that the name denotes a racial group varies across geographic contexts.When possible, scholars should select names where the probability varies across a limited range.This will help ensure that individuals perceive the race of the putative individual as intended.
We provide a slim R package, auditr, available at https://github.com/cdcrabtree/auditr to help researchers accomplish this.Relying heavily upon the wru package provided by Imai and Khanna (2016), the software takes a vector of last names, generates a matrix of name and county pairs, uses this matrix to return the probability that a name denotes one of four racial (or ethnic) groups (i.e., Asian, black, Hispanic, and white) for all spatial units, and then plots these values.This allows individuals to visually identify the extent to which the racial information provided by surnames varies across geographic contexts and to identify potentially problematic surnames.
Although this package does not solve all the potential problems that researchers might face when selecting appropriate last names for their audit studies, we think that it helps address a potentially important problem with current practices.Scholars can build on this work-and on Gaddis (2017a)-by examining how other factors condition individual perceptions of last names.
Notes 1 Gaddis (2017b) provides a brief history of this growing literature.Whereas we focus on studies on racial (or ethnic) discrimination conducted in America here (e.g., Butler 2014; Gell-Redman et al.Forthcoming), scholars are increasingly conducting audit studies in other countries and to detect other types of biases (e.g., Adida, Laitin, and Valfort  2014; Ahmed, Andersson, and Hammarstedt 2012, 2013; Baert 2016; Neumark, Bank,  and Van Nort 1995).
2 Pager (2007) reviews and addresses the major criticisms of audit studies.
3 It is probably the case that scholars who adopt the first strategy are often indirectly adopting the second strategy.This is because some of the earliest audit studies relied on these population-level lists (e.g., Bertrand and Mullainathan 2004).
4 We adapt our notation from Imai and Khanna (2016).
5 Researchers typically assume that P equals either 0 or 1.We assume instead that P is bound from 0 to 1.This means that individuals think that some names are more likely to be held by members of different races but does not require that they have perfect confidence about the correspondence between names and races.Our general argument, however, does not depend on this assumption.

Figure 1 :
Figure 1: P(R|S, L) plots.The predicted probabilities of a name signaling a race (or ethnicity) across geographic locations are shown.The vertical axis lists 20 last names commonly used in audit studies.The horizontal axis displays distribution of probabilities generated using Bayes' rule.The probabilities of a name belonging to a race sum to 1.The left panel plots P(white|S, L) for each name, the middle panel plots P(black|S, L), and the right panel plots P(Hispanic|S, L).