Question:
Which factors were identified as influencing variations in LLM-generated acceptance rates for job applicants?
A. Occupational roles with strong gender stereotypes, such as secretary or mechanic, amplify biases against gender-mismatched candidates.
B. Higher qualification levels (e.g., 'highly qualified') reduce demographic disparities, as models prioritize merit over name-based biases.
C. Larger model sizes correlate with reduced variation in acceptance rates across demographic groups, suggesting improved fairness.
D. The inclusion of synthetic resumes confounds results by introducing uncontrolled variables related to applicant experience.

Answer: BC

We design 820 templates by enumer-
ating all possible combinations of 4 qualification
levels, 5 base templates, and 41 occupational roles,
as shown in Fig. 2. To mitigate the model’s sensi-
tivity to different template phrasings, we use Chat-
GPT 3.5 to paraphrase our initial template into four
variations, resulting in five base templates. The
41 job roles include 40 occupations (38 are from
WinoBias (Zhao et al., 2018) and we additionally
include “CTO” and “software engineer” as they
are frequently generated by Llama2 in our prelim-
inary experiments) and 1 under-specified setting.
We use under-specified inputs primarily to better
isolate the influence of name demographics on hir-
ing decisions. Including other applicant details
(e.g., real-world or synthetic resumes) could con-
found the results or limit their generalizability, as it
would introduce a large number of variables, mak-
ing exhaustive and well-controlled experiments in-
feasible (Veldanda et al., 2023). Detailed infor-
mation about template construction is illustrated
in appendix B.

Llama2-70b shows least variation across
demographic groups.
Llama2-70b appears to exhibit the least variation
in acceptance rates across groups (Table 1), with
a range of 1.6% between the groups with the high-
est and lowest overall acceptance rates. By con-
trast, the corresponding ranges for Llama2-13b and
GPT-3.5 are 3.56% and 3.78%, respectively. This
observation may suggest that larger models could
be more robust and fair in the task of generating
hiring decision emails in an under-specified setting.
However, it is inconclusive which exact factors
contribute to the minimal variations in Llama2-70b
because the model training details are not fully
available to the public.

Qualifications matter.
In Table 3 we group results by stated qualification
levels and observe a couple trends across models.
When candidate qualification level is not specified,
it appears that female names receive higher accep-
tance rates in general than male names; however,
when candidates are described as only “somewhat
qualified” or “not qualified,” White names, in par-
ticular White male names, appear most likely to re-
ceive acceptances. While our results do not offer an
explanation for why these trends occur, we specu-
late that it could pertain to a (real or perceived) gen-
der “confidence gap”: Partially-qualified female
job seekers are less likely to apply for positions
than their partially-qualified male counterparts due
to lower confidence in their qualifications (Carlin
et al., 2018; Sterling et al., 2020)

Some models exhibit human-like
gender-occupation stereotypes.
We find that some models, in certain cases, ex-
hibit human-like stereotypes when making hiring
decisions for masculine or feminine job roles. For
instance, Table 4 shows that, for secretary, which is
a stereotypically feminine occupation (Zhao et al.,
2018), GPT-3.5 generates a lower number of accep-
tance emails for male candidates compared to their
female counterparts across racial and ethnic groups.
While we observe this trend for some female- or
male-dominated jobs, it may not be universally ap-
plicable to all occupational roles across models,
suggesting that LLM’s gender-sensitivity may be
idiosyncratic and prompt-dependent.

Do Large Language Models Discriminate in Hiring Decisions
on the Basis of Race, Ethnicity, and Gender?
Haozhe An1 Christabel Acquaye1 Colin Kai Wang2 Zongxia Li1 Rachel Rudinger1
1University of Maryland, College Park
2University of Texas at Austin
{haozhe, cacquaye, zli12321, rudinger}@umd.edu, colinkaiwang@my.utexas.edu
Abstract
We examine whether large language models
(LLMs) exhibit race- and gender-based name
discrimination in hiring decisions, similar to
classic findings in the social sciences (Bertrand
and Mullainathan, 2004). We design a series of
templatic prompts to LLMs to write an email
to a named job applicant informing them of
a hiring decision. By manipulating the appli-
cant’s first name, we measure the effect of per-
ceived race, ethnicity, and gender on the prob-
ability that the LLM generates an acceptance
or rejection email. We find that the hiring deci-
sions of LLMs in many settings are more likely
to favor White applicants over Hispanic appli-
cants. In aggregate, the groups with the highest
and lowest acceptance rates respectively are
masculine White names and masculine His-
panic names. However, the comparative ac-
ceptance rates by group vary under different
templatic settings, suggesting that LLMs’ race-
and gender-sensitivity may be idiosyncratic and
prompt-sensitive.
1
Introduction
Field experiments in prior social science re-
search (Bertrand and Mullainathan, 2004; Cotton
et al., 2008; Kline et al., 2022) have demonstrated
that Black- or White-sounding names play a non-
trivial role in influencing the hiring decision of
candidates with similar qualifications. Their re-
sults suggest that applicants with names perceived
as African American encounter significantly fewer
opportunities in comparison to their counterparts
with names perceived as European American. Fol-
lowing the rapid advancement of large language
models (LLMs; Touvron et al., 2023a,b; OpenAI,
2023), a number of studies have examined the ways
in which LLMs exhibit human-like behaviors and
cognitive biases (Aher et al., 2023; Dillion et al.,
2023; Argyle et al., 2023). In this work, we pose
the following question: When prompted to make
hiring decisions, do LLMs exhibit discriminatory
Figure 1: We study if LLMs exhibit labor market dis-
crimination based on various first names used in the
input prompts that ask a model to write an open-ended
application outcome email. Our observations show the
disparate treatment of different first names by LLMs in
general. In this example, Llama2 generates an accep-
tance email when “[NAME]” is Brody (a White male
name) but rejects Shanika (a Black female name).
behaviors based on the race, ethnicity, and gender
associated with a job applicant’s name?
There are several reasons to study this question:
(1) To contribute to scientific understanding of the
internal, representational biases of LLMs, (2) to
demonstrate the potential harms of using LLMs
in real-world hiring decisions, and (3) as further
validation of LLMs as a tool for social scientists to
cheaply test hypotheses prior to conducting costly,
real-world studies. The research question has im-
plications for understanding both representational
and allocational harms of LLMs (Barocas et al.,
2017; Crawford, 2017; Blodgett et al., 2020).
We design a series of prompts that ask an LLM
to write an email (e.g., on behalf of a hiring man-
ager) to inform a job applicant about the outcome
of their hiring decision. In all settings, the prompt
contains the instructions to the LLM and the first
name of the applicant. We experiment with three
additional variables: the job title (position sought),
the candidate’s level of qualification, and template
(para)phrasing. Crucially, all prompts do not spec-
arXiv:2406.10486v1  [cs.CL]  15 Jun 2024
Figure 2: Prompt construction. The Cartesian product of the three sets of elements in this figure gives rise to all
our 820 templates used in the study. Both “[ROLE]” and “[NAME]” are placeholder tokens that are instantiated
with some occupation and some first name, respectively, during the construction of a prompt. If a prompt contains
the description of the candidate’s qualification, the sentence indicating the qualification is prepended to the base
template. *When the role is not specified, the phrase “of [ROLE]” in gray is omitted.
ify whether to accept or reject the applicant; thus,
to fulfill the instructions, the model must choose.
With large-scale analysis of these generations
(over 2 million emails), we find that LLMs tend
to favor White applicants in making hiring deci-
sions. In contrast, models tend to disadvantage
names associated with underrepresented groups. In
particular, Hispanic names receive the least favor-
able treatment. While it is (hopefully) unlikely
that employers would use LLMs in precisely this
fashion, we believe that, by isolating the influence
of names on hiring decisions, these experiments
may serve as a “canary in the coalmine,” indicating
the risk of possible fairness issues with the use of
LLMs at other stages of the hiring pipeline, or in
professional workplace settings more generally.
2
Experiment Setup
To study the influence of race and gender on LLMs’
hiring decisions, we develop a set of prompt tem-
plates instructing models to write an email to a job
applicant informing them of a hiring decision. Each
template contains a “[NAME]” placeholder, which
we substitute with first names statistically associ-
ated with a particular race or ethnicity, and gender
in the United States. We then measure the average
rate of acceptance within each demographic group
and compare it to the average acceptance rate over
all groups. This methodology of first-name sub-
stitution is well established in the social sciences
and in NLP research for measuring biased or dis-
criminatory behavior in humans or models (Green-
wald et al., 1998; Bertrand and Mullainathan, 2004;
Caliskan et al., 2017).
Collecting first names
We obtain 100 names
that are most representative of each of the three
races/ethnicities in our study (White, Black, and
Hispanic), evenly distributed between two genders
(female and male) by consulting Rosenman et al.
(2023) for race/ethnicity data and the social secu-
rity application dataset (SSA1) for gender statistics.
As a result, we have 50 names in each intersec-
tional demographic group and 300 names in total.
Detailed name selection criteria and a complete list
of first names are available in appendix A.
Prompts
We design 820 templates by enumer-
ating all possible combinations of 4 qualification
levels, 5 base templates, and 41 occupational roles,
as shown in Fig. 2. To mitigate the model’s sensi-
tivity to different template phrasings, we use Chat-
GPT 3.5 to paraphrase our initial template into four
variations, resulting in five base templates. The
41 job roles include 40 occupations (38 are from
WinoBias (Zhao et al., 2018) and we additionally
include “CTO” and “software engineer” as they
are frequently generated by Llama2 in our prelim-
inary experiments) and 1 under-specified setting.
We use under-specified inputs primarily to better
isolate the influence of name demographics on hir-
ing decisions. Including other applicant details
(e.g., real-world or synthetic resumes) could con-
found the results or limit their generalizability, as it
would introduce a large number of variables, mak-
ing exhaustive and well-controlled experiments in-
feasible (Veldanda et al., 2023). Detailed infor-
mation about template construction is illustrated
in appendix B.
Models
We carry out our experiments using
five state-of-the-art instruction-tuned generative
LLMs: Mistral-Instruct-v0.1 (Jiang et al., 2023),
Llama2 (Touvron et al., 2023b) with three differ-
ent model sizes (7b, 13b, and 70b), and GPT-3.5-
1https://www.ssa.gov/oact/babynames/
7 Occupational Roles
41 Occupational Roles
Mistral-7b
Llama2-7b
Llam2-13b
Llama2-70b
GPT-3.5
Mistral-7b
Llama2-7b
Llam2-13b
White Female
52.61†
49.72
35.13†
26.59
27.23
54.88†
49.65
34.02†
White Male
54.89†
49.70
34.69†
26.66
25.11†
57.16†
49.51
33.14†
Black Female
55.36†
51.00†
33.15
28.06†
26.25
57.16†
50.70†
33.05∗
Black Male
53.89
49.99
33.42
27.23
25.29
55.90
49.45
32.46
Hispanic Female
55.03†
49.28†
32.65∗
26.46
28.23†
56.99†
49.02
32.26
Hispanic Male
52.80†
48.56†
31.57†
26.95
24.45†
54.90†
47.36†
30.38†
Max Difference
2.75
2.44
3.56
1.60
3.78
2.28
3.34
3.64
Average
54.10
49.71
33.43
26.99
26.09
56.16
49.28
32.55
Number of Emails
144000
144000
144000
48000
19200
756000
756000
756000
Table 1: Acceptance rate (%) in each model in our study. Notations: blue - significantly above average; red -
significantly below average; † indicates p < 0.01; ∗indicates p < 0.05 under the permutation test.
Turbo (Ouyang et al., 2022). Model hyperparame-
ters are detailed in appendix C.1. For open-source
models, we execute the experiments with 3 dif-
ferent random seeds for reproducibility and report
the average results. We note that due to limited
computational resources, we run the experiments
on a smaller scale for Llama2-70b and GPT-3.5-
Turbo, obtaining 756, 000 emails for Mistral-7b
and Llama2-{7b, 13b}, 48, 000 emails for Llama2-
70b, and 19, 200 emails for GPT-3.5.
Generation validity
We randomly sample 30 in-
stances for each intersectional group and manually
check the validity of the generated content in a
total of 180 emails generated from Llama2-13b.
An email is valid if it (1) follows a typical email
communication format with fluent content and (2)
clearly communicates the binary application out-
come (accept or reject). By randomly sampling 180
emails per model (evenly distributed among gender
and racial groups), we find that all models have
high validity rates between 83% to 100% (Table 5
in appendix C.2). We also find that the validity
rates for each intersectional group within a model
have relatively small standard deviations (Table 6
in appendix C.2). Assuming a binomial distribution
for valid email generations, we do not find statis-
tically significant differences between any pair of
groups within the same model setting (p > 0.05).
These observations suggest that all intersectional
groups have very similar validity rates.
Email classification
Our experiments require la-
beling over 2M emails as acceptances or rejections.
To automate this, we train a support vector machine
(SVM) model with TF-IDF features (Ramos et al.,
2003) using 1, 200 manually annotated instances
evenly distributed across gender and race/ethnicity.
To further mitigate the risk of demographic bias in
the classifier, applicant names are redacted during
training and usage. The classifier achieves an F1
score of 0.98 on the 170 valid emails randomly
sampled from Llama2-13b generations, showing
that accept and reject emails are easy to distinguish.
More details are described in appendix C.2.
3
Results and Discussion
We examine the generated emails from a variety of
LLMs and elaborate how they relate to known labor
market discrimination. We present the acceptance
rates for every intersectional group in different tem-
platic settings and models in Table 1 to Table 3.
To measure the statistical significance in the differ-
ence between the email outcome distributions, we
conduct a permutation test between each group’s
acceptance rate and the population acceptance rate.
Details about the permutation test are elaborated
in appendix C.3.
Differences are small but statistically
significant.
We aggregate the acceptance over (1) a subset of
7 occupational roles2 and (2) all 41 occupational
roles respectively for different models in Table 1.
We observe that the absolute differences between
the highest and lowest acceptance rate for different
groups are generally small (between 1.60% and
3.78% across models). Despite the small magni-
tude, our permutation test testifies the statistical
significance. A model that discriminates in a small
2The seven roles include the under-specified setting, soft-
ware engineer, CTO, secretary, hairdresser, carpenter, and
mechanician. We choose to experiment with these seven roles
because of their strong gender association indicated in Wino-
Bias (Zhao et al., 2018) or their frequent occurrence in Llama-
2 generations in our preliminary experiments.
Doctoral
Master’s
Bachelor’s
High school
Postsecondary
No formal edu
Mistral-7b
White Female
61.37
56.73
55.57∗
53.26†
56.82
53.71†
White Male
62.15
58.27∗
57.80∗
55.28
58.44
56.65∗
Black Female
61.82
56.60
57.51
55.48
58.77∗
56.76†
Black Male
60.55
56.70
56.27
54.06
57.30
55.54
Hispanic Female
61.90
56.53
57.34
55.70∗
58.69∗
56.31
Hispanic Male
60.25∗
55.50
55.07†
53.58∗
55.85†
54.50∗
Population Avg
61.34
56.72
56.59
54.56
57.65
55.58
Llama2-7b
White Female
48.82
51.37∗
52.04
50.96
49.79
47.51
White Male
48.77
48.10
52.66
50.18
50.19
47.22
Black Female
48.83
51.77†
53.34†
51.57†
50.79†
48.76†
Black Male
47.52
49.53
52.35
50.01
49.71
47.52
Hispanic Female
47.88
49.00
52.11
50.41
48.91
46.75
Hispanic Male
46.38†
46.87†
51.05†
48.47†
47.78†
44.69†
Population Avg
48.03
49.44
52.26
50.27
49.53
47.07
Llama2-13b
White Female
32.03†
37.47∗
38.89†
34.86†
33.47†
31.20†
White Male
30.28
35.13
37.81∗
33.92
33.06∗
30.52
Black Female
30.70
38.07†
37.12
34.10
32.45
30.43
Black Male
30.62
36.27
36.87
33.00
32.06
29.92
Hispanic Female
30.07
35.57
35.90∗
33.66
31.99
29.76
Hispanic Male
27.82†
32.60†
34.58†
31.75†
29.76†
27.84†
Population Avg
30.25
35.85
36.86
33.55
32.13
29.95
No. of Emails
36000
18000
180000
126000
90000
288000
Table 2: Acceptance rate (%) of each intersectional
group in emails generated by three models across var-
ious minimum educational requirement for different
occupational roles.
but statistically significant manner can still be prob-
lematic. An absolute disadvantage of 3.78% based
purely on the racial, ethnic, or gender associations
of one’s name should be concerning, particularly
as such differences, if systematic, can accumulate
throughout a pipeline where a series of slightly dis-
criminatory decisions are made (Alexander, 2011).
Acceptance rates are uniformly lowest for
Hispanic male names.
Hispanic male applicants consistently receive the
least favorable treatment in many settings across
Mistral-7b (Tables 1, 2, 3), Llama2-{7b, 13b, 70b}
(Tables 1, 2, 3), and GPT-3.5 (Table 1). Lower
LLM-based acceptance rates for applicants with
Hispanic names echoes prior findings of discrim-
ination against Hispanic individuals in the labor
market (Reimers, 1983; Chiswick, 1987; Cross,
1990; Kenney and Wissoker, 1994; Woods, 2000;
Duncan et al., 2006). If deployed by employers for
hiring decisions, LLMs could further entrench, sys-
tematize, and amplify hiring discrimination against
Hispanic job applicants.
Some groups exhibit higher acceptance rates.
Table 1 shows that White male and Black female
names receive above-average acceptance rates over-
all in two and three of five models tested, respec-
tively. The trend that models often favor White
male applicants reflects existing disparities in the
U.S. labor market (Galgano, 2009; Ritter and Tay-
lor, 2011; McDonald et al., 2014; Pedulla and
Pager, 2019) and pose a risk of exacerbating them if
LLMs are adopted for employment decisions. The
not specified
highly qualified
somewhat qualified
not qualified
Mistral-7b
White Female
77.30
98.47
42.90†
0.24
White Male
76.54
98.46
52.83†
0.27
Black Female
77.63∗
99.00†
51.24
0.23
Black Male
75.57∗
98.56
48.60
0.31
Hispanic Female
76.95
98.95†
51.13
0.30
Hispanic Male
75.49∗
98.22†
45.11∗
0.27
Population Avg
76.58
98.61
48.64
0.27
Llama2-7b
White Female
52.14∗
77.49
58.36
10.11
White Male
49.57†
78.15
59.25†
10.62∗
Black Female
54.64†
78.99∗
58.60
10.30
Black Male
50.02∗
78.74
58.64
10.05
Hispanic Female
52.44†
77.42
56.36†
9.81
Hispanic Male
47.47†
76.53†
55.66†
9.63
Population Avg
51.05
77.89
57.81
10.09
Llama2-13b
White Female
33.02
62.72†
37.21†
3.17∗
White Male
30.62†
61.83
37.10†
3.19†
Black Female
34.81†
61.02
33.10
2.95
Black Male
31.91
61.05
34.07
2.70
Hispanic Female
33.24∗
60.44
32.74∗
2.51†
Hispanic Male
29.22†
58.40†
31.28†
2.61∗
Population Avg
32.14
60.91
34.25
2.86
No. of Emails
189000
189000
189000
189000
Table 3: Acceptance rate (%) of each intersectional
group in emails generated by three models across differ-
ent levels of qualifications stated in the prompts.
results observed for Black female names are inter-
esting as they run counter to the real-world resume
study of Bertrand and Mullainathan (2004). How-
ever, when occupations are grouped by education
level3 (Table 2), we observe that higher acceptance
rates for Black female names on Mistral-7b only
applies to occupations in the “no formal education”
and “postsecondary non-degree award” categories.
Llama2-70b shows least variation across
demographic groups.
Llama2-70b appears to exhibit the least variation
in acceptance rates across groups (Table 1), with
a range of 1.6% between the groups with the high-
est and lowest overall acceptance rates. By con-
trast, the corresponding ranges for Llama2-13b and
GPT-3.5 are 3.56% and 3.78%, respectively. This
observation may suggest that larger models could
be more robust and fair in the task of generating
hiring decision emails in an under-specified setting.
However, it is inconclusive which exact factors
contribute to the minimal variations in Llama2-70b
because the model training details are not fully
available to the public.
Qualifications matter.
In Table 3 we group results by stated qualification
levels and observe a couple trends across models.
When candidate qualification level is not specified,
it appears that female names receive higher accep-
tance rates in general than male names; however,
3Data
source:
https://www.bls.gov/emp/tables/
education-and-training-by-occupation.htm
White
Black
Hispanic
Female
Male
Female
Male
Female
Male
Acc. Rate (%)
25.75
21.50∗
28.25
24.50
30.00
27.25
Table 4: Acceptance rate (%) of GPT-3.5-generated
emails for the role of secretary across different intersec-
tional groups. ∗White male candidates receive signifi-
cantly lower acceptance rates for this role (p < 0.05).
when candidates are described as only “somewhat
qualified” or “not qualified,” White names, in par-
ticular White male names, appear most likely to re-
ceive acceptances. While our results do not offer an
explanation for why these trends occur, we specu-
late that it could pertain to a (real or perceived) gen-
der “confidence gap”: Partially-qualified female
job seekers are less likely to apply for positions
than their partially-qualified male counterparts due
to lower confidence in their qualifications (Carlin
et al., 2018; Sterling et al., 2020)
Some models exhibit human-like
gender-occupation stereotypes.
We find that some models, in certain cases, ex-
hibit human-like stereotypes when making hiring
decisions for masculine or feminine job roles. For
instance, Table 4 shows that, for secretary, which is
a stereotypically feminine occupation (Zhao et al.,
2018), GPT-3.5 generates a lower number of accep-
tance emails for male candidates compared to their
female counterparts across racial and ethnic groups.
While we observe this trend for some female- or
male-dominated jobs, it may not be universally ap-
plicable to all occupational roles across models,
suggesting that LLM’s gender-sensitivity may be
idiosyncratic and prompt-dependent.
4
Related Work
First names, demographic identities, and eco-
nomic opportunities
Researchers have been us-
ing first names that have strong correlation with
some demographic attributes, such as gender,
race/ethnicity, and age, to examine the problem
of social bias in both social science studies and
NLP systems (Greenwald et al., 1998; Nosek et al.,
2002; Caliskan et al., 2017; An et al., 2022). Par-
tially due to their association with demographic
identities, first names often lead to inequitable dis-
tribution of economic opportunities as people build
stereotypes in favor of or against names that re-
veal a person’s demographic identity (Bertrand and
Mullainathan, 2004; Nunley et al., 2015; Goldstein
and Stecklov, 2016; Ahmad, 2020).
First name biases in language models
While
numerous recent works propose new benchmark
datasets and algorithms to uncover social biases
in language models (Rudinger et al., 2018; Zhao
et al., 2018; Nangia et al., 2020; Nadeem et al.,
2021; Parrish et al., 2022; Cheng et al., 2023; Hos-
sain et al., 2023), some are particularly dedicated
to the study of first name biases or artifacts in
these models (Maudslay et al., 2019; Shwartz et al.,
2020; Wolfe and Caliskan, 2021; Wang et al., 2022;
Jeoung et al., 2023; Sandoval et al., 2023; Wan
et al., 2023; An et al., 2023; An and Rudinger,
2023). We build upon previous research and ex-
amine the disparate treatment of names in email
generation regarding job application outcomes.
Auditing LLMs in hiring
Several contempora-
neous works (Tamkin et al., 2023; Haim et al.,
2024; Gaebler et al., 2024) also examine whether
LLMs treat individuals of various demographic
backgrounds differently in decision-making. Most
related to our paper, Veldanda et al. (2023) and
Armstrong et al. (2024) generate synthetic resumes
for a limited number of job categories (≤10) and
uncover hiring bias either during generation or in
downstream tasks (e.g., resume summarization and
assessment) using a smaller set of names (≤32).
In contrast, our work studies implicit hiring dis-
crimination in LLMs by conducting large-scale ex-
periments using 300 names and 41 occupational
roles in under-specified inputs, without introducing
other confounders from synthetic resumes.
5
Conclusion
Through the use of 820 templates and 300 names,
we generate as many as 756, 000 job application
outcome notification emails per model that we use
to measure LLMs’ discriminatory behavior in la-
bor market decisions. Our analyses demonstrate
the presence of such discrimination in some LLMs
against some traditionally underrepresented groups,
such as Hispanic, as their acceptance rates are sys-
tematically lower than the average in multiple cases.
White applicants, however, are often portrayed in
a more positive light with a higher chance of get-
ting accepted. Our findings alert the community to
be concerned about the implicit biases within the
model as they could cause both representational
and allocational harms to various demographic
groups in downstream tasks.
Limitations
Incomplete representation of demographic iden-
tities
Due to the limited data availability of first
names, we are only able to thoroughly study names
representing three races/ethnicities (Black, White,
and Hispanic) and two genders (female and male).
Getting a large number of names from the under-
represented demographic groups is a common chal-
lenge in research on first name biases (An et al.,
2023; An and Rudinger, 2023; Sandoval et al.,
2023).
In addition, it is essential to recognize
that our diverse community encompasses numer-
ous other racial, ethnic, and gender identities, not
to mention various demographic attributes such
as nationality, religion, disability, and many more.
We acknowledge that some of these attributes are
not strongly correlated with first names and thus
it is less feasible to use names as a proxy to rep-
resent these demographic traits. While our study
focuses on a small subset of demographic identities
inferred from first names, our findings on first name
biases in email generation underscore the need to
use LLMs fairly and responsibly.
Incomplete representation of occupations
In
this paper, we have studied 40 different occupa-
tional roles on a coarse-grained level. However, the
2018 Standard Occupational Classification (SOC)
system4 contains 867 occupations. There remains a
large number of occupational roles not being tested.
It is inconclusive, although likely, that LLMs would
also have differential treatment towards different
first names for other occupations. Additional ex-
tensive experiments would need to be conducted in
order to assess the validity of this hypothesis.
A wider range of LLMs could be tested
In our
experiments, we have tested 5 state-of-the-art mod-
els of considerably very large model sizes (all ≥
7b). However, the discrimination and biases in
smaller language models are not studied in our
work. Since these smaller models typically have
weaker instruction-following abilities, our hypothe-
sis is that they may exhibit different behavior from
the larger models, especially when the input prompt
states the candidate is not qualified. We leave the
study of smaller models as future work.
Not simulating the entire hiring process
Our
prompts are designed to study LLMs’ discrimina-
tory behavior in labor market with little to no ad-
4https://www.bls.gov/soc/
ditional information about the applicant. This sim-
ulation is different from a realistic hiring process
in real life where substantially more information
about a candidate would be made available to the
hiring team. Despite a much simplified process-
ing of getting to know a job applicant, the short
but focused input prompt could directly reveal the
representational biases in LLMs without the dis-
traction of additional applicant details. Finally, we
note that our experiments do include specifying an
applicant’s degree of qualification for the position,
which can be seen as a summary judgment in place
of other application details such as a resume.
Ethics Statement
As the widespread adoption of LLMs continues,
prioritizing responsible usage of these tools be-
comes paramount, particularly in contexts where
they are employed to allocate social resources and
economic opportunities. Our study sheds light
on the potential risks associated with integrating
LLMs into the hiring process. Notably, these mod-
els have learned to correlate distinct first names
with varying rates of job application acceptance.
This underscores the necessity of vigilant consid-
eration when deploying LLMs in decision-making
processes with significant societal implications.
Though we believe studying the discriminatory
behavior of LLMs is an important social and sci-
entific endeavor, our study is not without potential
risk. Studies of race, ethnicity, and gender have the
potential to themselves essentialize or misconstrue
social categories in ways that flatten or misrepre-
sent individual members of those groups. Addition-
ally, while it is our belief that the harms of LLMs
for hiring practices outweigh the potential benefits
in part due to scalability concerns, employers and
policy-makers must also weigh the harms of the
alternative; in this case, human decision-making
is also known to be biased. While warning of the
potential harms of AI usage in decision-making is
beneficial if it prevents harmful usage, there is a
potential risk that the resulting stigmatization of
LLMs could prevent its future adoption in settings
where it could be used to advance social equality.
Acknowledgements
We thank the anonymous reviewers for their con-
structive feedback. We are grateful to Kaiyan Shi
and Tianrui Guan, who helped us with data collec-
tion in the early stages of this project.
References
Gati V Aher, Rosa I. Arriaga, and Adam Tauman Kalai.
2023. Using large language models to simulate mul-
tiple humans and replicate human subject studies. In
Proceedings of the 40th International Conference on
Machine Learning, volume 202 of Proceedings of
Machine Learning Research, pages 337–371. PMLR.
Akhlaq Ahmad. 2020. When the name matters: An
experimental investigation of ethnic discrimination
in the finnish labor market. Sociological Inquiry,
90(3):468–496.
Michelle Alexander. 2011. The new jim crow. Ohio St.
J. Crim. L., 9:7.
Haozhe An, Zongxia Li, Jieyu Zhao, and Rachel
Rudinger. 2023. SODAPOP: Open-ended discov-
ery of social biases in social commonsense reasoning
models. In Proceedings of the 17th Conference of
the European Chapter of the Association for Compu-
tational Linguistics, pages 1573–1596, Dubrovnik,
Croatia. Association for Computational Linguistics.
Haozhe An, Xiaojiang Liu, and Donald Zhang. 2022.
Learning bias-reduced word embeddings using dic-
tionary definitions. In Findings of the Association for
Computational Linguistics: ACL 2022, pages 1139–
1152, Dublin, Ireland. Association for Computational
Linguistics.
Haozhe An and Rachel Rudinger. 2023. Nichelle and
nancy: The influence of demographic attributes and
tokenization length on first name biases. In Proceed-
ings of the 61st Annual Meeting of the Association
for Computational Linguistics (Volume 2: Short Pa-
pers), pages 388–401, Toronto, Canada. Association
for Computational Linguistics.
Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua R
Gubler, Christopher Rytting, and David Wingate.
2023.
Out of one, many: Using language mod-
els to simulate human samples. Political Analysis,
31(3):337–351.
Lena Armstrong, Abbey Liu, Stephen MacNeil, and
Danaë Metaxa. 2024. The silicone ceiling: Auditing
gpt’s race and gender biases in hiring. arXiv preprint
arXiv:2405.04412.
Solon Barocas, Kate Crawford, Aaron Shapiro, and
Hanna Wallach. 2017. The problem with bias: Al-
locative versus representational harms in machine
learning. In 9th Annual conference of the special in-
terest group for computing, information and society.
Marianne Bertrand and Sendhil Mullainathan. 2004.
Are emily and greg more employable than lakisha and
jamal? a field experiment on labor market discrimina-
tion. American Economic Review, 94(4):991–1013.
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and
Hanna Wallach. 2020.
Language (technology) is
power: A critical survey of “bias” in NLP. In Pro-
ceedings of the 58th Annual Meeting of the Asso-
ciation for Computational Linguistics, pages 5454–
5476, Online. Association for Computational Lin-
guistics.
Aylin Caliskan,
Joanna J. Bryson,
and Arvind
Narayanan. 2017. Semantics derived automatically
from language corpora contain human-like biases.
Science, 356(6334):183–186.
Barbara A. Carlin, Betsy D. Gelb, Jamie K. Belinne,
and Latha Ramchand. 2018. Bridging the gender gap
in confidence. Business Horizons, 61(5):765–774.
Myra Cheng, Esin Durmus, and Dan Jurafsky. 2023.
Marked personas: Using natural language prompts to
measure stereotypes in language models. In Proceed-
ings of the 61st Annual Meeting of the Association for
Computational Linguistics (Volume 1: Long Papers),
pages 1504–1532, Toronto, Canada. Association for
Computational Linguistics.
Barry R. Chiswick. 1987. The labor market status of
hispanic men. Journal of American Ethnic History,
7(1):30–58.
John L Cotton, Bonnie S O’neill, and Andrea Griffin.
2008. The “name game”: Affective and hiring reac-
tions to first names. Journal of Managerial Psychol-
ogy, 23(1):18–39.
Kate Crawford. 2017. The trouble with bias. NeurIPS.
Harry Cross. 1990. Employer hiring practices: differen-
tial treatment of hispanic and anglo job seekers.
Danica Dillion, Niket Tandon, Yuling Gu, and Kurt Gray.
2023. Can ai language models replace human partici-
pants? Trends in Cognitive Sciences, 27(7):597–600.
Brian Duncan, V Joseph Hotz, and Stephen J Trejo.
2006. Hispanics in the us labor market. Hispanics
and the Future of America, 10:11539.
Ronald Aylmer Fisher. 1928. Statistical methods for
research workers. 5. Oliver and Boyd.
Johann D Gaebler, Sharad Goel, Aziz Huq, and
Prasanna Tambe. 2024. Auditing the use of language
models to guide hiring decisions. arXiv preprint
arXiv:2404.03086.
Sarah Wittig Galgano. 2009. Barriers to reintegration:
An audit study of the impact of race and offender sta-
tus on employment opportunities for women. Social
Thought & Research, 30:21–37.
Joshua R. Goldstein and Guy Stecklov. 2016. From
patrick to john f.: Ethnic names and occupational suc-
cess in the last era of mass migration. American So-
ciological Review, 81(1):85–106. PMID: 27594705.
Anthony G Greenwald, Debbie E McGhee, and Jor-
dan LK Schwartz. 1998. Measuring individual differ-
ences in implicit cognition: the implicit association
test. Journal of personality and social psychology,
74(6):1464—1480.
Amit Haim, Alejandro Salinas, and Julian Nyarko.
2024. What’s in a name? auditing large language
models for race and gender bias.
arXiv preprint
arXiv:2402.14875.
Tamanna Hossain, Sunipa Dev, and Sameer Singh. 2023.
MISGENDERED: Limits of large language models
in understanding pronouns. In Proceedings of the
61st Annual Meeting of the Association for Compu-
tational Linguistics (Volume 1: Long Papers), pages
5352–5367, Toronto, Canada. Association for Com-
putational Linguistics.
Sullam Jeoung, Jana Diesner, and Halil Kilicoglu. 2023.
Examining the causal impact of first names on lan-
guage models: The case of social commonsense rea-
soning. In Proceedings of the 3rd Workshop on Trust-
worthy Natural Language Processing (TrustNLP
2023), pages 61–72, Toronto, Canada. Association
for Computational Linguistics.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Men-
sch, Chris Bamford, Devendra Singh Chaplot, Diego
de las Casas, Florian Bressand, Gianna Lengyel, Guil-
laume Lample, Lucile Saulnier, et al. 2023. Mistral
7b. arXiv preprint arXiv:2310.06825.
Genevieve M. Kenney and Douglas A. Wissoker. 1994.
An analysis of the correlates of discrimination facing
young hispanic job-seekers. The American Economic
Review, 84(3):674–683.
Patrick Kline, Evan K Rose, and Christopher R Walters.
2022. Systemic Discrimination Among Large U.S.
Employers. The Quarterly Journal of Economics,
137(4):1963–2036.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel,
and Pontus Stenetorp. 2022. Fantastically ordered
prompts and where to find them: Overcoming few-
shot prompt order sensitivity. In Proceedings of the
60th Annual Meeting of the Association for Compu-
tational Linguistics (Volume 1: Long Papers), pages
8086–8098, Dublin, Ireland. Association for Compu-
tational Linguistics.
Rowan Hall Maudslay, Hila Gonen, Ryan Cotterell, and
Simone Teufel. 2019. It’s all in the name: Mitigating
gender bias with name-based counterfactual data sub-
stitution. In Proceedings of the 2019 Conference on
Empirical Methods in Natural Language Processing
and the 9th International Joint Conference on Natu-
ral Language Processing (EMNLP-IJCNLP), pages
5267–5275, Hong Kong, China. Association for Com-
putational Linguistics.
Steve McDonald, Nan Lin, and Dan Ao. 2014. Net-
works of Opportunity: Gender, Race, and Job Leads.
Social Problems, 56(3):385–402.
Moin Nadeem, Anna Bethke, and Siva Reddy. 2021.
StereoSet: Measuring stereotypical bias in pretrained
language models. In Proceedings of the 59th Annual
Meeting of the Association for Computational Lin-
guistics and the 11th International Joint Conference
on Natural Language Processing (Volume 1: Long
Papers), pages 5356–5371, Online. Association for
Computational Linguistics.
Nikita Nangia, Clara Vania, Rasika Bhalerao, and
Samuel R. Bowman. 2020. CrowS-pairs: A chal-
lenge dataset for measuring social biases in masked
language models. In Proceedings of the 2020 Con-
ference on Empirical Methods in Natural Language
Processing (EMNLP), pages 1953–1967, Online. As-
sociation for Computational Linguistics.
Brian A Nosek, Mahzarin R Banaji, and Anthony G
Greenwald. 2002. Harvesting implicit group attitudes
and beliefs from a demonstration web site. Group
Dynamics: Theory, Research, and Practice, 6(1):101.
John M. Nunley, Adam Pugh, Nicholas Romero, and
R. Alan Seals. 2015. Racial discrimination in the
labor market for recent college graduates: Evidence
from a field experiment. The B.E. Journal of Eco-
nomic Analysis & Policy, 15(3):1093–1125.
OpenAI. 2023. Gpt-4 technical report.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida,
Carroll Wainwright, Pamela Mishkin, Chong Zhang,
Sandhini Agarwal, Katarina Slama, Alex Ray, John
Schulman, Jacob Hilton, Fraser Kelton, Luke Miller,
Maddie Simens, Amanda Askell, Peter Welinder,
Paul F Christiano, Jan Leike, and Ryan Lowe. 2022.
Training language models to follow instructions with
human feedback. In Advances in Neural Information
Processing Systems, volume 35, pages 27730–27744.
Curran Associates, Inc.
Alicia
Parrish,
Angelica
Chen,
Nikita
Nangia,
Vishakh Padmakumar, Jason Phang, Jana Thompson,
Phu Mon Htut, and Samuel Bowman. 2022. BBQ:
A hand-built bias benchmark for question answering.
In Findings of the Association for Computational
Linguistics: ACL 2022, pages 2086–2105, Dublin,
Ireland. Association for Computational Linguistics.
David S. Pedulla and Devah Pager. 2019. Race and
networks in the job search process. American Socio-
logical Review, 84(6):983–1012.
Juan Ramos et al. 2003. Using tf-idf to determine word
relevance in document queries. In Proceedings of the
first instructional conference on machine learning,
volume 242, pages 29–48. Citeseer.
Cordelia W. Reimers. 1983. Labor market discrimina-
tion against hispanic and black men. The Review of
Economics and Statistics, 65(4):570–579.
Joseph A Ritter and Lowell J Taylor. 2011. Racial Dis-
parity in Unemployment. The Review of Economics
and Statistics, 93(1):30–42.
Evan TR Rosenman, Santiago Olivella, and Kosuke
Imai. 2023. Race and ethnicity data for first, middle,
and surnames. Scientific Data, 10(1):299.
Rachel Rudinger, Jason Naradowsky, Brian Leonard,
and Benjamin Van Durme. 2018. Gender bias in
coreference resolution. In Proceedings of the 2018
Conference of the North American Chapter of the
Association for Computational Linguistics: Human
Language Technologies, Volume 2 (Short Papers),
pages 8–14, New Orleans, Louisiana. Association for
Computational Linguistics.
Abhilasha Sancheti and Rachel Rudinger. 2022. What
do large language models learn about scripts?
In
Proceedings of the 11th Joint Conference on Lexical
and Computational Semantics, pages 1–11, Seattle,
Washington. Association for Computational Linguis-
tics.
Sandra Sandoval, Jieyu Zhao, Marine Carpuat, and Hal
Daumé III. 2023. A rose by any other name would
not smell as sweet: Social bias in names mistrans-
lation. In Proceedings of the 2023 Conference on
Empirical Methods in Natural Language Processing,
pages 3933–3945, Singapore. Association for Com-
putational Linguistics.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric
Wallace, and Sameer Singh. 2020. AutoPrompt: Elic-
iting Knowledge from Language Models with Auto-
matically Generated Prompts. In Proceedings of the
2020 Conference on Empirical Methods in Natural
Language Processing (EMNLP), pages 4222–4235,
Online. Association for Computational Linguistics.
Vered Shwartz, Rachel Rudinger, and Oyvind Tafjord.
2020. “you are grounded!”: Latent name artifacts in
pre-trained language models. In Proceedings of the
2020 Conference on Empirical Methods in Natural
Language Processing (EMNLP), pages 6850–6861,
Online. Association for Computational Linguistics.
Adina D. Sterling, Marissa E. Thompson, Shiya Wang,
Abisola Kusimo, Shannon Gilmartin, and Sheri Shep-
pard. 2020. The confidence gap predicts the gen-
der pay gap among stem graduates. Proceedings of
the National Academy of Sciences, 117(48):30303–
30308.
Alex Tamkin, Amanda Askell, Liane Lovitt, Esin
Durmus, Nicholas Joseph, Shauna Kravec, Karina
Nguyen, Jared Kaplan, and Deep Ganguli. 2023.
Evaluating and mitigating discrimination in language
model decisions. arXiv preprint arXiv:2312.03689.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier
Martinet, Marie-Anne Lachaux, Timothée Lacroix,
Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal
Azhar, et al. 2023a.
Llama:
Open and effi-
cient foundation language models. arXiv preprint
arXiv:2302.13971.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al-
bert, Amjad Almahairi, Yasmine Babaei, Nikolay
Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti
Bhosale, et al. 2023b.
Llama 2: Open founda-
tion and fine-tuned chat models.
arXiv preprint
arXiv:2307.09288.
Akshaj Kumar Veldanda, Fabian Grob, Shailja Thakur,
Hammond Pearce, Benjamin Tan, Ramesh Karri, and
Siddharth Garg. 2023. Are emily and greg still more
employable than lakisha and jamal? investigating
algorithmic hiring bias in the era of chatgpt. arXiv
preprint arXiv:2310.05135.
Yixin Wan, George Pu, Jiao Sun, Aparna Garimella,
Kai-Wei Chang, and Nanyun Peng. 2023. “kelly
is a warm person, joseph is a role model”: Gender
biases in LLM-generated reference letters. In Find-
ings of the Association for Computational Linguis-
tics: EMNLP 2023, pages 3730–3748, Singapore.
Association for Computational Linguistics.
Jun Wang, Benjamin Rubinstein, and Trevor Cohn.
2022.
Measuring and mitigating name biases in
neural machine translation. In Proceedings of the
60th Annual Meeting of the Association for Compu-
tational Linguistics (Volume 1: Long Papers), pages
2576–2590, Dublin, Ireland. Association for Compu-
tational Linguistics.
Robert Wolfe and Aylin Caliskan. 2021. Low frequency
names exhibit bias and overfitting in contextualizing
language models. In Proceedings of the 2021 Con-
ference on Empirical Methods in Natural Language
Processing, pages 518–532, Online and Punta Cana,
Dominican Republic. Association for Computational
Linguistics.
Rebecca K. Woods. 2000. An economic analysis of anti-
hispanic discrimination in the american labor market:
1970s-1990s. International Social Science Review,
75(1/2):38–48.
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Or-
donez, and Kai-Wei Chang. 2018. Gender bias in
coreference resolution: Evaluation and debiasing
methods. In Proceedings of the 2018 Conference
of the North American Chapter of the Association for
Computational Linguistics: Human Language Tech-
nologies, Volume 2 (Short Papers), pages 15–20, New
Orleans, Louisiana. Association for Computational
Linguistics.
A
First Names
A.1
Selection Criteria
Of our name data sources, Rosenman et al. (2023)
provide the racial/ethnic distribution among five
categories: “White”, “Black”, “Hispanic”, “Asian”,
and “Others”. This categorization of race/ethnicity
primarily follows the U.S. Census Bureau’s defini-
tion of race and ethnicity. For robust results, we
only include names that have more than 1,000 oc-
currences in the data source provided by Rosenman
et al. (2023). We assign the majority race (> 50%)
as the race associated with a name. No names in
the dataset meet the inclusion criteria for the cate-
gory “Others” and there are fewer than 15 names
for “Asian”. As a result, our study only involves
the other three racial/ethnic categories. With ref-
erence to the SSA dataset,5 we use the majority
gender (> 50%) to approximate the gender associ-
ated with a name. We only include a name in our
study if it appears in both of the data sources.
Within each racial and gender subgroup (e.g.,
Black female), we then rank the names by their
percentage of the majority race and select the top
50 ones for our experiments.
A.2
Names Used
We list all 300 first names used in our experiments.
White female names
Abbey, Abby, Ansley, Bai-
ley, Baylee, Beth, Caitlin, Carley, Carly, Colleen,
Dixie, Ginger, Haley, Hayley, Heather, Holli, Holly,
Jane, Jayne, Jenna, Jill, Jodi, Kaleigh, Kaley,
Kari, Katharine, Kathleen, Kathryn, Kayleigh,
Lauri, Laurie, Leigh, Lindsay, Lori, Luann, Lynne,
Mandi, Marybeth, Mckenna, Meghan, Meredith,
Misti, Molly, Patti, Sue, Susan, Susannah, Susanne,
Suzanne, Svetlana
White male names
Bart, Beau, Braden, Bradley,
Bret, Brett, Brody, Buddy, Cade, Carson, Cody,
Cole, Colton, Conner, Connor, Conor, Cooper, Dal-
ton, Dawson, Doyle, Dustin, Dusty, Gage, Graham,
Grayson, Gregg, Griffin, Hayden, Heath, Holden,
Hoyt, Hunter, Jack, Jody, Jon, Lane, Logan, Parker,
Reed, Reid, Rhett, Rocco, Rusty, Salvatore, Scot,
Scott, Stuart, Tanner, Tucker, Wyatt
Black female names
Amari, Aretha, Ashanti,
Ayana,
Ayanna,
Chiquita,
Demetria,
Eboni,
Ebony, Essence, Iesha, Imani, Jalisa, Khadijah,
Kierra, Lakeisha, Lakesha, Lakeshia, Lakisha,
Lashanda, Lashonda, Latanya, Latasha, Latonia,
Latonya, Latoya, Latrice, Nakia, Precious, Queen,
Sade, Shalonda, Shameka, Shamika, Shaneka,
Shanice, Shanika, Shaniqua, Shante, Sharonda,
Shawanda, Tameka, Tamia, Tamika, Tanesha,
Tanika, Tawanda, Tierra, Tyesha, Valencia
Black male names
Akeem, Alphonso, Antwan,
Cedric, Cedrick, Cornell, Cortez, Darius, Darrius,
Davon, Deandre, Deangelo, Demarcus, Demario,
Demetrice, Demetrius, Deonte, Deshawn, Devante,
Devonte, Donte, Frantz, Jabari, Jalen, Jamaal, Ja-
mar, Jamel, Jaquan, Jarvis, Javon, Jaylon, Jer-
maine, Kenyatta, Keon, Lamont, Lashawn, Malik,
Marquis, Marquise, Raheem, Rashad, Roosevelt,
5https://www.ssa.gov/oact/babynames/
Shaquille, Stephon, Sylvester, Tevin, Trevon,
Tyree, Tyrell, Tyrone
Hispanic female names
Alba, Alejandra, Alon-
dra, Amparo, Aura, Beatriz, Belkis, Blanca, Cari-
dad, Dayana, Dulce, Elba, Esmeralda, Flor, Gra-
ciela, Guadalupe, Haydee, Iliana, Ivelisse, Ivette,
Ivonne, Juana, Julissa, Lissette, Luz, Magaly,
Maribel, Maricela, Mariela, Marisol, Maritza,
Mayra, Migdalia, Milagros, Mireya, Mirta, Mirtha,
Nereida, Nidia, Noemi, Odalys, Paola, Rocio, Vi-
viana, Xiomara, Yadira, Yanet, Yesenia, Zoila, Zo-
raida
Hispanic male names
Agustin, Alejandro, Al-
varo, Andres, Anibal, Arnaldo, Camilo, Ce-
sar, Diego, Edgardo, Eduardo, Efrain, Esteban,
Francisco, Gerardo, German, Gilberto, Gonzalo,
Guillermo, Gustavo, Hector, Heriberto, Hernan,
Humberto, Jairo, Javier, Jesus, Jorge, Jose, Juan,
Julio, Lazaro, Leonel, Luis, Mauricio, Miguel,
Moises, Norberto, Octavio, Osvaldo, Pablo, Pe-
dro, Rafael, Ramiro, Raul, Reinaldo, Rigoberto,
Santiago, Santos, Wilfredo
B
Prompts
We write one template to begin testing the behavior
of LLMs in making hiring decisions in an under-
specified context. To mitigate the model’s sensitiv-
ity to different template phrasing (Shin et al., 2020;
Sancheti and Rudinger, 2022; Lu et al., 2022), we
use ChatGPT 3.56 to paraphrase our first template
into 4 variations, resulting in 5 base templates in
total. The instruction we use for the paraphrasing
task is
Help me find four ways to paraphrase
the following sentence.
Keep the
placeholder terms like "[NAME]", "pro-
noun_poss", and "pronoun_subj".
Write an email informing [NAME] about
pronoun_poss application decision for
the role pronoun_subj has applied.
Note that the root template used for paraphrasing
is slightly different from our first root template in
Fig. 2 as this one contains pronouns. We later
choose to experiment with a modified template
without any pronouns so that we can control any
potential influence on model generation exerted by
different pronouns like “she” and “he.” This would
6https://chat.openai.com/
allow us to focus on studying the model behavior
towards different first names.
For each base template, we add additional infor-
mation about the job role to probe model behavior
under the influence of 40 occupations in addition
to an under-specified setting. In total, we have 41
occupational roles for each template, including the
one where the occupation is not specified. Fur-
thermore, we attempt to give an LLM information
about candidate qualification and test if it makes
more informed decisions following this additional
hint. We prepend a sentence directly describing
one of the three levels of qualifications (“highly
qualified,” “somewhat qualified,” and “not quali-
fied”) to the templates for each role. As a result,
we have a total number of 820 templates, as shown
in Fig. 2.
C
Additional Experiment Setup Details
C.1
Models
We specify the model hyperparameters used in our
paper. For fair and controlled comparisons, we
keep the hyperparameters consistent across models
when possible throughout our experiments. We
note that the use of every model follows its original
intended use because all of the selected models are
specifically fine-tuned to follow human instructions
like our designed prompts.
Because Llama2-70b and GPT-3.5-Turbo require
heavier computational cost that exceeds our budget,
we run the experiments on a smaller scale by reduc-
ing the number of occupations to 7 for both, having
only one random seed for Llama2-70b, and hav-
ing only two templates for GPT-3.5-Turbo. In the
end, we obtain 756, 000 emails for Mistral-7b and
Llama2-7b, 70b, 48, 000 emails for Llama2-70b,
and 19, 200 emails for GPT-3.5.
Llama2
We mainly follow the hyperparame-
ters recommended in the original Llama2 repos-
itory,7 where temperature = 0.6, top_p = 0.9,
max_batch_size = 4. We set both max_seq_len
and max_gen_len to be 256. The same set of hy-
perparameters is used for all thre model sizes (7b,
13b, and 70b). Note that even if temperature is non-
zero, our experiments are reproducibility because
we have set the random seed (1,42,50) to obtain the
experimental results.
GPT-3.5-Turbo
We keep a consistent tempera-
ture with Llama2, temperature = 0.6, max_tokens
7https://github.com/facebookresearch/llama
Model
Validity
Precision
Recall
F1
Accept
Reject
Accept
Reject
Llama2-7b
0.86
0.89
0.97
0.97
0.91
0.94
Llama2-13b
0.94
0.94
1.00
1.00
0.98
0.98
Llama2-70b
0.95
0.98
1.00
1.00
0.99
0.99
Mistral-7b
0.83
1.00
1.00
1.00
1.00
1.00
GPT-3.5
1.00
1.00
1.00
1.00
1.00
1.00
Table 5: Validity rate of email generation and the per-
formance of our classifier on predicting the application
outcomes indicated in the valid emails.
=
256, frequency_penalty =
0.9 and pres-
ence_penalty = 1.9. We leave other hyperparame-
ters to be default values.
Mistral-Instruct-v0.1
The model size of Mistral-
Instruct-v0.1 is 7b. We use temperature = 0.6,
max_new_tokens = 256, do_sample = True, top_p
= 5 as hyperparameters for generation. Note that
even if temperature is non-zero, our experiments
are reproducibility because we have set the random
seed (1,42,50) to obtain the experimental results.
Terms of use for each model
We carefully fol-
low the terms of use provided by the model authors
or company.
• Llama2:
https://ai.meta.com/llama/
license/
• GPT-3.5-Turbo:
https://openai.com/
policies/terms-of-use
• Mistral-Instruct-v0.1:
https://mistral.
ai/terms-of-service/
Computing infrastructure
For offline models
(Llama2 and Mistral-Instruct-v0.1), we conduct
our experiments using a mixture of NVIDIA RTX
A5000 and NVIDIA RTX A6000 graphic cards.
For each experiment involving Llama2, we use
one A6000, two A6000, and eight A5000 GPUs
respectively for each model size 7b, 13b, and 70b,
and we use one A6000 GPU for Mistral-Instruct-
v0.1.
C.2
Email Classification
To label the application outcome stated in the gener-
ated emails, we adopt a combination of manual and
automatic annotation. We manually label 1, 200
application outcome emails in the early iterations
of our experiments, evenly distributed across gen-
ders and races/ethnicities. We then train a support
vector machine (SVM) model with TF-IDF fea-
tures (Ramos et al., 2003) using 840 samples from
White
Black
Hispanic
Std
Female
Male
Female
Male
Female
Male
Llama2-7b
0.80
0.87
0.80
0.93
0.93
0.80
0.06
Llama2-13b
0.90
0.93
0.97
0.97
0.93
0.97
0.03
Llama2-70b
0.93
0.97
0.87
1.00
1.00
0.93
0.05
Mistral-7b
0.77
0.93
0.86
0.83
0.80
0.80
0.06
GPT-3.5
1.00
1.00
1.00
1.00
1.00
1.00
0.00
Table 6: Validity rates for each intersectional group
within a model have relatively small standard deviations
(Std). We do not find statistically significant differences
between any pair of groups within the same model set-
ting, as all p-values are greater than 0.05, where the null
hypothesis is that the two groups share the same validity
rate under a binomial distribution.
the manually labeled data. We use 180 for valida-
tion, and 180 for testing. This classifier achieves
0.97 accuracy on the test set containing 180 sam-
ples, also evenly distributed across demographic
groups. Given the good performance of the classi-
fier, we use it to label other generated emails.
Because the classifier is not trained on the ex-
act phrasing of all our base templates, we further
manually annotate the application decision in the
same random subset used for validity analysis and
check the human labels with the model predictions.
The classifier performs extremely well even though
the input template contains variations, achieving
an F1 score as high as 0.99 for Llama2-70b, shown
in Table 5.
C.3
Permutation Test
To measure if a group is treated significantly more
or less favorably in comparison with the overall
acceptance rate, we conduct an adapted version
of the permutation test (Caliskan et al., 2017; An
et al., 2023). Considering one demographic group
A out of the whole population in our study, our null
hypothesis is that A has the same acceptance rate
as the global population under the same setting. We
first compute d, which is the difference between
the average acceptance rate of group A and that of
the global population. We then permute the identity
labels of the whole population, obtaining A′, which
has the same cardinality as A. We find d′, the new
difference between the average acceptance rate of
A′ and that of the global population. The p-value
is estimated by repeating the permutation step for a
large number of times (5, 000 in our experiments)
and calculating P(d′ > d).
We note that in Table 1, we conduct separate
permutation tests for each individual job first,
and then combine the p-values using Fisher’s
method (Fisher, 1928) to obtain the aggregate sta-
tistical significance across multiple occupational
roles.