Mark T Holcombe

AI Usage In Recruitment
Part 1: The Ethical Minefield

Abstract

This meta-analysis collectively addresses the complex and ethically problematic intersection of Artificial Intelligence (AI) with hiring processes. Several sources highlight the prevalence of bias in AI recruitment tools, detailing instances where algorithms exhibit gender, racial, or age-based discrimination, sometimes even favoring certain demographics while others show the opposite.

Concerns are raised regarding the lack of transparency, or “black box” problem, in these AI systems, making it difficult to understand how decisions are reached and challenging to identify and rectify discriminatory practices. Various sources emphasize the urgent need for regulation, accountability, and robust auditing mechanisms to ensure fairness, protect human rights, and prevent the perpetuation of existing societal biases in AI-driven hiring. Finally, the meta-analysis discusses ongoing legal debates, potential liabilities for employers and AI vendors, and proposed solutions for developing and deploying AI tools responsibly in recruitment.

Introduction

Use of artificial intelligence (AI) in human resources (HR) is an ethical minefield. AI and Large Language Models (LLMs) are trained on human history that is heavily tainted with discrimination and bias. HR by nature deals with sensitive data; social security numbers, addresses, dates of birth, etc. The use of AI in hiring and recruitment processes is widespread, with an estimated 99% of Fortune 500 companies using some form of automation in their hiring process. While proponents suggest AI can enhance efficiency and reduce bias, empirical data reveals significant and varied forms of discrimination and bias in these tools.

AI Hiring: Empirical Data

• Gender Bias:

  • Amazon’s abandoned AI tool: In 2018, Amazon scrapped an experimental AI-based resume screening tool after internal audits revealed it was biased against women. The tool, trained on predominantly male CVs from a male-dominated industry, had learned that being a man was a key factor correlated with successful hires and downgraded female candidates.
  • Large Language Model Audits: A study auditing several mid-sized open-source LLMs for

 

gender bias using 332,044 real-world online job postings found that most models tended to favor men, especially for higher-wage roles. For instance, Ministral showed a female callback rate of 1.4%, while Llama-3.1, the most balanced model, had a 41.0% female callback rate, indicating a moderate bias against women.

  • This study found lower callback rates for women in male-dominated occupations (e.g., Construction and Extraction, Installation, Maintenance, and Repair) and higher rates in female- associated ones (e.g., Personal Care and Service), indicating occupational segregation.
  • Linguistic features in job ads strongly aligned with traditional gender stereotypes in model recommendations. For example, mentions of skills like “career counseling,” “writing,” and “recruitment” were linked to higher female recommendations, while “coding,” “hardware,” and “financial skills” corresponded to lower likelihoods. Women were associated with roles requiring “empathy” and “motivating,” while men were linked to “aggression” and “supervision”.
  • Models also tended to recommend women for lower-wage jobs, with a gender wage penalty ranging from 9 log points (Llama-3.1) to 84 log points (Ministral). Only Llama-3 yielded a 15 log points wage premium for women.
  • Resume Screening by LLMs: University of Washington research found that LLMs favorably ranked resumes male-associated names 52% of the time versus female-associated names only 11% of the time. Conversely, a newer study published in June 2025 found that leading AI hiring tools built on LLMs, including OpenAI’s GPT-4o, Anthropic’s Claude 4 Sonnet, Google’s Gemini 2.5 Flash, Gemma-3, and Mistral-24B, consistently favored female candidates over white and male applicants in realistic job screening scenarios.
  • Targeted Advertising: Facebook’s job ads for STEM careers were found to disproportionately reach men, despite gender-neutral language. Historically, Facebook Ads allowed business owners to specify gender for job ads, such as nursing and teaching roles only seen by women of a specific age group, before removing this function due to bias concerns.
  • Conflicting Findings: Some studies have shown that AI can reduce bias in job descriptions by making them gender-neutral. However, the overall impact on workforce diversity remains controversially discussed, with some sources indicating potential for more homogeneity due to systematic bias.

• Racial and Ethnic Bias:

  • Resume Screening by LLMs: University of Washington research showed that LLMs favored

 

white-associated names 85% of the time versus Black-associated names 9% of the time. Specifically, resumes with Black-associated names were only preferred 9% of the time, and Black men faced the most disadvantage, with their resumes being overlooked 100% of the time in favor of other candidates. This study focused on how LLMs ranked resumes based on perceived race and gender.

  • Conflicting Findings: The newer study from June 2025 indicated that tested LLMs (like GPT-4o, Claude 4 Sonnet, Gemini 2.5 Flash, Gemma-3, and Mistral-24B) showed biases consistently favoring Black over White This occurred even when explicit anti-discrimination prompts were used, suggesting these external instructions are “fragile and unreliable” and can be overridden by subtle signals like college affiliations. Models would rationalize biased outcomes with neutral explanations, a phenomenon termed “CoT unfaithfulness” (where the LLM’s step-by-step reasoning, aka chain-of- thought, does not match the way it actually derived the output) .
  • Accents and Non-Native Speakers: An Australian study warned that AI recruitment tools risk discriminating against applicants who speak with accents, particularly non-native English speakers. One company’s AI system, primarily trained on American data, had a word error rate for non-native English speakers with accents from other countries (e.g., China) that increased to between 12% and 22%, compared to under 10% for native US English This could lead to incorrect transcription and lower ratings by the algorithm.
  • Dialect-based Prejudice: LLMs have been found to exhibit dialect-based prejudice against African American English speakers, recommending them for less prestigious jobs.

• Age Bias:

  • In a case settled by the EEOC, iTutorGroup allegedly programmed its software in 2020 to automatically reject female applicants over 55 and male applicants over 60 for online tutoring positions. More than 200 qualified US tutor applicants were not hired due to their age.

• Disability Bias:

  • Prior research found that ChatGPT exhibits disability bias when sorting
  • AI-powered psychometric tests may be inaccessible for neurodivergent candidates or those using assistive Useage of some psychometric tests, such as the Myers-Brigg personality assessment, have long been viewed as unethical during the hiring process even before AI adoption.
  • Transcription tools can be biased against individuals with a speech
  • The standardized nature of AI hiring processes may disadvantage disabled individuals by not providing the necessary flexibility for equal opportunities.

 

• Other Forms of Bias and Concerns:

  • Proxy Characteristics: AI tools may make selections based on proxies for protected characteristics, such as residential zip code, languages spoken, membership in certain organizations, or educational institutions attended, leading to discrimination. They can also screen out applicants based on arbitrary characteristics like hand placement, pose, gestures, and tone during video interviews, potentially disadvantaging those with certain disabilities.
  • Black Box Problem: Many AI algorithms are “black boxes,” meaning their decision-making processes are opaque and difficult to understand, making it challenging to identify and prove
  • Training Data: A fundamental issue is that AI systems inherently mirror patterns in their training If these datasets reflect historical or societal inequities, the AI is likely to replicate or amplify them (“garbage in, garbage out”). This can perpetuate existing inequalities and lead to more homogenous workforces.
  • Privacy: AI tools can access more types of data, such as through facial recognition, and collect vast amounts of personal information (e.g., sexual orientation, pregnancy likelihood, physical attractiveness) that is often irrelevant to job qualifications, raising significant privacy concerns and increasing the risk of misuse and discrimination.
  • Lack of Scientific Validity: Many newly offered AI tools for assessment lack sufficient scientific validation regarding their underlying criteria for predicting job Once again, psychometric tests such as the Myers-Brigg have long been utilized withing the hiring proccess despite lacking any scientific validity. This means decisions might be based on unexplained correlations or physically determined attributes like tone of voice, rather than job-relevant skills.
  • Human Touch and Transparency: The increasing use of AI can lead to a “dehumanizing” interview experience, with applicants feeling disrespected due to lack of human interaction, inability to ask questions, or receive feedback. This lack of transparency about AI usage can also prevent applicants from requesting reasonable adjustments for disabilities.

 

These empirical findings underscore the critical need for robust regulatory frameworks, ongoing audits, and human oversight to mitigate bias and ensure fairness in AI-driven hiring processes. These issues will be addressed in following issues.

 

Bibliography

 

  • Arntz, (2025, July 10). McDonald’s AI bot spills data on job applicants. Malwarebytes.
  • Carnegie Mellon (2024, June 21). AI tools reshape job application process: A Q&A with CMU’s director of employer relations Sean McGowan on the use of generative AI in the job application process.
  • Chaturvedi, , & Chaturvedi, R. (2025, May 1). Who gets the callback? Generative AI and gender bias. arXiv. https://arxiv.org/abs/2504.21400
  • Deady, (2025, June 26). How are candidates using AI in the hiring process? SocialTalent.
  • Department for Science, Innovation & (2024, March 25). Responsible AI in recruitment. GOV.UK.
  • Diem, (2025, February 4). The ethics of AI in recruiting: Bias, privacy, and the future of hiring. JDSupra.
  • Fitzpatrick, , Basu, K., & Klar, R. (2024, October 4). AI hiring tools risk discrimination, watchdog tells Congress.

Bloomberg Government.

  • com. (2024, November 25). AI hiring: The hidden discrimination and how to counteract it.
  • Hunkenschroer, L., & Kriebitz, A. (2022, July 25). Is AI recruiting (un)ethical? A human rights perspective on the use of AI for hiring. AI Ethics, 3(1), 199–213. https://doi.org/10.1007/s43681-022-00166-4
  • Hunkenschroer, L., & Luetge, C. (2022, February 8). Ethics of AI-enabled recruiting and selection: A review and research agenda. Journal of Business Ethics, 178, 977–1007. https://doi.org/10.1007/s10551-022-05049-6
  • com. (2025, March 26). What employers think of job seekers leveraging Gen AI tools.
  • Johnston, (2025, February 28). Nearly two-thirds of job candidates are using AI in their applications, report says

—experts share their best practices. CNBC Make It.

  • Martucci, C., Erbaz, B., & Jutt, M. (2024, October 2). The role of AI in employment processes. The Journal of The Missouri Bar, 80(5).
  • Milne, , & Caliskan, A. (2024, October 31). AI tools show biases in ranking job applicants’ names according to perceived race and gender. UW News, University of Washington.
  • Richmond, (2024, June 21). How to avoid the pitfalls when using AI to recruit new employees. The Times.
  • Robinson, (2024, October 20). Why 80% of hiring managers discard AI-generated job applications from career seekers. Forbes.
  • Taylor, (2025, May 13). People interviewed by AI for jobs face discrimination risks, Australian study warns. The Guardian.
  • Thapa, , & Son, H. (2025, July 11). How deepfake AI job applicants are stealing remote work. CNBC.
  • (2025, July 11). Google AI overview: Jobs AI. [Unpublished text document].

 

  • University of (n.d.). AI in recruitment: Guidance for job applicants.
  • Walsh, (2025, May 22). Using AI in your job search: Helpful or hurtful? Bentley University Newsroom.
  • (n.d.). Artificial intelligence in hiring. Retrieved November 26, 2024, from https://en.wikipedia.org/wiki/Artificial_intelligence_in_hiring
  • Zappulla, (2024, November 19). Business leaders risk sleepwalking towards AI misuse. Ethical Corporation Magazine.
  • Zilber, (2025, June 24). AI-powered hiring tools favor black and female job candidates over white and male applicants: study. New York Post.