Table of Contents

• How does bias creep into AI systems used in HR?
• What kinds of human bias can AI actually reduce?
• Why data quality and training matter for fairness
• Can AI really improve performance management?
• What HR leaders should ask vendors about AI bias
• How regulation and lawsuits are shaping the future of AI in HR

Resources

• Kate’s AI vendor questionnaire
• AllVoices responsible AI trust center
• Kate’s Linkedin
• Claire’s Linkedin

How does bias creep into AI systems used in HR?

Bias in HR isn’t new. It shows up in hiring, promotions, pay equity, and performance reviews. What’s new is that AI is now involved in many of those processes — and without care, technology can replicate the same inequities at scale.

Kate O’Neil, co-founder of Opre, explained during our webinar that AI systems are only as fair as the data and rules they’re trained on.

“Bias doesn’t disappear just because you’re using AI. If you feed biased data into a model, you’re going to get biased outcomes.”
— Kate O’Neil, Co-founder of Opre

This is the uncomfortable truth for HR. AI can feel like a neutral tool, but the patterns it learns often reflect existing inequalities. If women or people of color have historically been rated lower in performance reviews, an AI trained on those reviews will perpetuate that bias.

Why HR can’t ignore this risk

• AI can scale bias faster than humans can check it
• Lack of transparency makes it harder to challenge AI-driven outcomes
• HR leaders are accountable for the fairness of systems, even if a vendor built them

Claire Schmidt added that this is why employee reporting channels matter more than ever. If someone suspects bias in an AI-powered process, they need a safe way to flag it.

“The technology is new, but the accountability isn’t. Employees will still look to HR to explain and fix unfair outcomes.”
— Claire Schmidt, Founder of AllVoices

What kinds of human bias can AI actually reduce?

While AI introduces risks, Kate was careful to note that it can also reduce bias — especially when compared to flawed human processes.

“We know humans are biased. Managers bring their own preferences and blind spots into performance conversations. AI, if designed responsibly, can help surface more objective patterns.”
— Kate O’Neil

Where AI can help level the playing field

• Performance reviews: Flagging inconsistent language in manager feedback
• Hiring: Reducing the weight of gut feel in resume screening
• Pay equity: Identifying unexplained gaps in compensation data
• Employee surveys: Synthesizing sentiment without amplifying the loudest voices

The point isn’t that AI is perfect. It’s that it can create more consistency and highlight issues that humans might overlook or downplay.

Still, Kate cautioned that this requires intentional design. If HR leaders assume “AI = objective,” they risk replacing one kind of bias with another.

Why data quality and training matter for fairness

The quality of the data fed into AI models is the biggest determinant of whether the output is fair. Kate emphasized that many HR systems are built on shaky foundations.

“If your performance reviews are full of vague language, or if you’ve never audited them for bias, you can’t expect an AI model trained on them to be any better.”
— Kate O’Neil

This means HR has to get serious about data hygiene before plugging AI into workflows.

Steps to improve HR data quality

• Audit performance reviews for biased language (e.g., “aggressive” used for women vs. “decisive” for men)
• Standardize criteria across roles to reduce manager subjectivity
• Provide training so evaluators understand how their input shapes AI models
• Regularly test outputs for disparate impact across demographics

Data cleanup isn’t glamorous, but it’s essential. Without it, HR risks automating inequities under the banner of efficiency.

Claire tied this back to transparency:

“If you can’t explain where the data came from, how the model works, and how you’re checking for bias, employees won’t trust the system.”
— Claire Schmidt

Can AI really improve performance management?

One of the most practical discussions centered on performance management. It’s an area every HR leader struggles with — and one where AI could either be a breakthrough or a liability.

Kate argued that AI has a real role to play in making reviews fairer.

“AI can’t replace the conversation between a manager and an employee. But it can highlight patterns that make those conversations more consistent.”
— Kate O’Neil

Potential uses of AI in performance management

• Checking review text for biased descriptors
• Identifying rating inflation or deflation trends across managers
• Highlighting discrepancies in feedback given to different demographic groups
• Suggesting standardized phrasing to reduce subjectivity

These applications don’t replace human judgment — they support it. The key is keeping managers accountable for decisions while using AI as a mirror to reveal blind spots.

The group agreed on one thing: AI should never be used to make final promotion or compensation decisions. Humans must stay in the loop for outcomes that shape careers.

What HR leaders should ask vendors about AI bias

One of the practical takeaways from the webinar was a set of questions HR leaders should pose to vendors before buying AI-enabled tools. Kate outlined the essentials.

“Don’t just ask what the tool can do. Ask how it was trained, how it’s monitored, and how you’ll know if it’s fair.”
— Kate O’Neil

Vendor questions HR should insist on

• What data was used to train this model?
• How do you audit for bias in outputs?
• Can you provide documentation of disparate impact testing?
• How do employees challenge or appeal AI-driven decisions?
• What regulatory frameworks do you comply with?

Asking these questions signals that HR isn’t just buying features — they’re buying accountability. Vendors that can’t answer transparently should raise red flags.

Claire noted that employees themselves will likely ask similar questions soon. HR has to be ready with answers.

How regulation and lawsuits are shaping the future of AI in HR

The session closed with a look at the regulatory and legal landscape. From New York City’s AI bias audit law to Colorado’s AI transparency bill, governments are starting to catch up.

Kate pointed to the recent Workday class action lawsuit as a sign of what’s coming.

“We’re going to see more litigation as employees challenge AI systems they believe are biased. HR needs to be ready — both legally and culturally.”
— Kate O’Neil

What HR leaders should do now

• Stay updated on evolving AI regulations in hiring and employment
• Work with legal to ensure compliance before adopting new tools
• Build audit and appeal processes into AI-enabled workflows
• Communicate openly with employees about their rights and protections

The message was clear: regulation isn’t a distant concern. It’s here now, and HR teams that ignore it put themselves — and their organizations — at risk.

Final word: AI and bias demand HR leadership

AI brings both promise and peril to HR. It can surface patterns humans miss, but it can also scale inequities faster than ever. The difference lies in how HR chooses to adopt it.

Kate O’Neil’s call to action is direct:

“Bias isn’t going away on its own. If HR wants AI to be fair, they have to be intentional about design, data, and accountability.”
— Kate O’Neil

And as Claire Schmidt reminded us, the trust employees place in HR won’t disappear just because a tool is involved.

“At the end of the day, people don’t blame the software. They blame the organization. HR has to own the fairness of these systems.”
— Claire Schmidt

By approaching AI adoption with clear problem definitions, data discipline, vendor accountability, and cultural transparency, HR leaders can make AI part of a more equitable future — not a faster path to the same old inequities.