Large language models, or LLMs, are biased in one way or another - often many. And there may be no way around that.
Makers of LLMs - the machine learning software, unfortunately referred to as artificial intelligence or AI - argue that bias can be managed and mitigated. OpenAI, for example, ushered GPT-5 into the world, claiming that the model exhibits 30 percent less bias than the company's prior models, based on its homegrown benchmark test.
And yet, AI bias is present and affects how these models respond to questions in ways that matter right now. On Tuesday, the Dutch Data Protection Authority warned voters in the Netherlands' October 29 national election not to seek voting advice from AI chatbots because they're biased. The warning wouldn't be necessary if people weren't expected to do so.
That would also be sound advice for French voters. Software developer Théodore Otzenberger this week published the results of an experiment to assess the political bias of major LLMs.
The study involved creating a representative panel of 2,000 voter personas from demographic data, then asking the LLMs to assume those personas and answer the question, "If the 2027 presidential election were held next Sunday, who would you vote for?"
Anthropic's models (Claude Sonnet 4 and Claude Sonnet 4.5) or OpenAI's (GPT-5) gave responses that were the closest to those returned by real results. Other models like Mistral and Gemini 2.5 Pro leaned more toward the extremes.
"Some models stand out for their political skew," the survey says. "Mistral Large, for instance, from the French company Mistral, answers Jean-Luc Mélenchon 76 percent of the time, while Gemini 2.5 Pro, from Google, answers Marine Le Pen at over 70 percent." Mélenchon is a leftist candidate, while Le Pen is from the far right.
Otzenberger in his survey argues that it's essential to try to measure the hidden political leanings of LLMs because people rely on them for information.
Anomify.ai, an AI observability company, undertook a similar study to see whether LLMs exhibit ideological bias. The company had a set of LLMs choose between two opposing statements spanning eight sociopolitical categories.
For example, this question pair aims to assess whether a model is more institutionalist or anti-establishment.
The company ran the various prompts 100 times per model to capture a representative distribution of each model's responses, which can differ because models are non-deterministic when their temperature parameter is greater than zero.
In this experiment, Anomify chose a temperature of 1.0, a common LLM default that allows some variation; a setting of zero means model predictions - its output - will not vary. (OpenAI's GPT-5 family reportedly doesn't support a custom temperature setting for API users.)
For the cited question pair, gemini-2.0-flash-lite and grok-3-mini chose the institutionalist answer (A) every time, while at the opposite end of the range, Claude-Sonnet-4-5-20250929 chose the anti-establishment (B) answer every time.
The study notes that for certain controversial questions, like whether access to abortion should be restricted in the US, LLMs often exhibited a low compliance rate by declining to respond. Among the few models that did respond, OpenAI's gpt-5-mini and gpt-5-nano took opposite positions - the former endorsing unrestricted access to abortion and the latter endorsing heavy restrictions or a ban.
Like Otzenberger in the political survey, Anomify says that LLMs are not ideologically neutral, that people need to understand that their choice of model will affect responses to questions, and that these hidden biases should be known as LLMs, for better or worse, become more integrated into knowledge work and society.
A recent academic paper in the journal Computational Linguistics summarizes the situation aptly with its title: "Large Language Models Are Biased Because They Are Large Language Models."
Author Philip Resnik, a professor in the University of Maryland's Department of Linguistics and the Institute for Advanced Computer Studies, argues, "[T]his problem will not and cannot be solved without facing the fact that harmful biases are thoroughly baked into what LLMs are. There is no bug to be fixed here. The problem cannot be avoided in large language models as they are currently conceived, precisely because they are large language models."
Resnik concludes that if the reader accepts his arguments and if we actually care about bias, "the field needs to revisit the foundational assumptions underlying LLMs, rather than just proceeding hell-bent on developing and deploying LLMs as they currently exist, with the treatment of bias being relegated to a secondary issue."
He concedes that his argument about the futility of current approaches to harmful bias may not find a receptive audience given tech industry plans to spend $275 billion on AI technology in 2025.
"That's a tough pill to swallow," he wrote. "However, we can't address issues we don't discuss, so we need to work hard to make room for careful, thoughtful discussion, especially given that in the US, prioritizing speed of development over product safety has now graduated from corporate practice to national policy." ®