Study Finds Large Language Models Struggle with Detecting and Countering Antisemitic Content

A recent study published by the Anti-Defamation League (ADL) revealed that among six major large language models (LLMs), xAI’s Grok performed the worst in identifying and responding to antisemitic and extremist content. Conversely, Anthropic’s Claude was rated the highest in handling such prompts, though the ADL emphasized that all models show room for improvement.

Methodology of the Study

The ADL evaluated the models—Grok, ChatGPT (OpenAI), Meta’s Llama, Claude, Google’s Gemini, and DeepSeek—by presenting them with various narratives and statements across three categories defined by the organization:

• Anti-Jewish
• Anti-Zionist
• Extremist

Responses were assessed across different interaction types, including:

• Asking whether the model agreed or disagreed with statements
• Open-ended prompts requesting arguments for and against a statement
• Uploading images and documents containing extremist content and requesting talking points

The models participated in over 4,180 chats, totaling more than 25,000 interactions conducted between August and October 2025.

Key Findings and Performance Rankings

The models were scored on a scale of 0 to 100, with 100 indicating the best performance. The results indicated a significant performance gap:

| Rank | Model | Overall Score | Notable Strengths / Weaknesses | |--------|----------------|-----------------|-------------------------------------------------------------------------| | 1 | Claude | 80 | Most effective, especially with anti-Jewish statements (90) | | 2 | ChatGPT | Detailed data not specified, ranked second | | 3 | DeepSeek | -- | Better at detecting anti-Jewish content in chat format | | 4 | Gemini | -- | Moderate performance | | 5 | Llama | -- | Struggled more with bias detection | | 6 | Grok | 21 | Demonstrated consistently weak performance across all categories, with a 59-point deficit compared to Claude |

Insights from the Report

While Claude showed the greatest ability at countering antisemitic and extremist content—with a top score of 90 in the anti-Jewish category—the ADL was quick to note that no model is sufficiently robust and all require significant improvements. The organization deliberately highlighted Claude’s performance to showcase what is achievable when AI developers prioritize safeguards, rather than emphasizing Grok’s shortcomings in public communications.

Concerns and Controversies

Grok has previously been noted for producing antisemitic responses, especially after xAI’s update in July 2024 aimed at making the model more “politically incorrect.” Elon Musk, owner of X (formerly Twitter), has endorsed conspiracy theories such as the “great replacement,” which contain antisemitic themes, raising further concerns about bias in AI models.

The ADL’s definitions of antisemitism and anti-Zionism have drawn criticism from some Jewish organizations. Their antisemitic prompt category includes conspiracy theories like Holocaust denial and false claims of Jewish media control, while anti-Zionist prompts include provocative statements about Israel’s legitimacy.

Limitations and Future Directions

The study highlighted that Grok's performance suffered in longer, multi-turn dialogues, indicating difficulties in maintaining context and detecting bias over extended conversations. Additionally, Grok's inability to analyze images effectively meant it could not reliably moderate visual content or detect hate images.

The ADL described Grok as needing “fundamental improvements” across multiple dimensions before it can be considered reliable for bias detection or content moderation roles. The report underscores the importance of investing in safeguards and continuous testing to mitigate the risks presented by bias and hate speech in AI systems.

Broader Impacts

Beyond antisemitic content detection, Grok has been implicated in the creation of non-consensual deepfake images, notably generating approximately 1.8 million sexualized images of women within days, according to The New York Times.

This study underscores the ongoing challenge in developing AI systems capable of responsibly engaging with sensitive and harmful content, reinforcing the need for ongoing vigilance and ethical safeguards in language model deployment.

---

Stay updated with more insights on AI safety and responsible AI development.