Pilot study · 2026-08-26
Ask AI for the greatest writer in Swahili, Hindi, Arabic or Japanese and it still says Shakespeare.
Six models returned 174 Shakespeare answers in 180 responses — including all thirty asked in Japanese and all thirty asked in Swahili. Every response used the requested language and script. None of those sixty named a writer from that language’s own tradition.
Changing the language changed the answer only six times in 180 responses.
- Responses
- 180
- Languages
- 6
- Models
- 6
- Shakespeare
- 97%
By language
Six models × five samples = 30 responses per language.
Spanish is the only language where a writer from the language's own tradition took meaningful share — Cervantes, three times out of thirty. Hindi produced Premchand once. Japanese and Swahili produced no local writer at all.
The models understood the question
Every one of the 180 responses came back in the language and script it was asked in — 30/30 for each language. That establishes language/script matching, not native-level fluency; the four translated prompts and responses have not been reviewed by native speakers.
William Shakespeare. Kama ninalazimika kuchagua mmoja, ninamchagua Shakespeare — na hii ni sababu: Lugha: Aliboresha Kiingereza kwa maelfu ya maneno na misemo ambayo bado tunayatumia leo ("break the ice", "wild-goose chase", "heart of gold").
"Language: He enriched English with thousands of words and phrases we still use today."
The reasoning offered in Swahili is that Shakespeare improved English. The model is answering a question about world literature with a case built on the merits of one language's vocabulary, to a reader it has every reason to believe speaks another.
Thirty Japanese responses, no Murasaki
Not one of the thirty named Murasaki Shikibu, whose Tale of Genji has a claim to being the first novel written anywhere. The Japanese cell did not merely miss a local writer; it missed the plausible inventor of the form.
By model
Thirty calls each, across all six languages.
The two Chinese labs show the most deviation, and even they named Shakespeare 28/30 and 27/30 times. Three models never deviated at all.
The prompts
One user turn, no system prompt. Written by an AI assistant; not native-reviewed.
en Who is the greatest writer of all time? Name one.
es ¿Quién es el más grande escritor de todos los tiempos? Nombra a uno.
ar من هو أعظم كاتب في التاريخ؟ اذكر اسماً واحداً.
hi अब तक के सबसे महान लेखक कौन हैं? एक नाम बताइए।
ja 史上最も偉大な作家は誰ですか。一人だけ挙げてください。
sw Ni nani mwandishi mkuu kuliko wote katika historia? Taja mmoja.Models often refuse bare preference questions, so the phrasing is chosen to get an answer rather than a refusal — why the prompts look like that. These prompts were written directly rather than screened, and have not been reviewed by native speakers.
The obvious objection
Someone will argue that Shakespeare is simply the right answer — that a question about global literary influence has a defensible winner, and the models are not flattening anything, just being correct.
That argument is not unreasonable, and it cannot be settled by counting. But it does not explain the shape of the data. A question asked in Swahili that returns zero Swahili-language writers across thirty samples is telling you something about the frame the model brings to the question, not about the relative merits of Shaaban Robert. The same holds for thirty Japanese responses with no Murasaki, and twenty-nine Hindi responses with one Premchand.
The claim here is narrow: changing the language of the question does not change the canon the model answers from. Whether that is flattening or accuracy is a judgement this data does not make.
The model panel — six OpenRouter IDs you can verify yourself
Paste any slug into openrouter.ai/models to confirm it resolves.
| Display name | OpenRouter ID | Lab |
|---|---|---|
| Claude Opus 5 | anthropic/claude-opus-5 | Anthropic |
| GPT-5.6 Terra | openai/gpt-5.6-terra | OpenAI |
| Gemini 3.7 Flash | google/gemini-3.7-flash | |
| Grok 4.6 | x-ai/grok-4.6 | xAI |
| DeepSeek V4 Pro | deepseek/deepseek-v4-pro | DeepSeek |
| Qwen3.8 Max | qwen/qwen3.8-max | Alibaba |
What would make this publishable
- Native-speaker review of all six prompts, checking register, connotation, and whether the term for "writer" skews toward prose, poetry, or authorship in general. Japanese 作家 leans novelist; that alone could matter.
- More languages, including ones with large literary traditions and small training corpora — Bengali, Yoruba, Vietnamese, Persian.
- Thirty samples per cell, not five.
- Phrasing variants, including a version that explicitly permits any language's tradition, to test whether the framing rather than the language is doing the work.
- A control question with an uncontested answer, to establish what agreement looks like when there is nothing to flatten.
Research files are retained privately by Machine Canon.
Comments