Human translators beat AI on only two of six content types in English-to-Chinese benchmark
A benchmark study of English-to-Chinese localisation has found that professional human translators finished outside the top five on four of six content types, and placed 10th of 15 on marketing copy, 22.2 points behind the best AI workflow. The research, a joint project by localisation provider EC Innovations and Jademond Digital, scored 774 outputs across six content types, seven workflow models and three task types.
Humans still won two categories, informational and SEO content, where factual precision and terminological consistency matter most and creative latitude is near zero. But the margin over the best machine workflow in those wins was only 2.8 points, which the authors describe as too narrow to base a sourcing decision on without replicating the test on your own content.
Post-edited Qwen won three categories outright, tied a fourth and placed second on the remaining two. Post-edited Google Translate scored 66.7 on SEO content, ahead of raw Qwen and every raw model in the study, suggesting that adding a post-editing layer to an existing machine translation pipeline with translation memory and a termbase gets further than switching to a raw large language model.
The study warns against treating category averages as procurement inputs. The four Chinese models tested span 9.2 points on SEO content and 22.3 points on technical content, while the two Western models differ by at most 8.4 points. Kimi finished last among the Chinese four in all six content types, so removing it, as any enterprise would after a short bake-off, shifts the category average by between 1.3 and 4.9 points.
The authors note that the benchmark measured localisation quality only, with no ranking, traffic or conversion data collected, and that the link between content quality metrics and search rankings is weaker than the industry generally concedes.
Reported by Search Engine Journal ↗
Topics: Measurement