Vol. IX · Data Plate 3 of 12language identification, character n-grams, fastText, CLD, short text, multilingual dataDE

Which Language.

Count the character sequences, compare with each language’s profile — easy for a paragraph, a coin toss for five letters.

A pretraining set is built language by language: English for one share, Chinese for another, code for a third. The sorting is done by a language identifier, usually a small classifier over character n-grams — fastText’s is a common choice — which assigns each document a language and a confidence. Documents below a confidence threshold are dropped or set aside.

Character n-grams work because languages differ in their letter sequences: “the” and “ing” for English, “ung” and “sch” for German, “ción” for Spanish, Han characters for Chinese. On a paragraph the evidence is overwhelming. On a short snippet — a title, a tweet, a line of code — it is thin, and the identifier guesses. The workshop’s identifier uses profiles of six classes, built from a few sentences each, and checks Han script first, as practical identifiers do.

Below: left, the identifier’s accuracy on held-out text cut into snippets of a chosen length, for each class. Right, its scores for a text you type.

What you will learn
Workshop
A character 1–3-gram identifier with profiles for English, Chinese, French, German, Spanish and Python code. Left: accuracy on held-out text cut into consecutive snippets of the chosen length (up to 30 per class). Right: scores for the typed text.

Plenty of evidence, or almost none

Averaged over the six classes, the identifier is right 60% of the time on five-character snippets, 89% at twenty characters and 100% at eighty. The errors at short lengths are between related languages — English read as Spanish, French as Spanish or German — and between code and prose, because a handful of characters shares n-grams with several profiles. Chinese is nearly always right, because the script alone decides it.

Real identifiers cover two hundred languages or more, and the hard cases are the ones with little text to learn from. Audits of web-crawled multilingual datasets found that for many low-resource languages, a large share of what was labelled as that language was something else — a related language, boilerplate, or nonsense. A confidence threshold trades one error for another: set high, it discards genuine text in under-represented languages and dialects; set low, it lets mislabelled text in. Either way, the languages with the least data pay most.

Code, mathematics and mixed-language text are awkward for language identifiers; pipelines usually route them separately (plate IX.7) rather than trust a prose classifier.

Terms

Language identification
Assigning each document a language, usually with a confidence.
Character n-gram profile
How often each short character sequence occurs in a language’s reference text.
Confidence threshold
The minimum confidence for a document to be kept as a given language.
Low-resource language
A language with little text available; identification errors weigh most on it.
Language identification is easy on paragraphs and unreliable on snippets, and its errors fall hardest on the languages with least data.

Sources

  1. Joulin et al., “Bag of Tricks for Efficient Text Classification”, EACL 2017 (fastText, with its language-identification models).
  2. Caswell et al., “Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus”, COLING 2020.
  3. Kreutzer et al., “Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets”, TACL 2022.