Language-Specific Loop Structure in mBERT Attention Graphs
Unlike other continuous physical phenomena like temperature or speed, color has no system of precise measurement in the common vernacular, so languages carve it into discrete categories and draw them differently. Russian splits blue into dark синий and light голубой, and Russian speakers are faster at telling shades across that boundary apart (Winawer et al., 2007).
This project set out to test whether color is distinct between languages in a multilingual model, with the two Russian blues as the motivation. What it found is that the language itself is distinct, not the color. Non-color adjectives like big, small, old and good show the same statistically significant difference between languages as the color terms do.
Following Kushnareva et al. (2021), whose pipeline I use as the feature extractor, each mBERT attention head over a sentence is treated as a weighted graph, and persistent homology summarizes its shape as connected components (H0) and loops (H1). The data are about 200 sentences per color term in English, Spanish and Russian, drawn from the news corpora (2019, 2020 and 2023) of the Leipzig Corpora Collection.
components (H0)loops (H1)
Figure 1. One attention head, read as a shape. Real weights from mBERT layer 2, head 7 (counting from 0) on a corpus line, “They are blue and have a very unique look.” A token pair joins once 1 − attention falls below the threshold.
Results
On all contexts the language effect is overwhelmingly large (z = 55–80). Two controls change what it means. On contexts matched for length, the H0 effect shrinks to z = 2.7–4.5 while the H1 effect holds at z ≈ 33–34, and the non-color adjectives show the same pattern at the same size.
color termsnon-color adjectives
Wasserstein distance
Bottleneck distance
Figure 2. Language effect (z against shuffled languages) for color terms and non-color adjectives, on all sentences and on sentences matched for length.
Figure 3. Language effect in each of mBERT's 144 attention heads (Wasserstein distance, color terms, length-matched sentences). Dots mark heads significant after Benjamini–Hochberg correction.
mBERT also splits Russian far more finely, at 1.8 wordpieces per word against 1.2–1.3 for English and Spanish. Controlling for tokenization removes the H0 effect, but the H1 effect survives (p = 0.001), for color and adjectives alike. It is small, though, only 2–4% above the variation within a single language’s sample. The two Russian blues do not separate at all (p = 0.57–0.97).
| W H0 | W H1 | B H0 | B H1 | |
|---|---|---|---|---|
| синий vs. голубой | 0.91 | 0.76 | 0.97 | 0.57 |
| {blue, azul, синий} | 0.096 | 0.068 | 0.146 | 0.044 |
| {blue, azul, голубой} | 0.52 | 0.34 | 0.52 | 0.98 |
Table 1. Do the Russian blues separate? Permutation p-values for the within-Russian split and the two blue translation triples. W = Wasserstein, B = bottleneck distance. Highlighted: p < 0.05.
Discussion
Whatever separates the languages in mBERT’s attention topology, it is not color-specific. What survives the controls, a language difference in loop structure, may reflect syntax or word order, or tokenization effects finer than the summaries used here.