Multilingual Metaphor Detection on the LCC Datasets
A sentence like “the foundation of our government is strong” is metaphorical because it talks about one thing in terms of another. This project fine-tunes mDeBERTa on the LCC Metaphor Dataset (Mohler et al., 2016). In the data, annotators score how metaphorical a source/target word pair is within a sentence. In “the War on Poverty,” for example, War is the source and Poverty the target. A pair counts as metaphorical if its mean score is at least 2 on the dataset’s 0–3 scale.
I work harder, get paid more, and paysource more taxestarget. literal
Results
On the held-out English test set (866 pairs, split by document so that no test document is seen in training), the model reaches F1 0.83, with precision 0.78 and recall 0.89, averaged over three seeds.
| Pred Literal | Pred Metaphor | |
|---|---|---|
| True Literal | 488 (56.4%) | 73 (8.4%) |
| True Metaphor | 39 (4.5%) | 266 (30.7%) |
Table 1. Confusion matrix for the joint fine-tuned model on the English test set (seed 0).
It leans toward labeling pairs as metaphorical. Most of its errors are literal pairs labeled as metaphors, and recall is higher than precision in every language.
| Precision | Recall | F1 | Acc. | N | |
|---|---|---|---|---|---|
| English | .776±.007 | .885±.012 | .827±.002 | .870±.001 | 866 |
| Spanish | .694±.016 | .786±.014 | .737±.003 | .855±.004 | 4,284 |
| Russian | .833±.002 | .883±.008 | .857±.003 | .889±.002 | 2,199 |
| Farsi | .618±.003 | .760±.010 | .682±.003 | .765±.001 | 2,652 |
Table 2. Test results of the joint fine-tuned model, as mean ± standard deviation (population) over three seeds. Metaphor is the positive class.
Scores follow how much the human annotators agreed with each other (Cohen’s κ 0.69 in Russian, 0.51 in English, 0.48 in Spanish and only 0.12 in Farsi).
Model comparison
Table 3 compares models on the English test set, from a sentence-level baseline to the fine-tuned multilingual model.
The bag-of-words baseline scores F1 0.59. It represents the whole sentence, so it cannot tell which two words are being judged, and the signal from the pair is diluted by everything else in the sentence.
| Model | Train | F1 | 95% CI |
|---|---|---|---|
| Sentence BoW + length, balanced | small | .588 | .544–.631 |
| All handcrafted, LR | small | .717 | .674–.756 |
| Frozen mDeBERTa spans, LR | small | .772 | .734–.809 |
| Frozen mDeBERTa spans, LR | large | .809 | .775–.840 |
| Frozen mDeBERTa spans, MLP | large | .812 | .778–.844 |
| Fine-tuned, English only | large | .833 | .800–.862 |
| Fine-tuned, joint (3 seeds) | large×4 | .827 | ≈.79–.86 |
| Annotator vs. annotator | .66 |
Table 3. English test F1 (866 pairs). “Small” and “large” are the English training bundles.
Handcrafted features of the source and target words raise F1 to 0.72, and frozen mDeBERTa vectors of the two words raise it to 0.77 on the same training data. Training on the large English bundle, with about six times as many pairs, adds roughly 0.04 (0.81 with logistic regression or an MLP). Fine-tuning adds another 0.02, reaching 0.833 when trained on English alone and 0.827 when trained jointly on all four languages. The confidence intervals of the best frozen model and the fine-tuned models overlap, so differences of that size on 866 test pairs are not conclusive.
Nonlinear classifiers improved little over logistic regression on the same features. Theory-driven features behaved as expected (very concrete source words are metaphors 50% of the time, versus 21% for very abstract ones), but added little on top of the encoder.