Evaluating BERTopic N-gram Configurations on Latvian Exile Press (1940s–2000s): the Coherence Paradox
Abstract
Simon Donig, Dinara Gagarina, Timur Mitrofanov (17. September)
Abstract
The 20th-century history of Latvia is a history of forced displacement, with approximately 175,000 Latvians forming global communities from the USA to Australia (Plakans 2021, 64). This "Latvia outside Latvia" created a robust media landscape comprising journals and magazines that served as a surrogate public sphere for those severed from their homeland. These were European voices echoing far beyond Europe’s borders, preserving their heritage through the press. To analyze these discourses, we employed BERTopic for topic modeling. While the topic modeling approach has long been criticized for low reproducibility and interpretability, frameworks like BERTopic are based on enhanced semantic coherence and are potentially better. In this study we check whether high coherence scores actually indicate meaningful topics. Analyzing about 7,000 English-language articles from 27 Latvian diaspora newspapers (1940s–2000s), we compare four n-gram configurations (unigram, bigram, trigram, and combined n-grams) using two coherence metrics (C_V and C_NPMI). Our findings reveal a paradox: bigram models achieving near-perfect coherence scores in many cases captured just formulaic content – telephone directories, contact information, and administrative listings – rather than historical themes. The combined n-grams model achieved optimal balance, while trigram models showed weakest characteristics. Our findings indicate that topic modeling captures sign patterns and statistical co-occurrences rather than semantic content directly. High coherence can signal structural regularity without guaranteeing historical meaningfulness.
Session P4: Digital Fingerprints: Detecting Human and Machine Authorship