Large language models

LLMs are a subclass of deep neural networks based on the Transformer architecture (Vaswani et al. Reference Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin2017), which were trained on large-scale corpora of raw text, often consisting of hundreds of billions to trillions of tokens. These models process input texts as sequences of subword tokens, which are embedded as vectors and passed through multiple layers of multi-head self-attention to capture contextual dependencies across the entire sequence. LLMs are typically developed as foundation models, general-purpose pre-trained models that can be adapted to a wide range of downstream tasks. These models are autoregressive, meaning they are trained to predict the next token in a sequence, including their own previously generated tokens.

Recent years have witnessed an unprecedented expansion in the scale and capability of LLMs. Notable models include GPT-3 and GPT-4 (Achiam et al. Reference Achiam, Adler, Agarwal, Ahmad, Akkaya, Aleman, Almeida, Altenschmidt, Altman and Anadkat2023; Brown et al. Reference Brown, Mann, Ryder, Subbiah, Kaplan, Dhariwal, Neelakantan, Shyam, Sastry and Askell2020), PaLM (Chowdhery et al. Reference Chowdhery, Narang, Devlin, Bosma, Mishra, Roberts, Barham, Chung, Sutton and Gehrmann2023), Llama (Touvron et al. Reference Touvron, Lavril, Izacard, Martinet, Lachaux, Lacroix, Rozière, Goyal, Hambro, Azhar, Rodriguez, Joulin, Grave and Lample2023), Mistral (Jiang et al. Reference Jiang, Sablayrolles, Mensch, Bamford, Chaplot, de Las Casas, Bressand, Lengyel, Lample, Saulnier, Lavaud, Lachaux, Stock, Le Scao, Lavril, Wang, Lacroix and El Sayed2023), Claude (Anthropic 2024) and DeepSeek (DeepSeek-AI et al. Reference DeepSeek-AI, Yang, Zhang, Song and Zhang2025). These models have demonstrated strong zero-shot and few-shot performance across a wide range of NLP tasks, often surpassing task-specific systems. Instruction-tuned variants, such as InstructGPT (Ouyang et al. Reference Ouyang, Wu, Jiang, Almeida, Wainwright, Mishkin, Zhang, Agarwal, Slama, Ray, Schulman, Hilton, Kelton, Miller, Simens, Askell, Welinder, Christiano, Leike and Lowe2022), Llama 3 Instruct and others, extend the capabilities of foundation models by fine-tuning them on curated datasets of prompts and responses. These models are optimized to follow human instructions more effectively across tasks like summarization, reasoning and dialogs. A key component of this fine-tuning process is known as reinforcement learning from human feedback (RLHF), in which human preferences are used to rank model outputs. A reward model is then trained on this data, guiding the base LLM to produce more aligned and helpful responses through reinforcement learning. This shift has propelled LLMs from general language modeling toward interactive systems capable of following complex instructions, reasoning and multi-turn dialogs.

Recent research has leveraged LLMs for solving complex domain-specific problems: for example, “BioMedLM” (Bolton et al. Reference Bolton, Venigalla, Yasunaga, Hall, Xiong, Lee, Daneshjou, Frankle, Liang, Carbin and Manning2024) for biomedical question answering, “MathPrompter” (Imani, Du, and Shrivastava Reference Imani, Du and Shrivastava2023) for improving mathematical reasoning, novel methods for historical text summarization, “HistBERTSum-Ext” (Lamsiyah, Murugaraj, and Schommer Reference Lamsiyah, Murugaraj and Schommer2023) and “HistBERTSum-Abs” (Murugaraj, Lamsiyah, and Schommer Reference Murugaraj, Lamsiyah, During, Theobald, Hämäläinen, Öhman, Bizzoni, Miyagawa and Alnajjar2025) were proposed for extractive and abstractive approaches, respectively, and “CodeT5+” (Wang et al. Reference Wang, Le, Gotmare, Bui, Li and Hoi2023) for advanced code generation and understanding. In education, models like “EduChat” (Dan et al. Reference Dan, Lei, Yiyang, Li, Yin, Lin, Ye, Tie, Zhou, Wang, Zhou, Zhou, Chen, Zhou, He and Qiu2023) enable personalized and compassionate intelligent education, serving teachers, students and parents, while in legal and policy analysis, LLMs have been applied to extract and reason over complex legislative documents (Yao et al. Reference Yao, Ke, Wang, Li and Hu2024). HiST-LLM (Hauser et al. Reference Hauser, Kondor, Reddish, Benam, Cioni, Villa, Bennett, Hoyer, Francois, Turchin, del Rio-Chanona, Globerson, Mackey, Belgrave, Fan, Paquet, Tomczak and Zhang2024) is a benchmark for evaluating LLMs on structured historical knowledge. It uses the Seshat Global History Databank (https://seshat-db.com/), which spans thousands of historical data points across various periods and world regions. The Qur’an QA 2023 Shared Task (Malhas, Mansour, and Elsayed Reference Malhas, Mansour, Elsayed, Sawaf, El-Beltagy, Zaghouani, Magdy, Abdelali, Tomeh and Farha2023) evaluated systems, including LLMs, for their ability to extract semantically accurate answers from religious and historical texts in Arabic, using the Holy Qur’an as a benchmark corpus.

Moreover, open-source ecosystems like Hugging Face’s Transformers (https://huggingface.co/transformers/) API and inference-acceleration libraries such as Unsloth (https://unsloth.ai/), Groq (https://groq.com/) and Replicate (https://replicate.com/) have democratized access to powerful LLMs, fostering experimentation in low-resource and domain-specific applications. The evolution of LLMs continues to reshape fields like education, healthcare and digital humanities through models specialized in question answering, summarization and RAG.

Retrieval-augmented generation

Building and fine-tuning LLMs “from scratch” is practically infeasible for most research groups around the world due to the enormous amounts of training data and computational resources they require. RAG (Lewis et al. Reference Lewis, Perez, Piktus, Petroni, Karpukhin, Goyal, Küttler, Lewis, Yih, Rocktäschel, Riedel and Kiela2020) therefore presents a fascinating fusion of classic retrieval techniques with LLM-based text generation. In essence, it enables custom text data to be integrated into an LLM by converting an input query (typically a question formulated in natural language) into an appropriate vector representation known as an “embedding” (Devlin et al. Reference Devlin, Chang, Lee and Toutanova2019; Mikolov et al. Reference Mikolov, Chen, Corrado and Dean2013). This query vector is then compared to a pre-indexed set of document vectors from a domain-specific corpus or application. The most similar document vectors are then subsequently fed into the LLM’s context to generate answers that are tailored to both the background knowledge of the LLM and the retrieved documents. Notable contributions in this area include Yahoo’s NGT (https://github.com/yahoojapan/NGT) and Facebook’s FAISS (https://github.com/facebookresearch/faiss) (Douze et al. Reference Douze, Guzhva, Deng, Johnson, Szilvasy, Mazaré, Lomeli, Hosseini and Jégou2025) index structures, which facilitate an efficient form of approximate nearest-neighbor (ANN) search among the query and document vectors. A general RAG workflow is depicted in Figure 1.

Figure 1. Baseline RAG workflow.

Keyword search challenges and NLP in historical research

Traditional keyword searches in humanities contexts often require scholars to engage in multiple rounds of query refinement, significantly increasing the time and cognitive effort spent on finding the relevant information. Oberbichler and Pfanzelter (Reference Oberbichler and Pfanzelter2021) explicitly discuss the problems posed by keyword-based corpus building in humanities research. They note that simple keyword queries often fail to capture nuanced or ambiguous terms, especially in historical texts, leading to both excessive noise and unintentional omissions. This serves as their motivation for introducing text-mining methods to construct a more representative, topic-specific newspaper corpus. Kroll, Mainzer, and Balke (Reference Kroll, Mainzer and Balke2022) demonstrate that when researchers attempt to retrieve narrative-level information, such as tracking a historical event across different sources, they must decompose their inquiry into a series of discrete keyword searches and then manually reassemble the results, a process that both fragments contexts and introduces a semantic gap. Hoeber et al. (Reference Hoeber, Harvey, Momeni, Pirmoradi and Gleeson2024) observed that humanities researchers typically reformulate their keyword queries multiple times to filter out false positives and recover relevant variants, underscoring how keyword-based search interfaces fail to support complex exploratory tasks.

Given these challenges, the extensive digitization of historical records has transformed how historians analyze large textual datasets by enabling the use of NLP techniques. NLP facilitates the automated processing and analysis of vast document collections, enabling researchers to address tasks that would be impractical to perform manually. Foundational techniques in NLP, particularly relevant to the analysis of historical texts, include word count analysis or dictionary-based named-entity recognition (NER). These methods provide essential tools for quantifying language use, identifying key actors or concepts and uncovering thematic trends across large document corpora. For example, word-count analysis has proven instrumental in tracking the rise and fall of specific terms across historical periods.

This approach has significantly contributed to numerous historical studies, including the groundbreaking quantitative analysis of Culturomics (Michel et al. Reference Michel, Shen, Aiden, Veres, Gray, Pickett, Hoiberg, Clancy, Norvig, Orwant, Pinker, Nowak and Aiden2011), which utilizes millions of digitized books to uncover patterns in cultural and linguistic trends. As a result, the Google N-Gram Viewer (https://books.google.com/ngrams/) was developed by Google in collaboration with various authors (Michel et al. Reference Michel, Shen, Aiden, Veres, Gray, Pickett, Hoiberg, Clancy, Norvig, Orwant, Pinker, Nowak and Aiden2011). This multilingual online tool enables users to explore linguistic and cultural trends by analyzing changes in the frequency of words and phrases over time. The tool utilizes n-grams extracted from a vast corpus of digitized books spanning the period from 1500 to 2022.

Greenfield (Greenfield Reference Greenfield2013) utilized Google’s N-Gram Viewer to explore theories on how cultural and physiological attributes have changed over two centuries between 1800 and 2000. The author found a significant increase in the usage of words associated with individualism (e.g., “choose” and “unique”) and self-focus (e.g., “self” and “get”), indicating a cultural shift toward individualistic values and personal autonomy. Conversely, words tied to community, social responsibility and traditional values (e.g., “obliged” and “give”) declined, reflecting a move away from collectivist norms. The authors attribute these changes to societal transformations, including industrialization, urbanization and advancements in technology and education, which emphasize independence and self-expression.

Another study (Lansdall-Welfare et al. Reference Lansdall-Welfare, Sudhahar, Thompson, Lewis, Cristianini and Gregor2017) analyzed a large corpus of historical British newspapers using NLP techniques, including n-gram frequency analysis, NER, named-entity disambiguation (NED), and temporal trend analysis to uncover broad trends in history and culture. These trends included patterns in gender bias, geographical focus, technology and politics, as well as precise identification of dates for key events. The authors presented that simple content analysis enables the detection of specific events, such as wars, epidemics, coronations and conclaves, with high accuracy. Advanced NLP techniques are neural-network-based methods that go beyond foundational approaches like counting word frequencies or rule-based extractions. Neural-based NER and neural topic modeling were employed to further enhance these works by moving beyond word counting to identify mentions of specific names, places and other entities, thereby providing deeper insights into the corpus. Several studies (Indukaev Reference Indukaev2021; Marjanen et al. Reference Marjanen, Zosa, Hengchen, Pivovarova and Tolonen2021) have focused on applying classical topic-modeling techniques, particularly LDA (Blei, Ng, and Jordan Reference Blei, Ng and Jordan2003) and NMF (Lee and Sebastian Seung Reference Lee and Sebastian Seung1999), to analyze discourse dynamics and conduct thematic analysis. More recently, the authors (Hills and Miani Reference Hills and Miani2023) have investigated unsupervised techniques for thematic analysis and topic modeling. They highlight three main approaches in Historical NLP: counting words or documents, analyzing the semantic meaning of words (e.g., sentiment analysis) and organizing data through methods like topic modeling, with a particular emphasis on LDA (van Zundert et al. Reference van Zundert, Koolen, Neugarten, Boot, van Hage and Mussmann2022). Thus, to summarize, recent studies in the historical domain continue to strongly rely on LDA and NMF (Bodrunova Reference Bodrunova2021; Chappelle, Auelua-Toomey, and Roberts Reference Chappelle, Auelua-Toomey and Roberts2024; Grant et al. Reference Grant, Sebastian, Allassonnière-Tang and Cosemans2021; Gryaznova and Kirina Reference Gryaznova and Kirina2021; Karamouzi, Pontiki, and Krasonikolakis Reference Karamouzi, Pontiki, Krasonikolakis, Bizzoni, Degaetano-Ortlieb, Kazantseva and Szpakowicz2024; Lin and Peng Reference Lin, Peng, Hämäläinen, Alnajjar, Partanen and Rueter2022; Maltseva et al. Reference Maltseva, Shilkina, Evseev, Matveev and Makhnytkina2021; Marjanen et al. Reference Marjanen, Zosa, Hengchen, Pivovarova and Tolonen2021; Oiva Reference Oiva2021; Uban, Caragea, and Dinu Reference Uban, Caragea, Dinu, Zong, Xia, Li and Navigli2021).

Despite their popularity, classical models often fail to capture nuanced semantics in historical texts, and neural topic modeling methods have gained popularity for capturing complex text relationships using deep learning. Neural topic modeling methods – such as Top2Vec (Angelov Reference Angelov2020) and BERTopic (Grootendorst Reference Grootendorst2022), which uses contextual embeddings and clustering to yield more coherent topics. However, only a handful of studies have explored neural approaches in historical topic modeling (Arseniev-Koehler et al. Reference Arseniev-Koehler, Cochran, Mays, Chang and Foster2022; Cvejoski, Sánchez, and Ojeda Reference Cvejoski, Sánchez and Ojeda2023; Ginn and Hulden Reference Ginn and Hulden2024; van Zundert et al. Reference van Zundert, Koolen, Neugarten, Boot, van Hage and Mussmann2022). Moreover, BERTopic has been shown to outperform classical models in various domains (Egger and Yu Reference Egger and Yu2022; Murugaraj, Lamsiyah, and Schommer Reference Murugaraj, Lamsiyah and Schommer2025; Orr, Van Kessel, and Parry Reference Orr, Van Kessel, Parry, Yates, Desmet, Prud’hommeaux, Zirikly, Bedrick, MacAvaney, Bar, Ireland and Ophir2024; Rajwal et al. Reference Rajwal, Pandey, Han, Sarker, Demner-Fushman, Ananiadou, Thompson and Ondov2024) yet their adoption in historical research remains scarce. We believe that this gap presents an opportunity for further exploration, motivating us to employ BERTopic, a cutting-edge topic-modeling method, and integrating it into an RAG framework to further enhance the accuracy and contextual relevance of its response generation. To the best of our knowledge, this is the first work that systematically investigates a combination of topic modeling and RAG to enhance historical newspaper analysis.

Data preparation

The data preparation step integrates data pre-processing and document embedding, ensuring that the documents are cleaned, transformed and represented in vector form for subsequent analysis. The input documents – in our use case, a collection of historical newspaper articles – are pre-processed and vectorized to apply a topic modeling algorithm, which generates topic representations. The first step involves Data Pre-Processing, including minimal normalization to remove unwanted patterns from the documents. To achieve this, we utilized Python’s Regex library to remove noisy patterns and irrelevant elements commonly found in unstructured text.

Special characters (e.g., #, $ and @) were removed, excessive whitespace was normalized, irrelevant alphanumeric codes were discarded and all text was lower-cased for consistency. By limiting this pre-processing to essential steps, we retain core information while ensuring the documents remain well-structured for embedding and retrieval tasks. We deliberately avoided aggressive pre-processing techniques, such as stopword removal or lemmatization, to preserve semantically relevant content and maintain the original textual structure. These pre-processing steps were specifically tailored to the characteristics of our historical newspaper dataset; other datasets may require different or additional cleaning procedures depending on their structure, quality and content. Our method is designed to be broadly applicable across various textual data types, not limited to historical sources. The core requirements for data preparation in our system are as follows. First, the input texts should be grammatically coherent and easily interpretable by modern language models to ensure language clarity. Second, the document collection must offer comprehensive topical coverage of the subject areas relevant to the expected queries. Finally, each document must have a unique identifier (UID), with optional metadata, such as title, date or category, to support traceability. These minimal requirements enable robust retrieval and generation without the need for domain-specific pre-processing.

We next employ Jina AI’s BERT-based embedding model to create Document Embeddings for the entire collection. This model leverages the capabilities of the “Bidirectional Encoder Representations from Transformers” (BERT) framework (Devlin et al. Reference Devlin, Chang, Lee and Toutanova2019) to create dense, high-dimensional vector representations that capture the semantic content and contextual nuances of each document. These embeddings frequently serve as a foundation for further downstream tasks, such as clustering, indexing and efficient retrieval, by enabling a machine-readable, context-aware representation of the document corpus.

Topic modeling

Using document embeddings generated with the Jina AI model, we leverage BERTopic (Grootendorst Reference Grootendorst2022) to extract latent topics within the collection. This state-of-the-art topic modeling technique has been shown to outperform traditional models like LDA (Blei, Ng, and Jordan Reference Blei, Ng and Jordan2003) and NMF (Lee and Sebastian Seung Reference Lee and Sebastian Seung1999) across several benchmarks (Cheddak et al. Reference Cheddak, Baha, Es-Saady and El Hajji2024; Egger and Yu Reference Egger and Yu2022; Mendonça and Figueira Reference Mendonça and Figueira2024). BERTopic introduces an innovative approach to topic extraction by employing a so-called “class-based term frequency-inverse document frequency” (C-TF-IDF) scoring technique (https://www.maartengrootendorst.com/blog/ctfidf/) Moreover, it also utilizes advanced sentence-based embedding models to represent documents as high-dimensional dense vectors, followed by dimensionality reduction using UMAP (McInnes et al. Reference McInnes, Healy, Saul and Großberger2018). Clusters are then identified using HDBSCAN (McInnes, Healy, and Astels Reference McInnes, Healy and Astels2017), which dynamically determines the optimal number of topics, enabling precise and adaptive topic modeling.

BERTopic’s combination of contextual embeddings, dimensionality reduction and its advanced topic representation method, C-TF-IDF, makes it highly effective in handling complex datasets compared to traditional approaches. C-TF-IDF builds on the traditional (TF-IDF) weighting scheme by incorporating cluster information, enabling more precise topic representations. It is implemented using the TfidfTransformer (https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfTransformer.html) library from Scikit-learn. Specifically, the C-TF-IDF method (Grootendorst Reference Grootendorst2022) aggregates all documents from a specific class c into a single virtual document. The score for a term t in class c is then calculated as the product of the normalized class-based term frequency $tf_{t,c}$ and the inverse term frequency across all classes $tf_t$ , as follows:

(1) $$ \begin{align} C\text{-}TF\text{-}IDF_{t,c} = tf_{t,c} \cdot \log \left(\frac{A}{1+cf_t}\right), \end{align} $$

where:

$$ \begin{align*} tf_{t,c} &= \frac{\text{Frequency of term } t \text{ in class } c}{\text{Total number of terms in class } c};\\ A &= \text{Average number of words per class;} \\ cf_t &= \text{Frequency of term } t \text{ across all classes.} \end{align*} $$

This process results in a list of topic representations, with each document being assigned to the closest topic based on its embedding. These topic representations consist of the top-k keywords that best capture the theme of each topic. We then save the BERTopic model using the built-in methods provided in the same library. The saved model includes the document embeddings, topics and all other associated parameters, making it easy for future use.

Vector indexing

We next create a vector index to store document vectors using the FAISS library (Douze et al. Reference Douze, Guzhva, Deng, Johnson, Szilvasy, Mazaré, Lomeli, Hosseini and Jégou2024), which is designed for efficient vector indexing and similarity search. Instead of the flat embedding storage structure commonly used in baseline RAG systems, we propose a more organized indexing framework called the “Topic-Document Index.” This framework leverages topic information from BERTopic to group document embeddings by their corresponding topics before storing them in a FAISS index. By utilizing BERTopic’s topic-document mapping, this structured approach reduces redundancies and improves retrieval efficiency through topic-aware searches. Clustering documents under their respective topics facilitates rapid and precise retrieval of the most relevant embeddings for a given query, significantly enhancing the effectiveness of the retrieval process.

The document index functions similarly to a dictionary, where topic IDs serve as keys, while their associated document embeddings are grouped as values. This structure provides flexibility to refine searches based on the context of the user’s query, thereby improving the precision and relevance of the results. By organizing documents according to their topics, the Topic-Document Index ensures that the retrieval process considers both the content of the documents and their thematic relationships. This approach delivers more contextually relevant responses to user queries, thereby enhancing the quality of the generated answers.

Retrieval of topics and documents

The Topic-RAG workflow is initiated when a user submits a query, as depicted in Figure 2 under the “Retrieval of Topics & Documents” We use the same embedding model employed for encoding the input documents and utilize the saved BERTopic model, as outlined in the “Data preparation” and “Topic modeling” sections, to generate query embeddings and topic representations for the user query.

While BERTopic typically assigns a single topic to a query, we enhance query interpretation by leveraging the soft-clustering probabilities computed during HDBSCAN clustering, as provided by BERTopic. These probabilities indicate the degree of association between the query and each topic, enabling the identification of multiple relevant topics while filtering out weaker associations. This approach prioritizes topics that significantly contribute to the query’s semantic meaning, ensuring that irrelevant or marginally related topics are excluded. To refine our topic selection, we first identify the dominant topic that is assigned to the query by using BERTopic. We then use the highest topic probability as a reference point to set a threshold, usually 50 percent of the dominant topic’s probability, to include additional relevant topics. In other words, we treat the top-assigned topic as an anchor, and include other topics whose probabilities are at least 50 percent of the dominant topic’s score. This strategy helps us to capture adjacent or semantically similar topics that may also be relevant to the query, without allowing irrelevant or noisy topics to enter the context. The 50 percent threshold represents a conservative yet inclusive setting that provided a stable balance between capturing secondary relevance and avoiding topic drift. Based on practical observations during development, we found that higher thresholds often excluded meaningful secondary topics, while lower thresholds risked introducing unrelated content. Once the relevant topics are selected, we use their indices to retrieve the corresponding documents from the Topic-Document index.

Unlike traditional methods that retrieve a fixed number of documents (e.g., the top-k), our framework provides flexibility by allowing users to specify the number of documents they wish to retrieve for a given query. This is achieved by further ranking the sets of documents within the filtered topics based on the Cosine similarities of the documents’ embedding vectors with the embedding vector of the query, as shown by the user prompt in Figure 3. This user-driven approach enables tailored document retrieval, allowing adjustments based on the specific needs and preferences of the query, rather than relying on a fixed top-k retrieval strategy. This flexibility is particularly advantageous when sending retrieved documents to the LLM-based generator model, ensuring a better understanding of and response to the query.

Figure 3. Example of a user’s input prompt, allowing the user to adjust the number of documents retrieved for the given query.

Additionally, we ensure traceability in the Topic-RAG framework by maintaining a link between the retrieved documents and their original sources. All relevant metadata, including user input, document IDs and topic IDs, are preserved throughout the RAG process. This traceability allows users to cross-check sources and verify information, a feature that is often absent in default RAG systems.

Response generation

In the final step, the retrieved documents, along with the original query and metadata, are sent to the LLM to generate a concise and coherent response, depicted in Figure 2 as the “Response Generation.” For this purpose, we utilize the instruction-finetuned version of Llama 3.1 8B, which is known for its strong benchmark performance across various NLP tasks. This model excels in metrics, such as Accuracy, Exact Match and F1 score, making it a robust choice for generating contextually rich and informative responses.

The selection of the 8B variant is a strategic decision, offering an optimal balance between performance and computational efficiency. Its smaller size enables deployment in diverse computing environments without requiring high-end hardware, ensuring accessibility. Additionally, its streamlined architecture facilitates rapid inference, providing a seamless user experience while maintaining high-quality outputs. While this balance makes it a versatile and efficient option, it is worth noting that larger models typically exhibit higher accuracy and provide more nuanced responses in complex scenarios, as evidenced by their superior benchmark scores (https://huggingface.co/meta-llama/Llama-3.1-8B).

Experimental setup

All experiments in this study were conducted on a personal laptop, specifically a MacBook Pro Max M3 with 36 GB of RAM, highlighting that further high-performance computing (HPC) resources are not necessary for performing our neural-network tasks. For topic modeling, we utilized BERTopic version 0.16.3 to extract and organize topics from document collections. The proposed method was implemented in Python, leveraging its extensive library ecosystem for NLP. For vector indexing, we employed FAISS (using the CPU version), a highly efficient library for vector similarity search that enabled the fast retrieval of relevant documents. Document embeddings were generated using the Hugging Face Jina AI base model. For the generation task within the Topic-RAG system, we used a quantized version of the Llama 3.1 8B Instruct model, which is optimized for generating contextually relevant responses based on the retrieved documents. This setup seamlessly integrated topic modeling, efficient document retrieval and high-quality response generation, forming the backbone of the proposed approach.

Dataset

The dataset used in this research comprises 4,711 digitized Swiss newspaper articles and adverts published between 1971 and 1986 on topics surrounding “nuclear power” and “nuclear safety.” The dataset is a subset of a keyword-based collection of ca. 160,000 articles written in French and German, which were extracted from the Impresso (https://impresso-project.ch/app/) corpus of historical media collections (Düring, Bunout, and Guido Reference Düring, Bunout and Guido2024; Ehrmann et al. Reference Ehrmann, Estelle, Clematide, Düring, Fickers, Roman, Kaplan, Romanello, Schroeder, Ströbel, Beek, Volk and Wieneke2020). We relied on French and German keywords to detect as many relevant articles with references to nuclear power as possible. An extensive OR-query was compiled of manually selected keywords and keyword suggestions generated via French and German corpus-based word embeddings (as described in Düring, Bunout, and Guido (Reference Düring, Bunout and Guido2024)). This approach yielded a list of 42 keywords, which included synonyms and spelling variations such as those caused by poor OCR (e.g., “nucliör” and “atornique”). We acknowledge that for a keyword-based approach, flaws in OCR and article layout recognition can lead to both false positive and false negative results.

Each of the articles and adverts in the dataset is assigned a UID, facilitating efficient tracking and retrieval. These UIDs serve as key references throughout the research, allowing for easy cross-referencing between different analyses and findings. The articles were originally published in French and German and translated into English using the Google Translate (https://pypi.org/project/googletrans/3.1.0a0/) library in Python. This library provides unrestricted access to the Google Translate API, enabling seamless bulk language translation and detection. To ensure the quality of the translations, we employed an LLM (gpt-3.5-turbo [https://platform.openai.com/docs/models/gpt-3.5-turbo]) to automatically assess semantic alignment between the original texts and their translated counterparts. The model was prompted to generate structured evaluation reports covering dimensions, such as factual consistency, fluency, terminology and tone. To validate these automated assessments, we manually reviewed approximately 100 randomly selected translation reports across source languages. These manual checks confirmed that the translations were generally correct, fluent and semantically faithful to the source texts. We did not observe any critical translation errors or omissions that would justify exclusion.

Moreover, the articles contained various formatting issues, such as unwanted patterns, characters, tags and other inconsistencies that could hinder analysis. We therefore initiated the data pre-processing phase discussed in the “Data preparation” section, which involved identifying and removing these extraneous elements. This included eliminating irrelevant patterns, cleaning up unnecessary characters and removing unwanted tags, ensuring that the dataset was refined and suitable for further analysis and processing.

Utilizing BERTopic’s self-optimizing framework, the model automatically inferred that 20 topics most effectively captured the latent semantic structure of the corpus. Rather than being manually predefined, this number emerged from the algorithm’s internal optimization process, which balances topic coherence and cluster compactness to yield a thematically rich, data-driven representation of the content. Among the identified topics, the top-10 ranked by document frequency are presented in Figure 4, while the corresponding top-10 topic terms are shown in Figure 5.

Figure 4. Top-10 topic distribution.

Figure 5. Top-10 topics and corresponding terms.

The articles cover a wide range of topics related to nuclear power, including discussions on the construction of nuclear power plants, energy policies and the authorization of new plants. They also address the aftermath of major nuclear accidents, such as the Three Mile Island incident and the Chernobyl disaster, highlighting the global impact and concerns about nuclear safety. Additional topics include radiation protection, public health concerns related to radioactive exposure, and debates on energy initiatives aimed at securing a reliable power supply. Furthermore, there were articles about the challenges of managing radioactive waste, including storage and disposal issues, along with international research and agreements concerning the reprocessing of nuclear fuel and waste management. These topics reflect the complexity and diversity of perspectives on nuclear power as represented in the media.

Synthetic Q&A generation

Due to the lack of annotations in our original dataset and to systematically evaluate our Topic-RAG system, we generated synthetic Q&A pairs automatically by using an LLM. This approach allowed us to conduct a controlled and scalable evaluation, while acknowledging that it does not replicate the complexity of questions typically crafted by domain experts or historians. As an initial step, our focus was on generating single-document-answerable questions to validate the core functionality of our pipeline before extending it to more complex cross-document scenarios. To this end, we instructed the LLM to generate five diverse Q&A pairs for the first 100 documents, resulting in over 500 Q&A pairs in total. These questions were designed to cover a wide range of types, including fact-based, inference-based, analytical, causal and opinion-based questions, addressing various aspects of the documents, such as key events, consequences, actions taken, involved parties, technical details and their impact on the population or environment.

To support the evaluation, we manually selected a representative subset of 50 Q&A pairs as ground truth for a detailed analysis. The 50 Q&A pairs were selected to reflect a representative range of question types and clarity levels rather than simply choosing the most easily answerable cases. While care was taken to ensure diversity, we acknowledge the potential for unintentional selection bias, which we plan to mitigate in future iterations. To ensure transparency and traceability, we extracted the exact lines from the original article that the LLM used as the basis for each answer, labeling these as supporting lines. These were verified manually to minimize hallucinations and to ensure alignment between the generated Q&A pairs and the source text.

We employed prompting techniques to create these relevant Q&A pairs, leveraging the recent Unsloth (https://github.com/unslothai/unsloth) library, which facilitates faster training and inference for LLMs. For this task, we utilized the Llama 3.1 8B Instruct (https://huggingface.co/unsloth/Meta-Llama-3.1-8B-Instruct) model to generate the synthetic Q&A pairs. A sample of the prompt design is shown in Figure 6. Since each Q&A pair was generated from a single document, no question required synthesis across multiple articles, and this constraint defines our current scope of evaluation.

Figure 6. Sample prompt for generating synthetic Q&A pairs.

Evaluation metrics

In this section, we outline the various evaluation metrics we employed to assess the performance of our proposed method. We discuss both traditional metrics, such as BERTScore and ROUGE, and more recent advanced metrics like UniEval, each providing valuable insights into specific performance characteristics.

BERTScore

BERTScore (Zhang et al. Reference Zhang, Kishore, Wu, Weinberger and Artzi2020) is a widely adopted automatic evaluation metric in various NLP tasks, and we use it for assessing the quality of text generated by LLMs. It uses contextual embeddings to represent the reference and candidate tokens and then computes token-level cosine similarity between them. BERTScore also calculates Precision ( $P_{\text {BERT}}$ ), Recall ( $R_{\text {BERT}}$ ) and F1 ( $F_{\text {BERT}}$ ) using the Cosine similarity values and by the greedy matching technique. Precision measures how well the tokens in the candidate text ( $\hat {x}$ ) are represented in the reference text (x). Recall measures how well the tokens in the reference text (x) are captured in the candidate text ( $\hat {x}$ ). F1 is the harmonic mean between the Precision and Recall. Scores range from 0 to 1, with higher values indicating better alignment (https://huggingface.co/spaces/evaluate-metric/bertscore).

Specifically, given the reference sentence $x = \langle x_1, x_2, \dots , x_n \rangle $ and a corresponding candidate sentence $\hat {x} = \langle \hat {x}_1,\hat {x}_2, \dots , \hat {x}_m \rangle $ , it uses contextual embeddings to represent the tokens using the pre-trained BERT-based embedding model. The reference sentence is first tokenized, and the embedding model converts it into a sequence of vectors $\langle \mathbf {x}_1, \mathbf {x}_2, \dots , \mathbf {x}_n \rangle $ . Similarly, the candidate sentence is tokenized and converted into embedding vectors $\langle \hat {\mathbf {x}}_1, \hat {\mathbf {x}}_2, \dots , \hat {\mathbf {x}}_m \rangle $ . Then, the similarity between each reference token $x_i$ and candidate token $\hat {x_j}$ is computed using their respective vectors ( $\mathbf {x}_i, \hat {\mathbf {x}}_j$ ) as follows (Zhang et al. Reference Zhang, Kishore, Wu, Weinberger and Artzi2020):

(2) $$ \begin{align} Cosine(\mathbf{x}_i, \hat{\mathbf{x}}_j) = \frac{\mathbf{x}_i \cdot \hat{\mathbf{x}}_j}{||\mathbf{x}_i|| \cdot || \hat{\mathbf{x}}_j||}. \end{align} $$

This reduces to $\mathbf {x}_i \cdot \hat {\mathbf {x}}_j$ as both $\mathbf {x}_i$ , $\hat {\mathbf {x}}_j$ are pre-normalized vectors. Then, for each token $x_i$ in x, it finds the token $\hat {x}_j$ in $\hat {x}$ that has the maximum Cosine similarity to calculate $P_{\text {BERT}}$ . Similarly, for each token $\hat {x}_j$ in $\hat {x}$ , it finds the token $x_i$ in x that has the maximum Cosine similarity to compute $R_{\text {BERT}}$ . Finally, $F1_{\text {BERT}}$ combines $P_{\text {BERT}}$ and $R_{\text {BERT}}$ using harmonic mean as follows:

(3) $$ \begin{align} P_{\text{BERT}} &= \frac{1}{|\hat{x}|} \sum_{\hat{x}_j \in \hat{x}} \max_{x_i \in x} \mathbf{x}_i \cdot \hat{\mathbf{x}}_j \end{align} $$

(4) $$ \begin{align}R_{\text{BERT}} &= \frac{1}{|x|} \sum_{x_i \in x} \max_{\hat{x}_j \in \hat{x}} \mathbf{x}_i \cdot \hat{\mathbf{x}}_j\end{align} $$

(5) $$ \begin{align} F_{\text{BERT}} &= 2\cdot \frac{ P_{\text{BERT}} \cdot R_{\text{BERT}}}{P_{\text{BERT}} + R_{\text{BERT}}}. \end{align} $$

Extensive experiments by the authors of Zhang et al. (Reference Zhang, Kishore, Wu, Weinberger and Artzi2020) have demonstrated that BERTScore, which measures token-level semantic similarity between a generated candidate text and a reference text, correlates strongly with human judgment. In this work, we use BERTScore to evaluate the semantic fidelity of generated answers against ground-truth references. By computing BERTScore Precision, Recall and F1, we aim to provide a nuanced evaluation of the effectiveness of our proposed method.

Quantitative evaluation with baseline method

In this section, we present a comparison of the performance of our proposed method, Topic-RAG, with a well-established baseline RAG method. We structure our evaluation into two main components: retrieval quality is assessed through lexical and semantic similarity between the query and retrieved documents, reflecting how well the system locates relevant material and generation quality is evaluated using BERTScore, ROUGE and UniEval as discussed in the “Evaluation metrics” section. By assessing both methods based on the same evaluation criteria, we aim to highlight the areas where our method excels and demonstrates superior results or introduces innovative benefits. The purpose of this comparison is to objectively evaluate the performance of our proposed approach relative to existing methods, highlighting both its strengths and areas for improvement to provide a clear and comprehensive understanding of its overall effectiveness.

We implemented a modified version of the original RAG architecture as proposed by Lewis et al. (Reference Lewis, Perez, Piktus, Petroni, Karpukhin, Goyal, Küttler, Lewis, Yih, Rocktäschel, Riedel and Kiela2020), and we refer to it as Base-RAG. For the retrieval component, we analogously utilized the FAISS (https://github.com/facebookresearch/faiss) library, which supports efficient vector indexing and several methods for similarity search. For the generation component, we employed the quantized version of Llama 3.1 8B Instruct to generate coherent and contextually relevant text based on the retrieved documents. This setup leverages the strengths of FAISS for fast and scalable retrieval and the Llama model’s robust capabilities for generating high-quality text.

Retrieval quality

To evaluate retrieval quality, we assessed both lexical and semantic alignment between the retrieved documents and the ground-truth documents across a set of 50 queries. The ground-truth documents refer to the original source passages used to generate synthetic Q&A pairs via an LLM (as described in the “Synthetic Q&A generation” section). For each query, we examine whether the retrieval system is able to return the same document that was originally used to generate the Q&A pair, thereby serving as a reference for evaluating retrieval accuracy. A boxplot summarizes a distribution by displaying its median, quartiles and range, making it easy to compare performance and variability across methods.

In comparing retrieval quality between TopicRAG and Base RAG, both semantic (BERTScore) and lexical (ROUGE) metrics consistently favor TopicRAG as the superior system. As shown in Figure 7, TopicRAG achieves markedly higher BERTScore values across all three submetrics, including Precision, Recall and F1. While BaseRAG’s scores cluster around the 0.78–0.81 range with wider interquartile ranges and longer lower whiskers, TopicRAG exhibits tighter distributions with many individual scores near 1.0. Notably, despite an unexpected dip in the median of TopicRAG’s Precision (approximately 0.83), its Recall and F1 scores remain substantially higher than those of BaseRAG, indicating that it retrieves more semantically relevant content overall. The dense concentration of high-scoring points and narrower variance further affirm TopicRAG’s reliability and semantic robustness. The difference is even more pronounced when evaluating lexical alignment using ROUGE metrics, as seen in Figure 8. Although TopicRAG consistently scores above 0.9 in ROUGE-1, ROUGE-2, ROUGE-L and ROUGE-Lsum, their medians typically vary, indicating moderate token-level overlap. In contrast, BaseRAG performs substantially worse: its ROUGE-1 median hovers near 0.3, while ROUGE-2 and ROUGE-Lsum are close to 0.1 or lower, with wide variability and multiple low outliers. This dramatic gap in ROUGE performance illustrates that BaseRAG frequently retrieves documents with little to no lexical overlap with the ground truth, undermining its effectiveness in scenarios that depend on token-level alignment.

Figure 7. Evaluation of retrieval quality using BERTScore for BaseRAG and TopicRAG across 50 queries.

Figure 8. Evaluation of retrieval quality using ROUGE for BaseRAG and TopicRAG across 50 queries.

Although ROUGE scores remain relatively low across the retrieved outputs, BERTScore results offer a compelling indication of TopicRAG’s retrieval quality. ROUGE evaluates exact token overlap between the retrieved document and the ground truth, and its low values suggest limited lexical matching. However, this does not necessarily imply poor retrieval, particularly in settings where documents may express similar meaning using different phrasings. In contrast, BERTScore, by leveraging contextual embeddings, provides a more robust measure of semantic similarity. The consistently high BERTScore values across Precision, Recall and F1 suggest that TopicRAG retrieves documents that are semantically close to the reference passages, even if their surface forms differ. This pattern validates TopicRAG’s effectiveness in retrieving content that captures the intended meaning of the source, supporting its utility in applications where semantic alignment is more critical than exact wording.

Generation quality

We evaluated our proposed method by comparing the generated answers with a set of 50 ground-truth Q&A pairs. These pairs were prepared using synthetic data-generation techniques and have been validated by human experts. We evaluated by comparing the performance of our method with the baseline using various metrics, including BERTScore (Precision, Recall and F1), ROUGE (R-1, R-2, R-L and R-Lsum) in terms of F1 score and UniEval. We visualized our results using box plots. Each box represents the inter-quartile range (IQR), which contains the middle 50 percent of the values. The box spans from the first quartile (Q1, 25th percentile) to the third quartile (Q3, 75th percentile), with a line at the median (50th percentile). The whiskers extending from the box indicate the range of the scores, excluding outliers. Circles outside the whiskers represent outliers, which are values that are significantly higher or lower than the rest of the data. The central line within each box indicates the median, which represents the middle value of the distribution.

The three box plots in Figures 9–11 visualize the performance of Topic-RAG versus Base-RAG using three different evaluation metrics: BERTScore, ROUGE scores and UniEval scores. Topic-RAG shows a high BERTScore precision, recall and F1 score compared to Base-RAG. Topic-RAG’s Recall reaches up to 0.95, while Precision and F1 remain above 0.85 for most of the samples. Base-RAG shows lower precision, recall and F1 scores, while the outliers (circles) occasionally indicate better performance, but the median is lower than for Topic-RAG. Moreover, Topic-RAG outperforms Base-RAG across all ROUGE metrics, particularly in ROUGE-1 and ROUGE-2, where the median scores for Topic-RAG are higher than those for Base-RAG, indicating that it generates responses with better word overlap at the unigram and bigram levels. For ROUGE-L and ROUGE-Lsum, Topic-RAG’s median is around 0.2–0.3, suggesting that it performs better in matching longer, more meaningful text sequences than Base-RAG, whose median is much lower. Topic-RAG also performs better than Base-RAG in terms of the UniEval metric. Topic-RAG has tighter box plots with fewer outliers, indicating a very stable performance across all queries. Overall, our approach outperformed the Base-RAG model, demonstrating the advantages of incorporating topic modeling in the retrieval phase.

Figure 9. Evaluation of generation quality using BERTScore for BaseRAG and TopicRAG across 50 queries.

Figure 10. Evaluation of generation quality using ROUGE for BaseRAG and TopicRAG across 50 queries.

Figure 11. Evaluation of generation quality using UniEval for BaseRAG and TopicRAG across 50 queries.

Additionally, the retrieval time of Topic-RAG was significantly reduced compared to the baseline method. This improvement is due to our method’s ability to avoid processing the entire document corpus. Unlike basic RAG approaches that perform exhaustive searches over every document to retrieve relevant information, our method employs a more efficient retrieval strategy. By utilizing topic indices and embedding-based retrieval, it focuses only on the subset of documents most relevant to the query’s topic representation. This focused approach reduces the amount of data that needs to be processed, resulting in faster retrieval and generation times without compromising the quality of the output response, as shown in Figures 12 and 13. We measured processing time for both single and multiple queries and also tracked the time based on the number of retrieved documents. Topic-RAG’s computation time remains low when retrieving a small number of documents, but increases as more documents are retrieved. In contrast, Base-RAG’s computation time remains relatively constant, as it performs exhaustive searches regardless of the number of retrieved documents. Our results demonstrate that Topic-RAG offers a substantial improvement in computational efficiency compared to Base-RAG, particularly in scenarios involving large document corpora. By focusing retrieval efforts on a smaller, topic-relevant subset of documents, Topic-RAG reduces processing time while maintaining high-quality responses. This makes it a more scalable and efficient solution for real-time RAG tasks.

Figure 12. Processing time for a single query.

Figure 13. Processing time for five queries.

Topic RAG+ to handle long documents

We also implemented an enhanced version of our Topic-RAG framework by introducing a chunking strategy to handle long documents more efficiently and improve retrieval performance. We refer to this improved version as Topic-RAG+. The key difference between this Topic-RAG+ and Topic-RAG lies in the way document embeddings are stored and managed.

In the traditional Topic-RAG, documents are processed and stored based on their overall topic, with each document treated as a single, coherent unit. The embeddings of entire documents are computed, and these embeddings are then indexed based on their topic association. This approach works well for smaller documents, where the entire content can be effectively represented in a single embedding. However, this method encounters limitations when dealing with long documents, as they may contain multiple subtopics, which can result in inefficiencies during the retrieval process.

To address the challenge of efficiently handling long documents and improving retrieval, we incorporated a semantic chunking strategy within the indexing process. Long documents are split into smaller, semantically meaningful chunks using the SemanticChunker of LangChain (https://python.langchain.com/docs/how_to/semantic-chunker/), an advanced tool that segments documents based on the semantic coherence of the subsequent sentences. SemanticChunker divides each document into semantically coherent segments by analyzing sentence-level embedding similarity. This allows it to detect subtle topic shifts, even within documents that discuss multiple related subjects, thereby improving alignment between chunks and the underlying thematic structure. This chunking process ensures that each chunk represents a specific part of the document, capturing a meaningful portion of the content.

First, we apply BERTopic to our corpus to perform topic modeling, ensuring that the global context and primary themes of the documents are preserved. Once the documents are assigned to topics, we map them accordingly, creating a document-topic mapping. In each topic, we apply the semantic chunking strategy to break down the documents into smaller, topic-relevant chunks. Each chunk is then embedded using the same embedding model as in Topic-RAG, and the resulting embeddings are stored again in the FAISS vector index for efficient retrieval. Additionally, we store essential metadata separately, including the document IDs (UIDs), the chunk IDs (CIDs), the chunk contents and the chunk embeddings. This process enables the retrieval system to focus on semantically meaningful chunks rather than entire documents, significantly improving retrieval efficiency and accuracy.

The chunking strategy improves the granularity of the retrieval process. Instead of retrieving entire documents that may contain irrelevant information, the system can focus on retrieving the most relevant chunks to the query, ensuring the responses generated are more focused and relevant. During retrieval, each chunk is indexed with this metadata, which also includes the chunk position, ensuring the system can maintain accurate tracking of the chunk’s location within the original document.

The chunking strategy also improves the scalability of the Topic-RAG+ framework. Since large documents are broken down into smaller chunks, the retrieval process becomes more efficient even when dealing with extensive document collections. This is particularly valuable for applications that require processing vast amounts of data, such as scientific research papers, legal documents or news archives. By working with chunks rather than entire documents, the system can scale more effectively to handle larger datasets without sacrificing retrieval performance or accuracy. The combination of semantic chunking and metadata-based storage ensures that Topic-RAG+ remains scalable, adaptable and capable of handling large datasets, making it a robust solution for complex, long-document retrieval tasks. By preserving both the global document context through topic modeling and the semantic integrity of each chunk, this approach ensures that long documents are handled effectively, making it ideal for large corpora like historical newspaper archives.

Case study: Qualitative evaluation from a historian’s perspective

In addition to the quantitative analysis of Topic-RAG provided in the previous section, we also employed Topic-RAG+ for a qualitative evaluation over nine complex queries (depicted in Table 1). These queries were formulated by our historian domain experts specifically based on this corpus and research objectives, reflecting the typical lines of inquiry historians follow when analyzing archival newspapers, rather than being derived from a particular published source.

Table 1. Qualitative assessment of Topic-RAG’s responses to complex historical queries

Note: “Task” outlines the historian’s informational intent, “Query type” categorizes the nature of the question and “Comments” provide expert feedback on the relevance and quality of the generated responses. a. https://www.bk.admin.ch/ch/d/pore/va/19790218/det296.html

Here, documents were dynamically retrieved based on user input (100 documents), ensuring that the majority of relevant documents were considered. The chunking strategy proved highly effective in efficiently handling such queries. The remainder of this section presents an additional case study evaluating these complex queries, conducted independently by expert historians. This qualitative assessment by domain experts complements our quantitative analysis and reinforces the strength of our method by examining the system’s ability to support historically grounded research inquiries.

For this purpose, we identified nine intentionally challenging queries that require a combination of analytical, opinion-based, factual and inference-based reasoning. From a historiographical perspective, it is essential to recognize newspapers not merely as carriers of factual information but both as reflections of their contemporary world and as active agents that promote interpretations of this world alongside the political and/or religious orientations of their creators. Especially for this case study, newspapers are less relevant for the retrieval of historical facts but rather as carriers of a past “Zeitgeist.” Traditionally, historians have sought to approach this goal by close reading and synthesizing large volumes of text. Scholars quickly reach their limits here: the abundance of archived materials has forced those working with traditional methods to either invest considerable time or make rigorous choices regarding the number of newspaper titles or the temporal coverage. LLMs present an exciting opportunity to augment and scale these research practices by delegating tasks. In the context of this case study, we have experimented with the following:

• • the classification of themes and viewpoints according to theoretical frameworks and analysis of the distributions across newspaper titles and over time;

• • the detection and mapping of opinion-based arguments expressed by different stakeholders, such as journalists, activists or experts;

• • the distinction between editorial text striving for neutral coverage and the explicit statement of opinions;

• • the detection of subtle expressions of opinion within seemingly neutral coverage, e.g., expressed through omissions and stylistic choices.

These tasks were translated into specific queries to support a research project on the representation of the debate over nuclear power technologies in Switzerland in the 1970s and 1980s. Note that these queries would be hard to answer for a human since they require in-depth analysis of a large number of texts and abstract reasoning, and pattern recognition. Results, as presented in Table 1, are overall promising, especially in light of queries for elusive concepts, such as “scientific experts,” “fear” or “environmentalism,” which keyword-based approaches can hardly capture. A closer look at the results serves to better illustrate the strengths and some shortcomings of the system.

Seemingly precise instructions as those in Query 2 (“Which articles take an explicit position in favor or against the Volksbegehren?”), yielded summaries of the viewpoints of stakeholders first in full text and then accompanied by a list representation of the same information.

Query 3, with the task of retrieving a list of scientific experts, yielded some good results but also clear mistakes. A positive example is the following, quoted from the output: “**Mr. Hans-Rudolf Lutz**, Director of Kaiseraugst SA, engaged in discussions with the municipality about the safety of the nuclear power plant.” Other output was flawed: poor article segmentation linked Dr. Josef Amstutz, superior general of the Bethlehem Mission Society, to concerns over nuclear waste. A closer inspection showed that poor article segmentation in the input newspaper data caused the system to incorrectly merge two distinct articles.

The response for Query 5 about the media creating a sense of fear among their readers yielded encouraging results which demonstrates the potential of Topic-RAG+ to generate relevant content that is too elusive for standard keyword queries: “The Neue Zürcher Zeitung wrote that this kind of anxiety makes minds sensitive to all negative information which is welcomed and promptly disseminated by the mass media, often inflated to the point of grotesqueness or distorted, which further increases the psychological effect.”

Similarly encouraging is the output for Query 8, which asks for experts on nuclear waste. The system correctly identified a number of experts and presented them together with a summary of their representation in the newspapers: “1. Otto Luscher, an engineer from Winterthur, Switzerland, who suggests that radioactive residues can be used for future applications, such as sterilizing medical and surgical accessories.”

There are, however, also clear limitations with the current setup, which require future work. In its current configuration, the system displays a clear preference and strong performance for summarization. The research-motivated queries, on the other hand, require higher levels of abstraction (ratios, evolution over time, etc.) as well as precise contextualization. To a researcher, it matters who exactly said what, where and when. In this regard, output in the form of lists and tables with rich contextual information originating from metadata is preferable to mere summaries. Our corpus is dominated by Topic 0, containing 1,966 articles, while the other topics contain between 23 and 326 articles. This seemingly resulted in a bias toward Topic 0 when linking queries to topics. More focused queries and more explicit instructions regarding the expected output may help to address the problems we have observed.

Overall, and despite these limitations, the system has shown its potential to comprehend complex queries, to map them to corresponding topics, and to perform complex reasoning. These first experiments demonstrate, however, that the system is in principle capable of delivering such output and may perform significantly better with adjusted prompting.