publications

2025

1. 

   Adaptive political surveys and GPT-4: Tackling the cold start problem with simulated user interactions

   Fynn Bachmann, Daan van der Weijden, Lucien Heitz, Cristina Sarasua, and Abraham Bernstein

   PLOS One, May 2025

   Abs HTML

   Adaptive questionnaires dynamically select the next question for a survey participant based on their previous answers. Due to digitalisation, they have become a viable alternative to traditional surveys in application areas such as political science. One limitation, however, is their dependency on data to train the model for question selection. Often, such training data (i.e., user interactions) are unavailable a priori. To address this problem, we (i) test whether Large Language Models (LLM) can accurately generate such interaction data and (ii) explore if these synthetic data can be used to pre-train the statistical model of an adaptive political survey. To evaluate this approach, we utilise existing data from the Swiss Voting Advice Application (VAA) Smartvote in two ways: First, we compare the distribution of LLM-generated synthetic data to the real distribution to assess its similarity. Second, we compare the performance of an adaptive questionnaire that is randomly initialised with one pre-trained on synthetic data to assess their suitability for training. We benchmark these results against an “oracle” questionnaire with perfect prior knowledge. We find that an off-the-shelf LLM (GPT-4) accurately generates answers to the Smartvote questionnaire from the perspective of different Swiss parties. Furthermore, we demonstrate that initialising the statistical model with synthetic data can (i) significantly reduce the error in predicting user responses and (ii) increase the candidate recommendation accuracy of the VAA. Our work emphasises the considerable potential of LLMs to create training data to improve the data collection process in adaptive questionnaires in LLM-affine areas such as political surveys.

2. 

   Alan’s Speakeasy - An Ecosystem for the Evaluation of Conversational Agents

   Luca Rossetto, Cristina Sarasua, Oana Inel, Juan Diego Bermeo, Pablo Sebastian Bolaños, Hugues Devimeux, Huiran Duan, Robin Hany, Athina Kyriakou, Dhivyabharathi Ramasamy, Varun Ghat Ravikumar, Konstantina Timoleon, Rosni Vasu, Ruijie Wang, Daan van der Weijden, Jan Willi, and Abraham Bernstein

   In Companion Proceedings of the ACM on Web Conference 2025 , Sydney NSW, Australia, May 2025

   Abs HTML

   Conversational artificially intelligent agents have received increasing attention in recent years. For many use cases, evaluating such agents requires a human-in-the-loop approach. In this demo paper, we present Alan’s Speakeasy, a Web-based ecosystem for the evaluation of conversational agents.

2024

1. 

   Playing with Fire: How the interplay between the Dutch House of Representatives and social media fuels rage

   Jeroen Bakker, B.A. de Graaf, Sander Prins, Mirko Schaefer, and Daan van der Weijden

   May 2024

   Abs HTML

   What are the dynamics between political debate in the House of Representatives and reactions, comments and framing on social media? This was investigated by researchers from Utrecht University’s Faculty of Humanities and Data School. Analyses of nearly 50 million messages on Twitter and Telegram provide insights into the interaction between political debate in the House of Representatives and online. Central to these analyses is how threatening and radicalising language moves between politics and social media.

2023

1. 

   The multifaceted and situated data center imaginary of Dutch Twitter

   Karin van Es, Daan van der Weijden, and Jeroen Bakker

   Big Data and Society, May 2023

   Abs HTML

   Data centers are material structures that take up space, use resources like water and energy, and possess a large carbon footprint. This paper examines the broader long-term discussion around data centers during the period 2020–2022 in the Dutch Twittersphere. Through an analysis of tweets and images, it identifies and reflects on the communities active in the discussion and the range of visions and imaginaries of data centers they produce. Unpacking these tweets and images over time traces not only the emergence of a ‘reactive imaginary’, critical of the promises of information technology (IT) industry and (local) governments, but also the blind spots of the discussion. It furthermore reveals an important role for journalism in the discussion by questioning the claims of the industry and contributing to a ‘visibility expansion’ of data center’s impact on Earth’s resources. The paper shows the multifaceted and situated nature of imaginaries and their role in shaping decision-making and policy.

2022

1. 

   Bursting the Burden Bubble? An Assessment of Sharma et al.’s Counterfactual-based Fairness Metric

   Yochem van Rosmalen, Florian van der Steen, Sebastiaan Jans, and Daan van der Weijden

   May 2022

   Abs HTML

   Machine learning has seen an increase in negative publicity in recent years, due to biased, unfair, and uninterpretable models. There is a rising interest in making machine learning models more fair for unprivileged communities, such as women or people of color. Metrics are needed to evaluate the fairness of a model. A novel metric for evaluating fairness between groups is Burden, which uses counterfactuals to approximate the average distance of negatively classified individuals in a group to the decision boundary of the model. The goal of this study is to compare Burden to statistical parity, a well-known fairness metric, and discover Burden’s advantages and disadvantages. We do this by calculating the Burden and statistical parity of a sensitive attribute in three datasets: two synthetic datasets are created to display differences between the two metrics, and one real-world dataset is used. We show that Burden can show unfairness where statistical parity can not, and that the two metrics can even disagree on which group is treated unfairly. We conclude that Burden is a valuable metric, but does not replace statistical parity: it rather is valuable to use both.