IRIS Colloquium I Additional date I Ana Barić: Recalibrating uncertainty for social science in the era of LLMs

We are delighted to offer this additional session. Our guest, Ana Barić, will share her research. She is a PhD student in NLP at TakeLab FER, University of Zagreb. Her research interests span from exploring model uncertainty in language models to tackling diverse topics in computational social science, such as sentiment analysis and bias detection.

Abstract

The high volume of pretraining data for LLMs can be seen as a double-edged sword. While it enables generalization, it also introduces noise and complexity when handling subjective tasks due to diverse perspectives captured in the data. This results in increased model sensitivity and output variation that is not easily interpretable. To address these challenges, uncertainty methods can help manage data ambiguity, improve robustness, and enhance interpretability, leading to more reliable and transparent insights.

In this talk, we will revisit the uncertainty framework for LLMs, focusing on estimation, evaluation, and calibration through the lens of computational social science. We will explore the uncertainty-confidence dilemma, how different uncertainty sources can be leveraged for subjective tasks, and the connection between uncertainty and human label variation while highlighting open challenges at the intersection of uncertainty and social science.

[Picture: SRF IRIS]