Qualitative Analysis and Large Language Models (LLMs)

The next event in the QUEST/NCRM/SCDTP seminar series will take place online on zoom with our usual format of three presentations on a linked theme and a discussion. Our theme is qualitative analysis and large language models (LLMs).

Large Language Models and the Future of Qualitative Research

Susan Halford and Les Carr

Susan Halford is Professor of Sociology at the University of Bristol, and co-Director of the ESRC Centre for Sociodigital Futures, and Les Carr is Professor of Web Science and a Director of the Web Science Institute at the University of Southampton.

The growth of Large Language Models has prompted widespread claims that these ‘tools’ will transform the future of qualitative research. Overall, the message is that qualitative researchers should equip themselves for a fast approaching future, or risk irrelevance. In this talk we examine these claims theoretically – drawing on work in the sociology of futures, and the social life of methods – and practically, by reporting on our own experiments with LLMs. Pulling these two elements together reveals a significant disjunction between the prophetic future claimed for LLMs and our own experience. Our conclusion is not to reject the use of LLMs in qualitative research, but rather to change the terms of the debate, fully aware of its consequences the futures in-the-making.

Harnessing AI for narrative insights: Usiing LLMs to analyse story completion data

Sarah Jenner

Sarah Jenner is a lecturer in Child and Adolescent Health and qualitative researcher based in the School of Health Sciences at the University of Southampton.

This talk will introduce a study conducted in collaboration between University of Southampton and Ipsos UK, which aimed to develop and test a novel method for analysing qualitative data using large language models (LLMs). We compared LLM-conducted analysis with human analysis of qualitative data collected using a story completion method. We also explored optimisation of LLMs for narrative analysis and evaluated their benefits and drawbacks. Beyond replication, LLMs provided additional insights into the data that enhanced the human analysis. Our study highlights the significant potential benefits of LLMs to the field of qualitative research and proposes that LLMs could one day be seen as valuable tools for strengthening research quality and increasing efficiency. The talk will also discuss ethical concerns surrounding responsible AI use in research and proposes a framework for using LLMs in qualitative analysis

Large Language Models in Qualitative Research: Uses, Tensions, and Intentions

Marianne Aubin Le Quere and Casey Randazzo

Dr. Marianne Aubin Le Quéré is a postdoctoral fellow at Princeton’s Center for Information Technology Policy, and Dr. Casey Randazzo is an Assistant Professor in the Department of Communication at UC Santa Barbara.

Qualitative researchers use tools to collect, sort, and analyze their data. Should qualitative researchers use large language models (LLMs) as part of their practice? LLMs could augment qualitative research, but it is unclear if their use is appropriate, ethical, or aligned with qualitative researchers’ goals and values. In this talk, we will present our findings from interviews with twenty qualitative researchers to investigate these tensions. Many participants see LLMs as promising interlocutors with attractive use cases across the stages of research, but wrestle with their performance and appropriateness. Participants surface concerns regarding the use of LLMs while protecting participant interests, and call attention to an urgent lack of norms and tooling to guide the ethical use of LLMs in research. We document the rapid and broad adoption of LLMs across surfaces, which can interfere with intentional use vital to qualitative research. We use the tensions surfaced by our participants to outline recommendations for researchers considering using LLMs in qualitative research and design principles for LLM-assisted qualitative research tools. We hope to use this talk to discuss how we can strengthen connections between social science and computational fields, especially when it comes to the development of AI research tools.