On January 12, 2024, Razan Jaberibraheem will present her PhD thesis at the Department of Computer and Systems Sciences (DSV), Stockholm University. The title of the thesis is “Towards Designing Better Speech Agent Interaction: Using Eye Gaze for Interaction”.

Razan Jaberibraheem at the “nailing ceremony” – it’s a Swedish tradition that PhD students use hammer and nails to put their printed thesis up on the department wall before the defence. Photo: Donald McMillan.

The defence takes place at DSV in Kista, starting at 13:00 pm.
Find your way to DSV

You can also take part in the defence online:
Link to Zoom
Contact Karey Helms to get the password.

The PhD thesis can be downloaded from Diva

PhD student: Razan Jaberibraheem, DSV
Opponent: Kerstin Fischer, University of Southern Denmark
Main supervisor: Barry Brown, DSV
Supervisor: Donald McMillan, DSV
 

Abstract

This research is about addressing the need to better understand interaction with conversational user interfaces (CUIs) and how human-technology ’conversations’ can be improved by drawing on the lessons learned from human-human interaction. It focuses on incorporating abstractions of complex human behaviour, specifically gaze, to enhance interactions with speech agents in conversations. Across four empirical studies, a mix of methods is used to look closely at the interaction between the user and the system.

I offer empirical and conceptual contributions for interaction designers and researchers. First, I present a novel speech interface, Tama, which is a gaze-aware speech agent designed to explore the use of gaze in conversational interactions with smart speakers. Second, I present the empirical contributions, that is, the studies that document the interactions with and around speech interfaces, including ongoing, non-system-directed speech. A moment-by-moment analysis of these interactions highlights the opportunities that the gaze offers as a modality to enhance the interaction with the speech agent, as well as the problems and limitations when such a modality is used. The third contribution is a conceptual contribution made by providing perspective on minimal anthropomorphic design. This produces interactions that are not human-like in terms of behaviour but do take advantage of the skills used in human interaction as a key to advancing interactions with speech agents.

Based on my research work and contributions, I reflect upon advancing interactions with speech interfaces, focusing on what different technologies can offer and the possibility of taking the next step in designing CUIs. I then discuss the need to bridge the work of different fields (i.e. conversation analysis (CA), human-computer interaction (HCI), and human-robot interaction (HRI)) to combine models and approaches from all these fields in order to guide designers building speech systems. I see three competing yet complementary interaction paradigms across CUIs. I call these paradigms Direct Speech Interaction, Agent-Mediated Interaction, and Para-Speech Interaction. Each of these paradigms has specific challenges and opportunities for interaction.