Comparison of human and AI analysis in an AI assistant and remote expert field trial

Artificial intelligence has begun to be used more in research work. It already offers tools that can be used naturally in research. AI assistants are also needed in factory work because they help fill the gap in the skills and preserve disappearing tacit knowledge. It quickly finds the right instructions, safety regulations and reports the work done afterwards. However, the reliability of AI has emerged in various studies. Can we trust that it does not hallucinate at any point? In this project, we studied whether it can be used reliably in the analysis of interviews in user studies. We obtained the material from a field trial in which both an AI agent and a remote expert via smart glasses assisted in turning a gear wheel.

Data to be analyzed was collected in a user study, in which five workers performed a lifting task using a crane. They used two novel technologies: (1) real-time guidance provided by a remote expert via see-what-I-see smart glasses (Iristick) and (2) guidance provided by an artificial intelligence assistant. Finally, Semi-structured interviews were conducted with all five participants and humans and AI analyzed them.

In this study, our research question was: How did the analyze results of AI agent and human factors experts differ from each other? How does the analysis of artificial intelligence differ from the analysis performed by trained researchers? We conducted a parallel analysis of the same interview data: a qualitative thematic analysis by the research team and an AI analysis (themes, summaries, direct quotes) produced with GPT-4o and 5 pro. The interviews were transcribed into text format, and this was given to artificial intelligence. AI analysis utilized ChatGPT’s deep analysis. Different analysis methods were compared using qualitative methods, including examining theme-to-theme correspondence, assessing importance, and mapping differences.

An AI assistant was implemented for the trial using ChatGPT version 4o as an AI agent. The agent was carried out in a restricted workspace for information security reasons. The agent was trained to follow the process defined in the lifting plan, including safety instructions, as well as other safety and lifting-related instructions and regulations. The test subjects were also shown images of the lifting plan, which the agent referred to during the assistance.
According to preliminary results, both analysis methods found several key research themes. The analysis performed by the researchers was considered trustworthy. Humans can see very different contexts and connections. On the other hand, AI is a very powerful tool. It can quickly go through even large amounts of data. With the help of AI, interview data can be quickly viewed through different research frameworks. Taking direct quotes from interviews and translating them into different languages was easy.