LLM based virtual assistant for human-robot interaction

Business Finland and European Commission have announced that industrial metaverse and virtual worlds are new focus areas. Main goal of the project is to build the first version of roadmap and proof-of-concept of an AI supported human-robot operation platform for industrial metaverse.
Utilization of large language models (LLM) such as ChatGPT have become popular tools in everyday life. As usage is progressing more within various industries, the question arises how to connect these into multi-user/device environments. For the LLM to function as an assistant to the operators, and provide value within the given task, the user-interface (UI) needs to be straightforward.
Motivation for the project thrives from the possibility to provide specific instructions to a novice operator about previously unknown task in a way that operator can effortlessly interact with an entity about the progress. The target is a LLM supported human-robot work allocation on a concept level.
Since the development of LLM based assistants in industrial metaverse context is a new topic, the project provides a novel approach for building a data pipeline from operator to a robot via a LLM as a conversational agent. The approach provides scaling options to multiple robots and operators.
The project utilizes ChatGPT4.0 APIs with a use-case specific context to provide the LLM a scope and OpenAI Whisper for text-to-speech and speech-to-text. LLM communicates with both a human operator and a robotic manipulator. The operator communicates via speech to the LLM either on a desktop application or via augmented reality (AR) headset, while LLM communicates through a socket to the manipulator, giving it commands to perform specific steps of the use-case. The robotic manipulator informs the LLM of its progress, which the LLM then articulates to the operator, thus providing interim information about the overall progress.
Project provides a pipeline and a proof-of-concept to manage a ChatGPT4.0 based virtual assistant in an assembly use case together with an operator and a robotic manipulator. The operator can interact with the assistant either on a desktop interface or with an AR-headset, while receiving information about the use case and how to progress in it. A robotic manipulator provides materials for the operator without operator needing to specify the robot, as the LLM watches over the process under the specified use case context.