Abstract

Today's industrial jobs require a skilled and trained workforce as tasks such as maintenance, service, and repair are becoming more complicated and more demanding. Therefore, both education and training for executing these tasks are becoming more important. This kind of training usually requires the presence of the master. But there are at least three factors that make the presence of the master difficult. Therefore, we are trying to create an assembly assistant that allows novices to be guided by asking questions when they encounter problems in assembly, just like there is a master by the side. But novices may have similar questions at different assembly stages, like “How to lock the screw?” but the answers are different. If we analyze the images during assembly, we can know the assembly progress. Therefore, we will implement a multi-modal assembly assistant that can observe the assembly process and respond to the assembler’s questions. In this talk, I will share the experience in building this multimodal dialogue system, including training dataset compilation and system construction. I will also provide a short overview of the major types of dialogue systems.

All faculties and students are welcome to join.