Jump to the main content block

[2022-Nov-23] Challenges of Sim-to-Real Learning for Deep Learning Based Intelligent Robotics

Institute of Information Systems and Applications


Prof. Chun-Yi Lee 李濬屹教授

NTHU CS, 清大資工系


Challenges of Sim-to-Real Learning for Deep Learning Based Intelligent Robotics


13:20-15:00 Wednesday 23-Nov-2022

QR Code:






Collecting data on a large scale is vital for developing cutting-edge artificial intelligence (AI) technologies, when they involve machine learning (ML) models such as deep neural networks that require to be trained using relevant data. On the one hand, collecting data of the real world, using cameras or microphones would allow AIs to better understand our everyday life and ultimately to behave naturally as we humans do or to help us in a natural fashion. On the other hand, due to growing concerns about security and privacy, it is becoming increasingly difficult to collect such real data.

This presentation aims to discuss a computational framework for real-data collection and learning, which effectively leverages a collection of AI models for self-navigating mobile robots. We will particularly focus on developing visual perception models that can see the real world through a camera -- as they play a pivotal role for a variety of AI-powered products and services, such as autonomous vehicles and smart cities, and are the main area of research that Elsa Lab has been contributing to. Visual perception models based on deep neural networks have achieved unprecedented accuracies in benchmark datasets. Installing such models would enable edge AIs to better perceive and understand their surrounding environments and act intelligently in the real world. However, they usually suffer from accuracy drops and insufficiency of effective data samples from the real world, leading to unsatisfactory performance and safety concerns in practical deployments.

To address the above-mentioned problems, we explore and incorporate the following key technologies into a framework: sim-to-real transfer, semantic segmentation based unsupervised domain adaptation (UDA), and mid-level representations. Specifically, sim-to-real transfer allows ML models to be trained in simulated environments first and migrated to the real world setting with ease. Semantic segmentation based unsupervised domain adaptation (UDA) further enables the above model migration process to become possible, even under practical and challenging scenarios where data collection in the real world involves labor-intensive manual preprocessing costs. Furthermore, mid-level representations are used to deliver various types of information from the perception module to the control module, and form the basis of modular frameworks for many learning-based systems. The main scientific challenge of this research direction is to integrate them into a unified solution, and improve the adaptation ability of AI models in the real world.


Chun-Yi Lee is an Associate Professor of Computer Science at National Tsing Hua University (NTHU), Hsinchu, Taiwan, and is the supervisor of Elsa Lab. He received the B.S. and M.S. degrees from National Taiwan University, Taipei, Taiwan, in 2003 and 2005, respectively, and the M.A. and Ph.D. degrees from Princeton University, Princeton, NJ, USA, in 2009 and 2013, respectively, all in Electrical Engineering. He joined NTHU as an Assistant Professor at the Department of Computer Science since 2015. Before joining NTHU, he was a senior engineer at Oracle America, Inc., Santa Clara, CA, USA from 2012 to 2015.

Prof. Lee’s research focuses on deep reinforcement learning (DRL), intelligent robotics, computer vision (CV), and parallel computing systems. He has contributed to the discovery and development of key deep learning methodologies for intelligent robotics, such as virtual-to-real training and transferring techniques for robotic policies, real-time acceleration techniques for performing semantic image segmentation, efficient and effective exploration approaches for DRL agents, as well as autonomous navigation strategies. He has published a number of research papers on major artificial intelligence (AI) conferences including NeurIPS, CVPR, IJCAI, AAMAS, ICLR, ICML, ECCV, CoRL, ICRA, IROS, GTC, and more. He has also published several research papers at IEEE Transaction on Very Large Scale Integration Systems (TVLSI), IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems (TCAD), Design Automation Conference (DAC), and Asia and South Pacific Design Automation Conference (ASP-DAC). He founded Elsa Lab at National Tsing Hua University in 2015, and have led the members from Elsa Lab to win several prestigious awards from a number of worldwide robotics and AI challenges, such as the first place at NVIDIA Embedded Intelligent Robotics Challenge in 2016, the first place of the world at NVIDIA Jetson Robotics Challenge in 2018, the second place from the Person-In-Context (PIC) Challenge at the European Conference on Computer Vision (ECCV) in 2018, and the second place of the world from NVIDIA AI at the Edge Challenge in 2020.

Prof. Lee is the recipient of the Ta-You Wu Memorial Award from the Ministry of Science and Technology (MOST) in 2020, which is the most prestigious award in recognition of outstanding achievements in intelligence computing for young researchers. He has also received several outstanding research awards, distinguished teaching awards, and contribution awards from multiple institutions, such as NVIDIA Deep Learning institute (DLI), The Taiwan IC Design Society (TICD), The Foundation for the Advancement of Outstanding Scholarship (FAOS), The Chinese Institute of Electrical Engineering (CIEE), Taiwan Semiconductor Industry Association (TSIA), Institute of Information & Computing Machinery (IICM), and National Tsing Hua University (NTHU). In addition, he has served as the committee members and reviewers at many international and domestic conferences. His researches are especially impactful for autonomous systems, decision making systems, game engines, and vision-AI based robotic applications. Prof. Lee is a member of IEEE and ACM. He has served as session chairs and technical program committee multiple times at ASP-DAC, NoCs, and ISVLSI. He has also served as the paper reviewer of NeurIPS, AAAI, IROS, ICCV, BMVC, IEEE TPAMI, TVLSI, IEEE TCAD, IEEE ISSCC, and IEEE ASP-DAC. He has been the main organizer of the 3rd, 4th, and 5th Augmented Intelligence and Interaction (AII) Workshops from 2019-2023, and the chair of the ACML Workshop on Machine Learning for Mobile Robot Vision and Control (MRVC) in 2021. He was the co-director of MOST Office for International AI Research Collaboration from 2018-2020. He will serve as the tutorial chair of the 18th International Conference on Machine Vision Applications (MVA 2023).

All faculty and students are welcome to join.

Click Num: