Abstract

Collecting data on a large scale is vital for the development of cutting-edge artificial intelligence (AI) technologies, especially those involving machine learning (ML) models, such as deep neural networks, which require training with relevant data. On one hand, the collection of real-world data, using devices such as cameras and microphones, would enable AI systems to better understand everyday life and ultimately behave or assist in a manner akin to human interaction. On the other hand, growing concerns about security and privacy make it increasingly difficult to collect such real-world data. As a result, the emergence of digital twins offers a promising direction for intelligent robots that employ deep learning models for tasks such as perception, planning, localization, and control.

This presentation will discuss a framework for data collection, training, and learning, utilizing the assistance of digital twins. This approach leverages a collection of AI models for self-navigating mobile robots. We will focus particularly on developing visual perception models that interpret the real world through a camera. These models play a pivotal role in a variety of AI-powered products and services, such as autonomous vehicles and smart cities. They are also a primary research focus at Elsa Lab. Visual perception models based on deep neural networks have achieved unprecedented accuracy in benchmark datasets. Implementing these models would enable edge AI systems to better perceive and understand their environments, and therefore acting more intelligently in the real world. However, they often experience drops in accuracy and lack sufficient effective real-world data samples. This can lead to unsatisfactory performance and safety concerns in practical deployments.

To address the aforementioned problems, we explore and incorporate the following key technologies into our framework: virtual-to-real transfer learning, foundation model-based domain adaptation, and the usage of mid-level representations. Virtual-to-real transfer learning enables ML models to train first in simulated environments and then migrate to real-world settings with ease. Foundation model-based domain adaptation facilitates this migration process, even under challenging scenarios where data collection in the real world involves labor-intensive manual preprocessing. Moreover, mid-level representations are employed to transmit various types of information from the perception module to the control module, forming the basis of modular frameworks in many learning-based systems. The primary scientific challenge of this research direction lies in integrating these elements into a unified solution, and improving the adaptation ability of AI models to real-world environments. We will present how these methodologies can be integrated with existing systems for intelligent perception, planning, localization, and control in deep learning-based intelligent robotics.

Bio.

Chun-Yi Lee is a Professor in the Department of Computer Science and Information Engineering at National Taiwan University (NTU), Taipei, Taiwan, and the supervisor of Elsa Lab. He received his B.S. and M.S. degrees from National Taiwan University in 2003 and 2005, respectively, and his M.A. and Ph.D. degrees from Princeton University, Princeton, NJ, USA, in 2009 and 2013, respectively, all in Electrical Engineering. Prof. Lee joined the Department of Computer Science at NTHU as an Assistant Professor in 2015. He was promoted to Associate Professor in 2019 and to full Professor in 2023. In 2024, Prof. Lee joined the Department of Computer Science and Information Engineering at National Taiwan University. Prior to 2015, he served as a senior engineer at Oracle America, Inc. in Santa Clara, CA, USA, from 2012 to 2015. Prof. Lee
founded Elsa Lab at National Tsing Hua University in 2015. Under his leadership, Elsa Lab has garnered several prestigious awards from global robotics and AI challenges. These include the first place at the NVIDIA Embedded Intelligent Robotics Challenge in 2016, the first place at the NVIDIA Jetson Robotics Challenge in 2018, the second place in the Person-In-Context (PIC) Challenge at ECCV 2018, the second place in the NVIDIA AI at the Edge Challenge in 2020, and the Best Solution Award (1st Place) in the Small Object Detection Challenge for Spotting Birds at MVA 2023.

Prof. Lee's research primarily focuses on deep reinforcement learning (DRL), intelligent robotics, computer vision (CV), and parallel computing systems. His contributions include developing key deep learning methodologies for intelligent robotics, such as sim-to-real and real-to-sim training, transfer techniques for robotic policies, digital twins, scene coordinate regression approaches, and domain adaptation techniques for semantic segmentation models. He has also advanced exploration approaches for DRL agents, multi- agent reinforcement learning (MARL) techniques, and generative modeling methodologies, including super-resolution and score-based generative models, as well as autonomous navigation strategies. His work has been published at major artificial intelligence (AI) conferences such as NeurIPS, CVPR, IJCAI, AAMAS, ICLR, ICML, ECCV, BMVC, CoRL, ICRA, IROS, GTC, MVA, and others. Additionally, his research appears in top AI journals, including IEEE Transactions on Pattern Recognition and Machine Intelligence (TPAMI), IEEE Transactions on Neural Networks and Learning Systems (TNNLS), Journal of Machine Learning Research(JMLR), and ACM Transactions on Evolutionary Learning and Optimization (TELO). In the realm of parallel systems, Prof. Lee has introduced innovative graphics processing unit (GPU) enhancement methodologies to improve efficiency. His work in this area has been published in IEEE Transactions on Very Large Scale Integration Systems (TVLSI), IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems (TCAD), the Design Automation Conference (DAC), the Asia and South Pacific Design Automation Conference (ASP-DAC), and International Conference on Computer Design (ICCD). 

Prof. Lee's research is particularly impactful in autonomous systems, decision-making systems, game engines, and vision-AI based robotic applications. In 2024, Prof. Lee was awarded the MARC Academia-Industry Collaboration Excellent Research Award by MediaTek Inc., and received the Garmin Scholar Fellowship from Garmin Inc. He was honored with the Academia Sinica Early-Career Investigator Research Achievement Award in 2022 and the Ta-You Wu Memorial Award from the Ministry of Science and Technology (MOST) in 2020. These prestigious awards in Taiwan recognize outstanding achievements in intelligence computing for young researchers. Prof. Lee has garnered numerous accolades, including outstanding research awards, distinguished teaching awards, innovation teaching awards, young scholar research awards, and contribution awards from institutions such as NVIDIA Deep Learning Institute (DLI), The Taiwan IC Design Society (TICD), the Foundation for the Advancement of Outstanding Scholarship (FAOS), The Chinese Institute of Electrical Engineering (CIEE), Taiwan Semiconductor Industry Association (TSIA), Institute of Information & Computing Machinery (IICM), and National Tsing Hua University (NTHU). In the academic community, Prof. Lee will serve as the Program Co-Chair at the International Conference on Machine Vision Applications (MVA 2025). He has served as the Area Chairs at NeurIPS 2023-2024, ICLR 2024, and ACML 2024, and as tutorial chair of MVA 2023. His roles have included committee member and reviewer at numerous international and domestic conferences. He has been session chair and technical program committee member multiple times at IROS, ASP-DAC, NoCs, ISVLSI, and MVA, and has held various chair roles at different international conferences. Prof. Lee has served as paper reviewer for NeurIPS, ICML, CVPR, ICLR, AAAI, ICCV, BMVC, ICRA, IROS, IEEE TPAMI, IEEE TVLSI, IEEE TCAD, IEEE ISSCC, and IEEE ASP-DAC on multiple occasions. He was the main organizer of the 3rd, 4th, and 5th Augmented Intelligence and Interaction (AII) Workshops from 2019 to 2023, and chaired the ACML Workshop on Machine Learning for Mobile Robot Vision and Control (MRVC) in 2021. Prof. Lee served as the director of the NVIDIA-NTHU Joint Innovation Center from 2023 to 2024, and served as the co-director of the MOST Office for International AI Research Collaboration from 2018 to 2020. From 2024, Prof. Lee serves as the co-director of the NVIDIA-NTU Joint Innovation Center. Prof. Lee is a professional member of IEEE and ACM.

 All faculty and students are welcome to join.