Abstract

Abstract
Fine-tuning large language models with long context windows pushes memory demand far beyond GPU capacity. CPU offloading relieves the GPU, but it shifts the bottleneck to system memory, which grows with context length and batch size. CXL-attached memory offers a scalable, lower-cost way to expand capacity, yet its higher latency slows fine-tuning when adopted naively, mainly in the CPU-side optimizer step. This talk analyzes where that slowdown comes from and presents two remedies: a PyTorch extension for tensor-level control of memory placement, and a CXL-aware allocation policy that keeps latency-sensitive data in local DRAM while placing latency-tolerant data on CXL memory. On 7B and 12B models, the approach improves throughput by up to 21% over naive CXL adoption and, with two CXL cards, stays within 1% of a DRAM-only system.

Bio.

Shuo-Han Chen is an Associate Professor in the Department of Computer Science and the Institute of Artificial Intelligence Innovation at National Yang Ming Chiao Tung University (NYCU), Taiwan, where he leads the COSMOS Lab. He received his Ph.D. in Computer Science from National Tsing Hua University in 2018 and was previously an Assistant Professor at NYCU and at National Taipei University of Technology. His research covers emerging non-volatile memory and storage technologies, memory and storage systems, and next-generation memory architectures.

All faculty and students are welcome to join.