Abstract

We present two novel infrastructural services based on Hadoop for big data storage and computing in a Taiwan's semiconductor wafer fabrication foundry. The two services include Hadoop data service (HDS) and distributed R language computing service (DRS), which have been built and operated in production systems for $4.5$ years. They evolve over time by incrementally accommodating users' requirements. HDS is a web-based distributed big data storage facility. Users simply rely on HDS to access data objects stored in Hadoop with the HTTP protocol. In addition, HDS is scalable and reliable. Moreover, HDS is efficient and effective by intelligently selecting either Hadoop distributed file system (HDFS) or database (HBase) for publishing data objects. Specifically, HDS is transparent to existing analytics and data inquiry applications, such as Spark and Hive. While HDS is a unified storage for supporting sequential and random data accesses for big data in the wafer fabrication foundry, DRS is a distributed computing framework for typical R language users. R users employ DRS to enjoy data-parallel computations, effortlessly and seamlessly. Similar to HDS, DRS can be horizontally scaled out. It guarantees the completion of computational jobs even with failures. In particular, it adaptively reallocates computational resources on the fly, minimizing job execution time and maximizing utilization of allocated resources.

Bio

Hung-Chang Hsiao is currently a full professor in Computer Science and Information Engineering, National Cheng-Kung University, Taiwan, since August 2012. He is currently also a Vice Center Director of Computing and Network Center, and a Head of Intelligent Big Data Integrated Solutions (IBDI) Division of Intelligent Manufacturing Research Center. Hung-Chang Hsiao received the PhD degree in computer science from National Tsing-Hua University, Taiwan, in 2000. Dr. Hsiao's research interests include distributed/parallel computing, big data, and randomized algorithm design and performance analysis.

All faculties and students are welcome to join.