Speaker:Runlong Zhang

HPCPerfOpt dataset and OptiAdvisor tool for utilizing large language models for performance optimization of parallel coding problems

Large language models(LLM) have received tremendous attention recently due to their potential and performance on various tasks. However, as powerful as LLMS may be, various studies have shown that LLMS may not be as capable of providing highly precise responses such as runnable and accurate code in specific domains like parallel programming. As such, this project is done in collaboration with Janesari labs of Iowa State University to explore the limitations of LLM6 in answering questions from the parallel programming domain. In addition, we also investigate the effects of utilizing retrieval augmented generation(RAG) on answering questions in this domain.

In this project, we work with Ph.D students from Janesari lab to develop a pipeline that (1) builds a datastore that houses relevant documents for answering questions. (2) Retrieves a certain number of relevant documents given some query. (3) Formats the relevant documents and the original query to a prompt which is then used to obtain the LLM response. Additionally, efforts have also gone into exploring methods that promote the scalability of small datasets, as well as into the optimization of the evaluation metrics used to gauge the quality of the model responses.

We show that in our preliminary results, the retrieval implementation is able to do well in fetching relevant documents based on the datastore we have constructed. In addition to this, testing shows that in smaller models, retrieval is able to generally improve the inference of the model with regard to the OpenMP domain. As for data augmentation, our methods have shown that while we achieved the desired effect of increasing the size of our dataset, the augmented samples do not show a significnnt difference when it cornes to model inference. Lastly, testing with evaluation metrics suggests that further fine-tuning of evaluation metric parameters is necessary to allow for a more accurate evaluation of the models tested.

Committee: Simanta Mitra (major professor) and Gurpur Prabhu