Speaker:Ge Luo

On The Advancement of Automatic Summarization Evaluation

Summarization is a classic Natural Language Process (NLP) task designed to alleviate information overload problems. It has also been used in other downstream tasks to save text input length. Thus, the summary quality plays an important role in the human reading experience and downstream task performance. This work focuses on evaluating the quality of machine-generated summaries automatically.

We present three works on the advancement of automatic summarization evaluation. In the first part, we propose to evaluate the overall quality of a summary by training a metric from synthesized summaries. By learning a preference rank of gold summary and inferior summaries synthesized by corrupting the gold summary, our method eliminates the need for a reference summary during inference time.

In the second part, we focus on the factual consistency of summarization. We first systematically analyze the shortcomings of the current methods in synthesizing inconsistent summaries. Then, employing the parameter-efficient finetuning (PEFT) technique, we discover that a competitive factual consistency detector can be achieved using thousands of real model-generated summaries with human annotations.

In the third part, we present a preliminary proposal to use the Large Language Model (LLM) for evaluating the factual consistency of the summarization task. First, we introduce a self-instruction approach to make LLM's reasoning match the factual label. Then, human annotators are instructed to edit LLM's unfaithful reasoning steps. Finally, the LLM is finetuned to follow the faithful corrections.

Committee Members: Forrest Sheng Bao (Major Professor), Ying Cai, Qi Li, Wensheng Zhang, Zhu Zhang

Join on Zoom: https://iastate.zoom.us/j/91868061066