NUBIA: NeUral Based Interchangeability Assessor for Text Generation

Abstract

We present NUBIA, a methodology to build automatic evaluation metrics for text generation using only machine learning models as core components. A typical NUBIA model is composed of three modules a neural feature extractor, an aggregator and a calibrator. We demonstrate an implementation of NUBIA which outperforms metrics currently used to evaluate machine translation, summaries and slightly exceeds/matches state of the art metrics on correlation with human judgement on the WMT segment-level Direct Assessment task, sentence-level ranking and image captioning evaluation. The model implemented is modular, explainable and set to continuously improve over time.

Muhammed Yusuf Kocyigit

PhD Student at Boston University

My research interests include better evaluating and improving LLMs, understanding the pre-training data of LLMs and computational social sciences