Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·Aug 24·5 min read
Source-Free MT Evaluation Is Not MT Evaluation
TL;DR
Source-free machine translation (MT) evaluation methods are inadequate as they often prioritize reference outputs over the original source, leading to biased assessments. This paper advocates for a shift towards quality estimation (QE) as a primary evaluation approach that respects source fidelity.
✦ Why It Matters
Reassess your MT evaluation methods to prioritize source fidelity over reference outputs for more accurate assessments.
Key Takeaways
- Shift from reference-based to source-grounded evaluation methods.
- Implement quality estimation as a primary evaluation approach.
- Develop hybrid metrics that prioritize source fidelity.
- Use references only as supplementary evidence in evaluations.
Full Summary
Current machine translation evaluation predominantly relies on reference-based metrics, which can misrepresent translation adequacy by favoring specific outputs over the original source meaning. The paper argues that adequacy should be assessed in relation to the source text, as references can introduce bias and errors.
It highlights that existing hybrid metrics are overly dependent on references, undermining the evaluation of source fidelity. The authors propose that quality estimation (QE) should be redefined as a primary method for evaluating translation adequacy, rather than a secondary option when references are absent.
They call for the development of hybrid metrics that prioritize the relationship between the source and the hypothesis, using references only as supplementary evidence. This approach aims to create a more accurate and fair evaluation framework for machine translation systems.