Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·Sep 12·5 min read
Does Linguistic Structure Enrichment Enhance Coherence Assessment? Not With Current Architectures
TL;DR
Enriching text with syntactic and rhetorical information does not improve coherence assessment in current language models. Instead, plain texts yielded higher accuracy in predicting incoherence.
✦ Why It Matters
Focus on using plain text for coherence assessment to improve detection of misleading information.
Key Takeaways
- Use plain text for better incoherence prediction accuracy.
- Explore the limitations of current language model architectures.
- Consider coherence as a tool for identifying misleading content.
- Access available code and models for further research.
Full Summary
Recent advancements in large language models have improved human-computer interaction, yet these models often generate texts that, while grammatically correct, lack semantic coherence. This study explores whether adding syntactic (structure-related) and rhetorical (persuasive techniques) information can enhance the prediction of incoherence in texts.
Experiments revealed that plain texts outperformed enriched ones, as the additional information conflicted with the models' existing architectures. Furthermore, zero-shot experiments on a Brazilian disinformation dataset indicated that coherence assessment could effectively identify misleading content.
The findings suggest that current architectures struggle with enriched linguistic structures, limiting their effectiveness in coherence evaluation. Code and models used in the study are publicly available for further exploration.