NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·3h ago
AI Research
arXiv CS.AI·Jun 9·5 min read
Beyond Agent Architecture: Execution Assumptions and Reproducibility in LLM-Based Trading Systems
TL;DR
Comparing the performance of large language models (LLMs) in financial trading is challenging due to inconsistencies in study methodologies. A reproducibility audit was conducted using a coded evidence matrix that analyzed 30 primary studies on LLM-based trading.
✦ Why It Matters
Engineers can enhance the reliability of LLM-based trading systems by standardizing methodologies and improving execution realism.
Key Takeaways
- Standardize methodologies for evaluating LLM performance in trading.
- Incorporate rigorous transaction-cost modeling in studies.
- Ensure consistent data provenance across research.
- Focus on execution timing to improve trading system reliability.
Full Summary
Large language models (LLMs) are increasingly utilized in financial trading, but their effectiveness is hard to evaluate due to differing methodologies across studies. A reproducibility audit was performed, employing a coded evidence matrix to systematically assess 30 primary studies focused on LLM-based trading systems.
Key factors examined included data provenance (the source of data), temporal split discipline (how data is divided over time), and transaction-cost modeling (how costs of trades are calculated). Results revealed substantial discrepancies in execution realism, with many studies lacking rigorous treatment of transaction costs and timing.
These inconsistencies can lead to misleading conclusions about the performance of LLMs in trading. For engineers and researchers, understanding these variations is crucial for developing more reliable trading systems and improving reproducibility in financial AI research.