NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·1h ago
AI Research
arXiv CS.AI·Jun 12·5 min read
Multi-Agent Reinforcement Learning from Delayed Marketplace Feedback for Objective-Weight Adaptation in Three-Sided Dispatch
TL;DR
Three-sided marketplaces face challenges in optimizing dispatch decisions due to delayed feedback on operational outcomes. A reinforcement learning system was developed at DoorDash to adapt dispatch objective weights using delayed signals, employing a store-level policy that selects multipliers for the dispatch optimizer.
✦ Why It Matters
Engineers can leverage delayed feedback in reinforcement learning to optimize decision-making in complex marketplaces.
Key Takeaways
- Implement reinforcement learning to adapt decision policies based on delayed feedback.
- Utilize store-level policies to optimize dispatch in complex systems.
- Incorporate Double Q-learning to improve value estimation accuracy.
- Focus on balancing delivery quality with operational efficiency.
Full Summary
Three-sided marketplaces, like food delivery services, struggle with optimizing dispatch decisions because feedback on performance, such as delivery speed and courier utilization, often arrives late. To address this, a reinforcement learning system was implemented at DoorDash that adapts the weights of dispatch objectives based on delayed marketplace feedback.
Instead of overhauling the existing combinatorial assignment optimizer, a store-level policy was created to select discrete multipliers that adjust the tradeoff between delivery quality and batching efficiency. The methodology involved training a shared value function using centralized offline data and decentralized execution, incorporating Double Q-learning targets and a conservative regularizer to mitigate value overestimation.
In a production switchback experiment, the offline-trained policy led to a significant increase in batching and a reduction in courier-side time costs, all while preserving customer-facing delivery quality. These findings demonstrate the potential of using real-world feedback to safely adapt decision-making policies in complex systems.