NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·May 26·5 min read
CITYREP: A Unified Benchmark for Urban Representations Across Cities, Tasks, and Modalities
TL;DR
Urban representation learning faced challenges due to limited evaluations that often led to inflated performance metrics. CITYREP is a unified benchmark designed to assess urban representations across multiple cities, tasks, and data types while minimizing spatial leakage.
✦ Why It Matters
Researchers can utilize CITYREP to ensure fair evaluations of urban representation models, improving their reliability and applicability.
Key Takeaways
- Adopt CITYREP for standardized urban representation evaluations.
- Use block-based spatial splits to avoid performance inflation.
- Explore variability in model performance across different urban contexts.
- Leverage the provided datasets and tools for reproducible research.
Full Summary
Urban representation learning aims to create general-purpose embeddings that capture the complexities of urban environments for various applications. However, existing evaluation methods typically focus on a narrow set of cities and tasks, often using random data splits that can skew results due to spatial leakage.
To address these issues, CITYREP was developed as a comprehensive benchmark that includes a spatial unit-agnostic evaluation framework, a standardized evaluation protocol using block-based spatial splits, and a multi-city, multi-task benchmark suite covering eight cities and eight tasks. The evaluation of 11 urban representation models demonstrated that random splits could inflate performance scores and alter model rankings.
Results indicated significant variability in model performance across different cities and tasks, emphasizing the need for generalization-aware evaluation methods. CITYREP is made available as a reproducible benchmark, complete with datasets and evaluation tools, to support fair comparisons and advance research in urban representation learning.