jundot / omlx

TL;DR

oMLX is a macOS application that enables continuous batching and tiered key-value (KV) caching for local large language models (LLMs), allowing users to manage model memory efficiently from their menu bar. It retains context across conversations, making it practical for coding tasks with tools like Claude Code.

✦ Why It Matters

Download oMLX today to enhance your local LLM management and improve coding efficiency.

Key Takeaways

Install oMLX via Homebrew for easy setup.
Utilize tiered caching for efficient model management.
Access a web interface for real-time monitoring.
Leverage continuous batching for improved performance.

How It Works

oMLX employs a two-tier caching mechanism where frequently accessed data is stored in RAM (hot tier) for quick retrieval, while less frequently used data is offloaded to SSD (cold tier). This allows the application to maintain context across requests, even when switching between different models or contexts.

The system also supports concurrent requests, enhancing its efficiency in handling multiple tasks simultaneously.

jundot / omlx

BerriAI / litellm

tensorflow / tensorflow

open-webui / open-webui

Introducing TimesFM 3.0 with multivariate time-series forecasting in BigQuery

BerriAI / litellm

tensorflow / tensorflow

open-webui / open-webui

Introducing TimesFM 3.0 with multivariate time-series forecasting in BigQuery