NVIDIA-NeMo / Switchyard

TL;DR

Switchyard is a Rust-based proxy and library designed for managing large language model (LLM) traffic, enabling seamless communication between different API formats. It allows developers to route requests across various LLM providers while collecting operational metrics.

✦ Why It Matters

Integrate Switchyard to streamline API interactions and improve your LLM traffic management today.

Key Takeaways

Install Switchyard using Cargo for easy setup.
Utilize API translation to maintain native formats across LLMs.
Implement custom routing algorithms for tailored traffic management.
Monitor operational metrics to optimize performance and reliability.

How It Works

Switchyard operates by translating requests and responses between different LLM API formats, allowing clients to maintain their native API while interacting with various backend models. It uses a routing mechanism that can be customized with different algorithms, enabling users to choose how requests are distributed among available models.

This includes options for random routing, classifier-based routing, and escalation routing, which can enhance the efficiency of model interactions.

NVIDIA-NeMo / Switchyard

BerriAI / litellm

tensorflow / tensorflow

open-webui / open-webui

Agentic data science with the Data Agent Kit

BerriAI / litellm

tensorflow / tensorflow

open-webui / open-webui

Agentic data science with the Data Agent Kit