💥 OpenAI Proxy Server

📄️ Quick Start

Quick start CLI, Config, Docker

You can find the Dockerfile to build litellm proxy here

Set model list, apibase, apikey, temperature & proxy server settings (master-key) on the config.yaml.

Features here are behind a commercial license in our /enterprise folder. See Code

Input, Output, Exceptions are mapped to the OpenAI format for all supported models

Track Spend, Set budgets and create virtual keys for the proxy

Requirements:

Routing

Create + delete keys through a UI

Alerts when a project will exceed it’s planned limit

Set base model for cost tracking azure image-gen call

Use JWT's to auth admin's into the proxy.

2 items

Add new models + Get model info without restarting proxy.

Use this to health check all LLMs defined in your config.yaml

2 levels of debugging supported.

LiteLLM supports Microsoft Presidio for PII masking.

LiteLLM supports similarity checking against a pre-generated list of prompt injection attacks, to identify if a request contains an attack.

Cache LLM Responses

3 items

- Modify data before making llm api calls on proxy

Use this to fail a request based on the output of an llm api call.

Cli arguments, --host, --port, --num_workers