Can LiteLLM edit a streaming answer before the user sees it?

Yes. LiteLLM v1.102.0 executes post_call guardrail pipelines on streaming responses and applies pipeline text rewrites to the stream itself. Tool-call rewrites are delivered into buffered chat, Responses and Messages streams. Earlier versions could only inspect a response after it finished, so anything streamed went out unchecked.

Do existing LiteLLM guardrails need rewriting for this?

No. LiteLLM v1.102.0 runs legacy post-call hooks as steps inside the new streaming pipeline, so checks written against the old interface keep running. The release also maps every guardrail scan id to its guardrail, stage and provider, which makes it clearer which hook produced which result.

What changed about LiteLLM's latency routing?

LiteLLM v1.102.0 ranks streaming latency routing by raw time-to-first-token instead of TTFT per token, and adds support for percentile-based TTFT routing. A routing entry with no latency samples is no longer treated as zero latency, and a refusing deployment is skipped when a non-transient error is retried.

Which OCR providers does LiteLLM v1.102.0 support?

LiteLLM v1.102.0 introduces an OCR core with adapters for Mistral, Azure Mistral, Azure Document Intelligence, Vertex Mistral, Vertex DeepSeek, and Reducto in both legacy and v3 form. The release also removes the older OCR pipeline that these adapters replace, and adds document fetching alongside the Azure Mistral adapter.

What does the in-container pgbouncer solve?

LiteLLM v1.102.0 ships pgbouncer inside the container so database connections are shared across workers rather than opened per worker, which is the usual cause of connection exhaustion at scale. The bundled pooler also follows rotating RDS IAM and Azure Entra tokens, so short-lived cloud database credentials keep working.

BerriAI · 2026-09-22 · major

LiteLLM v1.102.0 — guardrails finally run on streaming responses

LiteLLM v1.102.0 runs post_call guardrail pipelines on streaming responses, so text and tool-call rewrites apply mid-stream. Routing gains percentile-based TTFT selection, and a new OCR layer ships with adapters for five providers.

GitHub card for the BerriAI/litellm AI gateway repository

Post-call guardrails stop being an end-of-response check and start rewriting text while it streams.

Quick facts

Project	LiteLLM, by BerriAI
Version	v1.102.0
Released	22 September 2026
Guardrails	post_call pipelines on streams
Routing	Percentile-based TTFT routing
OCR adapters	Mistral, Azure Mistral, Azure Document Intelligence, Vertex Mistral, Vertex DeepSeek, Reducto
Database	In-container pgbouncer pooling

What is it?

Streaming guardrails are the headline of LiteLLM v1.102.0. Until now a post_call policy pipeline could only inspect a finished response, which left streamed answers unchecked. This release executes those pipelines on streams, applies text rewrites as the tokens go out, and delivers tool-call rewrites into buffered chat, Responses and Messages streams.

How does it work?

LiteLLM sits in front of many provider APIs and speaks OpenAI format to your code, so the gateway is where a policy can be enforced once for every caller. Legacy post-call hooks now run as steps inside that streaming pipeline, so existing checks keep working without a rewrite. Each guardrail scan id is mapped to its guardrail, stage and provider, and a Conduct Guard integration arrives with validated hooks and forwarded params.

Why does it matter?

Routing gets more honest about latency in this release: streaming latency routing ranks by raw time-to-first-token rather than TTFT per token, percentile-based TTFT routing is supported, and a deployment with no latency samples is no longer treated as instant. Operators also get an in-container pgbouncer that shares database connections across workers and follows rotating RDS IAM and Azure Entra tokens, plus a fresh OCR layer that replaces the legacy pipeline.

Who is it for?

teams running LiteLLM as a shared gateway

Frequently asked questions

Can LiteLLM edit a streaming answer before the user sees it?: Yes. LiteLLM v1.102.0 executes post_call guardrail pipelines on streaming responses and applies pipeline text rewrites to the stream itself. Tool-call rewrites are delivered into buffered chat, Responses and Messages streams. Earlier versions could only inspect a response after it finished, so anything streamed went out unchecked.
Do existing LiteLLM guardrails need rewriting for this?: No. LiteLLM v1.102.0 runs legacy post-call hooks as steps inside the new streaming pipeline, so checks written against the old interface keep running. The release also maps every guardrail scan id to its guardrail, stage and provider, which makes it clearer which hook produced which result.
What changed about LiteLLM's latency routing?: LiteLLM v1.102.0 ranks streaming latency routing by raw time-to-first-token instead of TTFT per token, and adds support for percentile-based TTFT routing. A routing entry with no latency samples is no longer treated as zero latency, and a refusing deployment is skipped when a non-transient error is retried.
Which OCR providers does LiteLLM v1.102.0 support?: LiteLLM v1.102.0 introduces an OCR core with adapters for Mistral, Azure Mistral, Azure Document Intelligence, Vertex Mistral, Vertex DeepSeek, and Reducto in both legacy and v3 form. The release also removes the older OCR pipeline that these adapters replace, and adds document fetching alongside the Azure Mistral adapter.
What does the in-container pgbouncer solve?: LiteLLM v1.102.0 ships pgbouncer inside the container so database connections are shared across workers rather than opened per worker, which is the usual cause of connection exhaustion at scale. The bundled pooler also follows rotating RDS IAM and Azure Entra tokens, so short-lived cloud database credentials keep working.

Try it

pip install litellm==1.102.0