Meet Redis LangCache: A Managed Semantic Cache That Cuts LLM API Costs by Up to 90% and Returns Cache Hits Up to 15x Faster

Redis LangCache is a managed service that sits between applications and large language models to match incoming prompts against previous answers by meaning rather than exact text. The company reports that this approach can reduce API costs by up to 90% and return cached responses up to 15 times faster than generating a new answer.

Deployment status

The tool is available now as a public preview on Redis Cloud. Teams access it through a REST API supported by Python and JavaScript SDKs. Redis notes that features and behaviour may change while the service is in preview.

The problem: paraphrases still trigger full LLM calls

Consider three requests to a customer-support assistant:

• “Can I get a refund after buying the monthly plan?”
• “Is the monthly subscription refundable?”
• “Can I cancel the plan and get my money back?”

The wording differs, but the question and answer are identical. Without a semantic cache, each version triggers a complete generation: input tokens processed, output tokens decoded, user waiting.

Prefix caching only removes part of that cost. When requests share a system prompt or context, the engine reuses the KV states computed for that prefix, but the request still reaches the LLM, new tokens still get processed, and the full answer still gets decoded. A prefix-cache hit is a cheaper generation call, not an avoided one.

How LangCache works

LangCache moves the cache outside the model and stores the generated response itself. The architecture is a two-call loop:

1. Before invoking the model, the app sends the prompt to POST /v1/caches/{cacheId}/entries/search.
2. LangCache generates an embedding for the prompt and runs a vector search over stored entries.
3. If a semantically similar entry clears the configured similarity threshold, the cached response is returned and no LLM call occurs.
4. On a miss, the app calls its chosen LLM as usual, then stores the prompt and new response through POST /v1/caches/{cacheId}/entries for future matches.

Embedding generation is handled by the service, with default models or bring-your-own. Cache behaviour is controlled through similarity thresholds, TTLs, and eviction policies, plus adaptive controls that tune precision and recall. Built on Redis’s vector database and exposed as a REST API, it works with any LLM provider and language. Hit rates and savings are monitored from the Redis Cloud console.

What a cache hit actually saves

A cache hit removes the input tokens, the output tokens, and the decoding latency of an additional model call. In a demo run comparing both paths on a paraphrased question, direct inference took 2.232 seconds and consumed 514 input tokens plus 250 output tokens. LangCache returned the earlier response in 0.37 seconds with zero LLM input or output tokens, roughly 6x faster in that run.

The Redis documentation is careful about how savings accrue. On a cached response you do not pay for output tokens, while input token costs are typically offset by embedding and storage costs. The suggested estimate is:

Est. monthly savings = (Monthly output token costs) x (Cache hit rate)

With $200 of monthly LLM spend, 60% of it on output tokens, and a 50% hit rate, that works out to $60 saved per month. Redis also publishes a savings calculator for annual estimates.

Redis’s public preview announcement cited up to 15x faster responses on cache hits and up to 70% lower token usage, while the current product page states savings of up to 90%. Customer Mangoes.ai reports a 70% hit rate on its patient-care voice app, cutting LLM spend by 70% with 4x faster responses. The actual result depends on how much safe repetition exists in the traffic.

Where semantic caching needs care

Deciding which questions can safely share an answer is a production concern, not a configuration detail. A threshold set too low returns a refund policy to a customer asking about upgrades. Set too high, nearly every paraphrase goes back to the model and the cache stops paying for itself. Production setups need well-tuned thresholds, expiration policies so stale answers age out, data isolation between tenants, and monitoring for incorrect matches.

LangCache covers these with access scopes, custom filtering, TTL and eviction controls, and monitoring through Redis Cloud. Data stays on the customer’s Redis servers, and Redis states it does not access that data or use it to train models.

Key takeaways

• Prefix caching cuts prompt-processing cost; semantic caching eliminates the LLM call entirely on a hit.
• LangCache is a two-call REST integration: search before the model, store after it.
• Savings come mainly from avoided output tokens; the docs give the formula output cost x hit rate.
• Redis claims up to 90% cost savings and up to 15x faster cache hits; a demo run showed 6x.
• Thresholds, TTLs, isolation, and false-match monitoring decide whether a semantic cache is safe.