🧠 Cognitive Engineering & Advanced RAG
Redis Vector Semantic Cache
Slashing LLM Latency from 2,000ms to 5ms and Reducing API Costs by 40% using Redis Vector Search (VSS) Semantic Caching
Traditional key-value caches (like Memcached or standard Redis string keys) rely on exact string equality (`MD5("How to reset password?") != MD5("How do I change my password?")`). In LLM production systems, users ask the exact same underlying question using hundreds of different wordings. A Semantic Cache converts input prompts into vector embeddings, performs ultra-fast HNSW similarity search in Redis, and returns cached LLM responses in <5ms when cosine distance is below a strict threshold (e.g. cosine distance <= 0.10).
Key Concepts
Sub-5ms Latency Response
Bypassing 1.5 - 3.0 second LLM API calls entirely for common queries by serving pre-computed responses straight from Redis RAM.
Cosine Distance Thresholding
Using strict cosine similarity thresholds (e.g., distance <= 0.10) to prevent false hits on subtly different user intents while capturing semantically identical prompts.
API Cost & Rate Limit Reduction
Eliminating 30-50% of redundant LLM token consumption in high-volume customer support workflows, directly protecting LLM API rate limits during traffic spikes.
TTL & Eviction Policies
Configuring Redis TTL (Time-to-Live) and LRU (Least Recently Used) eviction policies on vector keys to ensure outdated information is purged automatically.
Redis Semantic Cache Execution Flow Architecture
1. Production Python Redis Vector Semantic Cache Engine
Creating an HNSW Vector Index in Redis using `redis-py`, computing query embeddings, performing KNN search with distance thresholding, and storing cache entries with TTL.
cache/redis_semantic_cache.pypython
1import redis
2import numpy as np
3from redis.commands.search.query import Query
4from redis.commands.search.field import VectorField, TextField
5from langchain_openai import OpenAIEmbeddings
6
7# 1. Connect to Redis VSS
8r = redis.Redis(host='localhost', port=6379, decode_responses=False)
9embeddings_model = OpenAIEmbeddings(model="text-embedding-3-small")
10
11INDEX_NAME = "idx:semantic_cache"
12VECTOR_DIM = 1536 # OpenAI text-embedding-3-small dimension
13
14def init_redis_vector_index():
15 """Create HNSW Vector Index in Redis if not exists"""
16 try:
17 r.ft(INDEX_NAME).info()
18 except:
19 schema = (
20 TextField("prompt"),
21 TextField("response"),
22 VectorField(
23 "vector",
24 "HNSW",
25 {
26 "TYPE": "FLOAT32",
27 "DIM": VECTOR_DIM,
28 "DISTANCE_METRIC": "COSINE",
29 "INITIAL_CAP": 5000,
30 }
31 )
32 )
33 r.ft(INDEX_NAME).create_index(schema)
34
35def get_semantic_cache(prompt: str, threshold: float = 0.10):
36 """Check Redis VSS for semantically similar prompt"""
37 query_vec = embeddings_model.embed_query(prompt)
38 query_bytes = np.array(query_vec, dtype=np.float32).tobytes()
39
40 q = (
41 Query("*=>[KNN 1 @vector $vec AS score]")
42 .sort_by("score")
43 .return_fields("prompt", "response", "score")
44 .dialect(2)
45 )
46
47 res = r.ft(INDEX_NAME).search(q, query_params={"vec": query_bytes})
48 if res.docs:
49 doc = res.docs[0]
50 score = float(doc.score)
51 if score <= threshold:
52 print(f"⚡ [CACHE HIT] Cosine Distance: {score:.4f}")
53 return doc.response.decode('utf-8')
54
55 print("🐢 [CACHE MISS] Querying LLM...")
56 return None
57
58def set_semantic_cache(prompt: str, response: str, ttl_seconds: int = 86400):
59 """Store LLM response with prompt embedding and TTL"""
60 query_vec = embeddings_model.embed_query(prompt)
61 query_bytes = np.array(query_vec, dtype=np.float32).tobytes()
62
63 doc_id = f"cache:{hash(prompt)}"
64 r.hset(doc_id, mapping={
65 "prompt": prompt,
66 "response": response,
67 "vector": query_bytes
68 })
69 r.expire(doc_id, ttl_seconds) # Set 24h TTL2. TypeScript Redis Semantic Cache Implementation
Using node-redis to execute RediSearch KNN vector queries for Next.js / Node.js production backends.
lib/redis-semantic-cache.tstypescript
1import { createClient } from 'redis';
2import { openai } from '@ai-sdk/openai';
3import { embed } from 'ai';
4
5const redis = createClient({ url: process.env.REDIS_URL });
6await redis.connect();
7
8export async function checkSemanticCache(userPrompt: string, distanceThreshold = 0.10) {
9 // 1. Generate query embedding
10 const { embedding } = await embed({
11 model: openai.embedding('text-embedding-3-small'),
12 value: userPrompt,
13 });
14
15 const floatBuffer = Buffer.from(new Float32Array(embedding).buffer);
16
17 // 2. Query Redis VSS
18 const searchResults = await redis.ft.search(
19 'idx:semantic_cache',
20 '*=>[KNN 1 @vector $vec AS score]',
21 {
22 PARAMS: { vec: floatBuffer },
23 SORTBY: 'score',
24 RETURN: ['prompt', 'response', 'score'],
25 DIALECT: 2,
26 }
27 );
28
29 if (searchResults.total > 0) {
30 const topDoc = searchResults.documents[0].value;
31 const score = parseFloat(topDoc.score as string);
32
33 if (score <= distanceThreshold) {
34 return { hit: true, response: topDoc.response as string, score };
35 }
36 }
37
38 return { hit: false, response: null };
39}💡
Senior Architect Insight: A semantic cache is only as good as its threshold. Setting a loose distance threshold (e.g. > 0.20) will cause false cache hits, returning answers to fundamentally different questions. Start with a strict threshold (0.08 - 0.10) and monitor cache hit quality in production.