tensor_cache Benchmarks

The tensor_cache crate provides LLM response caching with exact, semantic (HNSW), and embedding caches.

Exact Cache (Hash-based O(1))

Semantic Cache (HNSW-based O(log n))

Put (Exact + Semantic + HNSW insert)

Embedding Cache

Eviction (batch processing)

Distance Metrics (raw computation, 128d)

Semantic Lookup by Metric (1000 entries)

Sparse vs Dense (80% sparsity)

Auto-Metric Selection

Redis Comparison

*Estimated: network latency dominates (99.9% of time).

Key Insight: For embedded use (no network), Redis is 3.5x faster for exact lookups. Over TCP (typical deployment), both are network-bound at ~143us. Our differentiator is semantic search (21us) which Redis cannot provide.

Analysis

• Exact cache: Hash-based O(1) lookup provides sub-microsecond hit/miss detection
• Semantic cache: HNSW index provides O(log n) similarity search (~21us for hit)
• Embedding cache: Fast O(1) lookup for precomputed embeddings
• Put performance: Consistent ~50us regardless of cache size (HNSW insert is O(log n))
• Eviction: Efficient batch eviction with LRU/LFU/Cost/Hybrid strategies
• Distance metrics: Auto-selection based on sparsity (>=70% sparse uses Jaccard)
• Token counting: tiktoken cl100k_base encoding for accurate GPT-4 token counts
• Cost tracking: Estimates cost savings based on model pricing tables

Cache Layers

Eviction Strategies

Metric Selection Guide