cache hit rate
Cache hit rate is a metric for the share of requests to a cache that are served directly from the cache.
Cache hit rate is the proportion of requests to a cache that are answered directly from it, known as hits, instead of from slower underlying storage or by recomputing the result. It is calculated by dividing the number of hits by the total number of requests, and the remaining share is called the miss rate. A higher value means more requests avoid the slower path.
The metric is used to evaluate caches across computing, including CPU caches, databases, web caches and content delivery networks. In prompt caching for large language model APIs, it refers to the share of input tokens read from the cache, and a higher rate lowers processing cost and response latency.
This entry is based on AIPOST articles and widely known facts. If something is wrong, please send us a correction request.
Articles covering this entry
Anthropic reports a cache hit rate above 99% and a CPU and memory footprint 200 times smaller than its old Git mirroring setup.
…Flash run used prompt context caching, which reuses previously processed input; the evaluation recorded a 98% cache hit rate across the open-weight model runs.