#LLM inference

1 articles tagged LLM inference

Tagged articles

Articles tagged LLM inference, newest first.

Developer at a desk with a transparent context window container holding stacked cards above a cutaway of model gears
AI in Practice

Temperature, Attention and Context Rot: Model Mechanics Behind Agent Design

Sampling, attention and the KV cache set the limits agent builders design around. Temperature divides logits before softmax, so keep it low for math and code. OOLONG scores drop from near 90 percent at 8k tokens to 10 to 40 percent at 512k. Keep tools minimal and append new context at the end to keep the KV cache valid. Protect safety instructions from context compaction and test that they survive.

Get new articles by RSS or podcast

Add the RSS URL to your RSS reader or the podcast URL to your podcast app to get every new article as soon as it is published. The podcast reads out the headline and key points.

How to subscribe