latency

Latency is the delay between sending a request and receiving its response, a key measure of response speed for AI models.

44 articlesLast mentioned

Latency is the time that elapses between sending a request or signal and receiving the corresponding response in computing and networking. It is usually measured in milliseconds and, together with throughput (the amount of work handled per unit of time), is a fundamental performance metric.

In AI, latency metrics include the time from submitting a prompt until the first token is produced (Time To First Token), the rate at which tokens are generated, and the time until the complete response is finished. In AI agent environments, hooks and policy checks inserted between model calls add latency, so a time budget is often set for each inspection step.

This entry is based on AIPOST articles and widely known facts. If something is wrong, please send us a correction request.

Articles covering this entry

  • AI Agent Tuning: When to Rewrite Prompts and When to Retrain WeightsAI Performance ·

    An evaluation and metric: exact match, pass rate, a model acting as judge, or cost and latency.

    Read article
  • Gyro-M vs. Gyro-S on a 16GB GPU: Higher Scores, Similar Coding ResultsAI Performance ·

    Gyro-M passes 41 of them, or 85.4%, with an average latency of 3,889 ms and a total runtime of 3 minutes 6 seconds.

    Read article
  • IBM's $11 Billion Confluent Bet: Real-Time Data as the Base for AI AgentsAI News ·

    Real-time data is not just about low latency, the delay before information arrives.

    Read article
  • safeagents Proxy Stops Coding Agents Tricked by Hidden README InstructionsAI Security ·

    Hugging Face documents 10 to 20 milliseconds of added latency per call; measured overhead in this setup was roughly 5 milliseconds, which is small enough to be…

    Read article
  • Orca Agent IDE: Why a Terminal Alone Falls Short for Claude Code WorkflowsAI in Practice ·

    The caveat is worth stating plainly: the latency has not been removed, only hidden behind a local interface, but for interactive work the result is that the lag…

    Read article
  • AI Agent Governance: Track Sessions by Cost, Time and Risk Before You BlockAI Security ·

    …or, Codex, GitHub Copilot, OpenCode, OpenClaw and Pi, relays events with identity attached, and returns an allow or block verdict within a tight latency budget.

    Read article
  • NVIDIA CEO Exit, Claude Fable 5.2 and Adobe MAX: A Week That Reshaped AIAI News ·

    …simple: when an agent loop only needs a yes-or-no, a category or a score, a decision endpoint cuts both latency and cost compared with a full completion model.

    Read article
  • Interaction-as-a-Judge: Computer-Use Agents Test AI-Built Games and Web AppsAI Performance ·

    …, a custom harness pauses the game between actions and advances it by 500 milliseconds after each one, so slower models are not penalized for inference latency.

    Read article
  • Anthropic Nasdaq Debut: What the Filing Numbers Say About the AI Bubble DebateAI News ·

    …s, vector databases, retrieval-augmented generation (RAG, a technique that lets a model pull in outside documents before answering), latency, or storage arrays.

    Read article
  • Temperature, Attention and Context Rot: Model Mechanics Behind Agent DesignAI in Practice ·

    Effort can be set to low, medium, high or max; higher effort improves accuracy at a steep cost in latency and spend.

    Read article
  • Bret Taylor: Legacy Firms Own the Context That Makes AI Agents Pay OffAI News ·

    …torage hierarchy: software mixes spinning disks, flash storage and RAM according to latency and cost targets rather than putting everything in the fastest tier.

    Read article
  • AI Alignment as a Game Theory Trap: Why One Lab's Good Intentions Fall ShortAI Ethics ·

    Raskin's fixes: compute cuts, latency penalties and executive criminal liability

    Read article
  • Pando: The Git Backend OpenAI Built for a 2,442% Surge in CommitsAI News ·

    Pando, the paper reports, cut median push latency by nearly 5x, lifted merge infrastructure reliability to 99.99%, and now serves nearly a petabyte of Git data…

    Read article
  • Prompt Audits and Cache Reuse Cut Claude Costs Up to 72.9%, Anthropic SaysAI in Practice ·

    Cache reads are billed at roughly one-tenth the price of base input tokens, and cache hits also cut response latency substantially.

    Read article
  • ShapeLearn Qwen3.8-27B on a 16GB GPU: Agent Tests Pass 4 of 5 at 128k ContextAI Performance ·

    The tiered reasoning benchmark consists of 48 questions split evenly across four difficulty levels, with an average latency of 6,324 ms per answer.

    Read article
  • ChatGPT Images 2.6 GIFs, Gemini Omni 1.2 Flash and Shifting AI Usage CapsAI News ·

    Response latency was fast and the cadence natural, though the voice carried a slightly muffled, call-center-headset quality.

    Read article
  • Claude Code as an AI Chief of Staff: A Three-Layer Knowledge BlueprintAI in Practice ·

    Multi-step runs that search Notion and fetch email threads can show noticeable latency.

    Read article
  • Napster CEO Lays Out Capture, Learn, Act, Own Plan for Video-First AI AgentsAI Startups ·

    On the uncanny valley, his position is that discomfort comes from poor execution and latency, or from asking a machine to perform human-only social behavior.

    Read article
  • ChatGPT Sites and MCP Events Bring Zapier-Style Automation Into ChatGPTAI News ·

    Workflows that need sub-minute latency, high-frequency schedules, or connectors to apps without event support still appear to be the territory of standalone pla…

    Read article
  • Self-Improving Agent Design: Two Loops, Versioned Harnesses, Sealed EvaluatorsAI in Practice ·

    …to maximize expected task performance subject to cost staying within budget, latency staying within target, and security staying at or above the required level.

    Read article
  • TinyFish Bets on AI Agents as Customers With 40-Cent Wallet BillingAI Startups ·

    …le logs the agent into an authenticated developer dashboard to chart search latency across the last 20 runs, reporting an average of 1.522 seconds, a median of…

    Read article
  • Compound's Multiplayer Email Agents and Headless Office Engine for Deal TeamsAI Startups ·

    Compound deliberately adds artificial latency so the agent waits for humans to finish their multi-part replies and then responds once, the way a thoughtful coll…

    Read article
  • Intelligence Plane: Why Memory and Ontology, Not the Model, Define AI AgentsAI in Practice ·

    Average latency per query 10.6 seconds 42.8 seconds

    Read article
  • ChatGPT for Financial Services: One Shared Model, Hosted Data, Human ReviewAI News ·

    Direct indexing sharply reduced latency and failure rates, and Kim says the built-in data bundle became the feature enterprise users praise most.

    Read article
  • NVIDIA Agent Toolkit Bets on Harness Engineering and Knowledge GraphsAI News ·

    At production scale they also hit throttling, rate limits and latency spikes, and those do not merely slow an agent down; they degrade its accuracy and reasonin…

    Read article
  • AI Coding Agents at Uber, Ford and Databricks: More Code, Same BottlenecksAI in Practice ·

    That latency justifies more caution on mobile than on servers, and it explains why consumer app teams see smaller gains from agentic coding than their backend p…

    Read article
  • GPT-6.1 Terra vs Claude Opus 5.5: Eight Builds Cost $7.64 Against $60.19AI Performance ·

    …rest of the interface is largely unchanged from GPT-6.1 Sol, but the granular slider gives users a direct way to trade latency and cost against reasoning depth.

    Read article
  • Mods for Claude Code: Custom Panels, Cache Timers and No-Code Add-OnsAI in Practice ·

    In a long session that means more latency and a lot of tokens spent re-reading text you already sent.

    Read article
  • Claude Design and Claude Code: Drag, Comment, and Ship React Without FigmaAI in Practice ·

    A sample scenario inside the widget investigates a question like why p95 latency spiked on a checkout flow overnight, pulling metrics and logs before proposing…

    Read article
  • Krea 3 and Grok Voice 3.0 Stand Out in Hands-On Tests of New AI ModelsAI News ·

    The weights are on Hugging Face and can be fine-tuned, meaning further trained for a specific task, and distilled versions are tuned for lower latency.

    Read article
  • Anthropic's Self-Replication Warning: What the 15 TV Box Test ShowsAI News ·

    Coordinating 15 slow devices piles up latency and adds a great deal of operational friction.

    Read article
  • Claude Haiku 5.5 and GPT-6 Intelligent UI Lead a Week of Cheaper, Faster AIAI News ·

    Unless latency is the real bottleneck, as in live user-facing apps, interactive coding or agents that control a desktop, the premium appears hard to justify.

    Read article
  • AI Agents Push AWS to Rethink Accounts, Permissions and GPU AllocationAI News ·

    Predictable tail latency

    Read article
  • OpenAI's New Decision Model Picks From a List in 150 ms: Seven Practical BuildsAI in Practice ·

    Typical latency About 150 ms About 1.6 seconds

    Read article
  • AI agent debugging: how Claude Code used traces to find a missing price filterAI in Practice ·

    …ace is a structured, nested record of one agent run: every conversational turn, tool call, prompt, model response, token count, latency figure, and dollar cost.

    Read article
  • cmux for Parallel AI Coding Agents: Remote Sessions That Survive a Closed LaptopAI in Practice ·

    Launch it in YOLO mode with , then type to start a low-latency spoken conversation inside cmux.

    Read article
  • DeepMind's New Robot Models Split Planning From Motion, but Dexterity LagsAI News ·

    Set latency requirements first.

    Read article
  • Shipping a Customer-Facing AI Agent: Memory, Least Privilege and EvaluationAI in Practice ·

    To find out why, Google Cloud Trace shows the latency, tool arguments and model execution spans for the whole request.

    Read article
  • GPT-6.1 Astra Pulled Over Safety: What It Means for How Companies Pick AI ModelsAI News ·

    …context of a request, then send it to whichever model best balances accuracy, latency and cost, whether an Anthropic model, an open-source model or IBM Granite.

    Read article
  • Why OpenAI's Agents Now Read App Structure Instead of ScreenshotsAI News ·

    …a custom harness is possible, but OpenAI trained its models on its own harness, so the official implementation brings advantages in accuracy, latency and cost.

    Read article
  • Claude Mods and Shared Artifacts Hint at Team Coding With AI AgentsAI News ·

    …multiple agents and continuously producing shared project information can also increase token use (tokens are the units of text a model processes) and latency.

    Read article
  • RunPod Flash Changes the Cost of Running an Image AI SaaSAI Startups ·

    Pinning a specific GPU favors predictable latency; allowing alternatives favors availability.

    Read article
  • Jev Tools for Claude Code: Six Projects and a Build DirectoryAI in Practice ·

    Its reported intent latency is roughly 250 to 350 milliseconds per spoken word, so the browser can begin responding before a complete sentence ends.

    Read article
  • Jev Email Triage Shows the Cost Case for AI Model RoutingAI Performance ·

    …ine checks: Across 150 yes-or-no decisions spanning five types of task, Jev’s median latency was 219 milliseconds per decision and the total charge was $0.0019.

    Read article

Found a mistake or missing an entry?

Wiki entries are based on AIPOST articles and widely known facts. If an entry is wrong, send us a correction request and we will check and fix it. Names from new articles are added to the wiki after the article is published.