LongMemEval is a benchmark for evaluating the long-term memory abilities of chat assistants. Released in 2024, it asks systems to answer 500 questions based on information scattered across long conversation histories spanning multiple sessions. It measures five abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention when there is no basis for an answer.
Systems under evaluation must either receive the entire history as context or store it in a memory system and retrieve the relevant parts before answering. The benchmark is used to compare memory designs for LLM agents and retrieval-based memory systems.