Context freshness: your assistant should know how old its facts are

Picture an assistant that has your calendar in its context. At 9am it read “dentist at 2pm.” At 10am the office called and moved it to 3pm, and your calendar updated. At 1pm you ask the assistant, “when do I need to leave?” and it tells you to leave at 1:40.

Nothing in that exchange looks broken. The model reasoned correctly. The retrieval worked. The answer was confident and specific. It was also wrong, because one of the facts it was reasoning over had quietly gone bad and nothing in the system knew.

I think this is one of the most underrated ways agents fail. People spend a lot of energy on whether the model is smart enough and whether retrieval finds the right document. Much less goes into a simpler question: how old is this fact, and is it still true?

This post is my attempt to work out what a fix looks like. The short version is that every fact an assistant holds should carry its age, different kinds of facts should go stale at different speeds, and the code that builds the prompt should rank facts on freshness alongside relevance.

Stale state is a quiet failure

Some bugs announce themselves. A crash, a timeout, an exception in the logs. Stale context is quiet. The system does exactly what it was built to do, using data that was correct when it was fetched. Nobody gets an error. The user just gets a wrong answer delivered in the same calm tone as a right one.

It also gets worse the more “memory” you give an agent. A chatbot with no memory can’t be stale about your calendar because it doesn’t know your calendar. Once you add a memory layer, a profile, a cached summary of your inbox, or a list of things the agent learned about you last week, you’ve added a pile of facts that each have an expiration date nobody wrote down.

Research on question answering has been circling this for a while. FreshQA, from Vu and coauthors, is a benchmark of 600 questions split into four groups: never-changing, slow-changing, fast-changing, and questions with false premises.¹ Their headline finding was that every model they tested, regardless of size, struggled on the fast-changing questions and the false-premise ones. A model’s weights are a snapshot. A fact that changed after training can leave the model confidently repeating an outdated answer.

RealTime QA took a similar angle by posting new questions about current events every week. One finding stuck with me: GPT-3 could often update its answer when retrieval handed it a newer document, but when the retrieved documents didn’t have enough information, it tended to fall back on an outdated answer.² Retrieval helps, then. It just doesn’t automatically know what “now” is.

Dhingra and coauthors looked at the same problem from the training side. Language models are trained on a snapshot of text, so facts like who holds an office drift out of date. Their fix was to train with the timestamp attached to each piece of text, which made the model better calibrated about facts from later periods.³ I like that paper because the core move is so simple. If time matters, write the time down next to the fact.

That’s the thread I want to pull on, but at the level of an assistant’s working context instead of a model’s weights.

Different facts rot at different speeds

Here’s the first idea. A fact’s useful lifetime depends on what kind of fact it is.

• Your next calendar event can change at any moment and matters to the minute.
• Your location changes a few times a day.
• The status of a project, like “essay draft is 80% done,” changes over days.
• A preference, like “likes short answers” or “is vegetarian,” might hold for months or years.
• Your name basically never changes.

If you treat all of these the same, you get one of two bad outcomes. Either you refresh everything constantly, which is slow and expensive, or you refresh nothing and the fast-moving facts go bad.

The web solved a version of this a long time ago. HTTP caching gives every cached response a freshness lifetime, and the spec defines a response as fresh when its age hasn’t passed that lifetime and stale once it has.⁴ The server says how long something stays good with Cache-Control: max-age=600, and the cache counts. DNS does the same thing with a TTL in seconds on every record.⁵ Nobody expects a DNS resolver to remember an IP address forever, and nobody expects it to look the address up fresh on every packet either. The record says how long it can be trusted.

So the first rule I’d use for agent context is: every fact type gets a TTL, and every stored fact records when it was last confirmed.

These are the numbers I’d start with. They’re guesses, not measurements, and I’d tune them by watching where the assistant gets things wrong.

I’ll explain why there are two columns in a second.

A TTL is a cliff, freshness is a slope

A TTL is binary. Before it expires the fact is fully trusted. After, it’s gone. That’s right for some things, but it throws away information. A calendar entry fetched 5 minutes ago and one fetched 5 hours ago are both “inside the TTL” if the TTL is 6 hours, and they really shouldn’t be trusted the same.

So I’d use both. A smooth freshness score that decays with age, and a hard cutoff past which the fact is not allowed in the prompt as current truth at all.

For the smooth part, exponential decay with a half-life is the obvious choice:

freshness = 0.5 ** (age / half_life)

At age zero the freshness is 1. After one half-life it’s 0.5. After two it’s 0.25. The half-life controls how quickly the ranking score falls. It is a heuristic, not a measured probability that the fact remains true.

This idea isn’t new to agents either. The Generative Agents paper from Stanford and Google, the one with 25 simulated characters living in a little town, retrieves memories using three scores: recency, importance, and relevance.⁶ Recency there is exactly an exponential decay, with a factor of 0.995 per sandbox game hour since the memory was last retrieved. Importance is a 1 to 10 rating the language model gives each memory when it’s created. Relevance is cosine similarity between embeddings. All three are scaled to 0-1 and weighted equally.

One difference from what I’m proposing: their recency is one decay rate for every memory. That makes sense in a simulation where memories are mostly “things that happened.” For an assistant that holds your calendar and your dietary restrictions in the same store, one decay rate can’t be right. The calendar needs a half-life of minutes, and the diet needs a half-life of months.

Ranking context: relevance, freshness, reliability, cost

Once every fact has a freshness score, building the prompt becomes a ranking problem. The context window is limited, and even when it isn’t, stuffing it has costs I’ll get to below. So for each candidate fact I’d compute:

score = relevance + freshness + reliability - cost

with a weight on each term.

• Relevance is how related the fact is to what the user just asked. Usually embedding similarity, which I wrote about in how a word gets meaning.
• Freshness is the decay score above.
• Reliability is how much you trust the source. A calendar API is more reliable than something the model inferred from a half-remembered conversation.
• Cost is the token count, because every token you add costs money and latency and pushes something else out.

Here’s a small version in Python.

from dataclasses import dataclass

# Half-life in hours per fact type. Guesses, not measurements.
HALF_LIFE = {
    "calendar": 0.5,
    "location": 2,
    "task_status": 48,
    "preference": 24 * 90,
}
# Past this age a fact is not "less useful", it is unsafe to state as current.
HARD_TTL = {
    "calendar": 6,
    "location": 12,
    "task_status": 24 * 14,
    "preference": 24 * 365,
}

@dataclass
class Fact:
    text: str
    kind: str
    age_h: float        # hours since we last confirmed it
    relevance: float    # 0..1, e.g. cosine similarity to the request
    reliability: float  # 0..1, how much we trust the source
    tokens: int

def freshness(f):
    if f.age_h > HARD_TTL[f.kind]:
        return 0.0
    return 0.5 ** (f.age_h / HALF_LIFE[f.kind])

def score(f, w_rel=1.0, w_fresh=1.0, w_trust=0.5, w_cost=0.2):
    return (w_rel * f.relevance + w_fresh * freshness(f)
            + w_trust * f.reliability - w_cost * f.tokens / 100)

facts = [
    Fact("Dentist at 3pm today", "calendar", 0.2, 0.9, 0.95, 8),
    Fact("Dentist at 2pm today", "calendar", 20, 0.9, 0.95, 8),
    Fact("Prefers short answers", "preference", 24 * 30, 0.4, 0.7, 5),
    Fact("Essay draft is 80% done", "task_status", 30, 0.7, 0.6, 9),
    Fact("Is at the library", "location", 3, 0.5, 0.8, 6),
]

for f in sorted(facts, key=score, reverse=True):
    print(f"{score(f):5.2f}  fresh={freshness(f):.2f}  {f.text}")

Running it prints:

 2.12  fresh=0.76  Dentist at 3pm today
 1.63  fresh=0.65  Essay draft is 80% done
 1.53  fresh=0.79  Prefers short answers
 1.36  fresh=0.00  Dentist at 2pm today
 1.24  fresh=0.35  Is at the library

The fresh dentist time wins, which is good. Now look at the fourth line. The stale “2pm” entry has a freshness of exactly zero, because it’s 20 hours old and the calendar TTL is 6. It still beats the location fact, because its relevance and reliability are high and the score is a sum.

That’s the bug I wanted to show. With an additive score, a fact that’s very relevant and from a trusted source can survive even when it’s dead. High relevance makes it worse, because the dead fact is exactly the one the model will lean on. The fix is to treat the hard TTL as a filter before ranking, not as one more term inside it:

usable = [f for f in facts if freshness(f) > 0]
refetch = [f for f in facts if freshness(f) == 0]

Now the ranking is the same four facts minus the stale dentist, and refetch holds ['Dentist at 2pm today']. What you do with the refetch list depends on the fact. For a calendar, call the API again before answering. For something you can’t cheaply re-check, you can still include it, but labeled, like “as of yesterday evening, unconfirmed.”

The weights in that function are made up. If I were building this for real I would want a test set of questions with known right answers at known times and tune the weights against it. A project I ran this summer taught me how easy it is to fool yourself with a scoring method that feels right, and that the only protection is comparing against simple baselines, including doing nothing. So the first baseline I’d check is “no freshness term at all,” and the second is “only the hard TTL filter.” If the smooth decay doesn’t beat the plain filter, it isn’t earning its complexity.

Where facts go in the prompt matters too

Ranking decides which facts make it in. There’s a second question about where to put them.

“Lost in the Middle,” by Liu and coauthors, tested what happens when the one document that contains the answer is moved around inside a long prompt.⁷ Performance was usually best when the relevant information was at the start or the end of the context and dropped when it sat in the middle. The numbers are striking. For GPT-3.5-Turbo with 20 documents, accuracy was 75.8% when the answer document came first and 53.8% when it was in the middle. With no documents at all, answering from memory, it got 56.1%. Burying the right document in the middle was slightly worse than giving it no documents at all.

FreshPrompt, the method from the FreshQA paper, uses a related trick for time. It sorts retrieved evidence by date from oldest to newest, so the most recent evidence sits at the end of the prompt, right before the question.¹ In their ablation on GPT-4, ordering by time or by search rank both beat random order, by up to 2.2% overall accuracy.

So the packing step I’d use is: top-scoring facts at the very start and the very end, lower-scoring ones in the middle, and within any one topic, the newest version last. It’s a small thing that costs nothing to do.

Tell the model how old things are

The cheapest fix in this whole post is also the one I’d do first. Put the age in the prompt.

Instead of:

User's next event: Dentist, 3:00pm

write:

Current time: 1:02pm, Sat Aug 8.
User's next event (from calendar, checked 12:50pm): Dentist, 3:00pm
User preference (learned Jul 2): prefers short answers

This borrows the time-aware paper’s idea of making dates explicit, but applies it at inference time; that paper does not validate this particular design.³ The model now has what it needs to say “as of ten minutes ago” or to notice that a fact from last week might not hold. Without the timestamp, a fact from five minutes ago and a fact from five weeks ago look identical in the prompt. The model has no way to tell them apart, so you can’t be surprised when it doesn’t.

This matters even more for voice. On a screen you can show a little gray “updated 3h ago” under a card, and the user’s eye does the work. In a voice reply there’s no gray text. Either the assistant says the age out loud or the user never learns it. And you can’t say it every time, because “as of 12:50pm, your dentist appointment, according to your calendar, is at 3pm” is exhausting to listen to.

My view is that the freshness score should decide this too. If the fact is fresh, just state it. If it’s in the decayed zone, say the age in a few words, like “last I checked, around noon.” If it’s past the hard TTL, either refetch or say plainly that you’re not sure. That keeps most answers short and puts the hedging only where it’s earned.

Update incrementally, don’t regenerate

The last piece is how facts get into the store in the first place.

The tempting design is to regenerate the whole context on a schedule. Every hour, re-summarize the inbox, re-read the calendar, rebuild the user profile, and paste it all in. This is simple and it’s what I’d build at a hackathon. It has two problems.

First, it’s wasteful. Most facts didn’t change in the last hour. Re-deriving “prefers short answers” every hour costs tokens and buys nothing.

Second, and worse, regeneration can make things less correct. If a summary is rebuilt by a model from raw data, each rebuild is a new chance for the model to drop a detail or invent one. A fact that was right in the last version can come out wrong in the next one for no reason except sampling.

The alternative is to store facts as separate entries keyed by what they’re about, and update one entry when something changes:

from datetime import datetime, timezone

store = {}  # key -> (value, source, confirmed_at)

def on_event(key, value, source, now):
    store[key] = (value, source, now)

on_event("calendar:dentist", "3pm", "calendar_api", datetime.now(timezone.utc))

If the calendar sends a change, you overwrite calendar:dentist. You don’t touch the preference entries. In the toy ranking above, the two dentist facts were separate rows, which is exactly the mess this avoids. With a keyed store, the 3pm entry replaces the 2pm one instead of sitting next to it and competing.

When you can’t get change events, polling still works, and the half-life tells you how often to poll each type. Facts with short half-lives get re-checked often. Facts with long ones almost never. The web has a nice pattern for the in-between case called stale-while-revalidate: serve the slightly stale copy right away and refresh it in the background, within a set window.⁸ For an assistant that would mean answering with the cached calendar if it’s only a little old, while kicking off a refetch, so the next question gets the fresh one. Past the window, you wait for the refetch.

Where this shows up in things I’ve built

None of my projects so far has had a real freshness layer, but looking back, the problem was sitting in a few of them.

DebateSim pulls real bills from Congress.gov and LegiScan. A bill’s text and status change as it moves through committees and votes. Any summary or analysis of a bill is a fact with a date on it, and an argument built on last month’s version of a bill can be about a version that no longer exists.

HAVEN, the watch I built at Milpitas Hacks, was designed so shelters broadcast bed and meal counts over a mesh network. A bed count is about as fast-moving as data gets. “12 beds open” is useful at 6pm and possibly harmful at 11pm if all 12 are gone and someone walks across town for one. If I took HAVEN further, every count shown on that tiny screen should carry how old it is, and old counts should fade out instead of sitting there looking current.

That’s the pattern I keep seeing. Freshness doesn’t belong to one kind of app. Any time a system shows or says a fact about a world that keeps moving, the fact needs a timestamp, and something needs to decide when the timestamp is too old.

What I’d try next

The part that worries me is how ordinary a stale answer can sound. An assistant can remember something accurately and still be wrong to use it today. If it remembers that I was busy last week, I don’t want that quietly shaping every suggestion it makes next month.

I’d start by testing a few clear cases: a changed deadline, a canceled plan, a preference that was only temporary. Does the assistant notice the newer information, and can I see why it chose one version over another? If that works, more elaborate scoring might be worth trying. What interests me about the future of assistant memory is whether it can become easier to correct as it remembers more.

Sources