$ cat ai-engineering/sleep-time-compute.md

Sleep-Time Compute for AI Agents

Background consolidation and speculative work for stateful agents with explicit consistency controls.

Jatin Bansal@blog:~/ai-engineering$ open sleep-time-compute

Reflection, compression, deduplication, embedding refresh, and speculative answering can improve an agent’s next response. They can also make the current response unacceptably slow. Sleep-time compute moves work that is useful but not immediately required onto a background path.

The name comes from Sleep-time Compute, which separates compute used while serving a query from compute performed between queries. The distinction is architectural, not biological: a sleep-time worker may run after an idle period, every few turns, on a schedule, or from a queue.

The key test is whether the current answer depends on the result. Required work remains on the hot path. Maintenance and speculative work can move off it when they have a clear consistency model and enough future reuse to justify their cost.

What belongs in the background

Good candidates share two properties: they are expensive relative to orchestration, and their output can benefit several later turns.

Consolidating episodes into summaries or beliefs
Merging duplicate memories and marking stale records
Rebuilding embeddings after a model change
Refreshing indexes or materialized views
Evaluating likely next questions and caching expensive intermediate results

Small writes, permission checks, and information needed to answer the active request do not belong in the queue. Deferring them may reduce displayed latency while making the answer incomplete or unsafe.

The database analogy is useful. Transaction processing serves current reads and writes; maintenance jobs compact logs, refresh views, and collect statistics. Those jobs improve later queries but need scheduling, resource limits, and isolation from foreground traffic.

The Letta formulation

The Letta write-up describes a foreground agent and a sleep-time agent sharing memory. The foreground path answers the user. The background agent receives a narrower tool set for memory maintenance and can use a cheaper model. This separation makes cost and authority explicit.

The associated paper studies shifting reasoning from test time to periods before a query arrives. Its reported benefits depend on the workload and prediction quality; they should not be treated as universal multipliers. The transferable engineering pattern is the split budget: reserve interactive compute for latency-sensitive work and spend background compute where it can be amortized.

A single process can implement the same boundary, but separate workers are easier to rate-limit and observe. The foreground service appends events and signals activity. A worker leases a bounded batch, transforms it, and commits results with version checks.

A minimal consolidation worker

This example uses a queue and an idle-or-cadence trigger. The consolidation function is deliberately separate from scheduling so it can be tested deterministically.

python

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
from dataclasses import dataclass
from queue import Queue, Empty
from threading import Event, Thread
from time import monotonic

@dataclass
class Episode:
    id: str
    text: str
    version: int

pending: Queue[Episode] = Queue()
stop = Event()

def consolidate(batch: list[Episode]) -> dict:
    """Call a model and return structured additions plus source IDs."""
    return {
        "summary": summarize([e.text for e in batch]),
        "source_ids": [e.id for e in batch],
        "source_versions": {e.id: e.version for e in batch},
    }

def commit(result: dict) -> bool:
    """Atomically fail if a source changed while the worker ran."""
    with memory_transaction() as tx:
        if not tx.versions_match(result["source_versions"]):
            return False
        tx.insert_summary(result["summary"], result["source_ids"])
        tx.mark_consolidated(result["source_ids"])
        return True

def worker(idle_seconds=60, max_batch=50, max_wait=900):
    batch: list[Episode] = []
    last_activity = monotonic()
    while not stop.is_set():
        try:
            batch.append(pending.get(timeout=1))
            last_activity = monotonic()
        except Empty:
            pass

        idle = monotonic() - last_activity >= idle_seconds
        overdue = batch and monotonic() - last_activity >= max_wait
        if batch and (idle or overdue or len(batch) >= max_batch):
            work, batch = batch[:max_batch], batch[max_batch:]
            if not commit(consolidate(work)):
                for episode in work:
                    pending.put(episode)

Thread(target=worker, daemon=True).start()

In a real system, summarize should produce validated structured output rather than unconstrained prose. The transaction should create provenance edges from the summary to its episodes. Failed batches need bounded retries and a dead-letter queue; blindly re-enqueuing a permanently bad record creates an infinite loop.

Do not delete raw episodes as part of consolidation unless retention policy requires it. A summary supports broad recall, while the source episodes support exact questions and audits. Marking episodes as consolidated is enough to keep them out of later batches.

Scheduling and resource isolation

An idle threshold alone fails under steady traffic: a user active every few seconds may prevent maintenance for hours. A turn-count or maximum-delay trigger provides a fallback. Conversely, a fixed short interval can run expensive jobs when there is no useful work.

Use several controls together:

minimum batch size or age, so tiny updates do not dominate orchestration;
maximum batch size, which bounds model context and transaction time;
rate and cost limits per hour;
lower worker priority or a separate compute pool;
back-pressure when the foreground service is under load.

The worker’s model can often be smaller than the foreground model because the operation is narrow and schema-constrained. Verify this with task-specific evaluations. A cheap model that merges contradictory facts incorrectly is not a saving.

Speculative pre-computation

Consolidation improves memory. A second use of sleep-time compute predicts likely next requests and pre-computes expensive work. A coding agent that has just diagnosed a failing test might prepare likely follow-ups such as locating related tests or drafting a focused patch.

Exact-answer caches are risky because even a small context change can invalidate them. Key cached work by the query representation and a version of every relevant dependency. On lookup, require both a strong query match and matching context versions. A miss should fall through to normal reasoning.

Prefer caching reusable intermediates over polished responses. Search results, parsed repository maps, or validated calculations are easier to reuse safely than a complete answer whose wording and assumptions may no longer fit.

Measure useful-hit rate rather than raw hit rate. A cache that frequently returns plausible but stale results reduces latency at the expense of correctness. Sample hits against fresh foreground computation and evict entries that do not hold up.

Consistency on shared state

The worker and foreground agent may read and write the same store. Without coordination, the worker can consolidate a record while the foreground path edits it, or publish a summary based on a mixture of versions.

Useful patterns include:

Single writer: the foreground appends immutable events; only the worker produces derived memory.
Optimistic concurrency: each record carries a version, and the worker retries when versions change before commit.
Snapshot processing: the worker reads from a database snapshot and publishes an output labelled with that snapshot’s revision.

Single-writer designs are easiest to reason about but may not fit interactive memory edits. Optimistic concurrency is flexible if transforms are idempotent. Snapshot processing is useful for large periodic rebuilds.

Every derived record should identify its source IDs and transform version. If a source is corrected or deleted, a reverse provenance index can find summaries that require recomputation.

Failure modes to plan for

Over-consolidation loses detail. Keep raw evidence and test exact historical questions, not just broad recall.

Background cost exceeds foreground savings. Track worker tokens, model cost, CPU time, useful outputs, and later reuse. Rate-limit speculative work first.

A stale cache serves confidently. Version the underlying context, apply TTL or LRU limits, and fall back on any mismatch.

Maintenance failures become invisible. Give the worker its own traces, queue-depth metrics, retry counts, and alerts. A week-old embedding index can degrade retrieval without causing an obvious foreground exception.

The worker has too much authority. Restrict it to memory and cache operations. It should not edit application code, contact users, spend money, or invoke destructive tools merely because it runs outside the interactive session.

No idle period ever occurs. Trigger on elapsed time or turn count as well as inactivity.

Sleep-time compute is worthwhile when state persists, future queries reuse the work, and interactive latency matters. It is usually overhead for one-shot, open-domain requests with no shared context. Start with one measurable maintenance job, compare foreground latency and retrieval quality before and after, and expand only when the background output is demonstrably reused.

Deployment checklist

Start with one queue and one transform. Give every job an idempotency key derived from its operation, source IDs, and transform version. If a worker crashes after committing but before acknowledging the message, a retry should observe the existing result instead of creating a duplicate summary.

Define queue ownership and retention. The foreground service should know whether enqueue failure blocks the request, degrades gracefully, or is retried through an outbox. Memory consolidation is normally best-effort, so an outbox written in the same transaction as the episode provides durability without adding a remote queue call to the response path.

Set service-level objectives for maximum queue age, successful pass rate, and cost per consolidated episode. Queue depth alone is ambiguous: a large queue can be healthy during a known import, while one old poison message can violate freshness with a nearly empty queue. Alert on age and repeated failure reason as well as depth.

Roll out workers with read-only or shadow mode first. Let them produce proposed summaries and compare those with existing retrieval results without publishing. Sample for contradiction, lost detail, and privacy leakage. When writes are enabled, begin with a narrow namespace and retain a kill switch that stops publication without preventing foreground memory reads.

Finally, rehearse rebuilding derived state. If summaries and indexes are truly derived, the team should be able to delete a test copy and regenerate it from source events with the same transform versions. A system that cannot rebuild its background products has made them primary data by accident.