$ cat ai-engineering/human-feedback-loops.md

Human-in-the-Loop Feedback Loops for LLM Systems

How to capture, sample, label, and route production feedback into evaluation and product changes.

Jatin Bansal@blog:~/ai-engineering$ open human-feedback-loops

A feedback widget has little value if its records are never linked to traces, reviewed, and routed into evaluation or product changes. A useful loop preserves the request context, selects a review sample deliberately, applies a versioned rubric, and records what changed because of the label.

Capture signals with trace context

The signals factor into three categories. A production loop that doesn’t capture all three is at least one axis blind.

Explicit feedback is what the user says directly. It may be a binary verdict, a structured category, or a free-text edit. Thumbs up and down reveal intent but rarely explain the defect. A follow-up category supplies more precise labels, although only about 10–30% of users complete it. A free-text edit provides a directional correction in the original context, making it the most useful input for annotation and preference-data pipelines. Capture each form with the same trace ID.

Implicit feedback. What the user does. Six common signals: regenerate (the user asked for another try; high precision negative), follow-up clarification (the user immediately pivoted to “no, I meant…”; high precision negative), abandonment (the user closed the session mid-stream; ambiguous but trend-significant), copy-without-edit (the user copied the output to clipboard verbatim; high precision positive), copy-with-edit (the user pasted, edited externally, returned; positive-with-correction; great supervised signal if you can capture the diff), and dwell time (the user spent significantly more or less time on a response than the per-category median; ambiguous but useful in conjunction). The 2025 analysis of human–LLM dialogues found implicit signals are noisier than explicit feedback per row but vastly higher volume; the right architecture exposes them as ordered ranks over the trace store, not as standalone labels.

Reviewer annotations. What internal humans say. The structural signal: a domain expert reviewing a sampled trace, scoring it against the same rubric the LLM judge uses, and either confirming the judge’s verdict (calibration data) or overriding it (new failure-mode candidate). This is the source of truth the loop calibrates against. Volume is constrained by human bandwidth; a reviewer can sustain 50–200 labels per day depending on rubric complexity; so the sampling policy is the load-bearing design decision (next section).

A subtler signal worth mentioning: public-channel feedback; support tickets, internal Slack threads about the assistant misbehaving, customer-success notes. This is structurally similar to explicit feedback but arrives without a trace ID, and the work of re-linking a complaint back to a specific turn is non-trivial. Tools like Langfuse’s user-feedback API and LangSmith’s feedback API include explicit traceId parameters so the application can attach feedback at the moment of capture, but support tickets and Slack threads usually don’t carry that ID, and the re-linking pass is what makes them usable. Worth doing for incidents; not worth doing at steady state.

Sample for information, not volume

A reviewer with 60 minutes a day and a 12,000-trace-per-day stream can label maybe 80 traces. The sampler is the difference between 80 of the most informative traces and 80 of the loudest. Five policies, picked in combination by what the loop is trying to learn.

Random sampling: A uniform random subset of all production traces. The floor below which the queue should never starve, because every other policy is biased toward what the system already knows is broken; only random sampling discovers unknown unknowns. Eugene Yan’s recommendation to aim for “a 50:50 split of passes and fails that spans the distribution of inputs” relies on random sampling as the recall mechanism. Allocate 10–30% of the annotation budget to random, even when the system is on fire and the targeted pulls are screaming.
Stratified sampling: Random within each error-analysis category so the long-tail categories aren’t drowned out by the dominant ones. The dominant category in a customer-support assistant might be “order status” at 40% of traffic; stratifying ensures the long-tail “billing dispute” or “refund processing” categories; which are usually where the failures concentrate; get proportional annotation time.
Negative-feedback sampling: Every trace with an explicit thumbs-down or a high-precision implicit negative (regenerate, abandon, immediate-pivot) goes into the queue with a high prior. The conversion rate from “user said it was bad” to “actually bad” is well under 100% but well above the random base rate; this is the cheapest place to find real failures. Cap the rate so a single noisy user can’t dominate the queue.
Uncertainty-based active learning: The LLM judge’s confidence on a trace; or the score’s distance from a threshold; is a usable uncertainty signal. Traces where the judge’s score sits at the rubric boundary (e.g. 2 on a 1–3 scale) are the rows where the model labels are most ambiguous and the human label most informative. The LLM-based active learning literature reports 50–80% reductions in annotation budget for the same downstream quality versus random sampling on this approach; the catch is that “informative for the judge” is not the same as “informative for the application,” and uncertainty sampling alone over-indexes on edge cases.
Targeted sampling on drift signals: When the drift detector flags a category or cluster as drifted, pull the next N traces from that slice into the queue ahead of the steady-state policies. This is the equivalent of an SRE on-call following the alert to the cluster that’s degrading; same shape, applied to the annotation queue.

The RLTHF paper (Wang et al., 2025) reports that combining LLM-based initial labelling with selective human review on the uncertain rows achieves full-human-annotation alignment quality with only 6–7% of the human annotation effort on the HH-RLHF and TL;DR benchmarks. The result is specific to RLHF preference-pair labelling, but the principle generalises: an LLM-first pass that pre-labels the queue and routes only the uncertain rows to humans is the production pattern that scales when annotation bandwidth is the bottleneck.

A practical mix that holds up: 25% random, 25% stratified by category, 30% negative-feedback (capped), 10% uncertainty-based, 10% targeted on the day’s drift alerts. Re-tune the weights quarterly based on which slice’s annotations turned into the most caught failures.

Keep labels consistent

The label is the metric. If the labels are inconsistent across annotators or across days, the downstream eval suite, judge calibration, and fine-tuning corpus all inherit the inconsistency. Four practices that production teams converge on.

Codebook discipline. The rubric is a versioned artifact; not a wiki paragraph that drifts with each onboarded annotator. Each label category has a written definition, a positive example, a negative example, and an edge case that’s explicitly not in the category. Versions are stamped on every label batch so that comparisons across time know which rubric version produced the labels. When the rubric changes, the affected categories’ historical labels are quarantined or re-labelled under the new codebook; not silently re-interpreted.

Benevolent-dictator workflow. Hamel Husain’s recommendation for most teams: appoint a single domain expert as the final judge of label quality and the rubric owner. Distributed annotation efforts without a single point of authority tend to drift into Cohen’s-kappa-below-0.5 territory within a quarter. The benevolent dictator’s job is to spot-check labels, resolve disagreements, edit the codebook, and own the inter-annotator agreement metric. When multiple annotators are necessary; for volume or for legitimately ambiguous categories; periodic alignment sessions with kappa measurement on a doubly-labelled subset keep the process honest.

Inter-annotator agreement as a process metric. Double-label 5–10% of the queue (the same trace labelled independently by two reviewers), compute Cohen’s kappa for two annotators or Krippendorff’s alpha for more, and track it as a dashboard line. Kappa below 0.6 on a category means the rubric is unstable; split the category, write better examples, or accept that the category is genuinely subjective and weight it differently in downstream metrics. The categories with the highest disagreement are usually the ones whose evals are most volatile and whose judge is most unreliable; treating IAA as a debugging tool catches the problem upstream of the eval suite.

Annotation latency budget. A label that lands 12 weeks after the trace is logged is too late to feed back into a meaningful product cycle. The target is single-digit days from “trace generated” to “label applied,” which constrains both the queue size and the policy mix. Queues that grow past their throughput will silently age into uselessness; cap the queue, drop the lowest-priority overflow, and ship the missing labels as a known limitation rather than letting the queue rot.

Route labels into product changes

A labelled trace is half the work. The other half is the routing decision: where does the label go, and who acts on it?

Into the eval suite: A confirmed failure that recurs in the wild is a new row on the golden set. The provenance; trace ID, date, annotator; gets stamped on the row, and the row gates merges until the underlying behaviour is fixed. This is the highest-leverage routing: it makes the failure expensive to re-introduce.
Into the judge calibration set: Every labelled trace is also a calibration data point for the LLM judge. Computing judge-human agreement on a rolling window catches judge drift before the eval suite numbers go bad. When agreement drops below a threshold (kappa < 0.6 against the human reference), the judge needs a re-tune or a model swap.
Into the prompt: Edits where the user rewrote the response are a direct prompt-improvement signal. The diff often clusters into a small number of recurring corrections; “use bullet points,” “cite the source,” “don’t apologise”; that are cheap to add to the system prompt as explicit rules.
Into the retriever: A negative trace where the right document was in the index but didn’t make it into the top-K is a retrieval failure, not a generation failure. The labelled trace becomes an example row for retriever evaluation, and over time the corpus of such rows is what drives changes to chunking, reranking, or hybrid search weights.
Into the model: Confirmed-failure-confirmed-fix pairs are the raw material for preference-pair fine-tuning via DPO and its variants. Most application teams don’t fine-tune anything in 2026, but for teams that do, this is the cleanest source of training data; preference pairs grounded in real production failures, validated by a domain expert against a versioned rubric.

The loop’s cycle time; from observation to a structural fix that prevents the failure recurring; is the metric the team should be tracking, not the raw label volume. Eugene Yan’s framing is right: whoever turns the data flywheel faster wins. A team that labels 5,000 traces a quarter and ships zero fixes is doing labelling-theatre; a team that labels 500 traces a quarter and ships 30 structural fixes is running the loop.

Code: capturing structured user feedback against a Langfuse trace in Python

The harness below captures explicit feedback (thumbs and structured reason) and implicit feedback (regenerate, dwell time, edit diff) against a Langfuse trace ID, stamps each with the trace context, and exposes a simple FastAPI endpoint. Install: pip install fastapi uvicorn langfuse pydantic.

python

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
# pip install fastapi uvicorn langfuse pydantic
import os
from datetime import datetime, timezone
from typing import Literal, Optional

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel, Field
from langfuse import get_client

app = FastAPI()
langfuse = get_client()

# --- schemas ---

class ExplicitFeedback(BaseModel):
    trace_id: str
    user_id: str
    rating: Literal["thumbs_up", "thumbs_down"]
    reason_category: Optional[
        Literal[
            "hallucination", "refused", "too_long", "too_short",
            "missed_point", "formatting", "other",
        ]
    ] = None
    free_text: Optional[str] = Field(default=None, max_length=2000)


class EditFeedback(BaseModel):
    trace_id: str
    user_id: str
    original_output: str
    edited_output: str
    edit_kind: Literal["copy_and_edit", "in_place_edit"]


class ImplicitFeedback(BaseModel):
    trace_id: str
    user_id: str
    event: Literal[
        "regenerate", "abandon_mid_stream", "copy_without_edit",
        "follow_up_pivot", "long_dwell", "short_dwell",
    ]
    dwell_ms: Optional[int] = None
    follow_up_text: Optional[str] = None


# --- routes ---

@app.post("/feedback/explicit")
def post_explicit(fb: ExplicitFeedback):
    # Score: -1 for thumbs_down, +1 for thumbs_up. Langfuse stores arbitrary
    # numeric scores against a trace_id; downstream queries filter by name.
    langfuse.create_score(
        trace_id=fb.trace_id,
        name="user_rating",
        value=1.0 if fb.rating == "thumbs_up" else -1.0,
        data_type="NUMERIC",
        comment=fb.free_text or "",
        metadata={
            "user_id": fb.user_id,
            "reason_category": fb.reason_category,
            "signal_type": "explicit",
            "captured_at": datetime.now(timezone.utc).isoformat(),
        },
    )
    return {"status": "ok"}


@app.post("/feedback/edit")
def post_edit(fb: EditFeedback):
    # An edit is supervisor signal. Stamp the diff as a score-with-data so
    # the trace store keeps the original + corrected pair together.
    langfuse.create_score(
        trace_id=fb.trace_id,
        name="user_edit",
        value=1.0,  # presence of an edit; magnitude is in the metadata
        data_type="NUMERIC",
        comment=fb.edited_output,
        metadata={
            "user_id": fb.user_id,
            "original_output": fb.original_output,
            "edit_kind": fb.edit_kind,
            "signal_type": "explicit_correction",
            "captured_at": datetime.now(timezone.utc).isoformat(),
        },
    )
    return {"status": "ok"}


@app.post("/feedback/implicit")
def post_implicit(fb: ImplicitFeedback):
    if fb.event == "long_dwell" and fb.dwell_ms is None:
        raise HTTPException(400, "dwell_ms required for dwell events")

    # Implicit signals carry a prior on satisfaction direction.
    polarity = {
        "regenerate": -1.0,
        "abandon_mid_stream": -1.0,
        "follow_up_pivot": -1.0,
        "short_dwell": -0.3,  # weaker negative
        "copy_without_edit": 1.0,
        "long_dwell": 0.5,    # weaker positive
    }[fb.event]

    langfuse.create_score(
        trace_id=fb.trace_id,
        name=f"implicit_{fb.event}",
        value=polarity,
        data_type="NUMERIC",
        metadata={
            "user_id": fb.user_id,
            "dwell_ms": fb.dwell_ms,
            "follow_up_text": fb.follow_up_text,
            "signal_type": "implicit",
            "captured_at": datetime.now(timezone.utc).isoformat(),
        },
    )
    return {"status": "ok"}


@app.get("/")
def root():
    return {"ok": True}


if __name__ == "__main__":
    import uvicorn
    uvicorn.run(app, host="0.0.0.0", port=8080)

the trace ID is required, not optional; feedback without a trace ID is feedback that can’t be investigated, and the API should reject it. Explicit, edit, and implicit signals land under different score names* (user_rating, user_edit, implicit_*) so downstream queries can filter cleanly; mixing them under one name will make the dashboards lie. The polarity weights for implicit signals are starting points*, not laws; recalibrate them quarterly against labelled traces (high-precision implicit signals should correlate strongly with confirmed-failure annotations; if they don’t, the weight is wrong). The metadata is structured*, not free-text; reason_category and signal_type are enums in the schema, which is what makes the trace store queryable a year from now. Free-text fields are useful for support escalation but can’t power dashboards.