Skip to content

LLM Interview Questions: Evaluation and Release Gates

A 12-question study guide to measuring LLM quality, building trustworthy evaluations, and deciding when a prompt or model change is safe to ship.

Ekanatha Reddy Urivakili15 min3,229 words

LLM interviews often focus on prompting, retrieval, or model architecture. In production, a harder question is whether a change made the system better—and whether you can tell before users pay the cost of a regression.

This guide focuses on LLM evaluation and release readiness: how to define quality, measure it, interpret noisy results, and operate changes safely. The questions progress from fundamentals to system design. For each, explain not only what you would measure, but also what the measurement cannot tell you.

Fundamentals

1. What does it mean to evaluate an LLM application?

What the interviewer is really testing: Whether you distinguish model capability from application quality, and whether you can translate a product requirement into observable behavior.

Model answer:

An LLM application is a pipeline, not just a model call. Its output can depend on instructions, retrieved context, tools, parsing, policies, and downstream code. I would evaluate the behavior users need from the whole pipeline, then use component-level checks to locate failures.

Start by defining a task and its failure modes. For a support assistant, those might include incorrect answers, unsupported claims, policy violations, and failure to abstain when evidence is missing. Quality is usually multi-dimensional: correctness, relevance, safety, format compliance, latency, and cost can all matter.

There is no universal “LLM quality” score. I would choose measures that correspond to the product’s risks, report them separately, and state what evidence supports each one.

Follow-up: How would you decide which failure modes deserve a hard release block?
Hint: Tie the decision to user harm, reversibility, and the ability to detect the failure after release.

2. How would you build an evaluation dataset?

What the interviewer is really testing: Whether you understand representativeness, labels, data leakage, and the maintenance cost of a test set.

Model answer:

I would begin with real task examples, then deliberately add cases that are rare but important: ambiguous requests, missing evidence, conflicting instructions, malformed input, and policy-sensitive requests. A dataset made only from easy, frequent cases can give a reassuring score while missing the failures that matter most.

Each case should record the input, relevant context, expected behavior, labels or reference answers, and the reason the case exists. For some tasks, a reference answer is appropriate; for open-ended tasks, it may be better to label requirements such as “must cite the provided policy” or “must abstain.”

Keep a held-out set for final comparisons. Watch for contamination: examples used to tune prompts or thresholds should not be presented as independent evidence of generalization. Version the dataset and review new cases rather than silently changing the test set.

Follow-up: What would you do when reviewers disagree about the correct answer?
Hint: Separate genuine ambiguity from reviewer error, and encode acceptable alternatives where appropriate.

3. When should you use exact match, task metrics, or a rubric?

What the interviewer is really testing: Whether you can select a measure that matches the output contract, rather than defaulting to a single convenient score.

Model answer:

Use exact match when the output has a canonical representation, such as a normalized identifier or a constrained classification label. For structured output, validate the schema and then score the fields that matter. For generated prose, exact string match is usually too brittle: two correct answers can have different wording.

Task-specific measures are preferable when they capture the actual outcome. Examples include precision and recall for extraction, or whether a response contains required evidence. A rubric can cover qualities that do not reduce cleanly to a deterministic metric, but it requires clear criteria and validation.

I would report a small set of interpretable measures, not hide trade-offs behind a single aggregate. If a summary score is necessary, document its weighting and keep the component scores visible.

Evaluation approachUseful whenMain limitation
Exact matchOutputs have a canonical formPenalizes harmless wording differences
Task-specific metricA task has measurable outcomesCan miss important unmeasured behavior
Human rubricQuality needs contextual judgmentReviewer time and consistency are limited
LLM-as-judgeMany open-ended examples need triageJudge preferences and errors can skew results

Minimal runnable example — exact match after normalization:

def normalize(text: str) -> str:
    return " ".join(text.lower().strip().split())

def exact_match(prediction: str, reference: str) -> bool:
    return normalize(prediction) == normalize(reference)

cases = [
    ("The answer is 42.", "The answer is 42.", True),
    ("  YES ", "yes", True),
    ("42", "forty-two", False),
]

for prediction, reference, expected in cases:
    actual = exact_match(prediction, reference)
    assert actual is expected
    print(f"{actual}: {prediction!r}")

Notice that normalization removes superficial differences, but it does not make semantically equivalent answers equivalent.

Follow-up: How would you measure whether a response grounded its claims in supplied documents?
Hint: Score claim support against evidence, rather than treating the presence of a citation as proof.

4. What are the risks of using an LLM as an evaluator?

What the interviewer is really testing: Whether you treat model-based judgments as measurements that need validation, not as ground truth.

Model answer:

A judge model can be useful for triage and consistent, rubric-based comparisons, especially when reviewing many open-ended outputs. But it can prefer certain styles, favor longer answers, miss subtle factual errors, or be sensitive to how candidates are ordered and described.

I would write a specific rubric, test it on examples reviewed by humans, and measure agreement on the dimensions that matter. For pairwise comparisons, randomize the order of the candidates and check whether reversing the order changes the result. I would also inspect disagreement cases instead of reporting only the judge’s aggregate score.

The judge is part of the evaluation system. Its model, prompt, and configuration should be versioned. A judge score is evidence—not a substitute for task-specific tests or human review of high-risk cases.

Follow-up: What would you do if a judge consistently prefers one candidate’s style?
Hint: Look for rubric ambiguity, order effects, and style-correlated differences unrelated to the task.

Intermediate

5. How do offline evaluations and online evaluations complement each other?

What the interviewer is really testing: Whether you understand that a curated test set and observed production behavior answer different questions.

Model answer:

Offline evaluations let you compare candidates on a repeatable set of cases before exposing users. They are useful for catching known regressions, testing edge cases, and making model or prompt experiments reproducible. Their weakness is that they only represent the cases and labels we thought to include.

Online evaluation reveals behavior under real traffic, including distribution shifts and user interaction patterns. It can use operational signals—such as completion, retries, or escalation—and sampled human review. However, online signals can be biased: users who abandon a response may not explain why, and a thumbs-up is not a precise correctness label.

I would use offline tests as a release filter and online evaluation as ongoing validation. A mismatch between them is a signal to investigate the dataset, the production distribution, or the choice of proxy.

Follow-up: Why might user satisfaction rise while answer correctness falls?
Hint: Consider presentation, expectations, and which users choose to give feedback.

6. How do you tell whether an evaluation change is meaningful?

What the interviewer is really testing: Whether you can reason about noisy samples and avoid shipping based on a small or selectively chosen set of examples.

Model answer:

I would compare candidates on the same examples wherever possible. Paired comparisons reduce variation caused by one candidate seeing easier cases than another. I would report the number and composition of examples, overall scores, slice-level results, and uncertainty—not just a difference between two point estimates.

For numeric metrics, an appropriate confidence interval or resampling method can show how much the result might vary with the sample. For qualitative judgments, I would review disagreement cases and assess reviewer consistency. A tiny average improvement can be less important than a regression in a high-risk slice.

Statistical uncertainty does not replace product judgment. The required evidence depends on the impact of failure, the cost of a mistake, and whether the change is reversible. Avoid repeatedly checking results and stopping as soon as a favorable score appears unless the experiment design accounts for that behavior.

Follow-up: What if the overall score improves but one important segment gets worse?
Hint: Decide segment-specific guardrails before examining results; do not let an average conceal a critical regression.

7. How would you evaluate a retrieval-augmented generation system?

What the interviewer is really testing: Whether you can separate retrieval failures from generation failures and evaluate the user-visible result as well as its components.

Model answer:

I would evaluate retrieval and generation separately, then evaluate the end-to-end response. For retrieval, useful questions include whether relevant evidence appears in the retrieved results and whether the selected context is sufficient for the query. For generation, test whether claims are supported by that context, whether the answer addresses the question, and whether the system abstains when evidence is inadequate.

Component tests help diagnose a bad answer. If the needed passage was never retrieved, changing the generation prompt is unlikely to fix the root cause. If the passage was present but the answer contradicts it, the issue is more likely in synthesis, instructions, or output handling.

I would include cases where no relevant source exists and where sources conflict. A system that answers confidently in those cases may look strong on ordinary answer-quality scores while failing its trust contract.

Follow-up: How would you test a change to chunking or retrieval ranking?
Hint: Hold the generation behavior constant where possible and inspect both retrieval measures and end-to-end outcomes.

8. What belongs in a regression suite for an LLM application?

What the interviewer is really testing: Whether you can design checks that are repeatable, layered, and useful in a deployment decision.

Model answer:

A regression suite should combine deterministic checks with evaluations that tolerate legitimate variation. I would include schema validation, tool or API contract tests, policy checks, known task cases, and representative quality evaluations. For stochastic outputs, test behavior against criteria or bounds rather than relying on a single exact string.

Keep the suite organized by risk and failure mode. A fast subset can run on every change; a more expensive evaluation can run before release or on a schedule. Record the model, prompt, dataset, and evaluation versions so a score remains interpretable later.

A regression should not automatically mean rollback: first determine whether the change is real, whether the affected cases matter, and whether the evaluation itself is stable. But a release gate should be explicit about which failures block deployment and who can approve an exception.

Realistic runnable example — a small, deterministic evaluation harness:

from collections import Counter

cases = [
    {
        "id": "policy-1",
        "slice": "grounded",
        "answer": "Refunds are available within 30 days.",
        "must_contain": ["30 days"],
    },
    {
        "id": "unknown-1",
        "slice": "no-evidence",
        "answer": "I do not have enough information to answer.",
        "must_contain": ["not have enough information"],
    },
    {
        "id": "policy-2",
        "slice": "grounded",
        "answer": "Refunds are always available.",
        "must_contain": ["30 days"],
    },
]

def passes(case: dict) -> bool:
    answer = case["answer"].casefold()
    return all(term.casefold() in answer for term in case["must_contain"])

counts = Counter()
for case in cases:
    counts[(case["slice"], passes(case))] += 1

for slice_name in sorted({case["slice"] for case in cases}):
    total = sum(counts[(slice_name, result)] for result in (True, False))
    passed = counts[(slice_name, True)]
    print(f"{slice_name}: {passed}/{total} passed")

assert counts[("grounded", False)] == 1

Notice that the harness makes slice-level failures visible; in a real suite, replace string checks with reviewed labels and task-appropriate evaluators.

Follow-up: How would you keep this suite useful as the product and its data change?
Hint: Version it, add reviewed production cases, and keep a stable set for comparisons.

Senior and system-design level

9. How would you design an evaluation system for frequent model and prompt changes?

What the interviewer is really testing: Whether you can make evaluations reproducible and useful across a changing pipeline, without turning them into a single opaque score.

Model answer:

I would model an evaluation run as a versioned experiment. Its inputs include the dataset, application configuration, prompt, model identifier, tool versions, evaluator versions, and relevant decoding settings. Its outputs include per-case results, aggregate measures, slice breakdowns, errors, and a link to the execution metadata.

Separate evaluation logic from serving logic, but run tests through the production path when the goal is end-to-end behavior. Keep deterministic unit checks fast and cheap; reserve larger or human-reviewed evaluations for changes where they add evidence. Store results so candidates can be compared on the same cases.

A run should be reproducible enough to explain what was tested, even if a hosted model can change behind an alias. Where the serving configuration permits it, use a pinned model version; otherwise record the actual available identifier and treat model drift as an external variable.

flowchart LR
    Data["Versioned evaluation cases"] --> Runner["Evaluation runner"]
    Config["Versioned app configuration"] --> Runner
    Runner --> Results["Per-case results"]
    Results --> Analysis["Metrics and slice analysis"]
    Analysis --> Gate["Release decision"]
    Gate --> Deploy["Canary or deploy"]
    Deploy --> Monitor["Production monitoring"]
    Monitor --> Data

Caption: A release gate uses versioned offline evidence and production monitoring as a feedback loop.

Follow-up: What if two teams need different quality criteria for the same shared model?
Hint: Separate shared model capabilities from application-specific acceptance criteria.

10. How would you release a change that improves quality but increases latency or cost?

What the interviewer is really testing: Whether you can reason about a multi-objective system and make trade-offs explicit.

Model answer:

First clarify which quality dimension improved and how it was measured. Then quantify latency and cost using the same workload assumptions and compare the result with user-facing constraints. An improvement is not automatically worthwhile if it breaches a hard latency target or makes the service uneconomic.

I would define the acceptable trade-off before the final experiment. For example, a team could require quality not to regress beyond a tolerance while enforcing latency and cost budgets. Any numeric limits must come from the product’s requirements and observed workload, not from a generic benchmark.

If the change remains promising, release it gradually and monitor both quality-related proxies and operational behavior. A fallback can limit exposure, but it must be tested: switching models or prompts can change answer behavior and may not preserve the same quality contract.

Follow-up: How would you compare candidates if one is cheaper but has a worse worst-case latency?
Hint: Examine the latency distribution and user impact, not only the mean and average cost.

11. How would you use human feedback to improve evaluation?

What the interviewer is really testing: Whether you can turn feedback into reliable evidence without treating every user signal as an accurate label.

Model answer:

Human feedback is valuable when it is collected with context. Reviewers need to see the user request, relevant evidence, the answer, and a clear rubric. For important decisions, use more than one reviewer or a defined adjudication process, and record why a response failed.

User feedback can help identify candidate cases, but it is often selective. People may respond more when an answer is surprising or frustrating, and positive feedback does not prove factual correctness. I would treat these signals as a source of investigation and sampling, not as unbiased ground truth.

Protect sensitive data during collection and review. Sample intentionally across task types and risk levels, and feed reviewed cases back into a versioned evaluation set. Track disagreement over time: rising disagreement may indicate ambiguous criteria, a changing task, or a genuinely difficult boundary.

Follow-up: How do you avoid training your evaluation set only on failures users happen to report?
Hint: Combine feedback-driven cases with deliberate sampling from ordinary and high-risk traffic.

12. How would you detect and diagnose a quality regression in production?

What the interviewer is really testing: Whether you can connect application behavior to observability while protecting user data and avoiding misleading proxies.

Model answer:

I would combine service health with sampled quality evidence. Operational signals can show that something changed—such as increased timeouts, retries, empty outputs, or escalation—but they do not directly establish that answers became less correct. To assess quality, sample responses for review or use validated automated checks, then segment results by relevant task, model, prompt, and retrieval configuration.

Trace the request across the application pipeline so a poor answer can be related to retrieval, model generation, parsing, or downstream actions. Use stable identifiers to correlate events, and avoid storing raw prompts or responses by default. Define access, retention, and redaction rules before enabling content capture.

The goal is to answer three questions quickly: what changed, which users or tasks are affected, and what action is safe—rollback, disable a feature, or continue investigating. Keep product-quality signals distinct from infrastructure health so a fast, successful response is not mistaken for a correct one.

Follow-up: What if quality review shows degradation but service metrics are healthy?
Hint: Treat them as separate signal families; good availability does not imply good answers.

Observability for LLM evaluation and serving

Use observability to diagnose failures, not to claim that telemetry alone measures truth. Keep request-level data minimal, control access to any captured content, and document retention. A structured log should make the pipeline and configuration identifiable without requiring the prompt itself.

Structured log example:

{
  "event": "llm.request.completed",
  "request_id": "req_01J...",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "model_id": "configured-model-version",
  "prompt_version": "support-v12",
  "retrieval_version": "index-2026-10",
  "latency_ms": 842,
  "input_tokens": 516,
  "output_tokens": 143,
  "outcome": "answered",
  "content_logged": false
}

Do not put raw prompts, completions, secrets, or unbounded user identifiers into metric labels. Use high-cardinality identifiers in logs or traces where access is controlled; keep metric dimensions bounded.

Signal groupUseful measures
RED for the serving serviceRequest rate, error rate, and duration, segmented by route or outcome
Model pipelineGeneration duration, timeout and retry counts, token usage, and parse failures
USE for resourcesCPU and memory utilization, saturation, and errors for the compute resources hosting the service
QualityReviewed pass rate, abstention behavior, and task-specific failures by stable slice

For distributed tracing, create spans around the request and meaningful pipeline stages. The example uses the OpenTelemetry Python API and SDK; install opentelemetry-api and opentelemetry-sdk to run it. Production deployments should configure an exporter and resource attributes for their telemetry backend.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import (
    SimpleSpanProcessor,
    ConsoleSpanExporter,
)

provider = TracerProvider()
provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("llm-serving")

with tracer.start_as_current_span("llm.request") as request_span:
    request_span.set_attribute("llm.model_id", "configured-model-version")
    request_span.set_attribute("llm.prompt_version", "support-v12")

    with tracer.start_as_current_span("retrieval") as retrieval_span:
        retrieval_span.set_attribute("retrieval.result_count", 4)

    with tracer.start_as_current_span("generation") as generation_span:
        generation_span.set_attribute("llm.outcome", "answered")

Notice that span attributes describe the request path without recording user content.

SLIExample SLO
Successful request availabilityAt least 99.9% over a rolling 30-day window; assumption, choose from product needs
Request latencyA defined percentile stays within the product’s latency budget; set the percentile and target from observed traffic
Output contract validityAt least 99% of responses pass schema or format checks; assumption, validate against application risk
Reviewed qualityA task-specific pass rate stays above an agreed threshold for the sampled evaluation set

Page on sustained service unavailability, elevated timeouts or errors, resource saturation likely to affect the SLO, and a confirmed high-impact quality or safety regression. Do not page on every score fluctuation from a small sample: define minimum sample sizes, review paths, and escalation thresholds appropriate to the risk.

Common mistakes candidates make

  • Treating one score as quality. An aggregate can hide a harmful regression in a small but important task segment.
  • Using exact match for open-ended answers. It confuses wording variation with correctness.
  • Trusting an LLM judge without validation. A judge is another model component with its own failure modes.
  • Confusing proxy signals with ground truth. Low latency and positive feedback do not establish factual correctness.
  • Changing the test set without tracking versions. A score is not comparable if the evaluation cases changed silently.
  • Ignoring the serving pipeline. A model-only test may miss failures caused by retrieval, parsing, tools, or application logic.
  • Shipping on an average alone. Report uncertainty and inspect important slices before making a release decision.
  • Logging content by default. Prompts and responses can contain sensitive information; define privacy controls first.
  • Failing to define a rollback decision. A release gate is incomplete if nobody knows what result blocks deployment or what happens next.

Quick-revision checklist

  • Define the task and the high-impact failure modes.
  • Build a versioned, representative dataset with edge cases.
  • Choose metrics that match the output contract and user risk.
  • Validate human rubrics and model judges against reviewed examples.
  • Compare candidates on the same cases and report uncertainty.
  • Evaluate components and end-to-end behavior separately.
  • Keep deterministic checks fast and expensive evaluations targeted.
  • Record model, prompt, dataset, evaluator, and configuration versions.
  • Set explicit quality, latency, and cost release criteria.
  • Use online evidence to complement—not replace—offline evaluation.
  • Instrument requests with privacy-aware logs, metrics, and traces.
  • Define who investigates regressions, who can approve exceptions, and how to roll back.

This article was generated with AI. #AIGenerated