Skip to content

AI Agents Interview Questions: Durable Execution, Checkpoints, and Replay

Twelve interview questions on making long-running AI agents recoverable, resumable, and observable without confusing saved conversation history with durable execution.

Ekanatha Reddy Urivakili14 min2,930 words

Fundamentals

1. What does durable execution mean for an AI agent?

What the interviewer is really testing: Whether you distinguish recovering a workflow from merely preserving its conversation.

Model answer: Durable execution means an agent run can make progress, stop, and later resume from persisted state without silently repeating completed work. The state must be sufficient to decide what happens next—not just reconstruct what the user and model said.

Conversation history is often one input to a run. A durable workflow also needs execution identity, completed-step results, pending work, deadlines, and any human approval state. If a process restarts, the system should recover the run’s status and continue from a known boundary.

Durability has limits. A saved checkpoint cannot undo an external side effect that happened before the checkpoint was committed. For side effects, combine durable progress with idempotency, reconciliation, or a compensating action.

Follow-up: What would you persist that would not appear in the chat transcript?
Hint: Think about step status, retries, deadlines, and authorization decisions.

2. What belongs in an agent checkpoint?

What the interviewer is really testing: Whether you can define a useful, bounded state contract instead of persisting arbitrary runtime memory.

Model answer: Persist the minimum data required to resume correctly:

  • A run ID, tenant or owner scope, workflow version, and current status.
  • Inputs or references to inputs, subject to retention and privacy rules.
  • Completed-step outputs, pending work, retry metadata, and the next transition.
  • Approval requests, deadlines, cancellation state, and a version or sequence number.
  • References to external resources, rather than open connections or in-memory objects.

Avoid blindly storing every prompt, model response, and intermediate token. That can increase cost, expose sensitive data, and make schemas difficult to evolve. Store enough provenance to explain decisions, but define retention and redaction deliberately.

A checkpoint should have an explicit schema. Persisting a serialized Python object or framework-specific memory structure as an opaque blob can make future migrations and incident inspection unnecessarily hard.

Follow-up: How would you handle a checkpoint too large for your primary database row?
Hint: Separate the indexed run record from versioned, access-controlled payload storage.

3. How is replay different from resuming from a checkpoint?

What the interviewer is really testing: Whether you understand the difference between re-running history and continuing from persisted progress.

Model answer: Resuming starts from a persisted checkpoint and executes the remaining work. Replay re-executes some or all of the recorded history, typically to debug a run, rebuild derived state, or verify a change.

Replay is difficult when inputs are nondeterministic. Model outputs can vary; tool responses can change; clocks, randomness, and external data can differ. A replayable system records the relevant inputs and outputs—such as the model request, model response, tool arguments, and tool results—or explicitly accepts that replay is an approximation.

Do not treat a replay as permission to repeat side effects. A replay mode should mock, suppress, or safely redirect writes. For production recovery, prefer resuming from committed results over asking the model and tools to reproduce them.

Follow-up: Which values would you capture to make a model step auditable?
Hint: Record the model configuration and request inputs as well as the returned output.

4. Why separate workflow state from conversational context?

What the interviewer is really testing: Whether you can keep user-facing memory distinct from execution correctness.

Model answer: Conversational context helps the agent interpret the user’s intent. Workflow state tells the runtime what has happened and what it must do next. They have different retention, access, size, and correctness requirements.

For example, a conversation summary might say that a user agreed to an action. The workflow should not use that summary alone as proof of an approval. It should refer to a durable approval record with an identity, timestamp, scope, and status.

Keep the boundary explicit: the agent may propose a transition, but the runtime validates and records it. This prevents a truncated context window or regenerated summary from changing the underlying workflow’s status.

Follow-up: What is one workflow fact you would never infer from a summary?
Hint: Choose a fact whose incorrect value could trigger a consequential action.

Intermediate

5. How should retries work when an agent step may have side effects?

What the interviewer is really testing: Whether you can explain why retries are not equivalent to exactly-once execution.

Model answer: A timeout does not prove that the operation failed. The remote system may have committed the request while the agent worker lost the response. Retrying with a new operation identity can therefore duplicate the side effect.

Give each logical operation a stable idempotency key, derived from the run and step identity. Persist the result when possible, and use the downstream service’s idempotency support when the effect is remote. If the downstream system has no such support, record enough information to reconcile uncertain outcomes before retrying.

For writes to the same database as the workflow state, put the domain change and completed-step record in one transaction. For a remote effect, use an outbox or a durable command record and design for duplicate delivery; a local database transaction cannot atomically commit a remote API call.

Here is a minimal example that saves step results before a later invocation can repeat the work:

import json
from pathlib import Path

state_path = Path("agent-run.json")

def load_state():
    return json.loads(state_path.read_text()) if state_path.exists() else {}

def save_state(state):
    temp_path = state_path.with_suffix(".tmp")
    temp_path.write_text(json.dumps(state))
    temp_path.replace(state_path)

def run_step(state, name, action):
    if name in state:
        return state[name]
    result = action()
    state[name] = result
    save_state(state)
    return result

state = load_state()
customer = run_step(
    state,
    "lookup-customer",
    lambda: {"customer_id": "c-17"},
)
print(customer)

Run it with python minimal_checkpoint.py twice. Notice that the second run reuses the stored result; this is a local demonstration, not a multi-worker or side-effect-safe persistence design.

Follow-up: What if the process crashes after the action succeeds but before save_state completes?
Hint: The checkpoint alone cannot determine whether an external action committed.

6. How would you pause an agent for human approval and then resume it?

What the interviewer is really testing: Whether you model approval as durable workflow state rather than a conversational interruption.

Model answer: Persist an approval request as a workflow transition. It should identify the run, the proposed action, the relevant context, the authorized approver or policy, and an expiry time. The run moves into a waiting state; it should not hold a worker thread or depend on an in-memory callback remaining alive.

When a reviewer responds, validate that the response matches the pending request and is still authorized. Record the decision as an immutable event or durable state transition, then enqueue a resume signal. The resumed worker reads the authoritative decision and continues from the checkpoint.

Use a correlation ID to connect the approval request to the resumed run. Handle duplicate responses idempotently, and define what happens on expiry, rejection, or cancellation. Approval should be scoped to the action shown to the reviewer—not an open-ended grant to perform new actions.

Follow-up: How would you avoid resuming twice if the approval notification is delivered more than once?
Hint: Make the approval transition conditional on the current pending state.

7. How do you prevent two workers from advancing the same run concurrently?

What the interviewer is really testing: Whether you understand worker ownership, optimistic concurrency, and stale workers.

Model answer: Use a durable claim mechanism, such as a lease with an expiry, or an atomic compare-and-swap on the run’s version. A worker claims a run, loads its version, and commits progress only if it still owns a valid claim and the version has not changed.

A lease by itself is not enough: a paused worker may resume after its lease expires while another worker is already processing the run. Use a monotonically increasing fencing token or version number. Downstream writes should reject progress from an older token where possible.

Keep claims short-lived and renewable. If a worker dies, the run becomes eligible for reclaim after the lease expires. Record claim changes so that duplicate processing and ownership bugs can be diagnosed.

Follow-up: What can go wrong if a lease expires while the original worker is paused by a long model call?
Hint: Consider what happens when both the original and replacement worker resume.

8. How do you evolve persisted agent state when the workflow code changes?

What the interviewer is really testing: Whether you plan for in-flight runs to outlive a deployment.

Model answer: Persist a workflow or state-schema version and define how the running service handles each supported version. Possible strategies include migrating state on read, supporting old versions until their runs drain, or routing an old run to a compatible executor.

Keep migrations explicit and test them against representative checkpoints. A new deployment should not assume every stored run was created by the current code. Renaming a state field or changing the meaning of a step can otherwise make resumptions fail or, worse, continue incorrectly.

For replay and audit, distinguish the version needed to interpret the stored state from the version used to execute new runs. Do not overwrite historical evidence just to match the latest schema.

Follow-up: When would you keep an old workflow version instead of migrating its state?
Hint: Consider migrations whose meaning is ambiguous or whose execution semantics changed.

Senior and system-design level

9. Design a durable runtime for long-running agent workflows.

What the interviewer is really testing: Whether you can connect state ownership, queues, workers, and recovery into one coherent design.

Model answer: Separate the control plane from execution. An API creates a run record and an initial durable task. Workers claim tasks, execute bounded steps, and commit results and the next task. A scheduler or queue makes eligible work discoverable; the database remains the source of truth for run state.

A useful outline is:

flowchart LR
    api["API"] --> store["Run store"]
    store --> queue["Task queue"]
    queue --> worker["Agent worker"]
    worker --> model["Model provider"]
    worker --> tools["Tool services"]
    worker --> store
    reviewer["Human reviewer"] --> store
    store --> queue

The store owns durable state; the queue wakes workers to process eligible transitions.

Workers should execute bounded units of work rather than keep a process alive for the entire run. Save progress between steps, use leases or version checks for ownership, and make every transition recoverable after a crash. A queue message is a delivery mechanism, not proof that a step completed.

Follow-up: What should happen if the database commits a next-step task but publishing its queue message fails?
Hint: Make state change and message publication recoverable through an outbox or equivalent mechanism.

10. Can you guarantee exactly-once execution of an agent step?

What the interviewer is really testing: Whether you separate delivery guarantees from the business effect.

Model answer: Across a database, queue, model provider, and external tools, exactly-once execution is generally not a realistic end-to-end guarantee. A worker can crash after a remote system commits an effect but before the worker records success.

Instead, design for at-least-once delivery and make effects idempotent where possible. Use a stable operation key, record state transitions atomically with local effects, and reconcile uncertain remote outcomes. If an effect is not repeatable, provide a queryable operation status or a manual recovery path.

Be precise about the boundary of any guarantee. You may be able to guarantee that one local database transition is committed once, but that does not imply one model call or one remote API effect.

Follow-up: How would you handle an external service that supports neither idempotency keys nor status lookup?
Hint: Describe the remaining uncertainty and the operational recovery path honestly.

11. How should an agent runtime handle cancellation, timeouts, and resource budgets?

What the interviewer is really testing: Whether you treat execution limits as part of workflow semantics rather than a worker-only concern.

Model answer: Give each run an overall deadline and each step a smaller timeout. Persist cancellation requests and check them at safe boundaries; cancellation cannot always interrupt an already-committed external effect. Propagate deadlines to model and tool calls where supported.

Define limits for model calls, tool calls, retries, and total elapsed time. A run that reaches a limit should enter a clear terminal or waiting state with a reason, not loop indefinitely. Retrying should respect the remaining budget, use bounded backoff, and avoid multiplying load during an outage.

Distinguish “cancel requested” from “cancelled.” If an operation is in flight, the runtime may need to wait, reconcile, or surface an uncertain result before it can truthfully mark the run cancelled.

Follow-up: What status should a run have when cancellation arrives after an external write was sent but before its response?
Hint: Avoid claiming success or safe cancellation until the outcome is known.

12. What would you measure to know whether a durable agent runtime is healthy?

What the interviewer is really testing: Whether you can observe service behavior, queueing, and execution correctness together.

Model answer: Measure request behavior at the service boundary, resource saturation at the worker and storage boundaries, and workflow outcomes across the full run. A healthy API can still hide a growing queue or a population of runs stuck waiting for resume.

Use traces to connect the API request, run transition, model call, tool call, and persistence operation. Put run and step identifiers in traces and structured logs, but do not use high-cardinality identifiers as metric labels. Avoid logging prompts, secrets, or sensitive tool payloads by default.

For a Python worker, create spans around the run and meaningful steps. Configure an SDK provider and exporter at application startup; the snippet below shows instrumentation, not exporter setup.

from opentelemetry import trace

tracer = trace.get_tracer("agent-runtime")

def execute_run(run_id: str, workflow_name: str):
    with tracer.start_as_current_span("agent.run") as span:
        span.set_attribute("agent.run.id", run_id)
        span.set_attribute("agent.workflow.name", workflow_name)

        with tracer.start_as_current_span("agent.step"):
            return run_next_durable_step(run_id)

Notice that trace context should cross queue and worker boundaries through propagation; creating spans alone does not connect separate processes.

Follow-up: Which run identifiers belong in traces but not metric labels?
Hint: Consider cardinality and the difference between investigation and aggregation.

Choosing a persistence and execution approach

These options solve related problems but make different trade-offs. The right choice depends on how long runs live, how much recovery logic you want to own, and what guarantees your side effects require.

ApproachBest fitMain trade-off
In-process loop with database checkpointsShort workflows with a small set of explicit pause pointsThe application owns retries, claiming, and recovery behavior
Queue plus worker and run storeWork that can be divided into bounded, independently retried stepsQueue delivery and stored workflow state must be reconciled
Durable workflow engineLong-running workflows with timers, signals, retries, and operational visibilityAdds a platform and its execution model to learn and operate
Event-sourced workflow stateStrong audit and rebuild requirementsMore modeling and projection complexity; events do not make external effects atomic

A queue is not a workflow engine: it delivers work but does not by itself define the state machine, timers, or recovery policy. Event sourcing is not automatically the right answer either. A conventional run table plus append-only transition records can provide adequate auditability with less machinery.

Common mistakes candidates make

  • Calling saved chat history “durability.” A transcript does not establish which steps committed or what to do next.
  • Promising exactly-once effects. Explain the failure window and the idempotency or reconciliation strategy instead.
  • Retrying after every timeout. The remote operation may have succeeded even when its response was lost.
  • Treating the queue as the source of truth. Messages can be duplicated, delayed, or lost relative to a database transaction.
  • Persisting opaque framework state. Durable state needs a schema, versioning strategy, and migration tests.
  • Holding a worker during human approval. Persist a waiting state and resume from a durable signal.
  • Ignoring stale workers. A lease should be paired with a version check or fencing strategy.
  • Using model output as an authoritative transition. Validate and commit transitions in application code.
  • Logging prompts and tool payloads indiscriminately. Observability must respect privacy and data-retention boundaries.

Observability and operational readiness

Instrument the runtime around the lifecycle of a run, not only around HTTP requests. For metrics, use RED for the service—rate, errors, and duration—and USE for constrained resources—utilization, saturation, and errors.

SignalUseful measurements
RED: API and worker operationsRun-start rate, failed-transition rate, step duration, and resume request latency
USE: workers and storageWorker utilization, queue or worker saturation, database connection utilization, and storage errors
Workflow healthOldest queued task, runs by status and age, retry count, expired leases, and approval wait duration

Log structured events for state transitions and recovery decisions. Keep identifiers useful for investigation while excluding secrets and sensitive content.

{
  "event": "agent.step.completed",
  "run_id": "run-42",
  "step_id": "create-shipment",
  "workflow_version": 3,
  "attempt": 1,
  "duration_ms": 184,
  "outcome": "success"
}

Example initial objectives below are proposed service targets, not universal guarantees. Set them from product requirements and observed behavior.

SLIExample SLO
Run acceptanceAt least 99.9% of valid run-start requests are durably accepted over the measurement window
Worker transition successAt least 99% of eligible transitions complete or reach a declared recoverable state within the defined service window
RecoveryRuns with expired worker ownership are detected and made eligible for recovery within the operational recovery objective

Page on conditions that require timely human action: a sustained failure to claim or commit work, rapidly growing queue age that threatens the recovery objective, repeated persistence errors, or a rise in runs stuck in an unexpected state. Alert on user-impacting symptoms and actionable failure modes; use dashboards or tickets for isolated slow runs and routine retry noise.

Quick-revision checklist

  • Explain why workflow state is more than conversation history.
  • Name the identifiers and version fields a checkpoint needs.
  • Distinguish resume from replay, and explain nondeterminism.
  • Describe the ambiguous outcome after a timed-out side effect.
  • Use stable idempotency keys and define a reconciliation path.
  • Model human approval as durable, scoped state.
  • Address duplicate workers with leases and stale-owner protection.
  • Explain how in-flight runs survive workflow code changes.
  • State the limits of exactly-once claims.
  • Connect run-level traces, structured logs, RED/USE metrics, and actionable alerts.

For implementation details, see the OpenTelemetry Python documentation and the Python sqlite3 documentation.


This article was generated with AI. #AIGenerated