The decision in one sentence
Start with prompting. Add retrieval when answers must reflect changing or attributable information. Fine-tune when measured failures are about consistent behavior that examples can teach—not facts that need to be looked up. Combine them only when evaluations show that each layer earns its operational cost.
That rule is deliberately conservative. Fine-tuning is compelling because it feels like teaching a model your domain. Retrieval is compelling because it seems to make answers current. Prompting is compelling because it is fast to change. Each can fail when applied to the wrong problem.
The useful question is not “Which technique is best?” It is: Which source of uncertainty is causing the quality failure, and what is the cheapest reliable intervention?
What the options do
Prompting supplies instructions, constraints, examples, and sometimes conversation context at inference time. The model’s weights do not change. It is the simplest way to specify a task, output format, and policy.
Retrieval-augmented generation (RAG) finds relevant external content and includes it in the model’s input. The knowledge remains outside the model—in documents, a database, or another authorized source—so it can be updated independently. Retrieval quality, access control, and source selection become part of the answer path.
Fine-tuning updates model weights using training examples. It can make a model more likely to follow a style, format, classification scheme, or repeated task pattern. It does not provide a dependable lookup mechanism for an evolving body of facts. Treat learned facts as model behavior, not as an authoritative data store.
These choices are not mutually exclusive. A production request may use a system prompt to define behavior, RAG to provide current evidence, and a fine-tuned model to produce a consistent domain-specific output. That combination is justified only if each part improves a measured outcome.
Why the choice matters
These techniques move cost and complexity to different parts of a system:
- Prompting puts more responsibility on the request: longer instructions consume context, and the model may still interpret them inconsistently.
- RAG adds a data ingestion and retrieval path. It can improve freshness and traceability, but introduces indexing, permissions, retrieval failures, and added latency.
- Fine-tuning adds dataset curation, training, model evaluation, and model-version management. It may shorten prompts or improve behavior, but creates another artifact to release and monitor.
Quality is similarly multidimensional. A concise answer can be wrong; a correct answer can lack evidence; a consistently formatted answer can still violate a policy. Evaluate the property you need rather than treating “quality” as one score.
A practical evaluation set should distinguish at least:
| Quality dimension | Example question |
|---|---|
| Task success | Did the response complete the requested task? |
| Factual correctness | Are its factual claims supported by ground truth? |
| Grounding | Are claims supported by the supplied sources? |
| Citation quality | Do citations point to relevant, valid evidence? |
| Instruction following | Did it honor format, tone, and constraints? |
| Safety and access | Did it avoid disallowed content and unauthorized data? |
| Operational quality | Was latency and cost within the request budget? |
How the mechanisms fit together
A request first needs a task and data classification. That determines whether the answer can be generated from instructions alone, needs current authorized evidence, or benefits from a behavior adapted through training. In a hybrid design, retrieval and prompting still operate at inference time even when the underlying model is fine-tuned.
flowchart TD
A["User request"] --> B["Classify task and data needs"]
B --> C["Build prompt and policy"]
B --> D["Retrieve authorized evidence"]
B --> E["Select deployed model"]
C --> F["Assemble model input"]
D --> F
E --> G["Generate response"]
F --> G
G --> H["Validate and return"]
I["Curated examples"] --> J["Fine-tuning and evaluation"]
J --> E
K["Documents and permissions"] --> L["Ingestion and indexing"]
L --> DCaption: Prompting and retrieval shape an inference request; fine-tuning changes the model artifact selected for that request.
The diagram separates two timelines. Document ingestion and model training happen before a user request. At inference time, the system assembles the prompt, retrieves evidence if needed, chooses a model version, and validates the result. A fine-tuned model does not eliminate the need for runtime retrieval when knowledge changes or must be cited.
A decision framework
1. Establish a prompted baseline
Use a capable general model, a concise instruction, a defined output contract, and representative examples where they clarify the task. Make a fixed evaluation set before adding infrastructure. Include common cases, edge cases, ambiguous requests, and known failure modes.
The baseline tells you whether a more elaborate approach is necessary. If a prompt change gets the task within quality and cost targets, stop there.
2. Diagnose the failure
Look at failed examples and classify their cause:
- Missing or changing information: retrieve it from a maintained source.
- Wrong or inconsistent output structure: improve the prompt or use a constrained output mechanism. If repeated examples still show a stable behavior gap, test fine-tuning.
- Weak task execution despite adequate context: compare prompt improvements with a fine-tuned candidate on the same evaluation set.
- Conflicting, stale, or inaccessible evidence: fix source quality, permissions, and retrieval before changing the model.
- Unclear request or insufficient evidence: ask a clarifying question or abstain rather than fabricating an answer.
This diagnosis is more valuable than starting with a technology preference. Fine-tuning cannot reliably repair a broken retrieval index; adding RAG cannot fix a task whose problem is inconsistent output style.
3. Choose by the required property
| Requirement | Start with | Consider next | Why |
|---|---|---|---|
| Stable task instructions or response format | Prompting | Fine-tuning if measured behavior still fails | Instructions are easy to change and review. |
| Current, searchable, or source-cited facts | RAG | Improve ranking, chunking, or source governance | The source of truth remains external and updateable. |
| Consistent task behavior across many examples | Prompting | Fine-tuning after controlled evaluation | Training can teach patterns that are awkward to repeat in every prompt. |
| Per-user or per-tenant authorization | Retrieval with enforced access controls | None as a substitute | Model weights are not an authorization boundary. |
| A small, stable, private dataset with no lookup requirement | Prompting or fine-tuning | Evaluate the operational burden | Training may be viable if examples are representative and licensing permits it. |
| Reliable answers that need both current evidence and a house style | RAG plus prompting | Fine-tune only for a proven behavior gap | Each component addresses a different failure class. |
4. Compare candidates on the same workload
A fair comparison uses the same request set, access rules, output schema, and success criteria. Compare prompt-only, RAG, and fine-tuned candidates only where each is applicable. Report per-case failures as well as aggregate results: a strong average can hide a serious regression on a sensitive category.
Include operational conditions. Evaluate retrieval with realistic document updates and permission boundaries. Evaluate fine-tuning on a held-out set that is separate from its training examples. Test prompt changes against old cases to catch regressions. Keep model, prompt, index, and evaluation-set versions identifiable.
5. Adopt the smallest system that meets the target
Do not choose a hybrid because it sounds more capable. A hybrid has more failure modes and release dependencies. If a prompt solves the task, fine-tuning is extra model lifecycle work. If retrieval supplies the required evidence, training facts into weights makes freshness and provenance harder.
Cost: calculate the whole path
A useful first-order inference estimate is:
monthly model cost =
requests after cache hits
× (input tokens × input price + output tokens × output price)
That is only one part of total cost. For RAG, add embedding generation, vector or database capacity, ingestion, reranking if used, and the extra input tokens containing retrieved material. For fine-tuning, include training jobs, data preparation and review, evaluation, artifact storage, and any serving premium or capacity commitment. These depend on the provider and deployment shape; compare actual quotes and usage rather than assuming training is cheaper.
Here is an illustrative calculation, not a vendor price or forecast. Assume 60,000 requests per day, 30 days per month, a 70% cache-hit rate for eligible requests, and hypothetical model prices of $0.30 per million input tokens and $1.20 per million output tokens. Assume 1,200 input tokens and 250 output tokens per uncached request.
That leaves 540,000 model calls per month. Under those assumptions, prompt-only model usage is approximately:
- Input: 648 million tokens × $0.30 per million = $194.40.
- Output: 135 million tokens × $1.20 per million = $162.00.
- Total: $356.40, excluding infrastructure and provider fees not represented by those assumptions.
If RAG adds 900 input tokens per call, the same hypothetical input rate adds about $145.80 per month. That does not mean RAG costs only $145.80 more: indexing and retrieval also consume resources. Nor does it mean RAG is uneconomic. Its value may be current answers, access control, or citations that prompting alone cannot supply.
Fine-tuning comparisons need the same discipline. A shorter prompt or cheaper model may reduce inference cost, but the saving must exceed training, evaluation, serving, and maintenance costs. Estimate break-even from measured per-request savings and expected traffic; do not infer it from a model’s training capability.
Minimal example: begin with prompting
This Python example makes a short prompt-only request through a local mock model. Replace mock_model with a provider adapter to exercise a real model; keep the prompt contract and evaluation separate from that adapter.
def mock_model(prompt: str) -> str:
# Replace with a provider call. The mock makes this example runnable.
return "Use the approved process and record the result."
def answer(question: str) -> str:
prompt = f"""You are an operations assistant.
Answer in one sentence. If the question needs current company policy,
say that you need an approved source rather than guessing.
Question: {question}
Answer:"""
return mock_model(prompt)
if __name__ == "__main__":
print(answer("How should I handle this routine request?"))
What to notice: the baseline defines behavior but does not pretend to contain current company policy.
Realistic production example: support answers with authorized evidence
Assume a SaaS product with 10,000–100,000 daily users, one or two engineering teams, and a modular monolith or a few services. A support-assistant endpoint runs in the application tier. It checks tenant and user authorization, retrieves relevant articles from a multi-AZ database-backed index, calls a model through a gateway, and returns citations. A queue handles document ingestion so a source update does not block a user request. CI deploys prompt and retrieval changes to staging before promotion.
The following runnable TypeScript example demonstrates the request boundary and a retrieval-backed answer path using in-memory adapters. It is not a production retrieval implementation: replace the mock index and model with your database and model-provider adapters, and enforce authorization within the retrieval query itself.
type Article = {
id: string;
tenantId: string;
title: string;
text: string;
};
type Evidence = Pick<Article, "id" | "title" | "text">;
const articles: Article[] = [
{
id: "refund-policy",
tenantId: "tenant-a",
title: "Refund policy",
text: "Eligible refunds are requested through the billing page.",
},
];
async function retrieve(
tenantId: string,
question: string,
): Promise<Evidence[]> {
const terms = question.toLowerCase().split(/\W+/).filter(Boolean);
return articles
.filter((article) => article.tenantId === tenantId)
.filter((article) =>
terms.some((term) => article.text.toLowerCase().includes(term)),
)
.slice(0, 3)
.map(({ id, title, text }) => ({ id, title, text }));
}
async function generate(prompt: string): Promise<string> {
// Replace with a provider call, including timeout and token limits.
const match = prompt.match(/Question: (.*)/);
return match?.[1]
? "The source says to request eligible refunds through the billing page. [refund-policy]"
: "I do not have enough information to answer.";
}
async function answer(
tenantId: string,
question: string,
): Promise<{ text: string; sources: string[] }> {
const evidence = await retrieve(tenantId, question);
if (evidence.length === 0) {
return { text: "I could not find an approved source for that.", sources: [] };
}
const prompt = [
"Answer using only the evidence below.",
"If it does not answer the question, say so.",
"Cite evidence using its ID in square brackets.",
`Evidence: ${JSON.stringify(evidence)}`,
`Question: ${question}`,
].join("\n");
const text = await generate(prompt);
return { text, sources: evidence.map((item) => item.id) };
}
answer("tenant-a", "How do I request a refund?")
.then((result) => console.log(JSON.stringify(result, null, 2)))
.catch((error: unknown) => {
console.error("answer_failed", error);
process.exitCode = 1;
});
What to notice: tenant filtering happens before evidence enters the prompt, while the prompt explicitly limits answers to retrieved evidence.
At production scale, keep the components modular rather than creating a distributed system by default. A request handler can call a retrieval module and a model gateway inside the same deployable service. Store document metadata, tenant ownership, and index state in the database; use a queue for idempotent ingestion and retries. Cache only results whose authorization and freshness semantics are explicit. A cache key should include relevant tenant or user scope and versions of the prompt, model, and retrieval configuration.
A model call should have a timeout and a token budget. If retrieval fails, choose a documented fallback: return an unavailable response, answer without retrieval only for a task known to be safe without it, or ask the user to retry. Do not silently drop evidence and keep presenting a grounded-answer guarantee.
Common pitfalls
- Treating a fine-tune as a knowledge base. Facts can be inconsistently recalled, difficult to update, and hard to attribute. Retrieve authoritative facts instead.
- Adding RAG before validating the corpus. Duplicate, stale, poorly segmented, or inaccessible documents can make answers worse. Fix source hygiene and permissions first.
- Optimizing only a judge score. Automated judges can miss unsupported claims, invalid citations, or access-control violations. Use human review for consequential cases and explicit checks for machine-verifiable properties.
- Training on examples that leak into evaluation. Keep training, validation, and held-out evaluation data distinct. Test for memorization and data rights before training.
- Comparing different workloads. Different prompts, token limits, or retrieval contexts make cost and quality comparisons misleading.
- Caching personalized output globally. A response grounded in one tenant’s evidence must never be served to another tenant.
- Assuming more context means more grounding. Irrelevant retrieved text consumes tokens and can distract the model. Measure retrieval relevance and answer support.
- Shipping prompt changes without regression tests. Prompt edits are code changes in all but name; version and test them accordingly.
When not to use each approach
Do not fine-tune when the primary need is fresh facts, citations, per-request access control, or a fast-changing policy set. Avoid it when you cannot obtain representative, permitted training examples or cannot maintain an evaluation and release process.
Do not use RAG when there is no trusted external corpus, the task does not require external information, or the retrieval system cannot enforce the required authorization. For a small fixed instruction set, retrieval may add more moving parts than value.
Do not rely on prompting alone when correctness depends on current private data, tenant-specific evidence, or verifiable citations. A prompt can request careful behavior; it cannot make missing evidence available or enforce authorization.
Observability and release safety
Treat the model call, retrieval, and any validation as observable parts of one request. Emit structured events with request and configuration identifiers, but do not log raw prompts, retrieved documents, or responses by default: they may contain personal, tenant, or confidential data.
A structured log event might look like this:
{
"event": "llm_answer_completed",
"request_id": "req-7f21",
"tenant_id_hash": "hmac:2c9a",
"model_version": "support-model-v3",
"prompt_version": "support-prompt-12",
"retrieval_version": "index-8",
"retrieval_count": 3,
"input_tokens": 1840,
"output_tokens": 196,
"cache_hit": false,
"outcome": "grounded",
"duration_ms": 842
}
Use a keyed hash or other approved privacy-preserving identifier where correlation is needed; do not expose raw tenant IDs in broadly accessible telemetry. Token and duration fields should come from provider or application instrumentation, not guessed values.
Metrics
Use RED for the request-serving service and USE for constrained resources.
| Lens | Metrics to track |
|---|---|
| RED: Rate | Requests per second by outcome and route; model calls per request |
| RED: Errors | Error and timeout rates for retrieval, model calls, and validation |
| RED: Duration | End-to-end and dependency latency, including p50, p95, and p99 |
| USE: Utilization | Database connections, worker concurrency, queue consumers, and cache capacity |
| USE: Saturation | Queue age and depth, connection-pool wait, rate-limit headroom |
| USE: Errors | Database, cache, queue, and worker errors |
Also track request tokens and estimated cost by model, tenant scope, and feature where policy permits. Track retrieval hit rate, citation validity, abstention rate, and offline evaluation results. Do not collapse these into a single “AI quality” dashboard.
Distributed tracing
In a Node.js service using OpenTelemetry, instrument the request and dependency boundaries with spans. Keep high-cardinality identifiers out of metric labels; trace attributes are useful for sampled diagnosis, but should still follow your data-handling policy.
import { trace, SpanStatusCode } from "@opentelemetry/api";
const tracer = trace.getTracer("support-assistant");
async function tracedAnswer(question: string, tenantId: string) {
return tracer.startActiveSpan("llm.answer", async (span) => {
span.setAttribute("llm.prompt_version", "support-prompt-12");
try {
const evidence = await tracer.startActiveSpan(
"retrieval.search",
async (retrievalSpan) => {
try {
return await retrieve(tenantId, question);
} finally {
retrievalSpan.end();
}
},
);
span.setAttribute("retrieval.result_count", evidence.length);
// Call the model gateway here; give that dependency its own child span.
return evidence;
} catch (error) {
span.recordException(error as Error);
span.setStatus({ code: SpanStatusCode.ERROR });
throw error;
} finally {
span.end();
}
});
}
SLIs, SLOs, and paging
Set targets from product requirements and observed baselines. The values below are proposed starting targets for this example architecture, not universal guarantees or benchmark results. Define the measurement window and exclusions with the product owner before treating them as commitments.
| SLI | Example starting SLO | Measurement |
|---|---|---|
| Request availability | 99.9% successful eligible requests over 30 days | Successful responses divided by eligible requests |
| Latency | 95% of requests complete within 3 seconds | End-to-end request duration |
| Grounded-answer behavior | At least 98% of evaluated answers cite a valid retrieved source when one is required | Versioned offline set plus sampled review |
| Tenant isolation | Zero confirmed cross-tenant evidence disclosures | Security tests and incident tracking |
Page on a sustained error-budget burn, a material rise in model or retrieval timeouts, queue age that threatens ingestion freshness, database saturation, or a confirmed tenant-isolation failure. Alert on spend thresholds through an owner-visible budget alert; page only when spend threatens a production limit or indicates a likely runaway loop. A single slow request or a small offline-evaluation dip should usually create a ticket or release block, not wake an on-call engineer.
Checklist
Before choosing or shipping an approach, confirm:
- The failure is classified as missing knowledge, behavior, policy, or infrastructure—not guessed from anecdotes.
- A prompted baseline and representative evaluation set exist.
- Retrieval is required for freshness, attribution, or authorized external evidence.
- Fine-tuning examples are representative, permitted, and separate from held-out evaluation data.
- Cost estimates include uncached inference, retrieval, ingestion, training, and operational work.
- Tenant authorization is enforced before evidence reaches the model; caches preserve the same boundary.
- Prompts, model artifacts, indexes, and evaluation sets are versioned and promoted through staging.
- Fallbacks, abstention behavior, timeouts, token budgets, and retries are explicit.
- Logs, traces, metrics, and alerts support diagnosis without unnecessarily exposing sensitive content.
- Every added component has a measured reason to exist.
Start simple, keep the evaluation honest, and invest in the technique that addresses the demonstrated failure. The best choice is the smallest system that meets the quality, cost, and trust requirements—not the one with the most moving parts.
This article was generated with AI. #AIGenerated