When building an AI application with access to domain-specific knowledge, there are three main approaches: prompting (include the information in the context window), RAG (Retrieval-Augmented Generation: retrieve relevant information at query time), or fine-tuning (train the model on the domain-specific data). Here is when each approach is appropriate.
The Approaches Explained
Prompting: the simplest approach. Include domain-specific information directly in the system prompt or user message context window. Best for: small knowledge bases that fit in the context window, one-off tasks, prototyping, and cases where the information changes frequently. Limitation: context windows, though now large (100K–1M+ tokens), are not infinite; costs increase linearly with context; the model retrieves from context less reliably when context is very long. RAG (Retrieval-Augmented Generation): a retrieval system (typically vector search over embeddings) finds the most relevant chunks from a large knowledge base and includes them in the prompt at query time. Best for: large knowledge bases (documentation, support articles, legal documents), questions where only a portion of the knowledge base is relevant to any given query, and cases where the knowledge base is updated frequently. How it works: documents are chunked and embedded (converted to vector representations); at query time, the query is embedded and the most similar document chunks are retrieved; those chunks are included in the prompt. The limitation: RAG quality depends heavily on retrieval quality — if the retrieval step does not find the right documents, the model cannot answer correctly. Fine-tuning: updating the model’s weights on domain-specific data so the knowledge is “baked in” to the model. Best for: changing the model’s style or tone, teaching specific structured output formats, improving performance on a very specific task type, or injecting stable factual knowledge that is unlikely to change.
The Decision Framework
Start with prompting: for any new application, start with a well-crafted prompt and include the relevant context. If that works, you’re done — prompting is the simplest, fastest approach. Upgrade to RAG when: the knowledge base is too large for the context window, the knowledge base changes frequently (daily or weekly), or you need to cite specific sources. Do NOT fine-tune when you can use RAG: fine-tuning is expensive (GPU time, data preparation), time-consuming, and creates a model that requires maintenance. Most applications do not need fine-tuning. Consider fine-tuning when: you need consistent structured output formats (JSON schemas, specific classification labels), you need to change the model’s default style significantly, you have a very specific narrow task with many labeled examples, or you need to improve latency/cost by using a smaller fine-tuned model instead of a larger general one.
Combining Approaches
In practice, the best-performing applications often combine approaches. A common pattern: fine-tune for style/format + RAG for knowledge. Example: a legal document assistant — fine-tuned to output in the firm’s specific citation style and to classify document types correctly, with RAG over the firm’s case law database for factual retrieval. Another pattern: RAG + prompting with few-shot examples in the system prompt to guide the model on how to use retrieved context. The cost reality: fine-tuning a model costs hundreds to thousands of dollars for meaningful training runs (GPT-4 fine-tuning costs approximately $0.03/1K training tokens as of 2025, so 10 million tokens = $300, before infrastructure overhead). RAG costs are lower but require vector database infrastructure and embedding computation. For most use cases — especially those where knowledge changes regularly — RAG with good prompting outperforms fine-tuning and is cheaper to maintain.
