Fine-tuning — training an existing foundation model on a custom dataset to change its behaviour — is often the first thing teams reach for when a general-purpose LLM doesn’t do exactly what they want. It is often not the right first step. Here is an honest picture of when fine-tuning helps.
What Fine-Tuning Does and Doesn’t Do
Fine-tuning adjusts the model’s weights based on examples — it changes what the model “knows” or how it behaves in a persistent way that doesn’t require prompting. What fine-tuning is good for: teaching the model a consistent output format (JSON schema, specific writing style, domain-specific language); improving performance on a narrow task with many examples of correct behaviour; making the model behave more like a specific persona consistently; and distilling a larger model’s capability into a smaller, cheaper, faster model for a specific use case. What fine-tuning does NOT do well: add new knowledge the base model doesn’t have (for knowledge addition, RAG is typically more effective and updatable); fix fundamental reasoning limitations (a fine-tuned model that wasn’t reasoning correctly will often still not reason correctly after fine-tuning); or replace a well-crafted system prompt for most use cases. The most common mistake: teams conclude that the base model’s output isn’t right and jump to fine-tuning, when the real issue is a poorly written system prompt or insufficient context. Prompt engineering and few-shot examples solve 80% of problems that people initially attribute to needing fine-tuning.
When Fine-Tuning Actually Makes Sense
Format consistency at scale: if your application processes thousands of requests and requires a very specific output format (a JSON schema with specific field names, a specific markdown structure), fine-tuning produces more reliable format adherence than prompting alone — particularly for edge cases and ambiguous inputs. Latency and cost at scale: a fine-tuned smaller model (e.g., a fine-tuned Haiku 4.5 or GPT-4o-mini) performing at the level of a larger general model on a specific task is both faster and cheaper. The economic argument for fine-tuning improves at scale. Style and persona: if your product requires a very specific tone, vocabulary, or persona that is difficult to maintain through prompting (particularly for short or context-sparse requests), fine-tuning that style produces more consistent results. Domain-specific accuracy: in narrow domains with very specific jargon, abbreviations, or reasoning patterns (medical coding, legal citation format, specific programming frameworks), fine-tuning on domain examples can improve accuracy beyond what prompting achieves. Classification tasks: binary or multi-class classification on specific categories, where you have labelled examples — fine-tuned smaller models often match larger general models and run at a fraction of the cost.
Practical Fine-Tuning Considerations
Dataset quality over quantity: a fine-tuning dataset of 500 high-quality, diverse, correct examples outperforms a dataset of 10,000 mediocre examples. The most common failure in fine-tuning projects is low-quality training data that contains inconsistencies, errors, or edge cases without coverage. Evaluation before and after: establish a proper evaluation benchmark before fine-tuning; compare the fine-tuned model to the base model on held-out examples; ensure you haven’t improved one metric while degrading another. The train/eval split: never evaluate on examples from your training set. A 90/10 split is a starting minimum; for small datasets, use cross-validation. Catastrophic forgetting: fine-tuning on a narrow task can degrade the model’s performance on other tasks. This is a real risk for general-purpose assistants that are fine-tuned for a specific use case. The practical implication: fine-tuned models are best for specific, bounded use cases — not as replacements for general assistants. Providers: OpenAI fine-tuning (GPT-4o, GPT-4o-mini) is the most accessible managed fine-tuning service; Anthropic offers fine-tuning for Claude (Enterprise tier); Hugging Face + Axolotl or LLaMA Factory for open-source models on your own infrastructure.
