LLM Fine-Tuning Course – From Supervised FT to RLHF, LoRA, and Multimodal

Unsupervised pre-training builds foundational knowledge

Models first undergo self-supervised learning on massive text corpora to understand language patterns before any task-specific adaptation occurs.

Supervised Fine-Tuning adapts models to specific tasks

SFT trains models on labeled datasets using either full fine-tuning (updating all parameters, requiring huge GPU memory) or partial fine-tuning (updating subsets).

Data preparation determines fine-tuning approach

Instruction fine-tuning uses prompt-response pairs for chat models, while non-instruction fine-tuning uses raw text continuation for specialized domains.

LoRA and QLoRA enable single-GPU training

Low-Rank Adaptation trains small adapter matrices instead of full weights, while QLoRA adds quantization for memory-efficient loading on consumer hardware.

Advanced alternatives extend beyond basic LoRA

DoRA (Weight Decomposition), IA3 (Infused Adapter), Adapter Layers, BitFit, and Prefix Tuning offer specialized approaches for different model architectures and constraints.

Layer freezing is obsolete for modern transformers

Freezing initial layers and retraining only final layers works for CNNs and early models like BERT but is ineffective for modern transformer-based LLMs.

RLHF uses reinforcement learning with PPO

Reinforcement Learning from Human Feedback employs Proximal Policy Optimization to reward models for generating human-preferred responses, famously used by OpenAI.

DPO offers a streamlined supervised alternative

Direct Preference Optimization has become the dominant modern technique, using preference datasets (chosen vs. rejected responses) without complex reinforcement learning loops.

Alignment requires specific dataset structures

Preference-based learning requires datasets containing questions, multiple possible responses, and human feedback indicating which response is preferred.