An applied course on reinforcement learning for large language models, covering the full workflow from autoregressive sampling and policy-gradient foundations to hardware-aware rollout systems and modern group-based policy optimization methods.

Course Curriculum

• Module 1: Language Model Preliminaries — autoregressive generation, token probabilities, and the distinction between supervised fine-tuning and reinforcement learning.
• Module 2: Reinforcement Learning and Sampling — rollout generation, sampling, exploration, and response-level reward signals.
• Module 3: Reinforcement Learning Math — expected rewards, policy gradients, advantages, baselines, Monte Carlo estimation, and importance sampling.
• Module 4: Reinforcement Learning on Hardware — rollout and update phases, sequential generation, parallel gradient computation, GPU memory constraints, batching, and old-policy mismatch.
• Module 5: vLLM Engine — efficient large-language-model inference and high-throughput rollout generation.
• Module 6: PPO and GRPO — value and group-relative baselines, probability ratios, clipping, and KL regularization.
• Module 7: GSPO and SAPO — sequence-level importance ratios and clipping, plus temperature-controlled soft gating for stable policy updates.
• Module 8: Hands-on Lab — GSPO fine-tuning on the GSM8K dataset in a Jupyter Notebook.