← Back to Library
AI/ML Document · July 2026 · Enhanced Edition
Advanced AI/ML Research: Phases 4-6 Research Summary
> Date: July 2026
Author: Anas Hussain (a1n4a)  ·  Date: July 2026  ·  Edition: Enhanced with Diagrams

Advanced AI/ML Research: Phases 4-6 Research Summary

Date: July 2026
Scope: Learning Paradigms, Foundation Models, AI Safety & Interpretability
Audience: ML researchers with strong fundamentals

Phase 4: Learning Paradigms

4.1 Reinforcement Learning from Human Feedback (RLHF)

RLHF — Reinforcement Learning from Human Feedback Step 1 Supervised Fine-Tuning (SFT on demos) Step 2 Reward Model Training (human pref pairs) Step 3 PPO Training with RM (policy optimisation) Aligned LLM Human Annotators rank model outputs Alternative: DPO Direct Pref. Optimisation (skips explicit RM)
Fig. 9 — RLHF Pipeline: (1) SFT on demonstrations, (2) train Reward Model from human preferences, (3) PPO to optimise policy against RM.

Reward Modeling

RLHF typically proceeds in three stages: supervised fine-tuning (SFT), reward model (RM) training from human preferences, and policy optimization (often PPO) against the learned reward. The reward model takes a prompt+response pair and outputs a scalar reward, trained via a Bradley-Terry preference model.

Key challenge: Reward overoptimization. The policy exploits the learned RM in ways that do not reflect true human preferences. Gao et al. (2023, arxiv:2210.10760) showed the gap between RM reward and true reward grows as a power law in KL divergence.

PPO (Schulman et al., 2017, arxiv:1707.06347)

PPO constrains policy updates via a clipped surrogate objective. In RLHF, PPO is applied as a bandit problem with a KL penalty from the SFT model. Open debate: Whether clipping is necessary or simpler KL-regularized reward maximization suffices (Ramanurthy et al., 2022, arxiv:2210.01241).

DPO (Rafailov et al., 2024, arxiv:2305.18290)

Eliminates the explicit reward model by reparameterizing the RLHF objective in terms of the policy. Key insight: Under Bradley-Terry, the optimal reward is a log-ratio of policies, so the policy implicitly represents the reward.

Extensions: IPO (Azar et al., 2024, arxiv:2310.12024), KTO (Ethayarajh et al., 2024, arxiv:2402.01306), Iterative DPO (Xiong et al., 2024, arxiv:2312.11456), TIS-DPO (Liu et al., 2024, arxiv:2410.04350).

GRPO (Group Relative Policy Optimization)

Gradient Descent on Loss Landscape Parameter w Loss L(w) Global Min Local Min Saddle Pt Start gradient steps (η) w_new = w - η · ∇L(w) η = learning rate
Fig. 5 — Gradient Descent: iteratively step opposite the gradient to minimise loss. Learning rate η controls step size.

Used in DeepSeek-R1 (2025, arxiv:2501.12948). Eliminates the need for separate RM and reference model. Groups multiple sampled completions per prompt, computes advantages within the group, updates policy without a critic network. Recent variants: Scaf-GRPO (arxiv:2510.19807), MO-GRPO (arxiv:2509.22047), CoDistill-GRPO (arxiv:2605.08873), N-GRPO (arxiv:2606.10768).

Constitutional AI (Bai et al., 2022, arxiv:2212.08073)

Replaces human preference labelling with AI-generated feedback guided by a written constitution. Two phases: supervised revision + RL from AI feedback. Extended to self-rewarding (Yuan et al., 2024, arxiv:2401.10020).

Preference Optimization Without Rewards

Key methods: DPO (policy=implicit reward), IPO (direct regression on odds), KTO (prospect-theoretic, no pairs), ORPO, SimPO. No consensus on which is superior across settings.

4.2 Meta-Learning

MAML (Finn et al., 2017, arxiv:1703.03400)

Learns an initialization for fast adaptation via few gradient steps. Differentiates through inner gradient descent (second-order). FOMAML often works nearly as well.

Reptile (Nichol et al., 2018, arxiv:1803.02999)

First-order: updates initialization toward post-adaptation parameters without inner-loop differentiation.

Few-Shot Learning Families

Model-based (memory-augmented), Metric-based (Prototypical Networks, Matching Networks), Optimization-based (MAML, Reptile, LEO).

Meta-Learning Bias-Variance Tradeoff

Bias-Variance Tradeoff Model Complexity → Error Bias² Variance Total Error Optimal Underfitting (High Bias) Overfitting (High Variance)
Fig. 12 — Bias-Variance Tradeoff: simple models have high bias (underfit); complex models have high variance (overfit). Total error = Bias² + Variance + Noise.

Too few inner steps = high bias; too many = high variance (meta-overfitting). Formalized by Chen et al. (2019), Bai et al. (2021).

Current frontier: Increasingly absorbed into LLM ICL paradigm. Still important for robotics, drug discovery.

4.3 Continual Learning

Catastrophic Forgetting (McCloskey & Cohen, 1989)

Scaled Dot-Product Attention Input X W_Q W_K W_V Q K V MatMul + Scale QKᵀ / √dₖ Softmax Weighted sum × V Attention weights (scores) Output
Fig. 3 — Scaled Dot-Product Attention: Q (query), K (key), V (value) projections. Score = softmax(QKᵀ/√dₖ) · V.

Sequential training overwrites prior knowledge due to shared representational capacity.

Elastic Weight Consolidation (Kirkpatrick et al., 2017, arxiv:1612.00796)

Quadratic penalty on Fisher-diagonal-weighted parameters. Diagonal approximation misses off-diagonal interactions.

Progressive Neural Networks (Rusu et al., 2016, arxiv:1606.04671)

Input Hidden 1 Hidden 2 Output x₁ x₂ x₃ x₄ ŷ₁ ŷ₂ 4 neurons 5 neurons 4 neurons 2 neurons → Forward Pass →
Fig. 1 — Feedforward Neural Network: 4-input, two hidden layers, 2-output. Each circle = neuron; each edge = learnable weight.

New column per task; prior columns frozen. Solves forgetting at linear parameter cost. Influenced adapters/LoRA.

Synaptic Intelligence (Zenke et al., 2017, arxiv:1703.04200)

Online importance estimation without task boundaries.

Stability-Plasticity Dilemma (Grossberg, 1987)

Inherent tension between retaining old knowledge and integrating new information.

Replay vs. Regularization

Replay: Experience Replay (simplest, most effective), Generative Replay (diffusion), Dark Experience Replay. Regularization: Weight (EWC, SI, MAS), Functional (LwF, KD), Representation (contrastive).

Current frontier: LLM continual learning via LoRA, activation steering, knowledge editing. Task boundary problem unsolved.

4.4 In-Context Learning

Three Hypotheses

  1. Bayesian inference over latent tasks (Xie et al., 2022, arxiv:2111.02080)
  2. Gradient descent analogy in forward pass (Von Oswald et al., 2023, arxiv:2209.14795; Akyurek et al., 2023, arxiv:2211.15661)
  3. Induction mechanism via specialized attention heads

For linear models, ICL = ridge regression or gradient descent. Non-linear picture incomplete.

Induction Heads (Olsson et al., 2022, arxiv:2209.11895)

Prefix-matching heads: find previous token occurrences, attend to following token. Appear spontaneously at scale threshold (phase change), explaining ICL emergence. Compose from previous-token head + pairing mechanism.

Mechanistic Interpretability of ICL

Function vectors (Todd et al., 2023, arxiv:2310.15213), error correction mechanisms, task-specific attention patterns.

Current debate: ICL as learning vs. pattern retrieval. Both mechanisms operate at different complexity levels.

Phase 5: Foundation Models

5.1 Scaling Laws

Neural Scaling Laws (Chinchilla & Kaplan) log(Compute / Parameters / Data) → log Loss L ∝ N^α L ∝ D^β L ∝ C^γ Chinchilla optimal N ∝ C^0.5, D ∝ C^0.5 Parameters (N) Data (D) Compute (C) All follow smooth power laws. Chinchilla: for a given compute budget, scale data and model equally.
Fig. 13 — Neural Scaling Laws: loss decreases as a power law with more parameters, data, and compute. Chinchilla scaling law: N ≈ D (compute-optimal).

Kaplan et al. (2020, arxiv:2001.08361)

Cross-entropy loss scales as power law with compute, data, and parameters. Optimal allocation: N_opt ~ C^0.71, D_opt ~ C^0.28. No plateauing across seven orders of magnitude.

Hoffmann et al. (2022, Chinchilla, arxiv:2203.15556)

Revised optimal allocation: N_opt ~ C^0.5, D_opt ~ C^0.5. Models should be smaller but trained on more data. Chinchilla 70B matched GPT-3 175B with 4x less compute. Discrepancy with Kaplan arises from methodology.

Subsequent: Llama 3 suggests compute-optimal ratio shifts at very large budgets. Data quality vs. quantity scaling is frontier. Multi-modal scaling laws actively studied (Cherti et al., 2023, arxiv:2307.13223).

Emergent Abilities Debate

5.2 Reasoning Models

Chain-of-Thought (Wei et al., 2022, arxiv:2201.11903)

CoT prompting adds intermediate reasoning steps, improving arithmetic, commonsense, and symbolic reasoning. Key questions: Why does it work? (externalizes working memory). Is reasoning faithful? (often post-hoc rationalization). Variants: Zero-shot CoT, Self-consistency, Complexity-based prompting.

Tree-of-Thought (Yao et al., 2023, arxiv:2305.10601)

Extends CoT to tree search with heuristic evaluation. Connection to classical AI (Newell & Simon). Best on verifiable puzzles.

o1/o3-Style Reasoning

  1. Process reward models (PRMs): Step-level supervision (Lightman et al., 2024, arxiv:2405.19020)
  2. Test-time compute scaling (Snell et al., 2024, arxiv:2408.03314): Pareto-optimal frontier between training and test-time compute
  3. RL on reasoning traces: DeepSeek-R1 (2025, arxiv:2501.12948) — pure GRPO produces emergent CoT

Open debate: Does extended reasoning generalize beyond math/code? Scale vs. architecture innovation.

5.3 Mixture of Experts (MoE)

Sparse MoE Layers (Shazeer et al., 2017, arxiv:1701.06538)

Sparsely-activated expert modules with noisy top-k gating, load balancing loss, capacity factor. Inherent load imbalance.

Routing Strategies

  1. Token-choice (Shazeer): load imbalance
  2. Expert-choice (Zhou et al., 2022, arxiv:2202.09368): perfect balance, may drop tokens
  3. Hash-based: deterministic, no imbalance
  4. Soft routing: no sparsity benefits

GShard (Lepikhin et al., 2021, arxiv:2006.16668)

MoE + automatic sharding across TPU pods. Expert parallelism with all-to-all. 600B parameter translation model.

Switch Transformer (Fedus et al., 2022, arxiv:2101.03961)

Transformer Encoder Block Input Embeddings + Positional Encoding Multi-Head Attention Q K V Attention(Q,K,V) = softmax(QKᵀ/√dₖ)V Add & LayerNorm Feed-Forward Network Linear → ReLU → Linear dim: 512 → 2048 → 512 Add & LayerNorm Output (to next block) Residual ×N stacked blocks
Fig. 2 — Transformer Encoder Block: Multi-Head Attention + residual connections + Layer Normalization + Feed-Forward Network.

Top-1 routing, up to 1.6T params (2048 experts), 7x speedup over dense.

Expert Choice Routing (Zhou et al., 2022, arxiv:2202.09368)

Experts select tokens instead. Perfect load balance, no capacity factor. Dropped tokens via residual.

Current Landscape

Mixtral 8x7B/8x22B, DeepSeek-V2/V3, Gemini, Llama 4.

Open directions: Expert specialization vs. redundancy, dynamic allocation, routing-aware LoRA, expert merging.

5.4 Alignment Tax & Post-Training

ML Training Pipeline Raw Data Dataset Pre- process Normalize Model Forward f(x;θ) Loss Compute L(ŷ, y) Optimizer Update θ←θ-η∇L ↻ Repeat for N epochs
Fig. 6 — The ML Training Loop: data flows forward to produce a loss, gradients flow back to update weights. Repeated for many epochs.

Alignment tax: Capability degradation during RLHF/alignment. Causes: KL constraint, reward over-optimization, preference bias, mode collapse. Evidence in InstructGPT, Claude, Llama 2. Mitigated by rejection sampling, mixed training, multi-objective RLHF.

Post-Training Pipeline (Llama 3, 2024, arxiv:2407.21783): SFT -> RLHF/DPO -> rejection sampling -> multistage training -> data iteration.

Open debate: Is alignment tax inevitable or suboptimal methodology? Evidence suggests it can be nearly eliminated.

5.5 Multi-Modal Foundation Models

Architectures: Connector-based (CLIP + LLM), Fusion-based (cross-attention), Unified (joint training). Key models: GPT-4V/o, Gemini, Claude 3 Vision, LLaVA family, Idefics2, Qwen-VL, InternVL.

Challenges: Resolution/detail loss, alignment without forgetting, saturating benchmarks.

Scaling: Similar power-law scaling to text-only (Cherti et al., 2023). Compute-optimal allocation across modalities unknown.

Essential Video & Paper Resources