Llama-2-7B Alignment

A production-grade Reinforcement Learning from Human Feedback pipeline using PPO, TRL, and LoRA.

Reward Model Simulator

Simulate the pairwise ranking loss by choosing the better response.

Prompt: Loading...
Response A
Loading...
VS
Response B
Loading...

Performance Gains

โœจ

Helpfulness

3.2 โ†’ 4.3
+35%
๐Ÿ›ก๏ธ

Safety

3.5 โ†’ 4.7
+35%
๐Ÿ’พ

GPU Memory

28GB โ†’ 8.4GB
-70%
๐Ÿ”€

Output Diversity

0.45 โ†’ 0.36
-20%

Pipeline Architecture

01

Supervised Fine-Tuning

Fine-tuning base Llama-2 on high-quality instruction data using PEFT LoRA adapters.

Rank: 16 Params: ~33M
02

Reward Modeling

Training on human preference data using Bradley-Terry pairwise ranking loss.

Agreement: 82%
03

PPO Alignment

Optimizing policy against the reward model with KL-divergence tracking via W&B.

KL Coeff: 0.05 Epochs: 4