Large Language Models (LLMs)

Topics

Notes

Linked

Causality Large Language Models (LLMs)
Better Think Thrice - Learning to Reason Causally with Double Counterfactual Consistency

How to make LLMs more causally consistent.

Open-Endedness Large Language Models (LLMs)
OMNI - Open-endedness via Models of human Notions of Interestingness

How to measure notions of interestingness without hand-coded formulas for learning agents

PPO - Proximal Policy Optimization
Large Language Models (LLMs)
Emergent Misalignment in LLMs

Fine-tuning LLMs on narrow tasks can unexpectedly produce misaligned behavior on unrelated tasks.

Large Language Models (LLMs)
Direct Preference Optimization (DPO)

Learn from preference dataset without the complicated RL setup

TRPO - Trust-Region Policy Optimization Bradley-Terry Model Distribution Shift Importance Sampling Maximum Likelihood Estimation RLHF - Reinforcement Learning with Human Feedback PPO - Proximal Policy Optimization KL Divergence Cross entropy
Large Language Models (LLMs) Reinforcement Learning
Group Relative Policy Optimization (GRPO)

Avoid learning an explicit value function in RL alignment setup

TRPO - Trust-Region Policy Optimization RLHF - Reinforcement Learning with Human Feedback Control Variates PPO - Proximal Policy Optimization KL Divergence