LLM Alignment and Safety: An Introduction to RLHF

When Claude/GPT emerges from pretraining, it is really just a "super autocomplete engine." Give it any opening, and it continues the text, potentially producing harmful content or answering the wrong question. It becomes a "useful assistant" through alignment, using training methods such as RLHF and DPO. Understanding alignment helps you understand why models sometimes "over-refuse" and how to use them more effectively.

1. The "Raw State" of a Pretrained Model

A model trained through pretraining (next-token prediction) is highly capable but "does not know the rules":

• It learns the statistical patterns of internet text, including misinformation, bias, and harmful content.
• It does not know "what it should say and what it should not say."
• It does not know how to "answer questions"; it only knows how to "continue text." If you ask "Hello," it may continue with another "Hello" rather than respond.

Alignment builds on this foundation and aligns the model with assistant behavior that is "helpful, honest, and harmless" (HHH).

2. What Alignment Does

The core of alignment is to make model behavior conform to human expectations: helpful by assisting users, honest by not inventing things, and harmless by not making matters worse.

It addresses three areas:

1. Instruction following: Teach the model to "answer questions" rather than merely "continue text."
2. Values: Refuse inappropriate requests, such as instructions for making drugs or producing discriminatory content.
3. Style: Use an assistant's voice rather than an arbitrary tone learned from the internet.

3. RLHF: The Classic Alignment Method

RLHF (Reinforcement Learning from Human Feedback) is the method popularized by ChatGPT. It has three steps:

1. SFT (supervised fine-tuning). Humans write a set of "high-quality question-and-answer" demonstrations, and the model learns to "respond like an assistant." This step turns the model from an "autocomplete engine" into a rudimentary assistant.

2. Train a reward model (RM). Humans rank several versions of model output, indicating that "this answer is good and that one is poor." These preferences train a "scoring model" that learns "what kinds of answers humans consider good."

3. RL optimization. Reinforcement learning (PPO) makes the primary model "pursue high scores from the reward model" by generating answers the RM prefers. This step pushes the model toward "human preferences."

RLHF works well, but it is complex, unstable, and expensive because training the RM and tuning PLO are difficult.

4. DPO: A Simpler Alternative

DPO (Direct Preference Optimization) is a simplified method that later became popular:

• It does not train a separate RM. Instead, it directly optimizes the primary model against preference data such as "A is better than B" using a simplified objective.
• It is mathematically equivalent to a form of RL but is simpler to implement, more stable, and less resource-intensive.

Many alignment pipelines now use DPO or one of its variants instead of RLHF. The results are similar, while the engineering barrier is much lower.

5. Alignment Is Both Safety and Capability

Alignment concerns not only "safety" by making models harmless, but also "capability" by making them useful:

• An unaligned model can continue text but cannot answer questions properly, follow instructions, or maintain a consistent style. It is almost unusable as an assistant.
• An aligned model follows instructions, uses an assistant's voice, and refuses what it should refuse. It becomes a usable product.

Alignment therefore both "adds safety constraints" and "adds assistant capabilities." A large part of the capable experience delivered by every flagship model you use comes from alignment, not from pretraining alone.

6. Over-Alignment: Model "Over-Refusal"

A side effect of alignment is over-refusal. In trying to remain "harmless," a model also refuses harmless requests.

Classic examples:

• "How do I kill a process?" in a programming context → The model may refuse because it misinterprets "kill" as harmful.
• "Help me write dialogue for a villain" → The model may refuse because it misinterprets "villain" as harmful.

This happens because the model sees many "refuse harmful content" examples during alignment and becomes overly conservative. Every provider is tuning this balance; Claude is relatively restrained, while some models are more sensitive.

As an application developer, you can handle over-refusal in several ways:

• Make the prompt's context more explicit. "In a programming context, 'kill process' means terminating a process. Help me write this code." Provide enough context to establish that the request is harmless.
• Switch to a more permissive model or version, because models use different alignment strategies.
• Define the role in the system prompt. "You are a programming assistant, and every request is made in a programming context."
• Self-host an open-source model. You control the degree of alignment yourself, but lose the protections provided by a commercial model and must take responsibility for safety.

7. Alignment and the Cat-and-Mouse Game of "Jailbreaking"

Users try to bypass alignment through "jailbreaks" and make models say things they should not. Alignment teams continually patch the models, while users continually discover new techniques. It is a classic cat-and-mouse game.

As an application developer:

• Do not rely on model alignment as your security boundary. Model alignment is "general-purpose protection," not your application's security policy. Your application needs its own security layer; see the article on prompt injection.
• Alignment is dynamic. When a model is upgraded, its alignment behavior may change. A prompt that works today may be refused tomorrow. Test it.

8. Practical Implications for Application Developers

Understanding alignment has practical value for developers:

1. Understand the boundaries of model behavior. Know which requests may be refused and why, so you do not waste time fighting the model.
2. Write better prompts. Give the model clear, harmless context to reduce over-refusal.
3. Choose models. Models differ in their alignment strategies; some are stricter and some are more permissive, so choose according to the use case.
4. Build your own safety layer. Do not entrust application security to model alignment.
5. Respect alignment during fine-tuning. Fine-tuning can "damage" alignment and make a model unsafe, so evaluate it.

Summary

Alignment through RLHF or DPO is the key training process that turns a pretrained "autocomplete engine" into a "useful assistant":

• It addresses: instruction following, values, and style.
• RLHF: SFT + reward model + RL; classic but complex.
• DPO: Direct preference optimization; simpler and more stable, and increasingly mainstream.
• Side effect: Over-refusal, in which harmless requests are incorrectly rejected.
• For developers: Understand the boundaries, write good prompts, build your own safety layer, and do not rely on model alignment for product security.

Remember this: Half the capability of the "useful model" you interact with comes from pretraining, and half comes from alignment. Once you understand alignment, you stop treating the model as a black box and asking, "Why does it behave like this?" Instead, you understand "why it behaves this way and how I should work with it." That is the difference between "being able to use AI" and "understanding AI."