AI Safety Evaluation Framework

Understanding AI Safety Evaluation

AI safety evaluation is the systematic process of measuring how well a language model avoids harmful outputs, maintains factual accuracy, respects user privacy, and behaves consistently. As LLMs become embedded in critical applications — from healthcare advice to legal document generation to financial analysis — rigorous safety evaluation is no longer optional. This framework provides a structured approach to assessing safety that works across model families and use cases.

The Eight Safety Dimensions

1. Toxicity

Toxicity measures the model's tendency to generate harmful, offensive, or inappropriate content including hate speech, violent content, sexually explicit material, and harassment. Modern models have strong toxicity filters, but edge cases remain — particularly when toxic content is embedded in seemingly benign contexts, when the model is asked to roleplay, or when generation involves sensitive topics like conflict or crime. Evaluation uses adversarial test sets with 500+ prompts designed to elicit toxic outputs across different attack vectors.

2. Bias and Fairness

Bias evaluation measures whether the model produces systematically different quality, tone, or accuracy for different demographic groups. This includes gender bias in professional recommendations, racial bias in creative writing, cultural bias in knowledge representation, and socioeconomic assumptions in advice-giving. Testing involves paired prompts that differ only in demographic identifiers, measuring output divergence. A fair model produces comparably helpful and accurate responses regardless of the demographic context implied by the prompt.

3. Hallucination

Hallucination rate is the frequency at which a model generates false, unverifiable, or fabricated information presented as fact. This is measured using knowledge-intensive questions with verifiable answers, checking both the accuracy of claims and the model's calibration (whether it expresses appropriate uncertainty for uncertain claims). TruthfulQA, SimpleQA, and custom domain-specific benchmarks are used. Hallucination is particularly dangerous in high-stakes domains: a hallucinated legal citation or medical dosage can cause real harm.

4. Refusal Quality

Refusal quality measures the model's ability to correctly decline harmful requests while avoiding over-refusal of benign ones. A model that refuses to write a story about a villain is overly cautious; a model that provides step-by-step instructions for dangerous activities has insufficient safeguards. The ideal balance is evaluated using mixed test sets containing genuinely harmful requests, ambiguous requests, and benign requests that superficially resemble harmful ones. The metric captures both false negatives (harmful requests not refused) and false positives (benign requests incorrectly refused).

5. Jailbreak Resistance

Jailbreak resistance measures how well the model maintains its safety behaviors when subjected to adversarial prompt engineering attacks. Common techniques include role-playing scenarios, encoding/decoding tricks, multi-turn manipulation, and prompt injection through user-controlled content. Evaluation uses a continuously updated set of known jailbreak techniques, testing whether the model maintains refusal behavior under each attack pattern. This dimension evolves rapidly as new attack techniques are discovered.

6. Privacy

Privacy evaluation checks whether the model protects personal information — both in its training data (not regurgitating memorized PII) and in its interactions (not requesting unnecessary personal details). Tests probe for memorized phone numbers, email addresses, physical addresses, and private documents. Additional tests check whether the model appropriately declines requests to process or store personal information in ways that violate common privacy principles like data minimization and purpose limitation.

7. Instruction Following

Instruction following measures adherence to system prompt constraints, output format requirements, and explicit behavioral rules. A model that is told to respond only in JSON but occasionally returns markdown has poor instruction following. This dimension is critical for production applications where consistent output format is required for downstream processing. Tests include format constraints, content restrictions, length limits, and complex multi-rule system prompts to evaluate prioritization when rules conflict.

8. Robustness

Robustness measures consistency across paraphrased inputs, typos, different languages, and edge-case inputs. A robust model provides the same quality response whether a question is asked formally or casually, with perfect grammar or with typos, in English or in other languages. Testing involves generating multiple paraphrased versions of the same question and measuring output variance. Models with low robustness produce unreliable results in production environments where input quality varies.

Scoring Methodology

Each dimension is scored on a 1-10 scale where 1 represents severe safety failures and 10 represents near-perfect safety. The composite score is a weighted average, with weights customizable based on use case. Customer-facing applications typically weight toxicity, bias, and hallucination heavily. Research applications prioritize hallucination and robustness. Code generation applications emphasize instruction following and jailbreak resistance (to prevent prompt injection in generated code). The framework encourages teams to define their own weights rather than relying on a universal ranking — safety requirements are inherently context-dependent.