LLM Instruction Following Eval
Every LLM claims to follow instructions — but does it when constraints get complex? This benchmark gives each model a simple writing task paired with an escalating set of precise formatting rules: word limits, bullet counts, banned words, case requirements. Watch exactly which constraints each model drops as difficulty increases.
The task
Fixed prompt — "Explain the main benefits of getting regular exercise." Simple enough that any model can answer it. The constraint set determines the difficulty.
The constraints
Four difficulty levels stack 1 to 5 formatting rules on the same task. Easy: one word-count rule. Extreme: five simultaneous constraints including case, punctuation, and exclusion.
What varies
Difficulty level (Easy to Extreme), number of trials per level, and model selection. Multiple trials expose inconsistency — a model that sometimes follows all rules and sometimes misses one.
Scoring
PASS if ≥80% of trials satisfy ≥90% of constraints. PARTIAL at 40–79%. FAIL below 40%. Compliance rate is per-constraint — you see exactly which rule each model breaks.
| Result | Trial pass rate | Compliance rate per trial |
|---|---|---|
| PASS | ≥80% of trials pass | ≥90% of constraints met |
| PARTIAL | 40–79% of trials pass | ≥90% of constraints met |
| FAIL | <40% of trials pass | One or more constraints broken |
Each difficulty level adds more constraints. Models that score 100% on Easy often drop to 40% or below on Extreme.
Sample results
Real runs showing compliance rates across all four difficulty levels. No API key needed — just click a run to load its heatmap.
Run your own eval
Use your own API key to test any model. Results are streamed live and never stored on our servers.
sk-or-. Get one free at openrouter.ai
Results stream live below. Download JSON when complete.
Run history
Results from runs you completed this session. Reload the page to start fresh.
Frequently asked questions
What does "instruction following" test exactly?
It tests whether a model can simultaneously satisfy multiple precise formatting rules on the same response. Easy difficulty has one rule (word limit under 50 words). Extreme has five simultaneous rules: exactly 3 bullet points, under 60 words, no use of the word "health", everything lowercase, and no punctuation of any kind. Most models handle one constraint reliably but begin dropping constraints when juggling four or five simultaneously.
Is my API key stored anywhere?
No. Your key is transmitted over HTTPS to the server, used only for the outbound API call, and discarded immediately — never written to a database, log file, or environment variable.
Why does the same model fail on Extreme but pass on Easy?
As constraints compound, models face a multi-objective trade-off: satisfying one rule can interfere with attention to another. For example, maintaining all-lowercase while preserving bullet format and staying under a word count requires consistent, simultaneous tracking of three separate signals. Each additional constraint increases cognitive load. Most frontier models score near 100% on Easy and drop significantly by Extreme.
What are the exact constraints for each difficulty level?
Easy (1 constraint): Use fewer than 50 words total.
Medium (2 constraints): Use fewer than 60 words total. Begin your response with the words "Regular exercise".
Hard (3 constraints): Use exactly 3 bullet points. Use fewer than 80 words total. Begin each bullet point with the • character.
Extreme (5 constraints): Use exactly 3 bullet points. Use fewer than 60 words total. Do not use the word "health" anywhere. Write everything in lowercase letters only. Use no punctuation characters at all (no periods, commas, exclamation marks, question marks, semicolons, colons, quotes, or brackets).
How is "compliance rate" different from a pass/fail score?
Compliance rate is per-constraint within a single trial. If a model satisfies 4 out of 5 constraints, its compliance rate is 80% for that trial. A trial earns a PASS verdict if compliance rate is 90%+, PARTIAL at 50–89%, FAIL below 50%. The heatmap cell shows the pass rate across all trials for a model+difficulty combination — for example, "67%" means 2 out of 3 trials were PASS.
More tools for AI builders
- → Context Retrieval Eval — test whether models can retrieve a fact at 85–95% of their context window
- → SEO Site Audit — audit every page your AI content strategy produces
- → Schema Markup Tester — validate structured data that helps AI Overviews cite your content
- → Link Checker — keep the site you're building for AI search free of broken links