We show where your model fails.
Research oriented evaluation frameworks for large language models, agentic AI and multimodality, built for the labs at the frontier.
A benchmark score is a summary. Summaries are useful, but they are also where failures go to hide. Two models can share a leaderboard rank and behave in entirely different ways the moment a task runs long, an input turns adversarial, or a tool returns something unexpected.Averages are dominated by the easy majority of a test set. The failures that cause incidents in deployment are, almost by definition, rare.
We take models to their limits on purpose. We push past the region where they look competent and into the region where they begin to break. Then we slow down and look closely. Every failure is reproduced, characterised and filed into a taxonomy that your researchers can reason about, rerun and fix.
“An error is something that happened once. A failure mode is something you can make happen again.”
Approach
Probe, push, catalogue, report. We begin at the edge of what a model has demonstrably done, then escalate difficulty and adversarial pressure step by step until behaviour degrades. At that point the real work starts.Each catalogued failure includes a minimal reproduction, trigger conditions, observed frequency, severity, and a hypothesis for the underlying mechanism.
Findings are clustered into failure modes rather than averaged into a score. Each engagement ends with a detailed research report and the versioned datasets and harnesses behind it, so every result can be rerun on your next checkpoint.Reproducibility is a design constraint, not an afterthought. If we can’t make it fail twice, we don’t report it as a mode.
We watch for reward hacking
Reward hacking is when a model finds a shortcut to a high score without really doing the task. For example, it changes the tests so they pass instead of fixing the code.When a score becomes the goal, it stops being a good measure.
If no one notices, training rewards the shortcut and the model learns it as normal behaviour. It learns to look successful instead of being successful, and that is how a model becomes misaligned. We look for these shortcuts early, before they become habits.
Domains
Most interesting failures sit at the intersections: an agent misreading a screenshot, or a model losing its instructions halfway through a long task.- Large language models
- Reasoning on unfamiliar inputs, calibration, performance on very long inputs, factuality and instruction hierarchy under adversarial pressure.
- Agentic AI
- Long tasks in sandboxed environments: tool misuse, compounding errors, reward hacking, unsafe actions and failure to recover.
- Multimodality
- Linking text and images correctly, visual hallucination, documents and charts, and consistency between what a model sees and what it says.
Services
Research. New evaluation methodology and studies of emergent failure, written to publication standard and developed with or for your team.
Evaluation. Custom test suites and regression tracking across versions.
Alignment. Behavioural evaluations for sycophancy, deception, specification gaming and policy adherence. We also check that a fix removes a failure rather than merely hiding it.A patched failure that reappears under a paraphrase was never fixed. We test for that explicitly.
Commitments
- Failures over averages: we report what breaks, not only what scores.
- Reproducible by construction: code and data ship with every finding.
- Adversarial by default: we assume the easy tests already pass.
- Confidential: access to unreleased models under NDA, on secure infrastructure.
- Independent: our conclusions are our own.
Correspondence
Instead of producing tasks at industrial scale, we focus on measuring quality and assessing failures.