LLM testing · AI agent testing · Evals
Testing AI features and agents before your users do
AI feature testing checks that an LLM feature or agent does what it should, refuses what it shouldn't, and doesn't get worse when you change the model or the prompt. I build evaluation suites for your use cases, red-team the feature, and turn the findings into tests that run on every release.
The problem
Classic tests expect the same output for the same input. LLM features don't work that way: answers vary, a prompt tweak fixes one case and breaks three others, and a model upgrade can quietly change behaviour across the board. Most teams find out from users.
What I test
- Hallucinations and unsupported claims, checked against your source data
- Prompt injection and jailbreaks, including instructions hidden in documents and tool results
- Quality regression after a model, prompt or retrieval change
- Agent behaviour: wrong tool calls, loops, actions taken without confirmation
- Data leakage: system prompts, other users' data, secrets in context
How I work
- Agree what 'good' means for your feature: a rubric and a set of real examples
- Build an evaluation set from your tickets, logs (anonymised) and edge cases
- Run evals and red-teaming against staging, triage the failures
- Add the evals to your CI so every prompt or model change is scored before release
- Report: what fails, how often, how severe, and what to fix first
FAQ
Which models and frameworks do you work with?
Any model behind an API, and agents built with common frameworks or your own code. The evals live in your repository.
Do you need production data?
No. Staging and synthetic or anonymised examples by default, as with all my work.
Shipping an LLM feature or an agent? In 15 minutes I'll tell you which failure modes I'd test first.
Book a 15-min callPrefer writing? Use the contact form. Replies within one business day, from me.