Promptfoo
VerifiedOpen-source CLI tool for evaluating LLM output quality and catching regressions. Test prompts, compare models, and run red-team adversarial tests. 5,000+ GitHub stars.
Frameworks Covered
Services
llm-evaluationprompt-testingred-teamingregression-testing
About Promptfoo
Promptfoo is an open-source CLI tool for evaluating LLM output quality and catching regressions before they reach production. It supports testing prompts across multiple models, running adversarial red-team tests, and generating evaluation matrices.
Features
- Evaluate prompts across multiple LLM providers
- Red-team adversarial test generation
- Regression testing for prompt changes
- Custom assertion types (JavaScript, Python, or natural language)
- Comparison matrices (model x prompt x test case)
- CI/CD integration
- Local cache for cost-efficient testing
- Supports OpenAI, Anthropic, Google, local models (Ollama)
Installation
npm install -g promptfoo
Loading reviews...