Colter Test

TL;DR: Run colter test https://your-store.com to send AI shopping personas through your store and score the experience. Add --json for structured output, --pdf for a report, or --fix to generate follow-up fixes automatically.

colter.test returns structured JSON built for agent workflows, regression checks, and CI jobs.

What Test Does

colter test simulates real shopping prompts across multiple model families and scores the outcomes. It answers a different question than Check: not just "is the protocol there?" but "does the agent succeed?"

Personas

Ten personas run by default, with the_comparer available as an opt-in persona.

Persona group	Focus
Platform shoppers	Protocol flows, browser flow, mobile flow
Intent shoppers	Security, pricing clarity, data quality, returns, edge cases

Scenarios

Typical scenarios include:

• discovery
• product info
• policy comprehension
• checkout readiness
• competitive comparison
• recommendation
• edge cases

Requirements

• colter auth login
• Pro plan with Test add-on, or Agency / Enterprise plan

CLI

colter test <url> [flags]

Common Flags

Flag	Purpose
--models LIST	Choose claude, gpt, gemini
--personas LIST	Filter personas
--scenarios LIST	Filter scenarios
--json	Structured output
--parallel N	Concurrent persona runs
--timeout DURATION	Per-test timeout
--budget AMOUNT	Max spend in USD
--threshold N	Exit non-zero below this score
--skip-check	Skip the pre-check phase
--check-json FILE	Reuse a saved check result
--fix	Generate fix plans for weak dimensions
--fix-threshold N	Cutoff used with --fix
--apply	Apply fixes after test when --fix is set
--dry-run	Generate fix content without writing
--pdf	Create a PDF report
--pdf-out PATH	Set the PDF path
--upload	Upload results
--headed	Show the browser window
--browser	Run the browser-based WebMCP flow
--api-url URL	Override API base URL

Examples

colter test https://store.example.com

colter test https://store.example.com --models claude,gemini --json

colter test https://store.example.com --threshold 70 --json

colter test https://store.example.com --fix --fix-threshold 75

colter test https://store.example.com --pdf --pdf-out report.pdf

Output Highlights

The JSON payload includes:

• overall score
• per-persona results
• per-scenario results
• per-model scores
• recommendations
• token and cost totals

CI

colter test exits with code 1 when the final score is below --threshold.

colter test https://mystore.com --threshold 70 --json

Recommended Flow

1. Run Check.
2. Run Test to see interaction failures.
3. Run Fix on the weak areas.
4. Re-run Test.
5. Use Lens for live traffic after launch.

Pricing

Test is:

• available as a paid add-on on Pro
• included on Agency
• included on Enterprise

Next Steps

• Lens
• Verify
• MCP Tools
• Evidence Packs
• Pricing
• API Reference