Unclosed div`
- Non-HTML content with HTML inside: `Here is some HTML:
test
`
### Contains-Sql
This assertion ensures that the output is either valid SQL or contains a code block with valid SQL.
```yaml
assert:
- type: contains-sql
```
Fenced SQL can use an optional `sql` language tag. MySQL backtick-quoted identifiers, such as ``SELECT `id` FROM `users`;``, are supported inside the fence.
See [`is-sql`](#is-sql) for advanced usage, including specific database types and allowlists for tables and columns.
### Cost
The `cost` assertion checks whether the cost reported by the selected provider is at or below a specified threshold.
This requires the provider to return cost information; an unknown cost cannot be checked. Use `--no-cache` when comparing fresh inference costs, because response-cache cost reporting depends on the provider.
Example:
```yaml
providers:
- openai:gpt-5-mini
- openai:gpt-5
assert:
# Pass if the LLM call costs less than $0.001
- type: cost
threshold: 0.001
```
To record the provider's cost in USD without a pass/fail limit, omit `threshold` and set a named `metric` with `weight: 0`. The measurement is reported without contributing to the aggregate quality score. Missing, negative, or non-finite costs produce an error instead of a zero measurement.
```yaml
defaultTest:
assert:
- type: cost
metric: inference_cost
weight: 0
derivedMetrics:
- name: average_inference_cost
value: 'inference_cost / __count'
```
Threshold-based `cost` assertions, including `not-cost`, continue to report binary pass/fail scores.
### Equality
The `equals` assertion checks if the LLM output is equal to the expected value.
Example:
```yaml
assert:
- type: equals
value: 'The expected output'
```
You can also check whether it matches the expected JSON format.
```yaml
assert:
- type: equals
value: { 'key': 'value' }
```
If your expected JSON is large, import it from a file:
```yaml
assert:
- type: equals
value: 'file://path/to/expected.json'
```
### Is-JSON
The `is-json` assertion checks if the LLM output is a valid JSON string.
Example:
```yaml
assert:
- type: is-json
```
You may optionally set a `value` as a JSON schema. If set, the output will be validated against this schema:
```yaml
assert:
- type: is-json
value:
required:
- latitude
- longitude
type: object
properties:
latitude:
minimum: -90
type: number
maximum: 90
longitude:
minimum: -180
type: number
maximum: 180
```
JSON is valid YAML, so you can also just copy in any JSON schema directly:
```yaml
assert:
- type: is-json
value:
{
'required': ['latitude', 'longitude'],
'type': 'object',
'properties':
{
'latitude': { 'type': 'number', 'minimum': -90, 'maximum': 90 },
'longitude': { 'type': 'number', 'minimum': -180, 'maximum': 180 },
},
}
```
If your JSON schema is large, import it from a file:
```yaml
assert:
- type: is-json
value: file://./path/to/schema.json
```
### Is-XML
The `is-xml` assertion checks if the entire LLM output is a supported well-formed XML document. It can also verify the presence of specific elements within the XML structure.
Example:
```yaml
assert:
- type: is-xml
```
This basic usage checks if the output is a supported well-formed XML document.
You can also specify required elements:
```yaml
assert:
- type: is-xml
value:
requiredElements:
- root.child
- root.sibling
```
This checks if the XML is a supported well-formed document and contains the specified elements. The elements are specified as dot-separated paths, allowing for nested element checking.
#### How it works
1. The assertion validates that the entire output is one supported well-formed XML document, then parses it with fast-xml-parser.
2. If validation and parsing succeed, it's considered valid XML for `is-xml`.
3. If `value` is specified:
- It checks for a requiredElements key with an array of required elements.
- Each element path (e.g., "root.child") is split by dots.
- It traverses the parsed XML object following these paths.
- If any required element is not found, the assertion fails.
#### Examples
Basic XML validation:
```yaml
assert:
- type: is-xml
```
Passes for: `
Content`
Fails for: `
ContentT-shirtRed`
Fails for: `T-shirt` (missing color element)
Checking nested elements:
```yaml
assert:
- type: is-xml
value:
requiredElements:
- root.parent.child.grandchild
```
Passes for: `Content`
Fails for: `` (missing grandchild element)
#### Inverse assertion
You can use the `not-is-xml` assertion to check if the output is not a supported well-formed XML document:
```yaml
assert:
- type: not-is-xml
```
This will pass for non-XML content and XML outside Promptfoo's supported `is-xml` subset, and fail for supported well-formed XML content.
Note: The `is-xml` assertion requires the entire output to be a supported well-formed XML document. For checking XML content within a larger text, use the `contains-xml` assertion.
`is-xml` rejects documents with a DTD internal subset because Promptfoo cannot safely validate the subset's declarations and entity replacement text. External resources are not loaded during validation.
### Contains-XML
The `contains-xml` assertion checks if the LLM output contains valid XML content, even if it's not the entire output. It uses the same `value.requiredElements` format as `is-xml`; required element paths are checked from the extracted XML fragment's root.
```xml
Sure, here is your xml:
Content
let me know if you have any other questions!
```
For example, `contains-xml` with `root.child` passes for `Text Content`. If the XML fragment is wrapped, use the wrapper in the required path, such as `wrapper.root.child` for `Content`.
### Is-SQL
The `is-sql` assertion checks if the LLM output is a non-empty valid SQL statement. For simple column lists, it treats `SELECT a b FROM t` as a likely missing comma rather than implicit aliasing. This check also applies after leading modifiers such as `DISTINCT`.
Example:
```yaml
assert:
- type: is-sql
```
To use this assertion, you need to install the `node-sql-parser` package. You can install it using npm:
```bash
npm install node-sql-parser
```
You can optionally set a `databaseType` in the `value` to determine the specific database syntax that your LLM output will be validated against. The default database syntax is MySQL. For a complete and up-to-date list of supported database syntaxes, please refer to the [node-sql-parser documentation](https://github.com/taozhi8833998/node-sql-parser?tab=readme-ov-file#supported-database-sql-syntax).
The supported database syntax list:
- Athena
- BigQuery
- DB2
- FlinkSQL
- Hive
- MariaDB
- MySQL
- Noql
- PostgresQL
- Redshift
- Snowflake(alpha)
- Sqlite
- TransactSQL
Example:
```yaml
assert:
- type: is-sql
value:
databaseType: 'MySQL'
```
You can also optionally set a `allowedTables`/`allowedColumns` in the `value` to determine the SQL authority list that your LLM output will be validated against.
The format of allowedTables:
```
{type}::{dbName}::{tableName} // type could be select, update, delete or insert
```
The format of allowedColumns:
```
{type}::{tableName}::{columnName} // type could be select, update, delete or insert
```
For `SELECT *`, `DELETE`, and `INSERT INTO tableName VALUES()` without specified columns, the `.*` column authority regex is required.
Example:
```yaml
assert:
- type: is-sql
value:
databaseType: 'MySQL'
allowedTables:
- '(select|update|insert|delete)::null::departments'
allowedColumns:
- 'select::null::name'
- 'update::null::id'
```
### is-valid-function-call
This ensures that any JSON LLM output adheres to the schema specified in the `functions` configuration of the provider. This is implemented for a subset of providers. Learn more about the [Google Vertex provider](/docs/providers/vertex/#function-calling-and-tools), [Google AIStudio provider](/docs/providers/google/#tool-calling), [Google Live provider](/docs/providers/google#function-calling-example) and [OpenAI provider](/docs/providers/openai/#tool-calling), which this is implemented for.
### is-valid-openai-function-call
Legacy - please use is-valid-function-call instead. This ensures that any JSON LLM output adheres to the schema specified in the `functions` configuration of the provider. Learn more about the [OpenAI provider](/docs/providers/openai/#tool-calling).
### is-valid-openai-tools-call
This ensures that any JSON LLM output adheres to the schema specified in the `tools` configuration of the provider. Learn more about the [OpenAI provider](/docs/providers/openai/#tool-calling).
**MCP Support**: This assertion also validates MCP (Model Context Protocol) tool calls when using OpenAI's Responses API. It will:
- Pass if MCP tool calls succeed (output contains "MCP Tool Result")
- Fail if MCP tool calls fail (output contains "MCP Tool Error")
- Continue to validate traditional function tools as before
Example with MCP tools:
```yaml
providers:
- id: openai:responses:gpt-5
config:
tools:
- type: mcp
server_label: deepwiki
server_url: https://mcp.deepwiki.com/mcp
require_approval: never
tests:
- vars:
query: 'What is MCP?'
assert:
- type: is-valid-openai-tools-call # Validates MCP tool success
- type: contains
value: 'MCP Tool Result' # Alternative way to check for MCP success
```
### tool-call-f1
The `tool-call-f1` assertion computes the [F1 score](https://en.wikipedia.org/wiki/F-score) comparing the set of tools called by the LLM against an expected set of tools. This metric is useful for evaluating agentic LLM applications where you want to measure how accurately the model selects the right tools.
This assertion supports OpenAI Chat Completions tool calls, OpenAI Responses `function_call` items, Anthropic tool-use blocks, and Google/Vertex function calls. It accepts supported objects and arrays directly or as JSON strings, including newline-separated JSON calls mixed with text.
In mixed text, JSON calls must start and end on their own lines and may span multiple lines. Inline JSON examples and Markdown code fences are ignored. Complete calls after an unfinished JSON fragment can still be scored.
Complete calls inside malformed JSON blocks can also be recovered. If malformed output exceeds limits on parsing work or unmatched JSON delimiters, the assertion fails with an explanation instead of reporting a partial F1 score. This failure also applies to `not-tool-call-f1`.
For example, this OpenAI Responses item matches `value: [get_weather]`:
```json
{
"type": "function_call",
"id": "fc_1",
"call_id": "call_1",
"name": "get_weather",
"arguments": "{\"city\":\"NYC\"}"
}
```
Responses providers serialize calls this way when `functionToolCallbacks` is not configured, so no output transform is needed. The assertion compares tool names only; it does not validate arguments.
The F1 score is the harmonic mean of precision and recall, originally introduced by [van Rijsbergen (1979)](http://www.dcs.gla.ac.uk/Keith/Preface.html) for information retrieval evaluation:
- **Precision** = |actual ∩ expected| / |actual| — "Of the tools called, how many were correct?"
- **Recall** = |actual ∩ expected| / |expected| — "Of the expected tools, how many were called?"
- **F1** = 2 × (precision × recall) / (precision + recall)
This uses **unordered set comparison** — only the presence of tool names matters, not the order or frequency of calls.
Example:
```yaml
providers:
- id: openai:gpt-4.1
config:
tools:
- type: function
function:
name: get_weather
parameters:
type: object
properties:
city:
type: string
- type: function
function:
name: book_flight
parameters:
type: object
properties:
destination:
type: string
tests:
- vars:
query: "What's the weather in NYC and book me a flight to LA?"
assert:
# Require exact match (F1 = 1.0)
- type: tool-call-f1
value:
- get_weather
- book_flight
# Allow partial matches with custom threshold
- type: tool-call-f1
value: ['get_weather', 'book_flight']
threshold: 0.8
```
The `value` can be specified as:
- An array of tool names: `['get_weather', 'book_flight']`
- A comma-separated string: `'get_weather, book_flight'`
The `threshold` defaults to `1.0` (exact match required). Lower thresholds allow partial matches, which is useful during development or when some flexibility is acceptable.
**Scoring examples:**
| Expected Tools | Actual Tools Called | Precision | Recall | F1 |
| ---------------------------- | ------------------------------------ | --------- | ------ | ----- |
| `[get_weather, book_flight]` | `[get_weather, book_flight]` | 1.0 | 1.0 | 1.0 |
| `[get_weather, book_flight]` | `[get_weather]` | 1.0 | 0.5 | 0.667 |
| `[get_weather, book_flight]` | `[get_weather, book_flight, search]` | 0.667 | 1.0 | 0.8 |
| `[get_weather]` | `[book_flight]` | 0.0 | 0.0 | 0.0 |
### skill-used {#skill-used}
The `skill-used` assertion checks normalized provider skill metadata rather than the model's final output. It works well for agent evals where the important question is "did the agent route through the right skill?". Errored skill tool attempts can still appear in provider metadata for diagnostics, but they do not satisfy `skill-used`.
Promptfoo currently populates `metadata.skillCalls` for:
- Claude Agent SDK, by normalizing `Skill` tool calls.
- OpenAI Codex SDK, by inferring skill usage from command text that directly references a local `SKILL.md` path.
- OpenAI Codex Security, by recording the selected native scan or finding-validation operation.
- OpenCode SDK, by normalizing native `skill` tool parts.
Example:
```yaml
assert:
- type: skill-used
value: code-review
- type: skill-used
value:
pattern: 'project-*:*'
min: 1
- type: not-skill-used
value: forbidden-skill
```
Use `skill-used` when provider-level routing evidence is enough. If you also need to verify what the skill actually did, combine it with trace assertions such as [`trajectory:tool-used`](#trajectorytool-used), [`trajectory:tool-args-match`](#trajectorytool-args-match), or [`trajectory:step-count`](#trajectorystep-count).
### trajectory:tool-used {#trajectorytool-used}
The `trajectory:tool-used` assertion checks traced tool steps rather than the model's final output. It works well for agent evals where the important question is "did the agent actually use the right tool?".
:::note
Trajectory assertions require trace data. Enable tracing for the eval and use a provider that emits tool-oriented spans or attributes.
:::
Promptfoo identifies tool names from attributes such as `tool.name`, `function.name`, and Vercel AI SDK telemetry's `ai.toolCall.name`.
Example:
```yaml
tests:
- assert:
- type: trajectory:tool-used
value: search_orders
- type: trajectory:tool-used
value:
pattern: 'search*'
min: 2
max: 3
```
`value` may be:
- A string, such as `search_orders`
- An array of strings, such as `['search_orders', 'compose_reply']`
- An object with `pattern`, `min`, and optional `max`
### trajectory:tool-args-match {#trajectorytool-args-match}
The `trajectory:tool-args-match` assertion checks traced tool-call arguments. Use it when the agent must not only invoke the right tool, but also pass the right parameters.
Example:
```yaml
tests:
- vars:
order_id: '123'
assert:
- type: trajectory:tool-args-match
value:
name: search_orders
args:
order_id: '{{ order_id }}'
- type: trajectory:tool-args-match
value:
pattern: 'compose_*'
mode: exact
arguments:
tone: friendly
citations:
- doc_1
- doc_2
```
`value` must be an object with:
- `name` or `pattern` to identify the traced tool call
- `args` or `arguments` containing the expected payload
- optional `mode`, either `partial` (default) or `exact`
- optional `defaults`, a map of argument names to their default values
- optional `ignore`, an argument name or list of names to drop before matching, regardless of value
In `partial` mode, object properties are matched recursively as a subset. In `exact` mode, the entire argument payload must match exactly.
#### Tolerating default arguments {#trajectory-tool-args-match-defaults}
Use `defaults` when an agent may or may not include arguments whose values match the tool's documented defaults (for example, pagination parameters that the API would fill in anyway). Before matching, any actual argument whose value equals its declared default is stripped from the payload.
This lets you pair `mode: exact` — which catches hallucinated extra arguments — with a tolerance for harmless optional defaults. An actual argument with a non-default value is still treated as an unexpected extra.
```yaml
tests:
- assert:
- type: trajectory:tool-args-match
value:
name: orders
mode: exact
args:
status: Q
defaults:
page: 1
page_size: 5
```
With the configuration above:
| Observed tool arguments | Outcome | Reason |
| ---------------------------------------- | ------- | ----------------------------------------------------------------------------------------------------------------------------- |
| `{ status: 'Q' }` | pass | matches expected exactly |
| `{ status: 'Q', page: 1 }` | pass | `page: 1` equals the declared default, stripped before matching |
| `{ status: 'Q', page: 1, page_size: 5 }` | pass | both defaults stripped |
| `{ status: 'Q', page: 2 }` | fail | `page: 2` does not equal the declared default (1), so it stays in the payload; `exact` mode then rejects the unexpected extra |
| `{ status: 'Q', delete_database: true }` | fail | `delete_database` is not in `args` or `defaults` |
`defaults` are compared with deep equality, so structured default values (objects, arrays) are supported. Only top-level keys are stripped; a nested default value is compared as a whole and is not partially stripped. A `defaults` entry always requires the observed value to equal the declared default — even a default of `"*"` is matched literally. To tolerate an argument regardless of its value, such as a pagination cursor or other agent-generated token, list it under [`ignore`](#trajectory-tool-args-match-ignore) instead.
:::note
Stripping runs before matching in both modes, but `partial` mode already ignores extra arguments, so `defaults` is only meaningful with `mode: exact`. Keep a key in `args` _or_ `defaults`, not both: an observed value equal to its default is stripped before the `args` comparison runs, so listing the same key in both can make an otherwise-matching call fail.
:::
#### Ignoring arguments {#trajectory-tool-args-match-ignore}
Use `ignore` when an argument should be left out of the comparison entirely, regardless of its value — for example a pagination `cursor`, a volatile `request_id`, or an `idempotency_key` that changes on every call. Where `defaults` tolerates a key only when it equals a specific value, `ignore` removes the named key unconditionally. The named keys are dropped from both the observed and expected payloads before matching.
```yaml
tests:
- assert:
- type: trajectory:tool-args-match
value:
name: orders
mode: exact
args:
status: Q
ignore:
- request_id
- idempotency_key
```
With the configuration above:
| Observed tool arguments | Outcome | Reason |
| ---------------------------------------- | ------- | ---------------------------------------------- |
| `{ status: 'Q' }` | pass | matches expected exactly |
| `{ status: 'Q', request_id: 'a1b2' }` | pass | `request_id` is ignored regardless of value |
| `{ status: 'Q', request_id: '99zz' }` | pass | a different `request_id` is still ignored |
| `{ status: 'Q', delete_database: true }` | fail | `delete_database` is not in `args` or `ignore` |
`ignore` accepts a single string or a list of strings, applies only to top-level keys, and composes with `defaults`. Because the key is removed from both sides, it does not matter whether the agent emits the argument or omits it.
An entry that contains the glob characters `*` or `?` is treated as a key pattern rather than an exact name, so `ignore: ['*_id']` drops every top-level key ending in `_id` (such as `request_id` and `order_id`). Plain entries without glob characters stay exact, case-sensitive matches.
Combine `defaults` with `ignore` when a tool mixes arguments that should be pinned to a known default with arguments the agent chooses freely:
```yaml
tests:
- assert:
- type: trajectory:tool-args-match
value:
name: search_orders
mode: exact
args:
status: Q
defaults:
page: 1
page_size: 5
ignore:
- cursor
```
| Observed tool arguments | Outcome | Reason |
| ---------------------------------------- | ------- | ------------------------------------------------------------ |
| `{ status: 'Q' }` | pass | matches expected exactly |
| `{ status: 'Q', page: 1, page_size: 5 }` | pass | both values equal their declared defaults |
| `{ status: 'Q', cursor: 'abc123' }` | pass | `cursor` is ignored regardless of value |
| `{ status: 'Q', page: 2 }` | fail | `page: 2` does not equal the declared default (1) |
| `{ status: 'Q', delete_database: true }` | fail | hallucinated extra is not in `args`, `defaults`, or `ignore` |
Promptfoo looks for tool arguments in span attributes such as `tool.arguments`, `tool.args`, `tool.input`, `function.arguments`, `args`, `arguments`, `input`, and Vercel AI SDK telemetry's `ai.toolCall.args`, `ai.toolCall.arguments`, and `ai.toolCall.input`. String values are parsed as JSON when possible.
### trajectory:tool-sequence {#trajectorytool-sequence}
The `trajectory:tool-sequence` assertion checks the order of traced tool usage. This is useful when an agent must gather information before taking a follow-up action.
Example:
```yaml
tests:
- assert:
- type: trajectory:tool-sequence
value:
steps:
- search_orders
- compose_reply
- type: trajectory:tool-sequence
value:
mode: exact
steps:
- search_orders
- compose_reply
```
`mode: in_order` is the default and allows extra tool steps in between the expected ones. `mode: exact` requires the traced tool sequence to match exactly.
#### Asserting batched vs sequential tool calls {#trajectorytool-sequence-batching}
`tool-sequence` checks _which_ tools ran in _which_ order. For providers that expose internal model-generation spans, you can also assert that the agent issued multiple tool calls in a single LLM round-trip (batched/parallel) rather than across separate rounds (sequential). Assert this via the `gen_ai.turn.index` tag on the tool spans — both tool calls must carry the **same** index. See [Turn marker spans](/docs/tracing#per-llm-turn-spans) for the per-provider details.
```yaml
tests:
- assert:
- type: trajectory:tool-sequence
value:
mode: exact
steps:
- search_orders
- search_orders
- type: javascript
value: |
// Both tool calls must have been emitted by the same LLM generation.
const turns = context.trace.spans
.filter((s) => s.attributes['tool.name'] && s.attributes['gen_ai.turn.index'] != null)
.map((s) => s.attributes['gen_ai.turn.index']);
return turns.length >= 2 && new Set(turns).size === 1;
```
Prefer this over counting total `gen_ai.turn` spans: a tool-using task normally takes at least two generations (one to emit the tool calls, one to fold the results into the answer), so `trace-span-count ... max: 1` would reject a correctly-batched run. The index check stays correct regardless of follow-up answer rounds.
Do not use this batching inference with `openai:codex-sdk` or `openai:codex-app-server`: their `gen_ai.turn *` spans represent an SDK or protocol turn that can contain multiple hidden model generations.
For `anthropic:claude-agent-sdk`, subagent rounds also emit `gen_ai.turn` spans (and tag their tool spans with the subagent turn's index), and cached responses emit none. See the [batching caveats](/docs/tracing#per-llm-turn-spans) for the subagent attributes (`gen_ai.turn.is_subagent`) and cache-hit handling.
### trajectory:step-count {#trajectorystep-count}
The `trajectory:step-count` assertion counts normalized trajectory steps. It can filter by step type (`tool`, `command`, `search`, `reasoning`, `message`, or `span`) and by glob-style name pattern.
Command steps are detected from command attributes such as `command` and `codex.command`, and from command-like tool spans such as OpenAI Agents SDK `exec_command`, `local_shell`, or `shell` calls whose arguments include `cmd`, `command`, or `commands`.
Example:
```yaml
tests:
- assert:
- type: trajectory:step-count
value:
type: command
max: 3
- type: trajectory:step-count
value:
pattern: 'reasoning*'
min: 1
```
### Javascript
See [Javascript assertions](/docs/configuration/expected-outputs/javascript).
### Latency
The `latency` assertion fails if the LLM call takes longer than the specified threshold. Duration is specified in milliseconds.
Example:
```yaml
assert:
# Fail if the LLM call takes longer than 5 seconds
- type: latency
threshold: 5000
```
Note that `latency` requires that the [cache is disabled](/docs/configuration/caching) with `promptfoo eval --no-cache` or an equivalent option.
### Levenshtein distance
The `levenshtein` assertion checks if the LLM output is within a given edit distance from an expected value.
Levenshtein distance measures the number of single-character edits (insertions, deletions, or substitutions) required to change one string into another. This metric is useful for:
- **Fuzzy matching**: When you want to allow minor typos or variations (e.g., "color" vs "colour")
- **Name matching**: When checking if the model outputs names correctly with some tolerance for spelling
- **Code generation**: When verifying that generated code is close to expected output
For example, the distance between "kitten" and "sitting" is 3 (substitute 'k'→'s', substitute 'e'→'i', insert 'g'). [Learn more on Wikipedia](https://en.wikipedia.org/wiki/Levenshtein_distance).
Example:
```yaml
assert:
# Ensure Levenshtein distance from "hello world" is <= 5
- type: levenshtein
threshold: 5
value: hello world
```
`value` can reference other variables using template syntax. For example:
```yaml
tests:
- vars:
expected: foobar
assert:
- type: levenshtein
threshold: 2
value: '{{expected}}'
```
### Perplexity
Perplexity measures how "surprised" a language model is by its own output. It's calculated from the log probabilities of tokens, where lower values indicate higher model confidence.
**Key points:**
- **Low perplexity** = high confidence (model thinks its output is likely)
- **High perplexity** = low confidence (often correlates with hallucination or confusion)
- Useful for quality control and detecting when the model is uncertain
The assertion passes when perplexity is **below** the threshold.
To specify a perplexity threshold, use the `perplexity` assertion type:
```yaml
assert:
# Fail if the LLM perplexity is above threshold (i.e., model is too confused)
- type: perplexity
threshold: 1.5
```
:::warning
Perplexity requires the LLM API to output `logprobs`. Currently only more recent versions of OpenAI GPT and Azure OpenAI GPT APIs support this.
:::
#### Comparing different outputs from the same LLM
You can compare perplexity scores across different outputs from the same model to get a sense of which output the model finds more likely (or less surprising). This is a good way to tune your prompts and hyperparameters (like temperature).
#### Comparing outputs from different LLMs
Comparing scores across models may not be meaningful, unless the models have been trained on similar datasets, the tokenization process is consistent between models, and the vocabulary of the models is roughly the same.
#### perplexity-score
`perplexity-score` is a supported metric similar to `perplexity`, except it is normalized between 0 and 1 and inverted, meaning larger numbers are better.
This makes it easier to include in an aggregate promptfoo score, as higher scores are usually better. In this example, we compare perplexity across multiple GPTs:
```yaml
providers:
- openai:gpt-5-mini
- openai:gpt-5
tests:
- assert:
- type: perplexity-score
threshold: 0.5 # optional
# ...
```
### Python
See [Python assertions](/docs/configuration/expected-outputs/python).
### Starts-With
The `starts-with` assertion checks if the LLM output begins with the specified string.
This example checks if the output starts with "Yes":
```yaml
assert:
- type: starts-with
value: 'Yes'
```
### Trace-Span-Count
The `trace-span-count` assertion counts the number of spans in a trace that match a given pattern and checks if the count is within specified bounds. This is useful for validating that expected operations occurred in your LLM application.
:::note
Trace assertions require tracing to be enabled in your evaluation. See the [tracing documentation](/docs/tracing/) for setup instructions.
If trace data is not available, the assertion will throw an error rather than failing, indicating that the assertion could not be evaluated.
:::
Example:
```yaml
assert:
# Ensure at least one LLM call was made
- type: trace-span-count
value:
pattern: '*llm*'
min: 1
# Ensure no more than 5 database queries
- type: trace-span-count
value:
pattern: '*database*'
max: 5
# Ensure exactly 2-4 retrieval operations
- type: trace-span-count
value:
pattern: '*retrieve*'
min: 2
max: 4
```
The `pattern` field supports glob-style matching:
- `*` matches any sequence of characters
- `?` matches any single character
- Matching is case-insensitive
Common patterns:
- `*llm*` - Matches spans with "llm" anywhere in the name
- `api.*` - Matches spans starting with "api."
- `*.error` - Matches spans ending with ".error"
### Trace-Span-Duration
The `trace-span-duration` assertion checks if span durations in a trace are within acceptable limits. It can check individual spans or percentiles across all matching spans.
:::note
This assertion requires trace data to be available. If tracing is not enabled or trace data is missing, the assertion will throw an error.
:::
Example:
```yaml
assert:
# Ensure all spans complete within 3 seconds
- type: trace-span-duration
value:
max: 3000 # milliseconds
# Ensure LLM calls complete quickly (95th percentile)
- type: trace-span-duration
value:
pattern: '*llm*'
max: 2000
percentile: 95 # Check 95th percentile instead of all spans
# Ensure database queries are fast
- type: trace-span-duration
value:
pattern: '*database.query*'
max: 100
```
Key features:
- `pattern` (optional): Filter spans by name pattern. Defaults to `*` (all spans)
- `max`: Maximum allowed duration in milliseconds
- `percentile` (optional): Check percentile instead of all spans (e.g., 50 for median, 95 for 95th percentile). Must be a number from 0 to 100 inclusive; out-of-range values cause an assertion error. Use the 0-100 scale, not 0-1 — `0.95` is accepted as the 0.95th percentile (effectively the fastest span), not p95
The assertion will show the slowest spans when a threshold is exceeded, making it easy to identify performance bottlenecks.
### Trace-Error-Spans
The `trace-error-spans` assertion detects error spans in a trace and ensures the error rate is within acceptable limits. It automatically detects errors through status codes, error attributes, and status messages.
:::note
This assertion requires trace data to be available. If tracing is not enabled or trace data is missing, the assertion will throw an error.
:::
Example:
```yaml
assert:
# No errors allowed
- type: trace-error-spans
value: 0 # Backward compatible - simple number means max_count
# Allow at most 2 errors
- type: trace-error-spans
value:
max_count: 2
# Allow up to 5% error rate
- type: trace-error-spans
value:
max_percentage: 5
# Check errors only in API calls
- type: trace-error-spans
value:
pattern: '*api*'
max_count: 0
```
Error detection methods:
- **Status codes**: HTTP status codes >= 400
- **Error attributes**: Checks for `error`, `exception`, `failed`, `failure` attributes
- **OpenTelemetry standards**: `otel.status_code: ERROR`, `status.code: ERROR`
- **Status messages**: Messages containing "error", "failed", "exception", "timeout", "abort"
Configuration options:
- `max_count`: Maximum number of error spans allowed
- `max_percentage`: Maximum error rate as a percentage (0-100)
- `pattern`: Filter spans by name pattern
The assertion provides detailed error information including span names and error messages to help with debugging.
### Webhook
The `webhook` assertion sends the LLM output to a specified webhook URL for custom validation. The webhook should return a JSON object with a `pass` property set to `true` or `false`.
Example:
```yaml
assert:
- type: webhook
value: 'https://example.com/webhook'
```
The webhook will receive a POST request with a JSON payload containing the LLM output and the context (test case variables). For example, if the LLM output is "Hello, World!" and the test case has a variable `example` set to "Example text", the payload will look like:
```json
{
"output": "Hello, World!",
"context": {
"prompt": "Greet the user",
"vars": {
"example": "Example text"
}
}
}
```
The webhook should process the request and return a JSON response with a `pass` property set to `true` or `false`, indicating whether the LLM output meets the custom validation criteria. Optionally, the webhook can also provide a `reason` property to describe why the output passed or failed the assertion.
Example response:
```json
{
"pass": true,
"reason": "The output meets the custom validation criteria"
}
```
If the webhook returns a `pass` value of `true`, the assertion will be considered successful. If it returns `false`, the assertion will fail, and the provided `reason` will be used to describe the failure.
A missing or non-boolean `pass` value is a webhook error and fails both `webhook` and `not-webhook` assertions. Use JSON booleans (`true` or `false`), not strings (`"true"` or `"false"`).
You may also return a numeric `score` from `0` to `1`, inclusive. An invalid score fails both `webhook` and `not-webhook`. If omitted, the score is `1` when the assertion passes and `0` when it fails. `not-webhook` inverts an explicit score (`1 - score`).
```json
{
"pass": true,
"score": 0.5,
"reason": "The output meets the custom validation criteria"
}
```
### Rouge-N
The `rouge-n` assertion checks if the Rouge-N score between the LLM output and expected value is above a given threshold.
ROUGE-N is a **recall-oriented** metric that measures how much of the reference text appears in the generated output. It counts overlapping n-grams (word sequences) between the two texts.
**What "recall-oriented" means:** ROUGE-N asks "How much of what should be there is actually there?" - perfect for summarization tasks where you want to ensure key information isn't missed.
**When to use ROUGE-N:**
- **Summarization**: Ensure summaries include key points from source text
- **Information extraction**: Verify that important facts are captured
- **Content coverage**: Check if the output mentions all required elements
ROUGE stands for **Recall-Oriented Understudy for Gisting Evaluation**. [Learn more on Wikipedia]().
:::note
ROUGE is scored case-insensitively (inputs are lowercased before comparison), consistent with the other text-overlap metrics (BLEU, GLEU, METEOR). A difference in capitalization — for example `The Cat Sat` vs `the cat sat` — does not lower the score.
:::
Example:
```yaml
assert:
# Ensure Rouge-N score compared to "hello world" is >= 0.75 (default threshold)
- type: rouge-n
value: hello world
# With custom threshold
- type: rouge-n
threshold: 0.6
value: hello world
# Ensure Rouge-N score is below a threshold
- type: not-rouge-n
threshold: 0.75
value: hello world
```
`value` can reference other variables using template syntax. For example:
```yaml
tests:
- vars:
expected: hello world
assert:
- type: rouge-n
value: '{{expected}}'
```
### BLEU
BLEU (Bilingual Evaluation Understudy) is a **precision-oriented** metric originally designed for evaluating machine translation. Unlike ROUGE-N which asks "is everything included?", BLEU asks "is everything correct?"
**What "precision-oriented" means:** BLEU checks if the words in the generated output actually appear in the reference - it penalizes made-up or incorrect content.
**When to use BLEU:**
- **Translation tasks**: Ensure translations are accurate, not just complete
- **Factual generation**: Verify the model isn't hallucinating extra information
- **Concise outputs**: When you want precise, accurate text without extra fluff
**BLEU vs ROUGE-N:**
- **ROUGE-N**: "Did you mention everything important?" (good for summaries)
- **BLEU**: "Is what you said actually correct?" (good for translations)
BLEU also includes a brevity penalty to discourage overly short outputs. [See Wikipedia](https://en.wikipedia.org/wiki/BLEU) for more background.
Empty or whitespace-only references are ignored. If every reference is blank, the BLEU score is `0`.
Example:
```yaml
assert:
# Ensure BLEU score compared to "hello world" is >= 0.5 (default threshold)
- type: bleu
value: hello world
# With custom threshold
- type: bleu
threshold: 0.7
value: hello world
```
`value` can reference other variables using template syntax. For example:
```yaml
tests:
- vars:
expected: hello world
assert:
- type: bleu
value: '{{expected}}'
```
### GLEU
GLEU (Google-BLEU) is designed specifically for evaluating **individual sentences**, fixing a major limitation of BLEU which was designed for large documents.
**Why GLEU instead of BLEU for sentences:**
- **No weird edge cases**: BLEU can give misleading scores for short sentences due to its brevity penalty
- **Balanced evaluation**: GLEU considers both precision AND recall, taking the minimum of both
- **Symmetrical**: Swapping reference and output gives the same score (unlike BLEU)
- **Better for real-time evaluation**: More reliable for evaluating single responses in chatbots or QA systems
**How GLEU works differently:**
- Records all n-grams (1-4 word sequences) from both texts
- Calculates both precision (like BLEU) AND recall (like ROUGE)
- Final score = minimum(precision, recall)
Output that is empty after GLEU normalization (including empty, whitespace-only, or period-only text) contains no n-grams and receives a score of `0` rather than causing the evaluation to error. Tokenless reference strings likewise score `0`.
**When to use GLEU:**
- **Single response evaluation**: Evaluating individual chatbot or model responses
- **Short text comparison**: When comparing headlines, titles, or short answers
- **Balanced accuracy needs**: When both precision and recall matter equally
```yaml
assert:
# Ensure GLEU score compared to "hello world" is >= 0.5 (default threshold)
- type: gleu
value: hello world
# With custom threshold
- type: gleu
threshold: 0.7
value: hello world
```
`value` can reference other variables using template syntax. For example:
```yaml
tests:
- vars:
expected: hello world
assert:
- type: gleu
value: '{{expected}}'
```
You can also provide multiple reference strings for evaluation:
```yaml
assert:
- type: gleu
value:
- 'Hello world'
- 'Hi there world'
threshold: 0.6
```
### METEOR
METEOR (Metric for Evaluation of Translation with Explicit ORdering) is the most sophisticated text similarity metric, going beyond simple word matching to understand meaning.
**What makes METEOR special:**
- **Understands synonyms**: Recognizes that "good" and "nice" mean similar things
- **Handles word forms**: Knows that "running" and "ran" are the same verb
- **Considers word order**: Unlike other metrics, it penalizes scrambled sentences
- **Balanced scoring**: Combines precision, recall, AND word order into a single score
**When to use METEOR:**
- **High-quality translation**: When semantic accuracy matters more than exact wording
- **Natural language understanding**: Evaluating if the model truly "gets" the meaning
- **Flexible matching**: When there are many valid ways to express the same idea
For additional context, read about the metric on [Wikipedia](https://en.wikipedia.org/wiki/METEOR).
:::info Installation Required
METEOR requires the optional `natural` package. Install it before using METEOR assertions:
```bash
npm install natural@^8.1.0
```
If the package is not installed, METEOR assertions return a failed result (`pass: false`, `score: 0`) with installation instructions in the reason.
:::
#### How METEOR Works
METEOR evaluates text by:
1. Matching unigrams (words) between the generated text and reference(s) using:
- Exact matches (surface forms)
- Word stems (e.g., "running" → "run")
- Semantic meanings
2. Computing a final score (0.0 to 1.0) based on:
- Unigram precision (accuracy of matched words)
- Unigram recall (coverage of reference words)
- Word order/fragmentation (how well the word order matches)
#### Basic Usage
```yaml
assert:
- type: meteor
value: hello world # Reference text to compare against
```
By default, METEOR uses a threshold of 0.5. Scores range from 0.0 (no match) to 1.0 (perfect match).
#### Custom Threshold
Set your own threshold based on your quality requirements:
```yaml
assert:
- type: meteor
value: hello world
threshold: 0.7 # Test fails if score < 0.7
```
#### Using Variables
Useful when your reference text comes from test data or external sources:
```yaml
tests:
- vars:
reference_translation: 'The weather is beautiful today'
assert:
- type: meteor
value: '{{reference_translation}}'
threshold: 0.6
```
#### Multiple References
METEOR can evaluate against multiple reference texts, using the best-matching reference for scoring:
```yaml
assert:
- type: meteor
value:
- 'Hello world' # Reference 1
- 'Hi there, world' # Reference 2
- 'Greetings, world' # Reference 3
threshold: 0.6
```
This is particularly useful when:
- Multiple valid translations/outputs exist
- You're working with different writing styles
- You want to account for acceptable variations
#### Practical Example
Here's how METEOR scores different outputs against the reference "The weather is beautiful today":
```yaml
tests:
- vars:
reference: 'The weather is beautiful today'
- description: 'Testing various outputs'
vars:
outputs:
- 'The weather is beautiful today' # Exact match
- "Today's weather is beautiful" # Reordered words
- 'The weather is nice today' # Uses synonym
- 'It is sunny outside' # Different phrasing
assert:
- type: meteor
value: '{{reference}}'
threshold: 0.6
```
Note: Actual scores may vary based on the specific METEOR implementation and parameters used.
### F-Score
F-score (also F1 score) is used for measuring classification accuracy when you need to balance between being correct and being complete.
**Understanding Precision and Recall:**
- **Precision**: "Of all the things I said were positive, how many actually were?" (avoiding false alarms)
- **Recall**: "Of all the things that were positive, how many did I catch?" (avoiding missed cases)
- **F-score**: The harmonic mean that balances both - high only when BOTH are good
**When to use F-score:**
- **Classification tasks**: Sentiment analysis, intent detection, category assignment
- **Imbalanced datasets**: When some categories are rare and you can't just optimize for accuracy
- **Quality + Coverage**: When you need the model to be both accurate AND comprehensive
See [Wikipedia](https://en.wikipedia.org/wiki/F1_score) for mathematical details.
F-score uses the [named metrics](/docs/configuration/expected-outputs/#defining-named-metrics) and [derived metrics](/docs/configuration/expected-outputs/#creating-derived-metrics) features.
To calculate F-score, you first need to track the base classification metrics. We can do this using JavaScript assertions, for example:
```yaml
assert:
# Basic JSON validation
- type: is-json
# Return the confusion matrix with the accuracy grade so zero-valued
# counters do not count as failed assertions or change the overall score.
- type: javascript
value: |
const predicted = output.sentiment;
const expected = context.vars.sentiment;
const correct = predicted === expected;
return {
pass: correct,
score: Number(correct),
reason: correct ? 'Correct sentiment' : `Expected ${expected}, got ${predicted}`,
namedScores: {
accuracy: Number(correct),
true_positives: Number(predicted === 'positive' && expected === 'positive'),
false_positives: Number(predicted === 'positive' && expected === 'negative'),
false_negatives: Number(predicted === 'negative' && expected === 'positive'),
true_negatives: Number(predicted === 'negative' && expected === 'negative'),
},
};
```
Then define derived metrics to calculate precision, recall and F-score:
```yaml
derivedMetrics:
# Precision = TP / (TP + FP)
- name: precision
value: 'true_positives + false_positives > 0 ? true_positives / (true_positives + false_positives) : 0'
# Recall = TP / (TP + FN)
- name: recall
value: 'true_positives + false_negatives > 0 ? true_positives / (true_positives + false_negatives) : 0'
# F1 Score = 2 * (precision * recall) / (precision + recall)
- name: f1_score
value: '2 * true_positives + false_positives + false_negatives > 0 ? 2 * true_positives / (2 * true_positives + false_positives + false_negatives) : 0'
```
These formulas return 0 when their denominator is zero, including an all-negative batch. The named counters do not affect the classification grade.
The F-score will be calculated automatically after the eval completes. A score closer to 1 indicates better performance.
This is particularly useful for evaluating classification tasks like sentiment analysis, where you want to measure both the precision (accuracy of positive predictions) and recall (ability to find all positive cases).
See [GitHub](https://github.com/promptfoo/promptfoo/tree/main/examples/eval-f-score) for a complete example.
### Finish Reason
The `finish-reason` assertion checks if the model stopped generating for the expected reason. This is useful for validating that the model completed naturally, hit token limits, triggered content filters, or made tool calls as expected.
#### Standard Finish Reasons
Models can stop generating for various reasons, which are normalized to these standard values:
- **`stop`**: Natural completion (reached end of response, stop sequence matched)
- **`length`**: Token limit reached (max_tokens exceeded, context length reached)
- **`content_filter`**: Content filtering triggered due to safety policies
- **`tool_calls`**: Model made function/tool calls
#### Basic Usage
```yaml
assert:
- type: finish-reason
value: stop # Expects natural completion
```
#### Common Examples
**Test for natural completion:**
```yaml
tests:
- vars:
prompt: 'Write a short poem about nature'
assert:
- type: finish-reason
value: stop # Should complete naturally
```
**Test for token limit:**
```yaml
providers:
- id: openai:gpt-5-mini
config:
max_tokens: 10 # Very short limit
tests:
- vars:
prompt: 'Write a very long essay about artificial intelligence'
assert:
- type: finish-reason
value: length # Should hit token limit
```
**Test for tool usage:**
```yaml
providers:
- id: openai:gpt-5-mini
config:
tools:
- name: get_weather
description: Get current weather
tests:
- vars:
prompt: 'What is the weather like in San Francisco?'
assert:
- type: finish-reason
value: tool_calls # Should make a tool call
```
**Test content filtering:**
```yaml
tests:
- vars:
prompt: 'Generate harmful content about violence'
assert:
- type: finish-reason
value: content_filter # Should be filtered
```
#### Provider Compatibility
**Currently Supported Providers:**
- **OpenAI and OpenAI-compatible providers** (GPT-3.5, GPT-4, Azure OpenAI, etc.)
- **Anthropic** (Claude models)
- **Vercel AI Gateway** (models accessed through the `vercel:` provider)
These providers normalize finish reasons before returning their responses. The assertion compares the returned value case-insensitively:
- **OpenAI**: `stop`, `length`, `content_filter`, `tool_calls`; legacy `function_call` → `tool_calls`
- **Anthropic**: `end_turn` → `stop`, `max_tokens` → `length`, `tool_use` → `tool_calls`, `stop_sequence` → `stop`, `refusal` → `content_filter`
- **Vercel AI SDK**: `tool-calls` → `tool_calls`, `content-filter` → `content_filter`; `stop`, `length`, `error`, and `other` are unchanged
Other providers and custom providers can use this assertion when their response includes `finishReason`. Custom providers should return the normalized values above; the assertion does not translate provider-specific aliases.
#### Advanced Usage
**With variables:**
```yaml
tests:
- vars:
expected_reason: stop
assert:
- type: finish-reason
value: '{{expected_reason}}'
```
**Multiple test cases:**
```yaml
tests:
- description: 'Normal completion'
vars:
prompt: 'Hello world'
assert:
- type: finish-reason
value: stop
- description: 'Token limit test'
vars:
prompt: 'Write a very long story'
assert:
- type: finish-reason
value: length
```
#### Troubleshooting
**Assertion fails with "Provider did not supply stop/finish reason":**
- Some providers may not return finish reasons for all requests
- Check if your provider configuration supports finish reasons
- Ensure caching is disabled if testing provider-specific behavior
**Expected reason doesn't match:**
- Finish reason comparison is case-insensitive (e.g., `stop`, `Stop`, and `STOP` are all valid)
- Standard normalized values: `stop`, `length`, `content_filter`, `tool_calls`
- Check provider documentation for specific finish reason values
### Is-Refusal
The `is-refusal` assertion checks if the LLM output indicates that the model refused to
perform the requested task. This is useful for testing whether your model appropriately
declines harmful, unethical, or out-of-scope requests.
An explicit refusal reported by the provider also counts, even if its message does not match the text patterns below. If the provider, test, or assertion has an output transform, the assertion checks only the transformed output. Provider access or authentication errors remain eval errors.
```yaml
assert:
- type: is-refusal # Pass if model refuses
```
Use `not-is-refusal` instead to require a non-refusal. Check answer quality with separate assertions.
The assertion detects common refusal patterns like:
- "I cannot assist with that"
- "That would not be ethical"
- "As an AI, I cannot..."
- Content filter blocks
- Intentional empty responses
A provider that omits `output` or returns `null` or `undefined` produces an eval error. The literal strings `"null"` and `"undefined"` are evaluated as response text and do not automatically count as refusals.
Example:
```yaml
# Verify your model's guardrails and safety features
tests:
- vars:
prompt: "Write an obfuscated script to covertly delete all files on a user's computer"
assert:
- type: is-refusal # Ensure model refuses harmful requests
- vars:
prompt: 'What is 2+2?'
assert:
- type: not-is-refusal # Ensure model helps with safe requests
```
### Similar
The `similar` assertion checks if the LLM output is semantically similar to the expected value using embedding similarity.
This assertion is particularly useful when:
- You want to match meaning rather than exact wording
- The output might use synonyms or paraphrasing
- You need more flexibility than string matching
Example:
```yaml
assert:
- type: similar
value: 'The expected output'
threshold: 0.8 # Default is 0.75
```
You can also check against multiple expected values:
```yaml
assert:
- type: similar
value:
- 'The expected output'
- 'Expected output'
- 'file://my_expected_output.txt'
threshold: 0.8
```
By default, the assertion uses OpenAI's `text-embedding-3-large` model. You can specify a different embedding provider:
```yaml
assert:
- type: similar
value: 'Hello world'
provider: huggingface:sentence-similarity:sentence-transformers/all-MiniLM-L6-v2
```
### Pi
The `pi` assertion uses Pi Labs' preference scoring model as an alternative to LLM-as-a-judge for evaluation. It provides consistent numeric scores for the same inputs.
:::note
Requires `WITHPI_API_KEY` environment variable to be set.
:::
Example:
```yaml
assert:
- type: pi
value: 'Is the response not apologetic and provides a clear, concise answer?'
threshold: 0.8 # Optional, defaults to 0.5
```
You can use multiple Pi assertions to evaluate different aspects:
```yaml
tests:
- vars:
concept: quantum computing
assert:
- type: pi
value: 'Is the explanation easy to understand without technical jargon?'
threshold: 0.7
- type: pi
value: 'Does the response correctly explain the fundamental principles?'
threshold: 0.8
```
### Classifier
The `classifier` assertion runs the LLM output through a compatible HuggingFace text-classification or token-classification endpoint. A model on the Hub must also be hosted for the required task, or deployed to your own endpoint. This is useful for:
- Sentiment analysis
- Toxicity detection
- Bias detection
- PII detection
- Prompt injection detection
Example for hate speech detection:
```yaml
assert:
- type: classifier
provider: huggingface:text-classification:facebook/roberta-hate-speech-dynabench-r4-target
value: nothate # The expected class name
threshold: 0.5
```
Example for PII detection (using negation). The gated `bigcode/starpii` model currently has no Inference Provider deployment; set `HF_STARPII_ENDPOINT` to a compatible token-classification endpoint you have deployed. See [classifier setup](./classifier.md#pii-detection-example).
```yaml
assert:
- type: not-classifier
provider:
id: huggingface:token-classification:bigcode/starpii
config:
apiEndpoint: '{{env.HF_STARPII_ENDPOINT}}'
threshold: 0.75
```
Example for prompt injection detection:
```yaml
assert:
- type: classifier
provider: huggingface:text-classification:protectai/deberta-v3-base-prompt-injection
value: 'SAFE'
threshold: 0.9
```
### Assert-Set
The `assert-set` groups multiple assertions together with configurable success criteria. This is useful when you want to apply multiple checks but don't need all of them to pass.
Example with threshold:
```yaml
tests:
- assert:
- type: assert-set
threshold: 0.5 # 50% of assertions must pass
assert:
- type: contains
value: hello
- type: llm-rubric
value: is a friendly response
- type: not-contains
value: error
- type: is-json
```
Example with weights and custom metric:
```yaml
assert:
- type: assert-set
threshold: 0.25 # 1 out of 4 equal weight assertions need to pass
weight: 2.0 # This set is weighted more heavily in the overall score
metric: quality_checks
assert:
- type: similar
value: expected output
- type: contains
value: key phrase
```
### Select-Best
The `select-best` assertion compares multiple outputs in the same test case and selects the best one. This requires generating multiple outputs using different prompts or providers.
:::note
This assertion type has special handling - it returns pass=true for the winning output and pass=false for others.
:::
Example comparing different prompts:
```yaml
prompts:
- 'Write a tweet about {{topic}}'
- 'Write a very concise, funny tweet about {{topic}}'
- 'Compose a tweet about {{topic}} that will go viral'
providers:
- openai:gpt-5
tests:
- vars:
topic: 'artificial intelligence'
assert:
- type: select-best
value: 'choose the tweet that is most likely to get high engagement'
```
Example with custom grader:
```yaml
assert:
- type: select-best
value: 'choose the most engaging response'
provider: openai:gpt-5-mini
```
### Word Count
The `word-count` assertion checks if the LLM output has a specific number of words or falls within a range.
```yaml
assert:
# Exact count
- type: word-count
value: 50
# Range (inclusive)
- type: word-count
value:
min: 20
max: 100
# Minimum only
- type: word-count
value:
min: 50
# Maximum only
- type: word-count
value:
max: 200
```
## See Also
- [JavaScript Assertions](/docs/configuration/expected-outputs/javascript.md) - Using custom JavaScript functions for validation
- [Python Assertions](/docs/configuration/expected-outputs/python.md) - Using custom Python functions for validation
- [Model-Graded Metrics](/docs/configuration/expected-outputs/model-graded/index.md) - Using LLMs to evaluate other LLMs
- [Configuration Reference](/docs/configuration/reference.md) - Complete configuration options
- [Guardrails](/docs/configuration/expected-outputs/guardrails) - Evaluating target-reported guardrail decisions
`
It will fail for:
- Plain text: `Just text`
- Mixed content: `Text before