Interpreting eval results

Use eval results to compare models, prompts, and configurations for your use case.

Access results

To open a configuration comparison, follow these steps:

1. On the Evals page, click Runs.
2. Find the group of eval runs that you want to compare.
3. Click Compare configs.

To inspect a suite run’s grader results, click the run row instead. The dashboard opens its run results page.

Results overview

The comparison view shows aggregate metrics for each eval configuration. Suite-linked runs also show a Score column with the suite score and passed-case count. The following table describes the available metrics:

Key metrics

Success rate

Use Success Rate to compare the percentage of recorded steps that finish with output. Use these checks when you review the metric:

• Compare the rate across configurations.
• Investigate failed steps before you deploy a configuration.

Latency

Use Avg Duration to compare the average duration per Record. Use these checks when you review the metric:

• Compare duration with output quality and success rate.
• Choose the shorter duration when configurations produce comparable output.
• Review the detailed step results when one configuration takes longer.

Cost

Use Total Cost and Avg Cost/Record to compare estimated spending. Use these checks when you review the metrics:

• Compare average cost when evals use different numbers of Records.
• Check whether a lower-cost configuration meets your quality target.
• Multiply average cost by planned Record volume to estimate spend.

Token usage

Use token counts to find configurations that consume more input or output tokens. The Baseline Comparison section includes input and output totals when available. The aggregate table shows output tokens.

Detailed comparison

Use the comparison view to inspect aggregate, Record-level, and step-level results:

1. Open the Summary tab to review the winner cards, aggregate metrics, and baseline comparison.
2. Open the Detail tab to compare the model, duration, cost, and output for each Record and step.
3. Filter the results by Record or Step Type to narrow the view.
4. Expand a step to inspect its input configuration and Record data.
5. Click Export CSV to download the detailed results.

The Summary tab shows cards for Highest Success Rate, Cheapest, and Fastest. The aggregate table marks configurations that lead on one of those metrics.

Interpreting results

Clear winner

If one configuration leads on success rate and meets your latency and cost targets, update your Flow with that configuration.

Cost tradeoff

If a higher-quality model costs more, compare its output with a lower-cost model on the Records that matter. Choose the higher-cost model only when the quality difference supports your use case.

Inconsistent performance

If a configuration succeeds on some Records and fails on others, use Detail to identify the affected steps and input patterns. Update the prompt or configuration, then run the eval again.

Similar metrics

If configurations have similar metrics, compare their outputs. Then choose the configuration that meets your quality target at the lower average cost or shorter average duration.

Keyword analysis

Use keyword analysis to count literal keyword or phrase occurrences in step outputs across a group of evals. Enter search terms and review total occurrences, the number of steps containing each term, and the occurrence rate. Enable case-sensitive matching when that distinction matters.

Common patterns

A lower-cost model meets the quality target

When a lower-cost configuration matches the baseline success rate, compare its outputs on the Records that matter. Use the lower-cost configuration when it meets your quality target.

The prompt affects output quality

When the same model produces different results with different prompts, improve the prompt before you switch models.

Temperature affects consistency

Compare temperature settings with the same model and prompt. Test a lower temperature for classification or extraction when you need more consistent output.

Act on results

After you choose a configuration, apply it and verify the change:

1. Update your Flow with the selected model, prompt, and settings.
2. Run the eval again to confirm that the configuration meets your target.
3. Monitor production logs after deployment to compare real-world behavior.

Next steps

Continue with these guides:

• Running an eval: run configuration comparisons.
• What are Evals?: understand evals, configurations, and metrics.
• Working with logs: review production executions after deployment.