Compare two AI responses against the same evidence

Give both responses the same job

A fair comparison begins with the same task, source material and output requirements. If one response received extra context, record that difference before interpreting the scores. Keep a copy of the task as well as the answers. Otherwise it becomes difficult to tell whether a missing item was a failure or simply never requested.

Decide what you will rate

Choose criteria that describe observable qualities. Source fidelity asks whether statements match the supplied facts. Coverage asks whether requested items appear. Structure asks whether the output follows the requested format. Usefulness considers whether the result suits the intended reader and makes uncertainty visible. These questions do different work; fluent prose should not compensate for incorrect numbers.

Use weights deliberately

Weights express the importance you assign to each criterion. For an extraction task, factual fidelity may matter more than presentation. Set the weights before comparing the responses when possible. If you change them later, explain why. A weighted score is a summary of your choices, not an objective measurement of a model’s overall quality.

Record evidence beside the rating

For each response, quote the passage that supports a rating or identify the missing requirement. A score of two without a reason is hard to revisit. “Invents a launch date that is absent from the source” gives another reviewer something to inspect. If you disagree with a colleague, return to the task and evidence before adjusting numbers.

Keep missing ratings visible

An unanswered rating is not a zero. Zero means you inspected a criterion and judged it to fail; blank means it has not been rated. Keep those states separate in your notes. Only compare completed ratings, and show how many responses you checked so a small sample does not look like a broad evaluation.

Look at the cases behind the mean

Two averages can be close while the underlying failures differ. One response may omit a required section, while another changes a date. Open the individual cases before choosing what to use. A tie does not require a forced winner. It may tell you that both answers need revision, or that you need a more relevant task to distinguish them.

Export a report you can question

Use the CSV when you want to inspect the data in a spreadsheet. Use the HTML report to review responses, criteria, weights and evidence together. The results describe these cases and these human ratings. They do not establish a universal ranking or guarantee future performance. Save the project backup if you want to continue on another device, and remove private source material before sharing any report.