Page content begins
On this page

Published

How much do interview scores drift between two reviewers?

A scoring review that measures disagreement between interviewers before changing the rubric.

How much do interview scores drift between two reviewers? - Hire Assistant Near Me research, published November 16, 2026
Research brief illustration: How much do interview scores drift between two reviewers?.

Key stats

8protocol stagesSource: Method
3sources checkedSource: Source list

Key takeaways

  • Freeze the candidate list before comparing scores.
  • Use the same rubric version for both reviewers.
  • Report the direction of the drift, not only the size.
  • Do not rank reviewers on a small sample.

Question and scope

Scope the review to interviews scored by two reviewers in the window. Exclude cases that lack a usable timestamp and report them separately. The hiring manager records the start date, stop date, and exclusion rule. A count without a denominator cannot support a staffing or quality decision. In the Hire Assistant Near Me hiring file, the note names the interviewer and the date. For this hiring review, keep the extract, the role brief version, and the reviewer list together.

The decision this review supports is whether the panel needs a shared scoring rubric. The recruiting lead should be able to use the result to change one rule, template, or review point. If the finding cannot lead to a decision, narrow the question. Write that boundary before selecting any case. The hiring manager should be able to repeat this check from the candidate record. For this hiring review, keep the extract, the role brief version, and the reviewer list together.

Scope the review to interviews scored by two reviewers in the window. Exclude cases that lack a usable timestamp and report them separately. The hiring manager records the start date, stop date, and exclusion rule. A count without a denominator cannot support a staffing or quality decision. Keep the evidence with the candidate file rather than in a private message. For this hiring review, keep the extract, the role brief version, and the reviewer list together.

Population and unit

Report the population size, the number excluded, and the number available for review. If the available set is too small, say so rather than widening the definition mid-review. The same rule should apply to every case in the window. Keep the frozen list with the final report. The hiring manager should be able to repeat this check from the candidate record. For this hiring review, keep the extract, the role brief version, and the reviewer list together.

The population is candidates with two independent score sheets. Freeze the list from the interview score file before counting anything. Assign each case a stable key so the extract and the evidence can be joined later. Record the extract time and the filter used. Keep the evidence with the candidate file rather than in a private message. For this hiring review, keep the extract, the role brief version, and the reviewer list together.

The unit of analysis is one case, not one message or one task. A case can contain several messages and still count once. The hiring manager removes duplicates with the case key before sampling. Document any merge so the count can be rebuilt. A second reviewer should see the same prompt, rubric, and evidence. For this hiring review, keep the extract, the role brief version, and the reviewer list together.

Definitions and measures

Report the rate with its denominator, observation window, and evidence standard in the same table. State all three beside the count. If the team changes a definition, begin a new series rather than joining old and new values. A longer chart is not worth a break in comparability. Keep the evidence with the candidate file rather than in a private message. For this hiring review, keep the extract, the role brief version, and the reviewer list together.

Define share of candidates whose two total scores differ by more than one point before collecting data. State the start event, the stop event, and any pause rule in plain language. The hiring manager writes one example and one counterexample so reviewers apply the same test. Ambiguous definitions produce confident but useless numbers. A second reviewer should see the same prompt, rubric, and evidence. For this hiring review, keep the extract, the role brief version, and the reviewer list together.

Report the rate with its denominator, observation window, and evidence standard in the same table. State all three beside the count. If the team changes a definition, begin a new series rather than joining old and new values. A longer chart is not worth a break in comparability. Record the decision and the next action before the case leaves the queue. For this hiring review, keep the extract, the role brief version, and the reviewer list together.

Evidence dictionary

Every dictionary entry should name the record's location, the person who can retrieve it, and the retention period. A required artifact that exists only in a personal inbox makes the measure impossible to reproduce. Replace it with a shared record or disclose the limitation. The dictionary belongs in the finding package. A second reviewer should see the same prompt, rubric, and evidence. For this hiring review, keep the extract, the role brief version, and the reviewer list together.

Version the dictionary with the date it changed. A later reviewer should know which definition produced which result. The hiring manager stores the dictionary beside the extract and the sample list. That package is what makes the review repeatable. Record the decision and the next action before the case leaves the queue. For this hiring review, keep the extract, the role brief version, and the reviewer list together.

The evidence dictionary lists the score sheets, the rubric, the interview notes, and the candidate file with the field, source, and acceptable condition for each. The hiring manager tests the dictionary on three cases before the full sample. Any artifact that cannot be found consistently is removed or marked optional. This keeps the review tied to records the team already keeps. The recruiting coordinator dates the entry and links the source document. For this hiring review, keep the extract, the role brief version, and the reviewer list together.

Sampling and collection

Collect the evidence in one pass and record retrieval failures. If a reviewer scored without recording evidence, mark the case and continue rather than stopping the review. The exception count belongs in the result. A review that silently drops hard cases will overstate performance. Record the decision and the next action before the case leaves the queue. For this hiring review, keep the extract, the role brief version, and the reviewer list together.

A full census is preferable when the population is small enough to review in the time available. For larger queues, state the sample size and the confidence the team can reasonably claim. Do not imply precision the sample cannot support. The hiring manager keeps the sample list with the original case keys. The recruiting coordinator dates the entry and links the source document. For this hiring review, keep the extract, the role brief version, and the reviewer list together.

A full census is preferable when the population is small enough to review in the time available. For larger queues, state the sample size and the confidence the team can reasonably claim. Do not imply precision the sample cannot support. The hiring manager keeps the sample list with the original case keys. Where the evidence conflicts, the file keeps both values and the reviewer's note. For this hiring review, keep the extract, the role brief version, and the reviewer list together.

Analysis and reporting

Break the result down by role, reviewer pair, and competency only when each group has enough cases to be meaningful. Avoid ranking people on a small sample. The recruiting lead reviews the finding before it is shared outside the team. Add the limitation beside the number, not in a footnote. The recruiting coordinator dates the entry and links the source document. For this hiring review, keep the extract, the role brief version, and the reviewer list together.

Break the result down by role, reviewer pair, and competency only when each group has enough cases to be meaningful. Avoid ranking people on a small sample. The recruiting lead reviews the finding before it is shared outside the team. Add the limitation beside the number, not in a footnote. Where the evidence conflicts, the file keeps both values and the reviewer's note. For this hiring review, keep the extract, the role brief version, and the reviewer list together.

Break the result down by role, reviewer pair, and competency only when each group has enough cases to be meaningful. Avoid ranking people on a small sample. The recruiting lead reviews the finding before it is shared outside the team. Add the limitation beside the number, not in a footnote. In the Hire Assistant Near Me hiring file, the note names the interviewer and the date. For this hiring review, keep the extract, the role brief version, and the reviewer list together.

Limitations

Do not turn a local count into an industry claim. The value of the review is that the team can repeat it and compare its own periods. If the method changes, the comparison breaks. Preserve the old definition so the history remains readable. Where the evidence conflicts, the file keeps both values and the reviewer's note. For this hiring review, keep the extract, the role brief version, and the reviewer list together.

Do not turn a local count into an industry claim. The value of the review is that the team can repeat it and compare its own periods. If the method changes, the comparison breaks. Preserve the old definition so the history remains readable. In the Hire Assistant Near Me hiring file, the note names the interviewer and the date. For this hiring review, keep the extract, the role brief version, and the reviewer list together.

Do not turn a local count into an industry claim. The value of the review is that the team can repeat it and compare its own periods. If the method changes, the comparison breaks. Preserve the old definition so the history remains readable. The hiring manager should be able to repeat this check from the candidate record. For this hiring review, keep the extract, the role brief version, and the reviewer list together.

Review and next action

Re-run the same measure after the change and compare like with like. If the result improves, keep the change and document why. If it does not, inspect the sample before assuming the change failed. The next review should test the rule, not the person. In the Hire Assistant Near Me hiring file, the note names the interviewer and the date. For this hiring review, keep the extract, the role brief version, and the reviewer list together.

The monthly review should end with one owner, one change, and one review date. The hiring manager picks the smallest change that addresses the most common gap. Update the template or evidence list instead of adding another report. Record the decision in the same file as the finding. The hiring manager should be able to repeat this check from the candidate record. For this hiring review, keep the extract, the role brief version, and the reviewer list together.

The monthly review should end with one owner, one change, and one review date. The hiring manager picks the smallest change that addresses the most common gap. Update the template or evidence list instead of adding another report. Record the decision in the same file as the finding. Keep the evidence with the candidate file rather than in a private message. For this hiring review, keep the extract, the role brief version, and the reviewer list together.

Sources (3)

  1. Society for Human Resource Management
  2. U.S. Equal Employment Opportunity Commission
  3. U.S. Bureau of Labor Statistics

Related research