How to design a scoring rubric

What you'll need

• A clear picture of the role's competencies (a role blueprint helps)
• The work-sample task the rubric will score, in near-final form
• 2–3 sample responses, real or drafted, at different quality levels
• Input from someone who does or manages the actual work

Step 1: Derive criteria from the role, not from the task

Start with the question the assessment exists to answer: which competencies does this role actually depend on? Each rubric criterion should trace back to one of them. If you start from the task instead, you end up scoring incidental features of the deliverable — formatting, length, tool choice — rather than the skill the task was built to reveal.

Keep the list short. Three to six criteria per rubric is usually right: enough to separate distinct skills, few enough that evaluators can hold all of them in mind while reading a single response. If two criteria always seem to rise and fall together, merge them; if one criterion hides two independent judgments, split it.

Step 2: Write observable behaviors, not traits

"Strong communicator" is a trait; "acknowledges the customer's problem before proposing a fix, and ends with a concrete next step" is a behavior an evaluator can point to in the response. Every criterion should be phrased so that two people reading the same response would look at the same evidence to score it.

A useful test: for each criterion, ask "what would I underline in a response to justify this score?" If the answer is a feeling rather than a sentence, a decision, or an artifact in the candidate's output, rewrite the criterion until it names the observable thing.

• Replace adjectives about the person with verbs about the work
• Name the artifact: what in the response demonstrates this?
• Avoid criteria that require guessing intent or personality
• One criterion, one judgment — no "clarity and accuracy" hybrids

Step 3: Choose between classic score entry and a level-based matrix

SkillCort supports two rubric styles. Classic score entry gives evaluators a numeric scale per criterion — fast and familiar, but it leans on each evaluator's private sense of what a 7 means. The level-based matrix defines explicit levels per criterion, each with an anchored description of the behavior that earns it, and evaluators score by clicking the level card that matches the response.

For anything with multiple evaluators or real stakes, prefer the matrix. The anchored descriptions do the calibration work up front: instead of every reviewer inventing their own scale, they are all matching responses against the same written standard. Classic entry remains reasonable for low-stakes, single-evaluator scoring.

Step 4: Anchor each level in concrete, distinguishable descriptions

In the matrix editor, write a description for every cell — each criterion at each level. Good anchors describe what a response at that level actually does: what a top answer includes, what an acceptable one gets right and misses, what a weak one fails to do. Vague gradations like "good", "very good", "excellent" force evaluators back into private judgment, which defeats the matrix.

Make adjacent levels genuinely distinguishable. The test is a borderline response: if an evaluator reading your anchors could argue equally well for two neighboring levels, sharpen the difference — usually by naming the one behavior that separates them. Three to five levels per criterion is typically enough; more levels mean finer distinctions than most evidence can support.

Step 5: Weight criteria to match what the role rewards

Not every criterion deserves equal say. SkillCort lets you weight each criterion (×N), and those weights flow directly into the task score and onward into role-fit. Weight the criteria the role most depends on — the judgment call, the quality of the written reply — above supporting ones like formatting or completeness.

Be deliberate here, because weights are where a rubric quietly encodes your hiring philosophy. A rubric that weights polish over judgment will surface polished candidates. Sanity-check the weights with someone close to the role: "if a candidate nailed criterion A but flubbed criterion B, would we still be interested?" The weights should agree with the answer.

Step 6: Decide whether candidates should see the rubric

SkillCort can make a rubric candidate-visible. Showing it tells candidates what good looks like, which is honest and tends to produce responses that address what you actually care about — you are testing the skill, not the ability to guess your criteria. Hiding it makes sense when knowing the criteria would trivially script the answer, as in some judgment scenarios.

Whatever you choose, choose it per rubric and on purpose. A visible rubric raises the bar on your anchor writing, since candidates will read it too — one more reason concrete, behavioral language matters. It also signals fairness: candidates who know the standard experience the assessment as an evaluation of their work, not a guessing game about your preferences.

Step 7: Test the rubric on sample responses before attaching it

Take your two or three sample responses and score them against the draft rubric — ideally with a second person scoring independently. You are testing the rubric, not the responses: do the anchors cleanly place each response at a level, or did you both hesitate at the same cell? Hesitation marks the anchor that needs rewriting.

Also check the arithmetic. With the weights applied, does the strong response comfortably outscore the weak one, and by a margin that feels right? If a mediocre response scores nearly as well as an excellent one, either the anchors are not separating quality or a weight is misplaced.

Step 8: Attach the rubric and understand the snapshot

When you attach a rubric to an assessment, it freezes into a snapshot. Later edits to the rubric in your bank never change a live assessment — every candidate in that assessment is scored against exactly the same standard, which is precisely what makes the scores comparable and defensible.

This means the moment before attaching is your last cheap chance to fix wording, levels, or weights for this assessment. Keep improving the bank version as you learn from real responses; those improvements will benefit the next assessment that attaches it, without disturbing the one in flight.