← Vault Index
frameworks / kit-interview-scorecard-design / 06-consultant-methodology.md
Source: frameworks/kit-interview-scorecard-design/06-consultant-methodology.md

Consultant Methodology — Scorecard Design Extraction

Where This Fits in the Engagement

Scorecard builds are triggered when a recruiting engagement reaches the interview design phase — after the role has been validated, the job description finalized, compensation benchmarked, and the interview team identified. The scorecard cannot be designed until these upstream deliverables exist. Once they do, the question is: how does this practitioner actually design evaluations?

The extraction interview is how you find out.

This is not a conversation about what a good scorecard looks like in theory. It's a structured interview that captures how the practitioner determines what to evaluate, how they build questions, how they prepare interviewers, and what the evaluation produces that feeds the debrief. By the end of the interview, you should be able to build a complete scorecard for a specific role without asking a follow-up question.

Before the Interview

Know What You're Building

Review the engagement context. Know:

If a prior scorecard template exists, read it before the interview. It tells you the structure the practitioner is comfortable with. The interview fills in the methodology behind the structure — why those focus areas, why that scoring scale, what works and what doesn't.

Confirm Logistics

Who needs to be in the room:

The extraction typically takes 45-60 minutes. If the practitioner has extensive experience with scorecard design, it may go faster. If they're building the methodology for the first time, budget 75 minutes.

Interview Structure

Part 1: Current Practice and Philosophy (10-15 minutes)

Start with how the practitioner currently approaches candidate evaluation.

Current state: "Walk me through how you currently evaluate candidates during the interview process. What does an interviewer have in front of them when they walk into an interview?"

Listen for whether there's a formal tool or an informal approach. Some practitioners have a structured scorecard they've used for years. Others have a mental checklist. Both are valid starting points — but they produce very different extraction paths.

Evaluation philosophy: "When you think about what makes a good hire versus a bad hire — what's the difference in how they were evaluated?"

This question surfaces the practitioner's beliefs about what evaluation should accomplish. Listen for themes like: consistency, evidence over impression, defensibility, structured versus gut-feel, predictive versus reactive.

Past failures: "Has there been a hire where the evaluation process missed something — the team thought the candidate was strong, but it didn't work out? What did the evaluation fail to catch?"

This question produces the most useful information in the entire extraction. The failure cases reveal what the scorecard needs to prevent. They also reveal what the practitioner has learned and what they've changed as a result.

Part 2: Focus Area Design (15-20 minutes)

This is the core of the extraction. The focus area methodology determines what the scorecard evaluates.

How focus areas are determined: "When you're designing the evaluation for a new role, how do you decide what focus areas to include? Where do you start?"

Listen for whether the practitioner:

All of these are valid approaches — but each produces a different build process. Capture the actual methodology, not what the practitioner thinks you want to hear.

Focus area depth: "For a given focus area — say, strategic thinking — what do you define for the interviewers? How do they know what to look for?"

Listen for whether the practitioner provides:

The answer tells you how much the scorecard needs to do. If the practitioner provides detailed descriptions and criteria, the scorecard documents what they already do. If they provide only names, the build process will need to develop the descriptions — which requires going back to the practitioner for validation.

Focus area assignment: "How do you decide which interviewer evaluates which focus area?"

Listen for the assignment logic:

Common focus areas the practitioner uses: "Across the searches you've done, are there focus areas that show up in almost every scorecard?"

This reveals the practitioner's core competency framework — the areas they always evaluate regardless of role. These are likely the strongest parts of their methodology and can inform the vault-level kit's guidance on common patterns.

Part 3: Questions and Scoring (15-20 minutes)

Question development: "How do you develop the specific questions for each focus area? Do you write them from scratch, pull from a bank, use a framework?"

Listen for:

Question quality criteria: "What makes a good interview question versus a bad one?"

This reveals the practitioner's quality standard. Listen for themes like: specificity, behavior-based vs. hypothetical, open-ended vs. closed, relevance to the role, avoiding leading questions.

Scoring methodology: "Walk me through your scoring system. What scale do you use? How is each level defined?"

Capture the exact scale, the exact definitions, and whether scores are weighted.

Justification requirements: "What do you expect interviewers to write beyond the score? How specific does the justification need to be?"

Listen for whether the practitioner requires:

Common scoring problems: "What's the most common mistake you see interviewers make on scorecards?"

This question produces flag notes and coaching points for the interviewer preparation materials.

Part 4: Presentation Evaluation (5-10 minutes, if applicable)

If a candidate presentation is part of the process:

"Walk me through how you evaluate the presentation portion. Is that standardized, or does it change per role?"

Capture:

Part 5: Debrief Connection (10 minutes)

How the scorecard feeds the debrief: "After all the scorecards are submitted, what do you do with them before the debrief?"

Listen for:

Debrief challenges that trace to the scorecard: "When a debrief doesn't go well — people can't explain their scores, or the discussion goes in circles — what usually went wrong?"

This question reveals what the scorecard needs to prevent. If interviewers can't explain their scores, the justification requirement isn't strong enough. If the discussion goes in circles, the focus areas may be too vague. The failure mode in the debrief is often a design flaw in the scorecard.

Part 6: AI Usage (5-10 minutes)

Current AI involvement: "Are you using AI anywhere in the scorecard design or evaluation process today?"

Listen for:

AI quality: "When you use AI for [whatever they mentioned], how much editing does the output need?"

This captures the agent process track — what AI can do reliably and where human review is essential.

What Must Be Confirmed Before Closing the Interview

Before ending the session, verify you have:

If any of these are missing when the interview ends, schedule a short follow-up (15-30 minutes) before the build starts.

Signals That the Extraction Is Incomplete

Generic competency language without role specificity: "We evaluate leadership, communication, and strategic thinking." → Ask: "For this specific role, what does leadership look like? What would a strong candidate demonstrate that's different from a generic leadership assessment?"

No description beyond the focus area name: "I just give them the topic and they know what to do." → Ask: "Walk me through a time when two interviewers evaluated the same focus area and came to very different conclusions. What would have helped them align?"

Scoring scale without definitions: "We use a 1-5 scale." → Ask: "What's the difference between a 3 and a 4? How does an interviewer know which to give?"

No justification requirement: "They just fill in the score." → Ask: "When someone gives a Strong Yes but can't explain why in the debrief, what happens? How do the other interviewers respond?"

Questions described as ad hoc: "I let them ask whatever they think is relevant." → Ask: "Has that ever produced a problem — someone asked something they shouldn't have, or two interviewers asked the same thing?"

AI used without quality review: "I have ChatGPT generate the questions." → Ask: "Do you review and edit those before they go to the interviewers? Have you caught anything that needed changing?"

After the Interview

Within 24 Hours

Write up your extraction notes mapped to the scorecard sections:

Produce the Gap Report

Work through the Required Inputs table in 01-context.md. Every item not captured in the extraction is a gap.

Advisor Review

The advisor reviews the extraction notes and gap report. They:

Build starts only after the advisor confirms the gap report is resolved.

Connection to the Engagement

Every scorecard extraction produces two things: a scorecard and advisor insight.

The scorecard is the deliverable. The extraction conversation often surfaces broader engagement signals — how the practitioner thinks about evaluation quality, whether their methodology is rigorous or ad hoc, where their process has gaps they haven't addressed, whether the interview team is sophisticated or needs significant preparation.

These signals inform the engagement strategy. A practitioner with a strong methodology needs a scorecard that documents and scales what they already do. A practitioner with a weak methodology needs a scorecard that introduces structure they don't currently have — and that's a different conversation with a different change management requirement.

Take notes on what you're observing about the practitioner's capability, not just what's going into the scorecard. The extraction interview is a diagnostic opportunity. Use it.