ted.neward@newardassociates.com | Blog: http://blogs.newardassociates.com | Github: tedneward | LinkedIn: tedneward
Interviewing people for your team:
What are you looking for?
How can you find it?
Evaluating what you find
Structuring an interview process
Making the offer
Onboarding the new hire
I am not an HR professional
You have some control over your interview process
You are open to rethinking your interview process
Never ask about:
Marital status
Health status
Family status
Race, religion, age, any other protected category
You may have your reasons for asking
I don't care
Neither does US law
Do you know what this is asking about?
originally, it was a "how do you think" sort of question
but quickly it turned into a trivia question
Cargo-culting others' interview processes is a BAD idea
even within the same company
even across the same position in different parts of the company
Worse, this is a "right answer" sort of question
if I've heard it before, there's no insight
if I haven't heard it before, how could I reason it out?
if I don't know the answer, do all your interviewers?
if I think I know, and it's different, how will your interviewers react?
This is a screen for "Hiring more people just like me"
Red flag to anyone who isn't like you
This isn't a DEI thing; this goes way wider than that
Wildly too generic
Wildly too personal
Drill deeper: Why? What did they say or do that you like?
Embrace the Popper Falsification Theorem here
"We won't hire someone unless everybody says yes"
This is just self-protection via blame-sharing
This is your hire, and therefore your decision
"We will wait (for years) for the right person" or "the best person"
What's the cost of not having somebody in that position?
If you can wait a year without somebody in that seat, do you really need to hire?
Will you ever know the "right person" when you see them?
How do you define "best"?
Recency bias
Most recent candidate is freshest in everyone's minds
Easiest to feel comfortable about
Capture all feedback immediately after interview
Do not revise feedback "later"
Do all loops within a short, fixed span of time
no latecomers
Keep in mind interview fatigue is a real thing
judges and sentences, mornings and afternoons
Potter Stewart, US Supreme Court judge, on pornography
If you can't define it
you can't recognize it when you see it
you've probably not spent enough time thinking about it
you can't defend your choices
chances are nobody agrees on your definition either
This really is just laziness
You have an opening on your team
Why not hire the first person you see?
Bets are decisions about future results made under some amount of uncertainty
"a choice made by thinking about what will probably happen"
"to risk losing (something) when you try to do or achieve something"
"to make decisions that are based on the belief that something will happen or is true"
Poker vs chess
Do not mistake odds for outcome or vice versa
Hiring somebody is always a bet
wagering future team productivity
against your imperfect knowledge about the individual
... and "it's the most important decision you'll ever make"
Every manager knows the old wives' tales about bad hires
"Caused the team to quit"
"Ruins the department"
"Destroys the product"
... but how many people have actually lived a bad hire?
Most "bad hire" stories are actually a "reluctance to fire" story
either on the part of the manager/leadership
or on the part of the company
"M-Soft Corp":
the old CEO "Bill": rise to dominance
the new CEO "Steve": screwed it all up
and the later replacement, "Satya": running it back
"Fruit Corp":
the old CEO "Steve": rise to success
the series of replacement CEOs: screwed it all up
and the later replacement, "Steve" (again): ran it back
now they have "Tim": jury's still out
neither ever went bankrupt, despite the mistakes of their CEOs
last example: "EyeBeeEmm Inc"
A psychologically-safe team can:
help you quickly identify a bad hire
survive a bad hire
A good relationship with your HR team can:
help smooth the performance issues
improve the termination process
But you also have to be prepared
strong culture of accountability
strong history of performance reviews
strong decision-making
ability to admit mistakes
Firing somebody....
... is an admission of a mistake
... on the part of the hiring manager
... and the interview team (but mostly the manager)
(If you're not comfortable admitting mistakes, maybe don't be a manager?)
Before any goal can be met, it has to be defined
... but don't assume everybody has the same goal!
Communication creates clarity
in other words, write it out
review it with your team
Focus on results (not processes)
Don't look for someone to mind the store
Look for someone who will own the store's profit/loss
Qualifications, skills, and attributes that are essential for success in the role
should be directly related to the job’s responsibilities/requirements
Avoid general or vague language
use specific and concrete language to describe what each criterion entails
avoid ambiguous terms or subjective descriptions
Comprehensible
Measurable
Actionable
BAD: "Synergize with the latest upward trends in disruptive technology paradigms"
Explain to me like I'm 15
GOOD: "Display solid understanding of core CS fundamental concepts"
BAD: "Writes good code" -- what does good code look like? Seriously!
GOOD: "Candidate code should have cyclomatic complexity score of less than ..."
GOOD: "Candidate should have a history of writing unit tests"
GOOD: "Candidate should be conversant with two different unit test frameworks"
BAD: "Keep up with latest technology trends"
GOOD: "Has learned one new technology every calendar year"
BAD: "Deep understanding of object orientation"
GOOD: "Has demonstrated experience with a modern O-O language"
BAD: "10+ years experience with Rust"
GOOD: ...
Understand user and business needs, and develop software solutions
Write clean, efficient, and well-documented code using programming languages like Java, C++, and Python
Conduct thorough testing and debugging of software applications
Create logical database designs and specialized software applications
Update software specifications throughout the production life cycle
Create documentation, flowcharts, layouts, diagrams, charts, and code comments
Gather and evaluate user feedback, and modify software for better usability
Provide support and guidance to other programmers and analysts
What "holes" are on my team? What does the team need most?
Lencioni's triplet:
Hungry: They have some level of ambition and/or career goals
Humble: They are as willing to admit what they don't know as they are what they do
Smart: They possess some demonstrable intelligence and "double-loop learning" capability
Going with our "gut feel" is dangerous
unconscious biases distort our thinking
different individuals prioritize different things differently
opinions are widely divergent across different individuals
reactions can be wildly distorted by environmental factors (eg: time of day, mood, etc)
We form our thoughts out of emotion, then use logic
Rubrics keep the evaluation process objective and fair, minimizing bias and subjectivity
Rubrics encourage consistency and can be calibrated
Rubrics can be kept in records and retrieved/subpoena'ed
Caveats:
A rubric will NOT make an interview 100% objective or algorithmic
A rubric WILL help eliminate many kinds of bias
theoretically, this is simple
create questions or topics of discussion
identify a scoring scale (1 to 5, 1 to 10, etc)
identify example answers or criteria at each score point
practically, this is hard
your HR partners can help
your peer managers can help
your boss can help
your team can help
in the end, this is your rubric; make sure it reflects your (team's) needs!
Focus on open-ended but guided questions
rubric should be judging content of answers, not candidate reactions
Don't rely on "trick questions"
if the candidate was given the rubric ahead of time, does it still work?
consider giving the candidates your rubric ahead of time!
Remember that not all questions/scores need be asked
the rubric is evaluating evidence (spoken or discovered)
interviews can help sharpen perceptions around a rubric score
Coming up with good questions and rubrics is not easy
don't attempt to do this the night before interviews start
don't throw one away once built
Recall: My goal is to find people who are:
Hungry
Humble
Smart
So what does a rubric look like for these?
Question: "Tell me about the most important accomplishments of your career. What are you most proud of?"
Great (3): The candidate speaks of a time when the team they were on accomplished a task. They rarely or never use the word "I" in the explanation and discussion.
Good (2): The candidate speaks of a time when they were on a team and accomplished a task. They use "I" as much as they do "we".
Yellow-flag (1): The candidate speaks of a time when they accomplished a thing and the team is mentioned perhaps once or twice.
Red alert (0): The candidate speaks of a time when they single-handedly rescued the team from its own ignorance.
Question: "How do you handle apologies, giving or accepting?"
Great (3): Candidate primiarly (66% or more) speaks of how to apologize to others.
Good (2): Candidate's answer focuses equally on accepting and offering apologies.
Yellow-flag (1): Candidate's answer focuses 66%+ on accepting apologies.
Red alert (0): The candidate never once speaks of having to apologize.
Question: "What's the newest technology thing you've learned? How did you learn it?"
Great (3): Candidate learned thing within the last year, using combination of sources
Great (2): Candidate learned thing within the last two years, using a single source
Yellow-flag (1): Candidate learned thing within last five years, from company-paid training class
Red alert (0): Candidate cannot identify any new thing they've learned within five years
This is an important step
... which almost nobody does
... because, admittedly, it's hard
Keep in mind:
the rubric should prevent any one person from being in every interview
the rubric should produce similar scores regardless of interviewer
the rubric can/should stretch across interviews
Rubrics can test for seniority via:
scores should be higher for more experienced folks
questions should be different for more experienced folks
Best heuristic I've found
find people currently in the role inside your company
ask them to anonymously self-evaluate
never, ever send these results to HR or their manager(s)
Alternative
take some folks currently in the role (inside or outside)
run them through the rubric (that is, interview them)
return the favor when asked
This is how we calculate the odds on the bet
What is "out there" to prove or disprove that this candidate has what we want?
Multiple "times" evidence is useful to gather:
Before the interview loop
During the interview loop
After the interview loop
Resume
Website/Portfolio
GitHub public repositories
What does Google turn up?
What does social media show us?
References/referrals
Well-worn, tried-and-true, trite, cliche option
Generally very vague
Sometimes yields interesting clues
emphasis/focus
attention to detail (or not)
work history
Usually required for HR reasons
Social media extension of the resume
Multiple parts:
Profile page
Work history
Endorsements
Skills
Feed
Articles
Comments on other articles
NOTE: LinkedIN combines social media with (employment) dating site
Content:
What's highlighted?
What's missing?
How does their writing look?
Infrastructure:
Domain name: Custom (DNS) or via another platform (GitHub, WordPress)?
Page source: Hand-crafted HTML? Angular? React?
Custom email?
Measure this against what they're applying for!
How many forked vs non-forked (owner) repositories?
What was most recent activity? What about a year ago? Two years ago?
Look for patterns: Started/abandoned/in-progress/"finished"
What is their preferred tool/language of choice?
Is there a non-trivial project that is 100% their own work?
Clone it; analyze it
Unit tests? Code coverage?
Build pipeline?
Documentation?
(Build some scripts to automate answering some of these questions!)
Any conference or user group talks?
Any articles or academic papers?
Any news stories?
Any interesting links of note?
Facebook, X/Twitter, Reddit, ...
How do they respond to newbie questions?
How do they respond to challenges?
Do they offer credit or praise to others?
Keep in mind: We're ALL terrible people on social media
Does somebody near you know them?
Are they currently at your same company?
Are they on your team?
What do they think?
Would they work with them again?
Would they hire them?
Would they work for them?
Make sure questions are specific and concrete
"What was something the candidate did very well?"
"What do you think the candidate needs to improve?"
Use this data to guide other evidence-gathering tactics
Coding tests
on-site/timed
take-home/untimed
pairing
Interviews with your team
1:1
n:1
Lunch
Code review
Other activities
Writing
Presentations
Editing
Just say no to LeetCode
or any other industrial platform
unless they let you customize the test
What does the job require?
language
frameworks
tasks
What is a fractional subset of that?
Goal:
something take-home (to reduce pressure)
requiring you no more than 1 hour
that allows you to see incremental progress
Each interviewer has well-defined "area"
don't ask the same questions twice except by intent!
Each interviewer has the rubric for scoring
it's acceptable to make notes during interview
but don't be too focused on notes that you miss the person
No more than 60 minutes, no less than 45
if you must go longer, give candidate break(s)
Tell the candidate ahead of time how many questions
Designed to get multiple perspectives simultaneously
More expensive (time) to hold/conduct
Otherwise still an interview (see previous slide)
Writing: Ask them to write some prose
specify a topic or ask them to
length no more than 2500 words
let the candidate choose the tool(s)
evaluate for both content and prose skills
Presentation: Ask them to present to you
specify a topic or ask them to
length no more than 30 minutes
recorded or live?
(live) audience/reviewers should be at least 6 people
evaluate for both content and presentation skills
Editing: prose? video?
provide them a sample, ask them to edit
Interview feedback
Rubric scoring
References/referrals
Vague statements should be avoided
BAD: "I liked them."
GOOD: "I liked that they always circled back to make sure the question was answered to my satisfaction. To me that shows an attention to detail we'll want when talking to our users and product owners."
All statements should be as objective as possible; focus on what was demonstrated and how it applies
BAD: "Jackson is a misogynist."
GOOD: "Even though I (a young woman) was giving the interview, and Peter was just shadowing, every time I asked a question, Jackson gave the response to Peter. Jackson asked Peter clarifying questions, but not me. This frustrated me, and makes me think Jackson would have a problem working productively with a younger woman."
Feelings are OK; conclusions based solely on feelings are not
Interviewers should look to keep the scores on the scale provided
Avoid "Can I give them a half-point here?"
Creating new scores complicates things
Doing so potentially hurts prior candidates
Topics not covered should be noted as "not covered", not zero
Why was the topic not covered? Is the interviewer aware of other evidence?
If the topic wasn't covered for reasons of time, why? What caused that?
Did the candidate surpass the target/floor score?
If so, cool! Now the score's utility drops to zero.
(Avoid using the rubric as a "best" measurement--it's a "past the post" measurement)
If not, send the email (and consider sending along the rubric scores)
This all presumes your company actually allows references/referrals
some don't for various HR reasons
also possible the reference/referral is prevented from saying anything meaningful
Before you contact anyone
What questions remain? What do you not yet know?
What evidence do you still need to find to answer those questions?
Craft your email or your questions with that in mind
Stay far, far away from anything HR-sensitive
Why their employment ended
"What they were like" as an employee
Relationships with other employees
Notable/lengthy absences
(This is why a lot of companies don't bother with them anymore)
... you know what your goals are
... you have evidence about the candidate
... so for some folks, it will be simple to infer the answer
many will not meet your floor requirements
Only now can you really embrace intuition
And this has to be entirely on you
Which one "vibed well" with the team?
Which one seems potentially successor-worthy?
Which one evidences human qualities you think are important?
Sleep on it (literally)
Then make your decision and don't look back
With ...
your team
your boss
the (former) candidate
HR
Asking ...
how much evidence turned out true? false?
rubric scoring vs performance reviews
interview assumptions vs actual job tasks
Refactor, recalibrate, reuse!
Interviewing is not as hard as others make it out to be
Hiring is important, but hiring right is critical
Keep researching! Being better as a manager is your job
HBR's 10 Must Reads on Managing People, Harvard Business Review
double-loop learning: "Teaching Smart People How to Learn"
The Ideal Team Player, Lencioni
"hungry, humble, and smart"
Thinking in Betts, Duke
Portfolio/Penguin; 2018, ISBN 9780735216365
"The Game-Changing Magic of Interviewing with Rubrics"
https://imagine.jhu.edu/blog/2023/11/16/the-game-changing-magic-of-interviewing-with-rubrics/
"Engineering Interviews: grading rubric"
https://medium.engineering/engineering-interviews-grading-rubric-8b409bec021f
this is Medium (the company's) engineering rubric, what they actually use
I don't agree with everything here, but rubric exemplars are good to discuss
I think publishing your rubric is a huge win
"HubSpot Engineering Performance Rubric"
https://github.hubspot.com/engineering-rubric/
"One Rubric Changed Box's Engineering Performance — Here's How"
https://review.firstround.com/one-rubric-changed-boxs-engineering-performance-heres-how/
interview rubrics often (and should!) look very similar to performance rubrics
"System Design Interview Rubric"
https://www.tryexponent.com/courses/system-design-interviews/system-design-interview-rubric
Architect, Engineering Manager/Leader, "force multiplier"
http://www.newardassociates.com
http://blogs.newardassociates.com
Sr Distinguished Engineer, Capital One
Educative (http://educative.io) Author
Performance Management for Engineering Managers
Books
Developer Relations Activity Patterns (w/Woodruff, et al; APress, forthcoming)
Professional F# 2.0 (w/Erickson, et al; Wrox, 2010)
Effective Enterprise Java (Addison-Wesley, 2004)
SSCLI Essentials (w/Stutz, et al; OReilly, 2003)
Server-Based Java Programming (Manning, 2000)