Interview, Fireside Chat
Hiring Engineers with Ammon Bartram
- Speaker Context: Amon Bartram, co-founder of SocialCam and TripleByte, brings experience from early hiring at Justin.tv and hiring decisions at acquired startup SocialCam.
- TripleByte Mission: A recruiting startup that evaluates engineers through full interviews, filters candidates, and matches them with companies based on high-probability success metrics.
- Defining "Good Engineers":
- Most companies mistakenly assume they are hiring "good engineers" generally, failing to realize Company A's definition differs significantly from Company B's.
- Noise Source: Mismatched definitions create noise; e.g., productive but non-academic engineers fail at companies prioritizing computer science theory, while academic engineers fail at companies prioritizing speed.
- Early-Stage Strategy: Small companies must define specific bottlenecks (e.g., productivity vs. security) and tailor hiring processes to that specific need rather than a generic "good engineer."
- Senior Stage: Larger organizations should build processes that identify diverse skill sets, accommodating both high-productivity practitioners and deep academic theorists.
- Interview Biases and Pitfalls:
- Self-Reflection Bias: Interviewers often ask questions about topics they personally master, inadvertently biasing the hire toward candidates with similar skills rather than company needs.
- Credential Disparity: Education source (CS degree vs. boot camp) is less predictive than real-world experience; a fresh CS grad without internships is effectively a junior engineer.
- Experience Proxy: Five years of experience correlates strongly with passing interviews, not necessarily higher technical skill, as the bar for offers lowers with senior-looking resumes.
- TripleByte Interview Methodology:
- Total Duration: Approximately two hours, divided into four 30-minute sections.
- Productivity Assessment: Candidates render well-specified algorithms into working, structured code; junior engineers often outperform seniors in this specific metric.
- Academic Skills: Separate evaluation of computer science fundamentals and problem-solving approaches.
- Debugging Evaluation: Candidates analyze and fix bugs in a large codebase to test experience-based skills often missed in traditional interviews.
- System Design: Remote assessment of designing and adapting production web systems under changing requirements to test trade-off analysis.
- Predictive Power: Data from 2,000+ interviews indicates that shorter, focused assessments are highly predictive of job success compared to longer formats.
- Interview Question Types and Noise:
- Standard Interviews: Relying on single "brain teaser" questions creates significant noise; inter-rater agreement is comparable to movie critics rating films.
- Consistency Issue: If companies were to re-interview colleagues blindly, roughly 40% of existing employees would likely be screened out due to process inconsistency.
- Predictive Sweet Spot:
- Answering hard questions correctly has a high false-negative rate (smart engineers failing hard problems).
- Solving "easy" problems (e.g., creating a checkers game) quickly is a more stable predictor of general engineering skill.
- Observing how a candidate struggles with a problem provides more signal than just the final answer.
- Preparation Advice:
- New Grads: Focus on breadth-first search (BFS) and hash tables, which represent ~40% of interview questions; practice writing code under strict time limits.
- Experienced Engineers: Practice is essential as performance varies; 80% of top engineers pass their interviews, but nearly no one passes 100%.
- Resources: "Cracking the Coding Interview" (for questions) and "Interview Cake" (for practice).
- Industry Trend: Companies like Facebook are offering interview prep classes, reducing the necessity for candidates to self-study but raising concerns about filtering for compliance over raw skill.
- Side Projects: Useful for resume screening to bypass filters but rarely impact the actual interview decision due to the difficulty of assessing quality and consistency remotely.
- Location Factors: Large tech companies (e.g., Google, Facebook) do not care about candidate location; small startups (<20 employees) show a 10-20% preference for local Bay Area candidates.
- Hiring Process Recommendations for Companies:
- Consistency First: The most critical improvement is ensuring all candidates face identical questions and evaluation criteria to reduce noise.
- Alignment: Companies must consciously decide what skills matter (e.g., academic rigor vs. practical output) and design the interview to measure those specific traits.
- Whiteboarding Limitations: Whiteboarding favors academic thinkers; practical coding in an IDE is better for measuring daily productivity.
- Trial Employment: While highly accurate, this method has high adverse selection bias as top candidates often decline unpaid trial periods.
- Interviewer Conduct and Culture:
- Hazing Warning: Interviews should not be used for hazing, being mean, or cutting candidates off; this damages company reputation and candidate experience.
- Unconscious Assistance: Interviewers often steer candidates toward answers; strict decision trees on what assistance is allowed are necessary for consistency.
- Panel Size: One-on-one with one shadow is the maximum recommended; larger panels increase stress without improving evaluation accuracy.
- Teaching Mode: When a candidate has clearly failed, interviewers should switch to "teaching mode" to explain solutions amicably rather than continuing evaluation.
- Retention and Cultural Fit:
- Key Indicator: Engineers excited about the company mission and the specific job are statistically more likely to have longer tenures than those chasing the highest paycheck.
- Skill Maintenance: Engineers should stay updated on modern tech stacks to remain viable in 10 years, even if day-one interviews do not heavily test the latest tools.
- Book Recommendations: Bartram advises against existing interview books, stating 80% of written advice lacks empirical support; practical coding assessments outperform past-project discussions.