Designing AI-resistant technical evaluations can be a challenging task, especially as AI capabilities continue to advance. In this article, Tristan Hume, a lead on Anthropic's performance optimization team, shares his journey in creating and refining a take-home test that has helped the company hire dozens of performance engineers. The test, which involves optimizing code for a simulated accelerator, has been a crucial tool in evaluating technical candidates. However, as AI models like Claude improve, the test has had to be continually redesigned to ensure it remains effective. Hume discusses the evolution of the test, from its initial design to the latest version, and the strategies he's employed to keep it ahead of the capabilities of the top AI models. He also introduces an open challenge, inviting anyone to try the original take-home test with unlimited time, highlighting the enduring advantage of human experts over current AI models at sufficiently long time horizons.
