Interview, Fireside Chat
What it takes to launch an AI safety startup
- Core Bottleneck: Max Nadeau identifies talent as the primary, possibly sole, bottleneck for advancing technical AI safety; existing organizations have numerous high-priority projects they cannot execute due to staff shortages.
- New Initiative: Coefficient Giving is launching "Project Tailwind," a specific funding stream designed to catalyze the creation of new AI safety nonprofits to fill identified gaps in the field.
- Funding Scale: Grants for new organizations range from $200,000 to $200 million, with Nadeau predicting grants exceeding $200 million will occur within the next one to two years; the organization explicitly states $200 million is not an upper limit.
- Pre-seed Grants: Ranging from $200,000 to $2 million for founders with preliminary ideas but potentially without a fully established team or detailed vision.
- Seed Grants: Ranging from $2 million to $20 million for teams with initial results and coverage of key operational bases.
- Large Grants: Up to $200 million (or potentially higher) available even for new organizations that demonstrate exceptional promise or track records.
- Strategic Approach: The organization is adopting a "hits-based giving" model borrowed from venture capital, accepting that most grants will yield no impact while a few high-impact successes justify the portfolio.
- Operational Speed: Decisions on pre-seed grants are targeted to occur in under one week following pitch days.
- Scouting: Use of "scouts" to identify top-tier founders globally rather than relying solely on inbound applications.
- Diligence Balance: The team aims to maintain due diligence standards while accelerating grant cycles by increasing the number of grantmakers.
- Case Study (Resolution): A $160 million grant was awarded to Resolution to establish a large research center led by Jeffrey Irving (ex-UK AI Security Institute, ex-DeepMind, ex-OpenAI).
- Key Criteria: The grant targeted a "luminous" founder with a plan to align superintelligence and utilize AI tools to accelerate safety research.
- Future Scaling: Nadeau notes that while top-tier founders currently require "hard-to-fake signals" of competence to secure massive initial grants, organizations can scale to this size later based on demonstrated results.
- Founder Profile Requirements: Successful founders must possess a unique combination of skills:
- General Startup Traits: Perseverance ("running through walls") and the ability to manage employees and funders.
- AI-Specific Traits: Discernment regarding "theory of impact," willingness to entertain speculative questions about non-manifested future risks, and high contextual awareness of existing field debates and models.
- Pivot Strategy: Founders are advised to be prepared for significant strategic pivots, citing the example of Redwood Research's leaders downsizing mechanistic interpretability work to focus on AI control after evaluating impact potential.
- Costs: Nadeau acknowledges that pivots can cause short-term pain and staff turnover but are sometimes necessary to achieve maximum impact in the absence of market forces.
- Nonprofit vs. For-Profit: Technical AI safety non-profits are argued to be more ambitious than for-profits because they are not constrained by the need to create profitable products or serve paying customers.
- Problem Selection: Non-profits can target speculative, existential risks (e.g., pandemic release by AI, loss of control) that venture capitalists ignore due to lack of a clear customer.
- Funding Landscape: The ecosystem for non-profits is diversifying beyond Coefficient Giving to include organizations like Macroscopic, Astralis, Lightcone Commons, and the OpenAI Foundation.
- Strategic Positioning vs. AI Companies: Third-party organizations are positioned to address risks that AI companies cannot or will not tackle due to "race to the bottom" pressures.
- Independent Auditing: Credible, disinterested assessment of AI company safety practices (e.g., the Hugging Face incident) requires outside actors.
- Speculative Research: Work on long-term, principled alignment techniques that may not have immediate commercial utility is best pursued outside corporate safety teams.
- Evidence Generation: Independent verification of AI capabilities and misalignment incidents provides critical data that internal company reports may obscure.
- Priority Project Categories: Project Tailwind is soliciting founders for organizations in six key areas:
- Independent Auditing: Systematic assessment of AI companies' safety procedures and training runs.
- New Alignment Techniques: Research into monitorability (e.g., chain-of-thought oversight) and control methods.
- Evidence Generation: High-signal, logistically complex experiments to verify AI capabilities and risks.
- Security & Verification: Addressing supply chain vulnerabilities, data poisoning, and hardware security.
- Public Goods: Centralized resources like specialized compute clusters or fellowship programs (e.g., MATS).
- Field Building: Matchmaking and integration support for individuals entering the AI safety career field.
- Application Process: Interested founders should review the list of 40-45 "project stubs" on the Coefficient Giving website, though the organization welcomes original proposals.
- Contact: Applications are submitted via a form at
cg.org/Tailwindor by direct email referral to Coefficient Giving's team. - Referrals: The organization invites listeners to nominate potential founders via email if they identify strong candidates who may not apply independently.
- Contact: Applications are submitted via a form at