SB 1047: California's AI bill & its potential to shape US AI policy | Nathan Calvin

Core Provisions and Scope of SB 1047

• The bill mandates safety assessments, third-party auditing, and liability for developers of advanced AI models that meet specific compute thresholds.
• Coverage applies only to models requiring $100 million or more in training compute, or those fine-tuned on existing large models with an additional $10 million in compute.
• As of the interview (August 19), no existing models met these criteria; the bill targets models expected in 2025 (estimated at 10^26 FLOPs).
• Covered developers must create a comprehensive safety and security plan detailing:
  • Measures to guard against cybersecurity attacks and model theft.
  • Protocols for shutting down all model copies in an emergency.
  • Testing results proving the model cannot cause "critical harm."
• "Critical harm" is statutorily defined as mass casualties or incidents resulting in $500 million or more in damages.
• Developers must publish the results of their safety tests to the public.
• Models must include guardrails preventing users from deploying the model in harmful ways.
• Developers face annual third-party audits to verify compliance with safety plans.
• Liability is triggered if a covered model causes critical harm or is used to cause such harm, subject to fines by the California Attorney General.
• The bill includes whistleblower protections for employees reporting noncompliance.

Legal Framework and Liability

• The bill codifies existing tort negligence principles ("duty of reasonable care") rather than creating a new regulatory agency or strict liability standard.
• Liability is not automatic; it applies only if a developer fails to take "reasonable care" to prevent "unreasonable risks."
• The statute clarifies that Section 230 or software licenses do not grant immunity from lawsuits when a computer model causes catastrophe under common law.
• The law does not require developers to submit statements under penalty of perjury (a provision removed in the most recent draft).
• The framework allows companies to demonstrate risks are low via testing, potentially avoiding liability if no harm occurs despite released models.

Legislative Status and Timeline

• As of August 23, SB 1047 had passed the California State Assembly and awaited Governor Gavin Newsom's signature by September 30.
• The bill cleared all six legislative policy committees with "commanding margins."
• The final hurdle was a vote in the Assembly and potential reconciliation with the Senate before the August 31 deadline for both chambers.
• Senator Scott Weiner was deeply involved in drafting the bill with technical input from the Center for AI Safety Action Fund and Economic Security California Action.

Support and Opposition Dynamics

• Major Supporters include AI safety researchers (e.g., Jeff Hinton, Stuart Russell, Joshua Bengio), labor unions (SEIU), nonprofits, and startups like Imbue and Notion.
• Public Opinion polls suggest roughly 75% of Californians support the bill.
• Primary Opponents include venture capital firm Andreessen Horowitz (a16z), tech trade associations, and individual figures like Yann LeCun.
• Nancy Pelosi opposed the bill, citing concerns about federal preemption and referencing a letter from Fei-Fei Li; she argued state legislation should not preempt potential federal action.
• Anthropic's Stance: Initially neutral, stating they would support the bill if amended; they requested the removal of a proposed new "Frontier Model Division" regulator and other cost-saving measures.
• Meta's Position: Criticized for potential liability regarding downstream uses of open-source models like Llama, though the company has not formally lobbied against the bill in the same capacity as a16z.

Key Criticisms and Rebuttals

• Innovation Stifling: Critics argue costs will force companies out of California or stifle startups.
  • Rebuttal: Companies already voluntarily adopt similar safety protocols; the bill targets only entities spending $100M+ on training, which are financially capable of compliance.
• Startups and Barriers: Concerns that the bill creates "regulatory capture" favoring incumbents.
  • Rebuttal: The $100M threshold effectively exempts small startups; costs for compliance are estimated at single-digit percentages of training costs.
• Open Source Harm: Critics fear the bill will discourage open-weighing large models due to liability for downstream misuse.
  • Rebuttal: The bill explicitly exempts the shutdown provision for models outside the developer's control; current open-source models (e.g., Llama 405B) fall below the $100M threshold.
• National Security: Objections that regulation hinders U.S. competitiveness against China.
  • Rebuttal: The bill aligns with industry self-standards; China also enforces strict domestic AI regulations; the risk of competitive disadvantage is deemed low given current industry practices.
• Federal Preemption: Arguments that AI regulation should only occur at the federal level.
  • Rebuttal: Congress has been stalled on data privacy and AI; states (like California with Prop 12) have the legal standing to regulate products sold within their borders.
• Arbitrary Thresholds: The $100M and 10^26 FLOP limits are criticized as arbitrary.
  • Rebuttal: The thresholds are necessary to create a clear legal line; they align with the Biden Executive Order and target the generation of models expected to emerge in 2025.

Strategic Implications and "California Effect"

• The bill leverages California's status as a massive market to influence global standards, similar to the "Brussels Effect" where local regulations force global compliance to maintain market access.
• State-level action is framed as a necessary precursor to federal legislation, building political coalitions and testing policy frameworks.
• The legislation is described as a "modest" step intended to establish a baseline of accountability before more extreme measures might be necessitated by a future catastrophe.
• The bill is designed to be robust against future algorithmic breakthroughs by focusing on "reasonable care" and flexibility rather than static technical requirements.