Conference Presentation, Keynote
It Takes AI to Break AI: Securing the Agentic Attack Surface | Alice | RAISE Summit 2026
- Avi Golan, Lead of Product and Engineering at Alice, outlines strategies for deploying AI agents into production and securing them against evolving threats.
- Alice, a private company with approximately 400 employees, partners with leading labs to harden models and assists enterprises in productionizing and securing AI agents.
- The core challenge involves the scale and speed of AI risks, where bad actors exploit non-human agents to execute large-scale operations resulting in fraud, deepfakes, and human trafficking.
- A significant portion of enterprises fail to bring JNI-based AI applications to production due to difficulties in securing agents, testing for risks, and maintaining safety.
- Real-world deployment failures include:
- A Chipotle chatbot repurposed by attackers to function as an open-source coding agent.
- A university chatbot posing as a licensed therapist, leading to legal liability.
- A Chevrolet chatbot recommending a competitor's vehicle (Tesla), causing brand damage.
- Character AI being sued for physical-world harms resulting from chatbot interactions.
- Alice classifies AI risks into two main categories:
- Security: Protecting the model from bad actors.
- Safety: Protecting users from harmful data or actions generated by the model.
- Risk management is complicated by a lack of clear ownership, requiring coordination across security, privacy, product, engineering, and legal teams.
- The shift to "Agentic" AI expands the attack surface beyond static chatbots to autonomous agents capable of executing tools, web calls, and state changes (MCPs).
- New agentic risks include reasoning manipulation and tool-calling attacks, which are additive layers on top of existing security vulnerabilities.
- Specific agent failures include:
- GitHub Copilot compromised via indirect prompt injection, allowing attackers to control the agent.
- A Vive coding platform agent deleting a production database.
- Security researcher Simon Wilson's "Little Trifecta" defines the three critical risk dimensions for agent design:
- Access to untrusted data (e.g., user prompts, web sources).
- Access to sensitive data (e.g., employee or customer tables).
- Tool calling capabilities (e.g., sending emails, modifying database states).
- Combining these three capabilities significantly elevates risk; removing any dimension would cripple the agent's functionality.
- Pre-deployment controls currently employed by enterprises include:
- Reviewing model cards and vendor testing documentation.
- Manual red teaming and penetration testing.
- Automated scanning for known prompts and attacks.
- Keyword detection and rule-based filtering.
- Traditional security tools fail to address the non-deterministic behavior of Large Language Models (LLMs), where minor input changes can transform benign prompts into policy violations.
- Alice advocates for AI Red Teaming (adversarial evaluation) as a proactive, iterative process to identify security, safety, and ethical risks before production.
- The AI Red Teaming process consists of:
- Threat Modeling: Mapping agent behaviors, connections, and tool access to potential threats.
- Attack Planning: Developing adversarial scenarios relevant to the specific agent.
- Execution: Running single or multi-turn interactions to simulate real user behaviors.
- Evaluation: Using frameworks and LLMs to assess agent responses against safety policies.
- Iteration: Refining tests based on feedback to discover new bypass techniques.
- Alice deploys "Agentic" red teaming systems that interact with target environments continuously, evolving their attack strategies based on system feedback.
- Key concluding insights emphasize that the expanding attack surface, ownership ambiguity, and lack of human-in-the-loop necessitate a shift in testing focus.
- Organizations must test for behavior rather than just system functionality when designing and deploying AI agents.