AI Engineer8 min
Your Spec Has Bugs. Catch Them Before Your Agent Codes. - Rajath Mahendrappa Kotyal, AWS
Rajath Mahendrappa Kotyal, Rajat Kothiyal
Conference Presentation, Lecture
Project Overview: AWS Automated Reasoning Group developed the "Kiro Specification Analyzer," a tool using formal methods and automated reasoning to validate specifications before coding agents generate code.
Motivation & Statistics:
- Public research indicates a 20–40% drop in LLM code accuracy when prompts or specifications are ambiguous.
- 60–90% of code initially generated as "valid" is found to be incorrect after execution.
- Deploying ambiguous specifications results in twice as many regressions.
- The tool aims to detect logical conflicts and gaps in natural language specifications prior to code generation, similar to design document verification.
Technical Methodology (Three-Step Process):
- Step 1: Refinement: The LLM attempts to self-correct wording and clarify ambiguity without formalization.
- Step 2: Formalization: If refinement fails, natural language rules are converted into Satisfiability Modulo Theories (SMT) formulas.
- Step 3: Automated Verification: SMT formulas are passed to a SAT solver to determine validity (true/false); if ambiguity or contradiction is detected, the LLM is instructed to re-evaluate the specific sentence.
Logical Interpretation & Clustering:
- When contradictions persist, the system samples LLM translations 7–10 times to analyze behavioral consistency.
- These interpretations are clustered by logical equivalence to identify specific assumptions made by the LLM.
- The system distinguishes between:
- Conflicts: Mutually exclusive rules (e.g., one rule mandates "backorder" for unavailable inventory, while another forbids it entirely).
- Gaps: Missing logic for specific scenarios (e.g., no rule covering "idle orders").
Human-in-the-Loop Resolution:
- The system presents clustered interpretations and options to the user for final decision-making, acknowledging the LLM lacks implicit domain knowledge.
- Resolution options include: retaining the rule as written, modifying it (e.g., "delete record"), or hiding specific logic from users.
- If no consensus is reached, the system utilizes LLM judges or abstains from implementing the specific rule to prevent errors.
- Final correctness guarantee relies on the user, as they possess the necessary domain context.
Conclusion & Takeaways:
- Users must rigorously check specification snippets for ambiguity before engaging coding agents.
- Clustering translations by logical equivalence is critical for identifying genuine LLM interpretation failures.
- The proposed workflow is: LLM translates → Tool checks for conflicts → Human decides.
Attribution: Work contributed by team members Noindrila, Jatin, Stefan, Nadia, and Remy Delmas.