newsfilter
← Interviews
AI Engineer8 min

Your Spec Has Bugs. Catch Them Before Your Agent Codes. - Rajath Mahendrappa Kotyal, AWS

Rajath Mahendrappa Kotyal, Rajat Kothiyal

Conference Presentation, Lecture
  • Project Overview: AWS Automated Reasoning Group developed the "Kiro Specification Analyzer," a tool using formal methods and automated reasoning to validate specifications before coding agents generate code.

  • Motivation & Statistics:

    • Public research indicates a 20–40% drop in LLM code accuracy when prompts or specifications are ambiguous.
    • 60–90% of code initially generated as "valid" is found to be incorrect after execution.
    • Deploying ambiguous specifications results in twice as many regressions.
    • The tool aims to detect logical conflicts and gaps in natural language specifications prior to code generation, similar to design document verification.
  • Technical Methodology (Three-Step Process):

    • Step 1: Refinement: The LLM attempts to self-correct wording and clarify ambiguity without formalization.
    • Step 2: Formalization: If refinement fails, natural language rules are converted into Satisfiability Modulo Theories (SMT) formulas.
    • Step 3: Automated Verification: SMT formulas are passed to a SAT solver to determine validity (true/false); if ambiguity or contradiction is detected, the LLM is instructed to re-evaluate the specific sentence.
  • Logical Interpretation & Clustering:

    • When contradictions persist, the system samples LLM translations 7–10 times to analyze behavioral consistency.
    • These interpretations are clustered by logical equivalence to identify specific assumptions made by the LLM.
    • The system distinguishes between:
      • Conflicts: Mutually exclusive rules (e.g., one rule mandates "backorder" for unavailable inventory, while another forbids it entirely).
      • Gaps: Missing logic for specific scenarios (e.g., no rule covering "idle orders").
  • Human-in-the-Loop Resolution:

    • The system presents clustered interpretations and options to the user for final decision-making, acknowledging the LLM lacks implicit domain knowledge.
    • Resolution options include: retaining the rule as written, modifying it (e.g., "delete record"), or hiding specific logic from users.
    • If no consensus is reached, the system utilizes LLM judges or abstains from implementing the specific rule to prevent errors.
    • Final correctness guarantee relies on the user, as they possess the necessary domain context.
  • Conclusion & Takeaways:

    • Users must rigorously check specification snippets for ambiguity before engaging coding agents.
    • Clustering translations by logical equivalence is critical for identifying genuine LLM interpretation failures.
    • The proposed workflow is: LLM translates → Tool checks for conflicts → Human decides.
  • Attribution: Work contributed by team members Noindrila, Jatin, Stefan, Nadia, and Remy Delmas.