newsfilter
← Interviews
AI Engineer12 min

Your Agent Hates Your CLI - Ayush Bhardwaj, Allos AI

Ayush Bhardwaj

Conference Presentation, Other
  • Core Observation: Agents frequently "secretly hate" CLIs by manually piping outputs through tools like head, tail, grep, or Python scripts to filter data, effectively treating the CLI's raw output as unstructured and noisy.

  • Statistical Evidence:

    • Analysis of 20,000 agent calls across 14 common CLIs (Git, Cargo, Docker, etc.) revealed that 25% of read-heavy commands and over 50% of specific commands (e.g., git show, cargo test at 62%) were followed by filtering pipes.
    • 83% of observed pipes fell into repetitive patterns involving the same CLI, subcommand, and pipe tools, indicating agents consistently need specific data subsets.
    • 55% of pipelines used only head, tail, or sort downstream, highlighting a demand for built-in range and limit controls.
  • Operational Risks:

    • Silent Failures: Piping to head or tail can truncate error messages, causing tools to report success even when the underlying process failed (e.g., a compile error hidden behind the last 40 lines of output).
    • Incomplete Context: Agents often receive truncated JSON reports or documentation pages, leading to incorrect reasoning or the creation of "workaround" scripts rather than filing bug reports.
    • Example: Cloud Code users reported a reference error that was hidden by tail showing only three harmless lines, masking the true failure from the model.
  • Case Studies & Solutions:

    • Sanity (Content Platform): After shipping a "give feedback" tool, agents moved from silent workarounds to explicit bug reports, revealing hidden friction points.
    • InfraCost: Agents were writing jq and Python pipelines over JSON dumps; moving filtering logic directly into the CLI (compact mode) increased test pass rates from 0/6 to 6/6 and reduced benchmark costs by 41%.
    • Author's Tool: Post-improvement, usage of manual pipes dropped from 82% to ~25% after adding selection flags (e.g., view, inspect) and a "skill" file to teach agents the new options.
    • DouglasDB: Implemented an "agent mode" that returns plain text tables and structured error JSON instead of formatted terminal output, reducing model read tokens by 59% (though cost savings were marginal at 0.5%).
  • Design Recommendations for Agent-Facing CLIs:

    • Fail Loudly: Ensure the final lines of any CLI output provide a summary of errors and a link/path to the full log if truncated.
    • Selective Output Flags: Ship built-in flags for ranges, limits, and compact formats (e.g., --limit, --compact) rather than relying on agents to pipe data.
    • Agent Mode: Detect agent execution via environment variables and adjust output formats (e.g., plain text vs. TUI tables, JSON errors) to match agent parsing capabilities.
    • Documentation as Interface: Treat skill files, help text, and error messages as the primary interface; agents follow these literally, so stale or contradictory documentation causes "bugs" in agent behavior.
  • Methodology for Discovery:

    • Transcript Analysis: Treat agent transcripts as product analytics; searching for a tool name followed by pipe characters (|) reveals the most common friction points.
    • Replay Testing: Replay recorded agent commands against new tool versions to verify if output changes affect the agent's subsequent actions or reasoning.
    • Automated Discovery: Tools like chat history on Cargo or GitHub can be used to search logs for these patterns; users should back up transcripts before running such queries as platforms may auto-delete inactive data.
  • Future Outlook:

    • Current Limitation: Agents currently struggle to design optimal tool interfaces in a single shot; they tend to replicate common human patterns (like manual piping) rather than optimizing for efficiency.
    • Recommended Approach: Humans should define the tool design based on transcript data, then use agents to validate the design against real-world execution logs.
    • Trend: MCP servers and tools built for agents often suffer from poor descriptions (97% flagged with issues) and context files written by agents that increase costs without improving task success.