AI Engineer12 min
Your Agent Hates Your CLI - Ayush Bhardwaj, Allos AI
Conference Presentation, Other
Core Observation: Agents frequently "secretly hate" CLIs by manually piping outputs through tools like
head,tail,grep, or Python scripts to filter data, effectively treating the CLI's raw output as unstructured and noisy.Statistical Evidence:
- Analysis of 20,000 agent calls across 14 common CLIs (Git, Cargo, Docker, etc.) revealed that 25% of read-heavy commands and over 50% of specific commands (e.g.,
git show,cargo testat 62%) were followed by filtering pipes. - 83% of observed pipes fell into repetitive patterns involving the same CLI, subcommand, and pipe tools, indicating agents consistently need specific data subsets.
- 55% of pipelines used only
head,tail, orsortdownstream, highlighting a demand for built-in range and limit controls.
- Analysis of 20,000 agent calls across 14 common CLIs (Git, Cargo, Docker, etc.) revealed that 25% of read-heavy commands and over 50% of specific commands (e.g.,
Operational Risks:
- Silent Failures: Piping to
headortailcan truncate error messages, causing tools to report success even when the underlying process failed (e.g., a compile error hidden behind the last 40 lines of output). - Incomplete Context: Agents often receive truncated JSON reports or documentation pages, leading to incorrect reasoning or the creation of "workaround" scripts rather than filing bug reports.
- Example: Cloud Code users reported a reference error that was hidden by
tailshowing only three harmless lines, masking the true failure from the model.
- Silent Failures: Piping to
Case Studies & Solutions:
- Sanity (Content Platform): After shipping a "give feedback" tool, agents moved from silent workarounds to explicit bug reports, revealing hidden friction points.
- InfraCost: Agents were writing
jqandPythonpipelines over JSON dumps; moving filtering logic directly into the CLI (compact mode) increased test pass rates from 0/6 to 6/6 and reduced benchmark costs by 41%. - Author's Tool: Post-improvement, usage of manual pipes dropped from 82% to ~25% after adding selection flags (e.g.,
view,inspect) and a "skill" file to teach agents the new options. - DouglasDB: Implemented an "agent mode" that returns plain text tables and structured error JSON instead of formatted terminal output, reducing model read tokens by 59% (though cost savings were marginal at 0.5%).
Design Recommendations for Agent-Facing CLIs:
- Fail Loudly: Ensure the final lines of any CLI output provide a summary of errors and a link/path to the full log if truncated.
- Selective Output Flags: Ship built-in flags for ranges, limits, and compact formats (e.g.,
--limit,--compact) rather than relying on agents to pipe data. - Agent Mode: Detect agent execution via environment variables and adjust output formats (e.g., plain text vs. TUI tables, JSON errors) to match agent parsing capabilities.
- Documentation as Interface: Treat
skillfiles, help text, and error messages as the primary interface; agents follow these literally, so stale or contradictory documentation causes "bugs" in agent behavior.
Methodology for Discovery:
- Transcript Analysis: Treat agent transcripts as product analytics; searching for a tool name followed by pipe characters (
|) reveals the most common friction points. - Replay Testing: Replay recorded agent commands against new tool versions to verify if output changes affect the agent's subsequent actions or reasoning.
- Automated Discovery: Tools like
chat historyon Cargo or GitHub can be used to search logs for these patterns; users should back up transcripts before running such queries as platforms may auto-delete inactive data.
- Transcript Analysis: Treat agent transcripts as product analytics; searching for a tool name followed by pipe characters (
Future Outlook:
- Current Limitation: Agents currently struggle to design optimal tool interfaces in a single shot; they tend to replicate common human patterns (like manual piping) rather than optimizing for efficiency.
- Recommended Approach: Humans should define the tool design based on transcript data, then use agents to validate the design against real-world execution logs.
- Trend: MCP servers and tools built for agents often suffer from poor descriptions (97% flagged with issues) and context files written by agents that increase costs without improving task success.