AI Engineer12 min
Your Agent Hates Your CLI - Ayush Bhardwaj, Allos AI
Conference Presentation, Other
- Plans to extract agent transcript data to identify missing tool capabilities based on repeated manual piping patterns, with an October audit recording approximately 1,000 calls for case study analysis.
- Predicts that agents will bypass friction silently or write cut-down commands by hand if tools lack built-in filtering, estimating that 83% of pipes in everyday CLIs follow a pattern repeated at least 10 times.
- Anticipates that providing agents a direct mechanism to complain will shift behavior from silent workarounds to explicit error expression, contrasting this with the "zero of six" success rate on hard questions previously seen versus "6 of 6" after filtering was integrated into Cloud Code's CLI.
- Forecasts that optimizing tools for agents reduces costs, noting a 41% drop in benchmark costs and a reduction in pipe reads from 82% to roughly 25% after introducing new flags and compact output modes.
- Proposes specific design improvements including flags to return data around matches, a compact JSON mode, and an inspect command to summarize sessions, aiming to replace output size caps with selection flags.
- Warns that stale documentation acts as a bug because agents persist in following it, necessitating updates to skill files, help text, and error messages to teach usage of new flags.
- Suggests using replayed agent commands against old and new builds to generate real-world test cases and establish appropriate ranges and limits, noting that 55% of measured pipelines rely solely on head, tail, or set downstream operations.
- Highlights that agent-built tools often incorrectly assume a human-centric design, citing that 97% of tool descriptions on real MCP servers contain issues and most servers merely represent existing APIs rather than optimizing for agents.
- References external studies indicating that context files written by agents do not improve task success, cost more to execute, and underperform compared to human-written files.
- Advises organizations to treat transcripts as critical product analytics, while warning that Cloud Code deletes inactive transcripts after 30 days, requiring users to backup data before analysis.
- Observes that 81% of the time the GitHub CLI limit flag is used, supporting the view that agent adoption of a flag depends on agents actively reaching for it.