AI Engineer14 min
Your SKILL.md Is 70% Stuff the Model Already Knows - Nikolay Petrov, Intel
Conference Presentation
- The session will introduce knowledge-type skills, utilizing evaluation tasks to validate deterministic performance improvements in scenarios comparing models with and without these skills loaded.
- Future optimization strategies include line-by-line brute force modifications, incorporating user feedback, and testing on new models to determine optimal modification sizes, aiming to increase solved instances from three out of five to all five.
- A deterministic evaluation process will split skills into atoms paired with questions and expected answers, estimating a fixed range of 25 to 30 atoms per skill for assessment.
- Two-arm evaluations will be conducted for each atom using a judge or panel to verify internal model understanding without answer leakage, distinguishing between new information and pre-existing knowledge at the sentence or item level.
- Performance outcomes are categorized by a four-outcome matrix where the ideal case involves solving specific questions or scenarios only after skill integration, while the fourth outcome—where poorly written skills hinder performance—is noted as rare in popular skills (one case in 400 items) but a risk for smaller, less popular skills.
- Cost-efficiency benchmarks indicate that evaluating 29 skills requires approximately $14 in Sonnet and Haiku calls.
- Knowledge strictness will be assessed through lenient checks for general ideas versus strict checks for exact details, parameters, and steps, highlighting a predicted performance gap in frontier models between general understanding (70%) and specific detail knowledge (22%).
- Smaller models currently demonstrate lower skill understanding compared to larger models like Sonnet, though overall understanding is expected to increase with newer models and training advances.
- Significant performance differences between skills are anticipated, as some are designed to be directional rather than knowledge-providing, while removing redundant information from a skill should maintain scoring to enable context size reduction.
- The Document framework's novelty functionality will serve as an initial vector for skill optimization, with plans to evolve the framework to include context impact assessments, routing, description comparisons, and naming conventions.
- The initiative aims to encourage the audience to develop higher-quality skills following the evaluation and optimization methodologies described.