AI Engineer14 min
Your SKILL.md Is 70% Stuff the Model Already Knows - Nikolay Petrov, Intel
Conference Presentation
Skill Architecture and Types
- Skills are defined as portable, reproducible functionalities that pack repeatable prompts into agent-ready units.
- Two distinct categories exist:
- Knowledge Skills: Introduce new capabilities or information to the model.
- Process Skills: Provide a repeatable flow of actions without adding new information.
- Validation methods include informal "feeling" checks and deterministic evaluation suites (evals).
- Evals function by comparing two experimental arms:
- Arm A: Model runs without the skill loaded.
- Arm B: Model runs with the skill and its context loaded.
- Success metrics include task completion rates, token consumption, and execution time.
Evaluation Methodology: The "Atoms" Approach
- Skills are decomposed into 25–30 discrete "atoms," where each atom represents a specific instruction or item.
- Each atom is paired with a question and an expected answer for rigorous testing.
- Judging is performed by a panel of different model families to prevent answer leakage.
- Outcomes are mapped to a four-quadrant matrix based on model capability with and without the skill:
- Known: Model answers correctly both with and without the skill (skill adds no value).
- Degraded: Model answers correctly without the skill but fails or underperforms with the skill (instruction interference).
- Novel: Model fails without the skill but succeeds with it (the ideal knowledge-transfer scenario).
- Missed: Model fails to answer correctly in both scenarios (skill implementation is ineffective).
- Empirical data from Anthropic skills indicates that performance degradation (the "Degraded" quadrant) is rare, observed in only 1 case out of ~400 items, but remains a risk for smaller, less popular skills.
- Visualization techniques allow for line-by-line or item-by-item coloring to identify exactly which parts of a skill contribute new knowledge versus existing model capabilities.
Granularity and Model Limitations
- Evaluations can be lenient (general concepts) or strict (exact details, parameters, values).
- Strict evaluation reveals significant gaps in frontier models:
- Anthropic Opus shows a drop from 70% correct on general concepts to 22% on specific details.
- Larger models (e.g., Sonnet) generally demonstrate better understanding of skill details than smaller models.
- While newer models excel at generalization, they still require explicit data for solving specific problems.
- Testing across three popular skills showed that understanding varies significantly by skill type, even within the same model family.
Optimization Strategy and Tooling
- Skills can be optimized by identifying and removing redundant information that the model already possesses, thereby reducing token usage and context size.
- This optimization provides a deterministic alternative to brute-force testing of instruction combinations.
- The process serves as an initial vector for skill refinement before final production evals.
- The
documeframework supports this workflow, offering tools for:- Comparing agent contexts, skills, and MCP tools.
- Assessing context impact and routing efficiency.
- Comparing skill descriptions, naming conventions, and novelty.
- The cost of running this extensive evaluation across 29 skills using Sonnet and Haiku was approximately $14.