newsfilter
← Interviews
AI Engineer14 min

Your SKILL.md Is 70% Stuff the Model Already Knows - Nikolay Petrov, Intel

Nikolay Petrov

Conference Presentation
  • Skill Architecture and Types

    • Skills are defined as portable, reproducible functionalities that pack repeatable prompts into agent-ready units.
    • Two distinct categories exist:
      • Knowledge Skills: Introduce new capabilities or information to the model.
      • Process Skills: Provide a repeatable flow of actions without adding new information.
    • Validation methods include informal "feeling" checks and deterministic evaluation suites (evals).
    • Evals function by comparing two experimental arms:
      • Arm A: Model runs without the skill loaded.
      • Arm B: Model runs with the skill and its context loaded.
    • Success metrics include task completion rates, token consumption, and execution time.
  • Evaluation Methodology: The "Atoms" Approach

    • Skills are decomposed into 25–30 discrete "atoms," where each atom represents a specific instruction or item.
    • Each atom is paired with a question and an expected answer for rigorous testing.
    • Judging is performed by a panel of different model families to prevent answer leakage.
    • Outcomes are mapped to a four-quadrant matrix based on model capability with and without the skill:
      • Known: Model answers correctly both with and without the skill (skill adds no value).
      • Degraded: Model answers correctly without the skill but fails or underperforms with the skill (instruction interference).
      • Novel: Model fails without the skill but succeeds with it (the ideal knowledge-transfer scenario).
      • Missed: Model fails to answer correctly in both scenarios (skill implementation is ineffective).
    • Empirical data from Anthropic skills indicates that performance degradation (the "Degraded" quadrant) is rare, observed in only 1 case out of ~400 items, but remains a risk for smaller, less popular skills.
    • Visualization techniques allow for line-by-line or item-by-item coloring to identify exactly which parts of a skill contribute new knowledge versus existing model capabilities.
  • Granularity and Model Limitations

    • Evaluations can be lenient (general concepts) or strict (exact details, parameters, values).
    • Strict evaluation reveals significant gaps in frontier models:
      • Anthropic Opus shows a drop from 70% correct on general concepts to 22% on specific details.
      • Larger models (e.g., Sonnet) generally demonstrate better understanding of skill details than smaller models.
    • While newer models excel at generalization, they still require explicit data for solving specific problems.
    • Testing across three popular skills showed that understanding varies significantly by skill type, even within the same model family.
  • Optimization Strategy and Tooling

    • Skills can be optimized by identifying and removing redundant information that the model already possesses, thereby reducing token usage and context size.
    • This optimization provides a deterministic alternative to brute-force testing of instruction combinations.
    • The process serves as an initial vector for skill refinement before final production evals.
    • The docume framework supports this workflow, offering tools for:
      • Comparing agent contexts, skills, and MCP tools.
      • Assessing context impact and routing efficiency.
      • Comparing skill descriptions, naming conventions, and novelty.
    • The cost of running this extensive evaluation across 29 skills using Sonnet and Haiku was approximately $14.