Lecture, Conference Presentation
Arjun Guha: How Language Models Model Programming Languages & How Programmers Model Language Models
- The research agenda focuses on improving LLMs for low-resource languages like OCaml, analyzing user interaction patterns, and interpreting model internals during programming tasks, with plans to integrate these three threads.
- Current evaluation benchmarks such as Human Eval (167 problems) and MBPP (approx. 400 problems) are criticized for being saturated, consisting almost exclusively of Python despite multilingual training data, and containing approximately 10% faulty problems, prompting the creation of new language-agnostic benchmarks.
- Early 2022 Codex model performance is cited as approximately 45% on Python benchmarks, while current state-of-the-art models approach 90% saturation; however, performance on lower-resource languages remains significantly lower.
- A small vanilla model named QuenCoder 2.5 3D Instruct trained specifically on OCaml data achieved a 10% success rate on a language-agnostic Human Eval benchmark in its base form, rising to 17% after training, compared to 72% for GPT-5 Mini.
- Interpretability experiments using activation steering on a model with 36 layers and 2,000-dimensional vectors aim to alter language output by adding a "minus Python plus OCaml" patch to the residual stream, successfully shifting default language generation from Python to OCaml.
- Activation steering techniques are expected to correct type prediction errors in Python and TypeScript, achieving up to 60% accuracy from a zero baseline, suggesting a shared internal representation of types across languages.
- A study from early 2023 involving 120 students with basic Python knowledge and no prior LLM experience utilized the Codex model on trivial programming tasks from exams; results indicated a wide distribution of success rates where many students failed to verify code or generate effective prompts without explicit editing capabilities.
- Data from the student study includes a unique benchmark derived from over 2,000 prompts, noting that approximately 25% of prompts from failed student trajectories were actually high-quality yet still failed to produce correct code.
- Future plans include further work on reinforcement learning for code LLMs in low-resource languages and refining benchmarks by systematically adjusting prompt detail levels to observe predictable performance degradation curves.
- Technical resources for these experiments include the NNSight project and the NDIF NSF-sponsored GPU service, which facilitate querying large models like the 4 billion parameter LAMA model without local hosting costs.