AI Digest — May 25, 2026 (Morning)

A recent study on large language model pretraining examines the assumption that stronger teachers yield better students in knowledge distillation. The researchers varied architecture sizes and training token budgets to create different teacher-student relationships and found that even small and undertrained teachers can improve larger students with proper loss mixing. However, they also observed that a stronger teacher is not always better, as excessive parameters or training tokens can saturate or reverse distillation gains. The study further notes that distillation improves generalization more readily than in-domain fitting. These findings challenge the common belief that distillation pretraining always requires a strong teacher.

A new study proposes a distributional theory of how hypernymy is encoded geometrically in language representations. The theory is based on the assumption that words closer on the WordNet hypernym graph co-occur more often. The researchers characterize the spectrum of the resulting embedding Gram matrix of word2vec embeddings and prove that the leading eigenvectors produce a hierarchical splitting geometry. The study confirms these predictions in word2vec embeddings and shows that the same signature extends to other models. The results indicate that hierarchical concept geometry in LLMs emerges from the spectral structure of pairwise word statistics.

A recent analysis by Epoch.ai reveals that memory has become a significant component of AI chip costs, accounting for nearly two-thirds of the total. This shift is largely driven by the increasing complexity and data requirements of modern AI models. As AI chips continue to evolve, the cost and efficiency of memory will play a crucial role in determining their overall performance. The findings are based on data insights from the AI chip industry, highlighting the growing importance of memory in AI chip design. This trend has significant implications for AI chip manufacturers and researchers.

A recent study on arXiv examines the concept of constraint decay in large language models (LLMs) used for back-end code generation. The researchers found that LLMs can struggle to maintain constraints and generate high-quality code, particularly in complex scenarios. This fragility has significant implications for the reliability and trustworthiness of AI-generated code. The study highlights the need for further research into improving the robustness of LLMs in code generation tasks.

Researchers found that local Large Language Models (LLMs) perform better when taught to ask clarifying questions before providing answers. This approach allows the models to gather more context and provide more accurate responses. The system, which involves prompting the model to ask questions, has shown promising results in improving the overall performance of local LLMs. This development could lead to more efficient and effective use of LLMs in various applications. The study highlights the importance of contextual understanding in language models.

DeepSeek Reasonix is a native coding agent with high caching and low cost, developed by esengine. It is designed to improve coding efficiency and reduce costs. The agent is part of the DeepSeek project, which aims to create AI-powered coding tools. The release of DeepSeek Reasonix has garnered significant attention, with 499 points and 209 comments on the announcement. The technical details of the agent are available on the esengine GitHub page.

Apple has introduced the Perceptual Image Codec, a learned image compression technique. The codec is designed to improve image compression efficiency while maintaining perceptual quality. It is available on Apple's GitHub repository, ml-pico, along with a research paper detailing its development and technical aspects. This codec could potentially be used in various applications, including image and video processing. The release provides insights into Apple's approach to image compression using machine learning.

A new theoretical framework models Large Language Model (LLM) training as information transmission over a noisy channel, based on the Shannon-Hartley theorem. This perspective, known as the Shannon Scaling Law, captures the interaction between the learning signal and intrinsic noise in LLMs. The law is validated through experiments on Pythia and OLMo2 models under various perturbations, outperforming classical scaling laws. It accurately predicts performance degradation and loss basins, even for unseen models. The framework provides a unified explanation for non-monotonic phenomena in LLM training, such as catastrophic overtraining and quantization-induced degradation.

Researchers introduced SkillOpt, a systematic controllable text-space optimizer for agent skills, which trains skills as the external state of a frozen agent. SkillOpt uses a separate optimizer model to make edits on a single skill document based on scored rollouts and a held-out validation score. The approach is evaluated across six benchmarks, seven target models, and three execution harnesses, outperforming human, one-shot LLM, and other skill optimization methods. SkillOpt shows significant improvements in accuracy, with an average lift of +23.5 points in direct chat and +24.8 points inside the Codex agentic loop. The optimized skill artifacts also retain value when transferred across model scales and execution environments.