Intelligence at the Edge of Chaos

This repository contains the official implementation for our ICLR 2025 paper, Intelligence at the Edge of Chaos. In this work, we investigate how the complexity of rule-based systems, specifically elementary cellular automata (ECA), impacts the intelligence of large language models (LLMs) trained on sequences generated by these systems. We train LLMs on a spectrum of ECA patterns—ranging from uniform and periodic to highly chaotic—and evaluate their performance across various downstream tasks, including logical reasoning and chess move prediction. Our experiments demonstrate that LLMs exposed to intermediate, structured complexity significantly outperform those trained on overly simple or excessively chaotic patterns. These results highlight a critical insight: structured complexity serves as an essential catalyst for developing advanced artificial cognition. Our findings propose that intelligence emerges most effectively at the boundary between order and chaos.

Table of Contents

1. Overview
2. Setup
3. Pre-training
4. Downstream Tasks
   • ARC
   • Chess
   • NIM Game
5. Model Configurations