Less is More: Recursive Reasoning with Tiny Networks

This repository is a reproduction and experimental verification of the paper “Less is More: Recursive Reasoning with Tiny Networks” by Alexia Jolicoeur-Martineau (2025). This repository is not an official implementation. It is maintained solely for research reproduction. For the official version, please visit SamsungSAILMontreal/TinyRecursiveModels

Experiments

Reproduction Results

Method	Params	Sudoku	Maze	ARC-1 (@2)	ARC-2 (@2)
TRM-Att	7M	77.71	78.70	41.00	3.33
TRM-MLP	5M	84.80	/	/	/

Model Checkpoints on Hugging Face

TinyRecursiveModel-Maze-Hard

TinyRecursiveModels-Sudoku-Extreme-att

TinyRecursiveModels-Sudoku-Extreme-mlp

TinyRecursiveModels-ARC-AGI-1

TinyRecursiveModels-ARC-AGI-2

The file pretrain.py has been slightly modified to handle missing evaluators gracefully:

    try:
        evaluators = create_evaluators(config, eval_metadata)
    except Exception as e:
        import traceback
        print("No evaluator found:", repr(e))
        traceback.print_exc()
        evaluators = []

In addition to evaluation during training, a standalone evaluation script run_eval.py has been added. This script allows loading checkpoints and running evaluation separately. We report exact accuracy for Maze and Sudoku, and pass@k for ARC.

torchrun --nproc_per_node=8 run_eval.py
# or evaluate all tasks
bash eval_scripts.sh

All experiments were conducted on 8 × H GPUs with a global batch size of 4608.

ARC-AGI-1

run_name="pretrain_att_arc1concept_8"
torchrun --nproc-per-node 8 --rdzv_backend=c10d --rdzv_endpoint=localhost:0 --nnodes=1 pretrain.py \
arch=trm \
epochs=200000  \
data_paths="[data/arc1concept-aug-1000]" \
arch.L_layers=2 \
arch.H_cycles=3 arch.L_cycles=4 \
+run_name=${run_name} ema=True

Runtime: 37h

ARC-AGI-2

run_name="pretrain_att_arc2concept_8"
torchrun --nproc-per-node 8 --rdzv_backend=c10d --rdzv_endpoint=localhost:0 --nnodes=1 pretrain.py \
arch=trm \
epochs=200000  \
data_paths="[data/arc2concept-aug-1000]" \
arch.L_layers=2 \
arch.H_cycles=3 arch.L_cycles=4 \
+run_name=${run_name} ema=True

Runtime: 49h

Sudoku-Extreme:

run_name="pretrain_mlp_t_sudoku"
torchrun --nproc-per-node 8 --rdzv_backend=c10d --rdzv_endpoint=localhost:0 --nnodes=1 pretrain.py \
arch=trm \
data_paths="[data/sudoku-extreme-1k-aug-1000]" \
evaluators="[]" \
epochs=100000 eval_interval=5000 \
lr=1e-4 puzzle_emb_lr=1e-4 weight_decay=1.0 puzzle_emb_weight_decay=1.0 \
arch.mlp_t=True arch.pos_encodings=none \
arch.L_layers=2 \
arch.H_cycles=3 arch.L_cycles=6 \
+run_name=${run_name} ema=True

run_name="pretrain_att_sudoku"
torchrun --nproc-per-node 8 --rdzv_backend=c10d --rdzv_endpoint=localhost:0 --nnodes=1 pretrain.py \
arch=trm \
data_paths="[data/sudoku-extreme-1k-aug-1000]" \
evaluators="[]" \
epochs=100000 eval_interval=5000 \
lr=1e-4 puzzle_emb_lr=1e-4 weight_decay=1.0 puzzle_emb_weight_decay=1.0 \
arch.L_layers=2 \
arch.H_cycles=3 arch.L_cycles=6 \
+run_name=${run_name} ema=True

Runtime: 40min

Maze-Hard:

run_name="pretrain_att_maze30x30"
torchrun --nproc-per-node 4 --rdzv_backend=c10d --rdzv_endpoint=localhost:0 --nnodes=1 pretrain.py \
arch=trm \
data_paths="[data/maze-30x30-hard-1k]" \
evaluators="[]" \
epochs=50000 eval_interval=5000 \
lr=1e-4 puzzle_emb_lr=1e-4 weight_decay=1.0 puzzle_emb_weight_decay=1.0 \
arch.L_layers=2 \
arch.H_cycles=3 arch.L_cycles=4 \
+run_name=${run_name} ema=True

Runtime: 2h