DeepSeek published a paper in Nature on September 17, 2025 demonstrating that large language models can develop sophisticated reasoning capabilities through reinforcement learning alone, without requiring human-labeled reasoning examples. The model, DeepSeek-R1, achieved 79.8% pass@1 accuracy on the American Invitational Mathematics Examination 2024, outperforming the average human competitor. DeepSeek released the model weights, training code, and data samples under an MIT license.
The Core Claim: Reasoning Without Demonstrations
The paper's central argument challenges a foundational assumption in AI development. Since OpenAI's GPT-4 and the widespread adoption of chain-of-thought prompting, the field has operated on a premise: teaching LLMs to reason requires showing them how humans reason. You provide step-by-step examples, the model learns to imitate them, performance improves.
DeepSeek's researchers tested what happens when you skip this step entirely.
They took DeepSeek-V3 Base (their pre-trained foundation model) and applied Group Relative Policy Optimization (GRPO) directly, using only binary feedback: correct answer or incorrect answer. No demonstrations. No reasoning traces. No human-annotated intermediate steps. The prompt template was minimal: think inside these tags, answer inside these other tags.
The result was DeepSeek-R1-Zero, a model that learned to reason by figuring out what worked through trial and error.
What R1-Zero Actually Learned
The training dynamics tell the more interesting story. At initialization, the model scored 15.6% on AIME 2024. After 10,400 training steps (1.6 epochs), it reached 77.9%. But the trajectory was not linear, and the behavioral changes were not obvious.
The researchers tracked the average response length throughout training. Early in training, responses averaged around 2,500 tokens. By the end, they exceeded 17,500 tokens. The model did not learn to be verbose for its own sake. It learned that longer, more exploratory responses correlated with correct answers on hard problems, so it generated more of them.
More striking was what the researchers call the "aha moment." Around training step 8,000, the frequency of the word "wait" in model outputs increased sharply. The model had not been taught to pause and reconsider. It had discovered, through reinforcement, that producing phrases like "Wait, wait. Wait. That's an aha moment I can flag here. Let's reevaluate this step by step" preceded correct answers more often than not.
The paper includes an example response where the model, mid-solution, writes: "Wait, wait. Wait. That's an aha moment I can flag here." The researchers describe this as "anthropomorphic" reasoning that emerged without explicit instruction.
Other emergent behaviors included verification (checking answers before submitting), reflection (identifying mistakes and backtracking), and systematic exploration of alternative solution approaches. None of these patterns were specified in the training objective.
Why This Matters for AI Development
The traditional pipeline for training reasoning models has three stages: pre-training on text, supervised fine-tuning on demonstrations, then reinforcement learning from human feedback. DeepSeek's approach eliminates the middle step for reasoning tasks.
This has two implications.
First, scalability. Human reasoning demonstrations are expensive and slow to produce. A mathematics PhD can write perhaps a dozen high-quality solution traces per day. Pure RL requires only problems and answers, both of which exist in vast quantities for domains with verifiable solutions.
Second, ceiling removal. When models learn to imitate human reasoning, their performance is bounded by human reasoning ability. A model trained on human solutions to AIME problems can, at best, match human performance on similar problems. A model that discovers its own reasoning strategies can, in principle, find approaches humans have not considered.
The researchers phrase this carefully: "Rather than explicitly teaching the model how to solve a problem, we simply provide it with the right incentives and it autonomously develops advanced problem-solving strategies."
The Practical Model: DeepSeek-R1
R1-Zero had problems. It mixed languages mid-response (a consequence of training on multilingual data). Its outputs were difficult to read. It could reason but not write. For these reasons, DeepSeek developed R1, a more practical variant.
The R1 training pipeline has four stages. First, a small supervised fine-tuning stage using "cold-start" data: a few thousand examples of responses that demonstrate human-aligned thinking in conversational format. Second, reinforcement learning with rule-based rewards plus a language consistency penalty. Third, rejection sampling and SFT incorporating both reasoning and non-reasoning data. Fourth, a final RL stage using model-based preference rewards alongside rule-based reasoning rewards.
The staged approach preserved the reasoning capabilities of R1-Zero while making the model usable in practice. On AlpacaEval 2.0 (a user preference benchmark), R1 scores 87.6%, compared to R1-Zero's 24.7%. On Arena-Hard, R1 achieves 92.3% versus R1-Zero's 53.6%. Reasoning performance remained comparable: 79.8% on AIME 2024 versus 77.9%.
The final R1 model also beats GPT-4o (2024-05-13) on most benchmarks listed in the paper, though the researchers describe its safety level as "comparable" to GPT-4o rather than superior.
Technical Implementation: How GRPO Works
GRPO is a simplification of Proximal Policy Optimization (PPO), the algorithm OpenAI used for RLHF in ChatGPT. PPO requires training a separate value network to estimate expected future rewards. GRPO eliminates this requirement.
For each question, GRPO samples a group of outputs (16 in DeepSeek's implementation) from the current policy. Each output receives a reward (correct or incorrect). The algorithm computes relative advantages by comparing each output's reward to the group mean, normalized by standard deviation.
The policy update then increases the probability of above-average responses and decreases the probability of below-average ones, with a clipping mechanism to prevent updates that are too large. A KL divergence penalty keeps the updated policy from drifting too far from a reference policy.
The key practical choices: learning rate of 3×10⁻⁶, KL coefficient of 0.001, sampling temperature of 1.0, and replacing the reference model every 400 steps. Maximum response length was 32,768 tokens initially, increased to 65,536 tokens after step 8,200 (which caused the performance spike visible in training curves).
Reward Design: The Critical Constraint
The success of pure RL depends entirely on reward reliability. The researchers emphasize this repeatedly.
For reasoning tasks with verifiable answers (mathematics, coding competitions, logic puzzles), DeepSeek used rule-based rewards exclusively. Math problems require answers in a specific format (inside a box), enabling automated verification. Code problems use a compiler with test cases. No neural reward model is involved.
The paper explains why: "Neural reward models are susceptible to reward hacking during large-scale RL." A model-based reward system becomes exploitable as training progresses. The policy model learns to produce outputs that score highly according to the reward model without actually solving problems correctly.
For general tasks without verifiable answers (writing, open-ended questions), DeepSeek could not avoid model-based rewards. They limited RL to a few hundred steps for these tasks and relied primarily on supervised fine-tuning. The paper acknowledges this as a limitation: "Scaling up pure RL methods remains an open challenge" for tasks without reliable reward signals.
Distillation: Transferring Reasoning to Smaller Models
The paper includes results on distilling R1's reasoning capabilities into smaller models. These "distilled" models are trained by generating reasoning traces from R1, then using supervised learning to teach smaller models to produce similar traces.
The distilled models outperform their instruction-tuned counterparts (versions fine-tuned conventionally without R1-style reasoning). DeepSeek released these alongside the full R1 weights, noting they "will greatly contribute to the research community by providing a valuable resource for understanding the mechanisms underlying long CoT reasoning models."
The specific distilled model names and sizes are not detailed in the main paper text, but the researchers indicate they made "several smaller models" publicly available.
Limitations the Authors Acknowledge
The paper dedicates substantial space to limitations, an unusual choice for a Nature publication.
Structure output and tool use: R1 cannot reliably produce structured outputs (JSON, specific formats) or use external tools (calculators, search engines). The researchers note this is "not hard" to address with additional RL environments, suggesting a future version.
Token efficiency: R1 exhibits "overthinking" on simple problems, generating thousands of tokens when dozens would suffice. Unlike approaches such as Monte Carlo tree search (which explore multiple paths explicitly), R1's computational cost scales with output length regardless of problem difficulty.
Language mixing: The model sometimes reasons in English even when prompted in other languages, a consequence of DeepSeek-V3 Base's training distribution favoring English and Chinese.
Prompt sensitivity: Few-shot prompting (providing examples before the question) degrades R1's performance, unlike most LLMs. The researchers recommend zero-shot prompting exclusively.
Software engineering tasks: The long evaluation times for coding tasks (requiring compilation and testing) limited RL iterations. R1 shows minimal improvement over DeepSeek-V3 on software engineering benchmarks such as SWE-bench. Future versions will use rejection sampling or asynchronous evaluation.
Safety Considerations
The paper includes an explicit ethics statement acknowledging that enhanced reasoning capabilities create new risks. R1 can be jailbroken to produce dangerous content (the example given is explosive manufacturing plans), and its reasoning ability makes such content "more operational and executable."
Safety benchmark performance is described as "moderate level (comparable with GPT-4o)." The researchers note that a "risk control system" raises safety to a "superior standard," but this system is not described in detail.
The model is released under an MIT license with no usage restrictions beyond standard terms.
What This Paper Does Not Prove
The paper's claims are narrower than headlines might suggest.
The approach works for tasks with verifiable answers. Mathematics, competitive programming, and STEM problems with deterministic solutions are amenable to rule-based rewards. Creative writing, nuanced analysis, and tasks requiring judgment are not. The paper does not claim otherwise.
The emergent reasoning is not human-like in origin, even if it resembles human reasoning superficially. The model discovered that producing certain linguistic patterns (self-correction, verification, "wait") correlates with correct answers on the training distribution. Whether this constitutes "understanding" in any meaningful sense remains contested.
The safety analysis is preliminary. The paper acknowledges vulnerability to jailbreaks and fine-tuning attacks. A model that reasons well about chemistry can reason well about dangerous chemistry.
The Larger Research Context
DeepSeek is a Chinese AI company founded in 2023. The DeepSeek-V3 base model underlying R1 was released in December 2024. This Nature paper represents peer-reviewed validation of claims the company made in a January 2025 preprint.
The paper cites 36 references, drawing heavily on OpenAI's work (GPT-4, RLHF) and Google DeepMind's research on chain-of-thought prompting. The GRPO algorithm was introduced in DeepSeek's earlier work on mathematical reasoning (DeepSeekMath, February 2024).
Publication in Nature (rather than a machine learning venue such as NeurIPS or ICML) signals the journal's assessment that this research has implications beyond the AI research community.
What Comes Next
The paper closes with a forward-looking statement: "The future holds immense potential for solving any task that can be effectively evaluated by a verifier, regardless of its complexity for humans. Machines equipped with such advanced RL techniques are poised to surpass human capabilities in these domains."
The qualifier is important: "tasks that can be effectively evaluated by a verifier." The hardest problems in AI may be precisely those where constructing reliable verification is impossible.
DeepSeek-R1-Zero, DeepSeek-R1, data samples, and distilled models are available at github.com/deepseek-ai/DeepSeek-R1.
