Thanks to Upendra Jadon for prompting this comparison. This is a dated review of claims and artifacts available around January 31, 2025—not a permanent model leaderboard.
Start by separating products from models
The original comparison mixed several distinct things. ChatGPT is an application that can expose different models and tools. GPT-4 and GPT-4o are model families with versioned snapshots. DeepSeek-V3 is a general mixture-of-experts model described in a late-2024 technical report. DeepSeek-R1 is a separate reasoning-model release built from DeepSeek-V3-Base. Qwen2.5-Max was presented by the Qwen team in January 2025 as a large-scale mixture-of-experts model delivered through Qwen Chat and Alibaba Cloud's API.
| Artifact | What the primary source established | What not to infer |
|---|---|---|
| Qwen2.5-Max | Qwen's January 28, 2025 post reported an MoE model pretrained on more than 20 trillion tokens, post-trained with SFT and RLHF, and made available through Qwen Chat and Alibaba Cloud API as qwen-max-2025-01-25. | The post did not release Qwen2.5-Max weights or prove that it was best for every task. |
| DeepSeek-V3 | DeepSeek reported 671 billion total parameters, 37 billion activated per token, a 128K context length, 14.8 trillion pretraining tokens, and released model weights plus inference resources under stated licenses. | Released weights and a paper do not disclose the complete training corpus, all data processing, or organizational governance. |
| DeepSeek-R1 | A separate reasoning release trained from DeepSeek-V3-Base. DeepSeek released R1/R1-Zero weights and six smaller distilled checkpoints. | R1 is not another name for V3, and deployment instructions for one checkpoint do not apply to all of them. |
| GPT-4 / GPT-4o | Proprietary OpenAI models. GPT-4 was introduced March 14, 2023; the Qwen report compared a named GPT-4o snapshot. | “ChatGPT,” “GPT-4,” and “GPT-4o” are not interchangeable evaluation labels. |
What Qwen's benchmark report showed
The Qwen team reported Qwen2.5-Max instruct-model results against DeepSeek-V3, GPT-4o, and Claude 3.5 Sonnet on a selected set of evaluations. It reported higher Qwen2.5-Max scores than DeepSeek-V3 on Arena-Hard, LiveBench, LiveCodeBench, and GPQA-Diamond, and described the result on MMLU-Pro as competitive. For base models, it compared Qwen2.5-Max with open-weight alternatives because proprietary base models were unavailable.
That is useful evidence about a defined vendor-run snapshot. It is not evidence that Qwen2.5-Max “beats ChatGPT” across all domains. A valid comparison must record the exact model snapshot, prompt template, sampling settings, tools, context, output limit, metric, evaluator, test date, and dataset version. It should also test the organization's own languages, documents, failure costs, safety constraints, latency, and workload.
What DeepSeek-V3's report showed
DeepSeek's V3 repository reports a 671B-total/37B-activated MoE model with a 128K context window. It reports 14.8 trillion pretraining tokens and 2.788 million H800 GPU hours for the full reported training process. Those are developer-reported training facts—not an independent audit and not the total cost of creating, serving, supporting, and governing a production model.
The repository publishes detailed benchmark tables. The results vary by task: V3 leads some listed comparisons and trails others. Even within one table, conclusions depend on the named model snapshot, metric, output limit, and evaluation setup. Report individual results with those qualifiers rather than saying it universally outperformed GPT-4, Gemini, or any model family.
“Open” is not one property
Evaluate openness on separate axes:
| Axis | Question |
|---|---|
| Weights | Are the exact checkpoint parameters downloadable? |
| Inference code | Is runnable code available, and under what license? |
| Architecture and report | Are model details, evaluation settings, and known limitations documented? |
| Training code and data information | Could a skilled team reconstruct a substantially equivalent training process? |
| License | Are commercial use, modification, redistribution, and derived models allowed, and under what conditions? |
| Operational transparency | Can a deployer inspect system prompts, retrieval, tools, moderation, logging, updates, and incident handling? |
DeepSeek-V3 is accurately described as open-weight with published technical material and inference code. That provides more deployment and inspection options than an API-only model, but it does not eliminate unknowns, safety risks, or governance work. Generated chain-of-thought text is not a complete or necessarily faithful disclosure of a model's internal reasoning process.
Compare cost with a dated workload model
There is no durable “27 times cheaper” answer without a defined denominator. For an API comparison, record date, region, exact model alias, input tokens, cached-input rules, output tokens, batch or priority tier, rate limits, and discounts. For self-hosting, include accelerators, memory, storage, networking, utilization, energy, orchestration, observability, security, support, and engineering time.
Run the same representative workload and report cost per successful task alongside quality, latency, throughput, reliability, and incident burden. A lower token price can cost more if the system needs longer outputs, retries, additional retrieval, or more human review.
Use the official deployment path for the exact checkpoint
Do not use the old python run.py or from deepseek import DeepSeekModel examples; they are not official DeepSeek-R1 interfaces. First select the exact checkpoint and read its model card and license. The full 671B V3/R1 artifacts and the smaller R1-distill checkpoints have materially different hardware and serving requirements.
- Pin the model identifier or revision, license, serving engine, container, and dependencies.
- Follow the current official repository or model-card instructions for that checkpoint. DeepSeek documents its own distributed V3 demo and lists community engines; the R1 repository gives separate examples for distilled checkpoints.
- Measure memory, throughput, latency, output quality, and failure behavior on target hardware before committing to a topology.
- Harden the entire application: dependency provenance, network egress, authentication, secrets, logs, telemetry, prompt and retrieval inputs, tool permissions, and update procedures.
- Confirm privacy with data-flow tests and controls. “Runs locally” does not itself mean “no data leaves the machine.”
A defensible procurement test
- Define representative tasks, languages, inputs, expected answers, and costly failure modes.
- Blind the outputs and use domain-qualified human review where judgment matters.
- Test exact model versions under the same prompts, tools, retrieval, token budgets, and sampling settings.
- Measure task success, unsupported claims, citation fidelity, harmful failures, latency, throughput, availability, and total cost.
- Document data handling, retention, model-update policy, license obligations, security controls, exit plan, and residual risk.
The January 2025 reports established that Qwen2.5-Max and DeepSeek-V3 were serious technical alternatives. They did not establish a universal winner. The right conclusion is a reproducible, dated decision for a defined workload—not a permanent ranking based on vendor charts.
Historical comments from Datanizant
No public comments on this article
No approved public comments were included in the WordPress export for this article.