New DeepSeek model catches up to Claude and GPT for pennies, but too often makes up facts

DeepSeek updated its open-weight V4.1 Flash model to revision 1017. On paper, it's a small point release, but the update noticeably improves the model's efficiency. The release page pushes the slogan "wish-driven programming made real," an obvious nod to "vibe coding" – the model is pitched as a tool for open-ended tasks described in broad strokes.

The low price and strong benchmark scores come with a catch, though. DeepSeek V4.1 Flash 1017 (DSF-1017) hallucinates far more often than closed competitors, confidently serving up made-up information.

The large language model market is split into two camps right now.

On one side are closed commercial models from the frontier labs: expensive to run, but consistently excellent results.

On the other are Chinese open-weight models, which trade some reliability for a much lower running cost, especially in long-running agentic workloads.

DSF-1017 is built on a Mixture-of-Experts (MoE) architecture. The model has 552 billion parameters in total, but only 16 billion are active when processing each request. That approach keeps the "knowledge" of a large model while saving a lot of compute. The model is multimodal, accepts both text and images as input, and offers a context window of 1 million tokens, which has become the standard.

According to DeepSeek internal tests, the model sits between GPT-5.6 Sol and Claude Opus 5, and even beats Anthropic's model in agentic tasks. The headline achievement of this revision is per-task efficiency and token usage. Compared to V4 Flash, the new version uses one-quarter of the HBM and one-eighth of the SSD storage for its KV cache, spending only 890 bytes per token entry.

The KV cache stores intermediate attention data for context that has already been processed. The longer the conversation or agent session, the more space it takes up. Shrinking it directly cuts the need for expensive video memory and lets the same hardware serve more requests.

To illustrate the point, DeepSeek published a cost-per-task graph based on the DeepSWE dataset from September 2, adding its own results for both DSF and DSF-1017. Both versions land in the top-left corner of the chart, where a high task completion rate meets a low price. Even in low-effort mode, DSF-1017 delivers a completion rate comparable to the "max" settings of pricey competitors. DeepSWE is only one of many benchmarks, but the result is still impressive.

The model is open-weight, so it can be run on any infrastructure. Even DeepSeek own off-peak pricing looks very modest:

  • $0.15 per million input tokens

  • $0.6 per million output tokens

  • $0.003 per million input tokens on cache hits

  • $0.75 per 1,000 search queries

Search costs less than at competitors – the closest one, Parallel, charges $1 for the same volume. Third-party inference providers have undercut the DeepSeek API more than once, so the final price could end up even lower.

Independent analytics platform Artificial Analysis ranks DSF-1017 (max) at #2 on its intelligence index among large open-weight models. The model scored 45, while the overall leader, Claude Opus 5.5 (max with fallback), got 58. DSF-1017 is also one of the fastest models in its class, and only Xiaomi MiMo V2.6-Pro beats it on cost per task.

Trouble starts where a model needs to know the limits of its own knowledge. Users generally prefer models that decline to answer when they lack supporting facts, and DSF-1017 handles that poorly.

In the Artificial Analysis Omniscience benchmark, the model ranked among the worst. It answered questions correctly 43% of the time, which is among the best for open-weight models, though closed competitors still score higher. That alone wouldn't be a problem if the model skipped questions it had no basis for. Instead, it made up an answer in 67% of such cases.

The main Omniscience index awards points for correct answers and deducts them for wrong ones, while non-answers don't count. DSF-1017 scored just 5 points, compared to Kimi K3-max's 20, with the leading closed models at 42 and above. That doesn't make the model incompetent – quite the opposite. It just means tasks should be chosen with care.

DSF-1017 works best for tasks whose output can be checked automatically:

  • programming

  • OCR with review

  • data format conversion

In these scenarios, the agent can run verification steps against its own work and try again if something fails. Open-ended research, looking up facts, and interpreting or drafting documents where precise wording matters are better left to frontier models or a human. Hallucinations are hard to catch in those tasks, and the results won't be very trustworthy.

Other strong open-weight models show a similar pattern. A few months ago, Moonshot AI released Kimi K3, praised for its efficiency and low price at near-frontier intelligence. It also frequently produces hallucinated data even for short, precise fact look-ups, and Artificial Analysis figures back that up.

Open-weight models still play an important role. They make AI agents more accessible in absolute dollar terms and help keep the cost of compute in check as frontier labs potentially raise prices.

More news
Tags: