SambaNova Systems, the Intel-backed AI chip startup, is demonstrating that older Nvidia GPUs can still compete at the frontier of AI inference, if you pair them with the right accelerator.
Third-party benchmarks from Artificial Analysis show that SambaNova’s heterogeneous computing platform, which combines four Nvidia H200 GPUs with 16 of the company’s own SN50 RDU (Reconfigurable Dataflow Unit) accelerators, achieves 763 tokens per second on the MiniMax M2.7 model at short context lengths. At longer contexts, the system sustains more than 450 tokens per second.
How disaggregated inference works
The key innovation is a technique called disaggregated inference, splitting the two phases of AI model execution across different hardware. The computationally intensive prefill phase, during which the model processes the input prompt and generates key-value caches, runs on the H200 GPUs. The memory-bandwidth-bound decode phase, where output tokens are generated one by one, runs on SambaNova’s SN50 RDUs.
This approach extends the useful life of GPU fleets that might otherwise be considered obsolete for modern AI workloads. Instead of replacing H200s entirely, data center operators can add SambaNova racks as decode accelerators, handling the step that typically becomes the bottleneck in conversational AI and agentic applications.
SambaNova is planning larger configurations with 128 and eventually 256 accelerators to maintain high token rates at high throughput.
Market context
Disaggregating prefill from decode has become a major trend in AI infrastructure. Nvidia demonstrated a similar approach with its NVL72 rack, and AMD, AWS, and Cerebras have all announced competing platforms. SambaNova’s differentiator is its air-cooled design, the SN50 rack can be deployed in existing data centers without the liquid cooling infrastructure that Nvidia’s upcoming Rubin GPUs require.
The approach is particularly relevant for agentic AI workloads, long-running code assistants, autonomous research tools, and multi-step reasoning chains, where decode latency dominates the user experience.
Funding and valuation
SambaNova completed the first close of a $1 billion Series F funding round, led by General Atlantic, giving the company an $11 billion valuation. Intel is a strategic backer. The company is ramping production of its fifth-generation SN50 RDU.
Its first announced large-scale customer is Together AI, with deployment partner Vector Core Compute.
Sources: The Register (July 8)