OpenAI Unveils Jalapeño: First Custom AI Inference Chip Delivers Massive Speed and Efficiency Gains

OpenAI has shared the first official performance numbers for Jalapeño, its purpose-built custom chip designed specifically to accelerate AI model inference. Developed in close collaboration with Broadcom, the silicon focuses entirely on serving trained large language models, delivering substantial improvements in response times and energy efficiency compared to traditional hardware setups.

The announcement marks a major milestone in OpenAI’s push to build and optimize its own infrastructure stack, directly targeting the massive compute and power demands of running hundreds of millions of daily queries across ChatGPT and its enterprise developer platforms.

30-Second TL;DR

• Core News: OpenAI revealed early benchmark results for Jalapeño, its first in-house AI inference accelerator.
• Key Detail: The chip achieved 1.5x to 1.9x more work per watt and 1.7x to 3.6x lower end-to-end latency across massive open-weight models.
• What It Means: Jalapeño is captive data-center silicon aimed at slashing operating costs and boosting generation speeds, with initial deployment set for late 2026.

Built for Inference, Not Training

Unlike general-purpose GPUs that handle both heavy model training and daily generation workloads, Jalapeño is an Application-Specific Integrated Circuit (ASIC) engineered exclusively for inference.

When an AI model generates responses, the workload splits into two phases:

• The Prefill Phase: The system ingests and processes the prompt, requiring raw compute power.
• The Decode Phase: The model generates output token by token, where memory bandwidth and communication speeds often create severe bottlenecks.

Jalapeño was architected to keep model data-specifically the Key-Value (KV) cache-closer to where calculations happen. OpenAI designed the chip as an interconnected system encompassing memory, custom interconnects, and specialized networking, minimizing data transfer delays across chips when running multi-hundred-billion-parameter models.

Benchmark Highlights: Speed and Efficiency

OpenAI evaluated Jalapeño using SemiAnalysis’s public InferenceX benchmark suite across three massive open models: GPT-OSS 120B, DeepSeek R1 670B, and Kimi K2.5 1T.

Key Jalapeño Performance Gains

Metric	Measured Improvement	Workload Focus
Peak Throughput Efficiency	1.5x – 1.9x more work per watt	Data center power optimization
End-to-End Latency	1.7x – 3.6x faster generation	Faster overall response delivery
Interactive Performance	2.1x – 4.1x higher throughput	Real-time chat & reasoning
Single-Stream Decode Speed	~1,400 tokens/sec (GPT-OSS 120B)	High-speed conversational AI

On Kimi K2.5 1T, the largest 1-trillion-parameter model evaluated in the test suite, OpenAI recorded a 3.4x reduction in end-to-end latency alongside a 1.5x gain in peak throughput efficiency compared to baseline commercial configurations. Notably, these gains were reached without enabling speculative decoding or multi-token prediction optimizations.

Designed in Nine Months with AI Assistance

One of the most notable aspects of the Jalapeño project is the velocity of its development. OpenAI stated that the engineering team moved from initial design to physical tape-out in nine months, using internal AI tools to verify logic, lay out circuits, and optimize software compilers.

Since announcing Jalapeño, our first custom inference chip, we’ve been testing it and the system around it.

The results show a major advance: more intelligence from every watt and faster responses, delivering both higher throughput and lower latency in one architecture without… pic.twitter.com/vj7VOrA8pP

— OpenAI (@OpenAI) August 25, 2026

The physical hardware was co-developed with Broadcom, which contributed specialized ASIC physical design and networking IP, with manufacturing handled on TSMC’s advanced process nodes.

Complementing, Not Replacing, Commercial GPUs

OpenAI leadership made it clear that Jalapeño is not intended to immediately replace external hardware suppliers. The company confirmed it will continue purchasing and deploying high-density GPU clusters from Nvidia, AMD, and cloud partners to handle its massive training runs and diversified infrastructure needs.

Instead, Jalapeño serves as captive silicon-proprietary hardware integrated solely inside OpenAI’s private data centers to power ChatGPT queries and developer API requests.

Availability and Future Roadmap

Jalapeño represents the first generation of an ongoing hardware initiative. OpenAI confirmed that second-generation silicon is already well into development, with third-generation architectures in the planning phase.

Production qualification and software optimizations are currently underway, with initial internal deployment of Jalapeño hardware clusters scheduled to begin before the end of 2026.

Source