OpenAI and Broadcom unveiled a new LLM-optimized inference chip on Wednesday, as the generative AI vendor seeks to reduce token costs for businesses and gain a competitive advantage over rival Anthropic.
The chip, named Jalapeño, is OpenAI’s first intelligence processor. It is the first step for the ChatGPT maker toward building a full stack behind its models and products. OpenAI designed the chip from the start to work with all LLMs. The vendor said it will release a detailed technical report on the model in the next few months but said the chip’s architecture reduces data movement and balances compute, memory and networking resources to achieve peak performance.
The chip is a strategic move for the AI vendor, which has been trying to diversify its infrastructure by partnering with Nvidia, Cerebras and now Broadcom. However, having its own chip now gives it options, as more businesses are conscious of inference costs and rising token prices. OpenAI, for the first time in a long while, also appears to have grabbed the spotlight from Anthropic, which has been releasing models and services rapidly and outpacing OpenAI.
“[Enterprises] are seeing explosive cost switches associated with tokenomics and token consumption,” said Chirag Dekate, an analyst at Gartner. “As the token consumption rises, their cost structures are increasing.”
Providing Options
With this deal with Broadcom, OpenAI is appealing to businesses looking for a way to ease the rising costs that come with the use of more tokens.
“This enables OpenAI to change the economics of converting watts to tokens, and that also translates to higher margin efficiencies and margin gains,” Dekate said. “If the model ecosystem turns into a price war, this gives OpenAI much more leeway and bandwidth to tweak and tune the pricing without affecting profitability.”
While OpenAI will use other underlying infrastructure from either Nvidia or Cerebras where it seems fit, having its own AI chip provides it with much of the flexibility that Google has with TPUs, AWS with its Trainium chips, and Alibaba with its Zhenwu M890 chip.
The Broadcom Factor and Challenges
For Broadcom, the semiconductor designer gets to collaborate with a major model company and co-innovate with it.
“By Broadcom engaging in this, they’re able to change the economics, and they can position themselves to deliver better efficiency at a different price point,” Dekate said.
While Jalapeño appears promising, little is known about the chip, as its technical details have not yet been released. It is unclear how it will integrate with other systems. And it is unclear whether Broadcom and OpenAI will be able to make it a sustainable product capable of maturing through generations of chip design.
“The longevity of [custom-designed microchips] is hard to predict,” Dekate said. “Getting into infrastructure is really hard. You need to have a multiyear pipeline, multiyear, multi-generation view of executing and scaling.”
He added that with competitors Nvidia and AMD established as longtime chipmakers, OpenAI and Broadcom will need to find a way to sustain chip supply.
“It’s too premature, but overall, this is a really good direction,” Dekate continued. “It is a logical direction for the model providers to take, especially frontier model providers like OpenAI and Anthropic. I would anticipate Anthropic will have something similar cooking up soon, too.”
