OpenAI is reportedly in talks with Broadcom to build a custom inferencing chip.
A Reuters report claims the AI upstart and the chip design firm have staged confidential discussions about custom silicon, with Taiwan Semiconductor Manufacturing Company involved as the likely foundry for the effort.
Just why OpenAI wants its own inferencing chip is not known, but it’s not hard to guess why such a move appeals: the startup has enormous cloud bills – some of them comped by partners like Microsoft – and might fancy running its own hardware instead. It certainly wouldn’t be alone in finding on-prem operations are considerably cheaper than renting cloudy resources.
Developing silicon tuned to its own services could be another motive. AI applications guzzle energy, and mutual optimization of hardware and software could mean OpenAI’s services become more efficient.
OpenAI has also reportedly tried to convince investors to build giant datacenters dedicated to running AI services. Perhaps those theoretical bit barns will be cheaper to build and/or run with custom silicon inside.
Diversifying suppliers could be another motive. The world’s foundries can only crank out so much stuff, and rely on supply chains that are sometimes tenuous. OpenAI would not be immune to those vagaries but could at least reduce its dependence on third-party suppliers of finished product.
The Register can’t imagine OpenAI wants to get into the mucky business of hardware sales – an industry that requires all sorts of bothersome investments in the real world and would therefore bloat its headcount. But as inferencing is a workload best run physically close to users – because latency sucks – a play that puts devices deep into networks can’t be ruled out. That’s how content delivery networks and the likes of Netflix operate. An architecture that places an OpenAI inferencing box on the network edge is not a fantastic notion.
Custom inference chips are not novel. AWS has one called Inferentia. Google’s Tensor processing units and Microsoft’s Maia silicon can handle inferencing and training workloads.
The suggestion that OpenAI is talking to Broadcom could be one reason the chip design firm’s shares popped a little in late trading. Broadcom’s most recent quarterly earnings predicted it would sell $12 billion of AI silicon this financial year alone – a billion bucks more than its previous forecast – but investors still appeared disappointed. Teaming with the hottest name in AI software would likely get Wall Street more excited. ®
