A Shakeout is Coming for Inference Startups

If you tuned into Nvidia’s and Microsoft’s recent quarterly earnings calls, you probably heard their leaders discuss how inference—a fancy word for operating an artificial intelligence model—is becoming a bigger part of the computing workloads in their data centers. In other words, chatbots like ChatGPT Enterprise are serving real customers. That contrasts with what was going on for much of last year, when major tech firms were developing, or training, the models, which also uses a lot of computing capacity.

With so much inference going on, a host of startups has sprung up to provide servers for inference computing for developers using their own in-house models (such as AI video firm Pika) or free, open-source ones like Llama 2 and Mixtral. We sometimes refer to these inference providers as AI server resellers because they take on the hassle of renting and maintaining servers to power models for their customers. (Some of these companies, like Together AI, also offer servers for training models.)

Unfortunately for the half a dozen or so startups that collectively raised more than $700 million in venture capital to pursue this business, a shakeout is coming.