American startup Groq, founded by ex-Google engineer Jonathan Ross, has unveiled a specialized chip designed to accelerate the execution of pre-trained AI models, aiming to be ten times more cost-effective than traditional GPUs. The chip, named LPU (Language Processing Unit), focuses on optimizing the execution of large language models at runtime rather than during training, addressing a specific need in the AI infrastructure landscape.
Emphasizing real-time responsiveness, the LPU chip is tailored to prevent visitor attrition on websites by ensuring prompt responses from AI chatbots. With the capability to produce 300 tokens per second per user, the chip can answer queries 75 times faster than a human. Groq has demonstrated this prowess through its online platform, GroqChat, showcasing the efficient execution of a Meta Llama-2 70B model on its chips.
While research suggests that only large companies may have the resources to develop their own AI models, Groq aims to cater to the broader market of businesses seeking to execute pre-trained models efficiently. The company acknowledges that its current focus does not directly address the intermediate challenge of fine-tuning pre-trained models with specific company data.
The LPU chip, based on Groq's in-house architecture called TSP (Tensor Stream Processor), operates through a parallel processing approach designed to perform a fraction of processing on 320 data items while consuming energy equivalent to 20 processing units, presenting significant efficiency gains compared to GPUs.
In terms of products, Groq offers the GroqChip for server integration, the GroqCard PCIe 4.0 card for networking, the GroqNode server with high processing power, and the GroqRack cluster, combining multiple servers for enhanced computing power. Accompanying these hardware offerings is the GroqWare Suite, a comprehensive software package including a compiler for model adaptation, a Linux library for chip control, and the GroqView tool for testing settings.
Amid its innovative hardware and software solutions, Groq acknowledges the need to recompile large language models for execution on the LPU chip, a process that currently takes about five days for each model variation. Nevertheless, the company has begun marketing its products in various forms, making strides to establish its presence in the AI infrastructure market.
