Training AI models like Google’s Gemini Ultra, costing $191 million, and OpenAI’s GPT-4, costing $78.4 million, is becoming increasingly expensive, as shown in Stanford’s 2024 AI Index Report.
Millions of dollars are needed, and the expense of training sophisticated AI models like Google’s Gemini Ultra and OpenAI’s ChatGPT is rising quickly.
The cost of the computer power needed to train them is skyrocketing as computational needs rise. AI companies are reconsidering their approach to training generative AI systems in response. These frequently involve methods to lower computing expenses in light of the present growth trajectories.
The following chart, created by Dorothy Neufeld of Visual Capitalist and based on data from Stanford University’s 2024 Artificial Intelligence Index Report, illustrates how training expenses for sophisticated AI models have increased.
How Training Cost is Determined
To evaluate the costs of training AI models, which were based on cloud computing leasing fees, The AI Index worked with research firm Epoch AI. The length of time the model was trained, the pace at which the hardware was used, and the cost of the training hardware are among the important variables that were examined.
Although many have hypothesized that the cost of training AI models has increased, there isn’t enough thorough evidence to back up these assertions. One of the uncommon sources for these estimates is the AI Index.
Ballooning Training Costs
The training costs of popular AI models since 2017 are displayed below, with inflation taken into account:
The cost of training OpenAI’s GPT-4 last year was projected to be $78.4 million, a significant increase over Google’s PaLM (540B) model, which had cost $12.4 million the year before.
To put things in perspective, Transformer, an early AI model created in 2017, cost $930 to train. The architecture of many big language models in use today is shaped in part by this paradigm.
At an astounding $191 million, Google’s AI model, Gemini Ultra, is even more expensive. The model beats GPT-4 as of early 2024 in several criteria, most notably the Massive Multitask Language Understanding (MMLU) test. This benchmark is an essential tool for assessing large language models’ performance. For example, it is well-known for assessing knowledge and ability to solve problems in 57 different academic areas.
Training Future AI Models
To address growing expenses, AI businesses are coming up with innovative ways to train language models in light of these constraints.
These encompass a variety of techniques, like building smaller models with specialized functions. Some businesses are experimenting with producing artificial intelligence (AI) systems using their data. But a definite breakthrough is still a ways off.
These days, artificial intelligence models that use synthetic data have demonstrated the ability to generate gibberish in response to specific commands, a phenomenon known as “model collapse.”
Recently, GreatGameIndia reported that in a post on X, Ethereum co-founder Vitalik Buterin said OpenAI’s GPT-4 has passed the Turing test, citing recent preprint research from the University of California, San Diego.
One Response
What you don’t mention is the “Data Annotation” industry, which employs tens of thousands of people to explain the data to new / naive AIs. My guess is that’s where the money goes.