OpenAI is experiencing delays in the development of its latest flagship AI model, codenamed Orion, which will eventually be known as GPT-5. The company has been working on the model for 18 months, aiming to achieve the desired results, but has faced significant challenges. One major issue is the insufficient amount of data available worldwide to make the model intelligent enough, according to a report by the Wall Street Journal.
Struggles with Training and Costs
OpenAI has conducted at least two large-scale training runs, each requiring several months of data processing to complete Orion. Despite these efforts, each run has encountered new issues, and the system has not produced the expected results. In its current form, Orion performs better than OpenAI’s existing systems, but the developers claim it has not advanced enough to justify the massive costs of keeping the new model running. Training for six months can cost approximately $500 million in computing resources alone.
Two years ago, OpenAI and its CEO Sam Altman made a big splash with the release of ChatGPT, signaling that AI would soon penetrate all aspects of human life and significantly improve them. Analysts predicted that tech giants would invest up to $1 trillion in AI over the coming years. The greatest expectations are directed toward OpenAI, the company that has sparked the AI boom.
In October 2023, OpenAI completed a funding round that valued the company at $157 billion, largely due to Altman’s promise of a “significant leap forward” with GPT-5. This model is expected to revolutionize various fields, including scientific research, and simplify everyday human tasks, such as making appointments or booking plane tickets. Researchers also hope that GPT-5 will learn to question its own assumptions, reducing its tendency to “hallucinate” and provide incorrect or irrelevant answers.
Expectations for GPT-5
If GPT-4 can be compared to the cognitive abilities of a smart high school student, GPT-5 is anticipated to demonstrate the problem-solving skills of a PhD-level expert in certain fields. However, there are no clear standards for determining whether a new generation model can be labeled as GPT-5. Models are generally tested on their performance with math and programming problems, but the final verdict is usually intuitive. As the development of large language models is as much an art as a science, this decision has yet to be made.
Models are tested during lengthy training runs, in which they are exposed to trillions of tokens, or fragments of words. A large-scale training run can take months of work and involve tens of thousands of Nvidia AI accelerators. GPT-4 cost $100 million to train, and training future models is expected to cost well over $1 billion. A failed training run is akin to a failed rocket test, as the researchers attempt to minimize the chances of failure by conducting smaller-scale experimental runs.
In mid-2023, OpenAI conducted a trial training run that tested the potential Orion architecture. However, the experiment did not yield significant results. It soon became apparent that a full-scale training run would be too time-consuming and expensive. The results of the Arrakis project further highlighted the challenges in developing GPT-5, and developers began making technical changes to strengthen Orion. They realized that achieving the desired outcome would require a lot of diverse, high-quality data, and the information available on the public internet would not suffice.
Data Shortages and New Strategies
AI models typically improve as they are exposed to more data, such as books, academic papers, and other trusted sources that help the AI express itself more clearly and handle a wider variety of tasks. Previous OpenAI models were trained with data from news articles, social media posts, and other sources. However, to make Orion smarter, more data was required, and there simply wasn’t enough. To solve this issue, OpenAI decided to generate its own data: they hired people to write code and solve complex math problems, explaining each step of their approach. Additionally, they brought in theoretical physicists to help guide the AI through the toughest challenges in their field.
This process, however, is incredibly slow. GPT-4 was trained on 13 trillion tokens. For context, a thousand people writing five thousand characters per day could generate a billion tokens in just a few months. So, OpenAI began developing synthetic data, where other AI systems were tasked with generating data to train the new AI model. However, research revealed that AI-to-AI data generation could lead to crashes or produce meaningless results. To address this, OpenAI turned to another model, o1, to handle the data generation more effectively.
Time Pressures and Internal Competition
By early 2024, OpenAI realized that time was running out. GPT-4 had been around for a year, and competitors were catching up. In fact, the new Anthropic model was believed by some to be outperforming GPT-4. Orion’s development stalled, and OpenAI was forced to pivot to other projects. A lighter version of GPT-4 and a new video generator named Sora were released. This internal shift created competition among developers for limited computing resources.
The competition in the AI industry grew so intense that large tech companies began to withhold research papers about their breakthroughs, a practice that became increasingly common. As a result, fewer papers were being published in the scientific community, and corporations started to treat their research results as trade secrets. Researchers, too, became more cautious, often avoiding public places like coffee shops or airports where someone might overhear their discussions.
A New Approach and Setbacks in 2024
In early 2024, OpenAI was ready to attempt Orion again, with a more refined dataset. In the first few months of the year, researchers conducted a few small training runs to fine-tune their strategy. By May, they were prepared to launch the large-scale training for Orion, which was planned to run through November. However, early in the process, a problem emerged: the data was not as diverse as expected, which could limit the potential quality of the AI’s training. This issue did not become apparent in the smaller-scale pilots but was evident once the full-scale run began. By that point, OpenAI had already invested significant time and money, making it difficult to start over.
Researchers made efforts to source a more diverse range of data during the training process, but it remains uncertain whether this strategy will prove successful. These ongoing difficulties with Orion led OpenAI to explore a new approach to making large language models smarter: reasoning. Reasoning would enable the AI to solve complex problems even if it hasn’t been trained on them directly. This is the same approach used in OpenAI’s o1 model, where multiple answers are generated and analyzed to find the best one.
However, there is no certainty that this will work. According to some researchers at Apple, “reasoning” models likely only interpret the data they were trained on, rather than solving entirely new problems. For instance, if small changes are made to the original problem that are unrelated to its solution, the quality of the AI’s response may deteriorate significantly.
The High Cost of Reasoning
Incorporating reasoning into AI models is expensive. OpenAI must pay for the generation of multiple answers instead of just one. Noam Brown, a research fellow at OpenAI, explained that if an AI model spends just 20 seconds thinking during a poker game, the costs increase dramatically—comparable to training the model 100,000 times longer. Orion may eventually adopt a more advanced and efficient reasoning model, and researchers are pursuing this approach in the hope that it will complement large datasets. These datasets could come from other OpenAI models, with the final results being refined using human-generated material.
At this stage, the future of Orion is uncertain, concludes NIX Solutions. The challenges OpenAI has faced in developing GPT-5 highlight the complexity of creating a next-generation AI model. Yet, OpenAI remains hopeful. The company is committed to continuing its work on Orion, and we’ll keep you updated as more integrations and breakthroughs become available. The journey to create GPT-5 is far from over, and it is clear that the stakes are incredibly high for both OpenAI and the AI industry as a whole.