Large Models: an Exclusive Party for the Upper Crust

Credit：Visual China

By Zhang Shuai and Shaw Wan

BEIJING, July 11 (TiPost)— “As long as you have a graphics processing unit (GPU), I can buy the whole server,” said a buyer. Zhang Yang, head of a cloud computing service provider, witnessed customers going on a shopping spree since March. “At that time, customers were in such a rush that all they cared about was to hoard the devices. They didn’t have any requirements for the products, nor did they mention cloud networking or data storage. They didn’t even know how to put them into good use,” said Zhang.

It wasn’t until April that some of these buyers started to figure out what kind of devices they truly needed. They went on a wrong track of hoarding a bunch of GPUs, when training a large model demands massive distributed computing power that often pairs with a full set of services.

The computing power industry involves various parts, such as artificial intelligence (AI) chips, servers, optical modules, data centers and cloud computing platforms, forming the driving force for the digital economy. Because of such scale and complexity, only a small number of enterprises are able to afford to join the race. Since training a large model is the start point of the large model ecosystem basic, enough computing power is the admission ticket to the industry.

The birth of the AI chatbot ChatGPT in last November has proven the partnership between Microsoft and Open AI a success. Training models in the cloud turned out to make the cut. For companies, cloud service providers can provide scalable computing resources, including hardware and software, which saves businesses from building their own infrastructure. They can also serve talents in the industry, such as R&D engineers, algorithm engineers and individual developers. Since they are often backed up by tech giants, they are rich in financial support, talents and data, enabling them to crack on with the big model training.

The frenzied ChatGPT wave

The companies that are qualified for the race are mostly established giants. For example, the super computer behind ChatGPT is Azure, a Microsoft-backed cloud computing platform that was initially released 14 years ago . At the current stage, to catch up with ChatGPT or to get ahead in the race, companies are competing against each other with their strategies and technologies that were developed in the past years.

“The large model training is obviously hyped. The industry should be more rational and avoid taking advantage of the concept to lure investment. I think the companies that want to scramble for a piece of the pie shouldn’t start from scratch. There are possible chances for them, but there are also great challenges,” said a person in charge of large-scale model products of a tech giant.

From the perspective of academia, OpenAI does not embody revolutionary innovation. It is more like “engineered innovation” of the artificial general intelligence (AGI) products. The engineered production process involves different phases of big model training, including the research, engineering, products and organization.

“It is also hard to pull the engineered production off. However, it at least proved that having more computing power and more data would work,” said Han Kai, principal engineering manager at Microsoft.

Although engineered production has proven to be a success, it was hard for many other companies to choose the path in the first place because the huge input didn’t promise a bright future. Chinese enterprises tend to follow others’ steps, which is also why ChatGPT wasn’t born in China. 

Challenges in big model training

There are at least three major challenges to achieve the engineered production of cloud computing.

The first is computing power. “For example, GPT-3, which has 175 billion parameters, required 314 zettaFLOPS of computing power in order for it to be trained. However, as one GPU only delivers 312 teraFLOPS of deep learning performance, it takes 32 years to train one large model. Therefore, it is necessary to introduce the distributed training, where we use multiple devices and multiple GPUs to train large models,” said Chen Xi, an expert at Chinatelecom Cloud, a cloud computing platform backed by China Telecom.

Data storage can also be a problem. The video memory of a single GPU can no longer load a model with hundreds of billions of parameters. It takes about a few terabytes to fully load so many parameters. That number can grow even bigger, when intermediate results generated during the training process, such as gradients and optimizer states, are also taken into consideration. Thus, hundreds of GPUs are needed.

Therefore, companies generally adopt a way of pipeline parallelism, and run different layers of the model in GPUs of different nodes. In this way, a group of nodes only need to load a limited number of parameters, reducing the pressure on the memory.

As the big model training task is broken into a sequence of processing stages, there will be a large amount of communication between clusters, resulting in high requirements on the bus and its bandwidth. The amount of data transferred can reach up to hundreds of gigabytes.

Apart from these three major challenges, the fast growth of large model parameters and the slow development of chip technology is also hindering the industry. In recent years, with the introduction of the transformer, the number of model parameters has increased by 15 times every two years. However, the development of chip technology lags behind. The computing power of a single GPU grew by less than 4 times, with the chip process decreased from 7 nanometers to 4 nanometers.

Large model training requires not only computing power, but also storage, security, and the training framework. A complete set of platforms or services are needed to provide support. “We feel like there are not many service providers who can satisfy the needs of large model training. And the overall supply of high-performance computing power is relatively tight,” said Chen.

Opportunities for Chinese chip makers

As the Chinese companies are trying to jump on the bandwagon of large model training, the demand for chips soar. Although Chinese chip makers are trying to catch up with the top chip designers, they are not the first choice of many computing power platforms.

“At present, when everyone is working on large model training, time is of the essence. What the industry needs is high-end products, so that they can avoid stability or maturity problems. That’s why the Chinese chips are left out,” said Zhang Yalin, the chief operating officer of Enflame, a Shanghai-based  AI start-up developing cloud-based deep learning chips for AI training platforms.

The American chip maker Nvidia is the dominant supplier for the reasoning and training of large models in China. Chinese tech giant Baidu once purchased tens of thousands of Nvidia A800s within just half a year.

According to Nvidia’s financial results  for first quarter fiscal 2024, the revenue of Nvidia’s data center business was 4.28 billion dollars, a record high with an increase of 14 percent compared with the same period last year. In May , shares of the company soared, taking the company’s valuation above one trillion dollars.

However, in terms of reasoning, there are still business opportunities for Chinese chips. “I think Chinese chip makers should take a different path, by starting with reasoning and fine-tuning. They can later cooperate with research institutes from universities and national laboratories to move on to large model training,” Zhang said.

The development of AI chips is faster than Moore’s Law, which could also lead to a decline in growth in some cases, according to Xie Guangjun, the vice president of Baidu. The temporary shortage of computing power is because the computing power cannot keep up with the demand. The overall imbalanced supply chain is also part of the cause.

As of now, no Chinese chips can replace the high-end chips produced by Nvidia, such as the A100. Several Chinese chip makers planned to release similar products later this year. As the shortage of Nvidia chips persists, Chinese chips are likely to grab a slice of the cake after next year, once they are able to meet the requirements.

“Although Internet companies care more about the price-performance ratio, they need to pay more attention to the total cost of ownership when it comes to computing power. For example, the GPU cluster of my company can support 1,000 GPUs, whose performance is similar to that of Nvidia’s cluster of 600 GPUs. But our products can also be competitive, as long as we can provide more cost-effective and customized services,” said Zhang.

If Enflame’s products were to be favored by the Internet clients, they needed to have 1.5 times the performance of Nvidia products and twice the price-performance ratio in the desired scenarios and businesses, Zhang added.

As early as June 2021, Baidu AI Cloud began to plan the construction of a new GPU cluster of high-performance. Together with NVIDIA, it completed the design of the InfiniBand network architecture, which can be equipped with over 10,000 GPUs to provide EFLOPS-level computing power. Thanks to this cluster, Baidu released its ChatGPT-style AI bot called Wenxin Yiyan, or ERNIE Bot, in March.

更多精彩内容，关注钛媒体微信号（ID：taimeiti），或者下载钛媒体App

​Read More