In 2023, “generative AI” was undeniably the hottest term in the tech industry.

The launch of the generative application ChatGPT by OpenAI has sparked a frenzy in the market, prompting various tech giants to join the race.

As per a report from TechNews, currently, NVIDIA dominates the market by providing AI accelerators, but this has led to a shortage of their AI accelerators in the market. Even OpenAI intends to develop its own chips to avoid being constrained by tight supply chains.

On the other hand, due to restrictions arising from the US-China tech war, while NVIDIA has offered reduced versions of its products to Chinese clients, recent reports suggest that these reduced versions are not favored by Chinese customers.

Instead, Chinese firms are turning to Huawei for assistance or simultaneously developing their own chips, expected to keep pace with the continued advancement of large-scale language models.

In the current wave of AI development, NVIDIA undoubtedly stands as the frontrunner in AI computing power. Its A100/H100 series chips have secured orders from top clients worldwide in the AI market.

As per analyst Stacy Rasgon from the Wall Street investment bank Bernstein Research, the cost of each query using ChatGPT is approximately USD 0.04. If ChatGPT queries were to scale to one-tenth of Google’s search volume, the initial deployment would require approximately USD 48.1 billion worth of GPUs for computation, with an annual requirement of about USD 16 billion worth of chips to sustain operations, along with a similar amount for related chips to execute tasks.

Therefore, whether to reduce costs, decrease overreliance on NVIDIA, or even enhance bargaining power further, global tech giants have initiated plans to develop their own AI accelerators.

Per reports by technology media The Information, citing industry sources, six global tech giants, including Microsoft, OpenAI, Tesla, Google, Amazon, and Meta, are all investing in developing their own AI accelerator chips. These companies are expected to compete with NVIDIA’s flagship H100 AI accelerator chips.

Progress of Global Companies’ In-house Chip Development

• Microsoft

Rumors surrounding Microsoft’s in-house AI chip development have never ceased.

At the annual Microsoft Ignite 2023 conference, the company finally unveiled the Azure Maia 100 AI chip for data centers and the Azure Cobalt 100 cloud computing processor. In fact, rumors of Microsoft developing an AI-specific chip have been circulating since 2019, aimed at powering large language models.

The Azure Maia 100, introduced at the conference, is an AI accelerator chip designed for tasks such as running OpenAI models, ChatGPT, Bing, GitHub Copilot, and other AI workloads.

According to Microsoft, the Azure Maia 100 is the first-generation product in the series, manufactured using a 5-nanometer process. The Azure Cobalt is an Arm-based cloud computing processor equipped with 128 computing cores, offering a 40% performance improvement compared to several generations of Azure Arm chips. It provides support for services such as Microsoft Teams and Azure SQL. Both chips are produced by TSMC, and Microsoft is already designing the second generation.

• Open AI

OpenAI is also exploring the production of in-house AI accelerator chips and has begun evaluating potential acquisition targets. According to earlier reports from Reuters citing industry sources, OpenAI has been discussing various solutions to address the shortage of AI chips since at least 2022.

Although OpenAI has not made a final decision, options to address the shortage of AI chips include developing their own AI chips or further collaborating with chip manufacturers like NVIDIA.

OpenAI has not provided an official comment on this matter at the moment.

• Tesla

Electric car manufacturer Tesla is also actively involved in the development of AI accelerator chips. Tesla primarily focuses on the demand for autonomous driving and has introduced two AI chips to date: the Full Self-Driving (FSD) chip and the Dojo D1 chip.

The FSD chip is used in Tesla vehicles’ autonomous driving systems, while the Dojo D1 chip is employed in Tesla’s supercomputers. It serves as a general-purpose CPU, constructing AI training chips to power the Dojo system.

• Google

Google began secretly developing a chip focused on AI machine learning algorithms as early as 2013 and deployed it in its internal cloud computing data centers to replace NVIDIA’s GPUs.

The custom chip, called the Tensor Processing Unit (TPU), was unveiled in 2016. It is designed to execute large-scale matrix operations for deep learning models used in natural language processing, computer vision, and recommendation systems.

In fact, Google had already constructed the TPU v4 AI chip in its data centers by 2020. However, it wasn’t until April 2023 that technical details of the chip were publicly disclosed.

• Amazon

As for Amazon Web Services (AWS), the cloud computing service provider under Amazon, it has been a pioneer in developing its own chips since the introduction of the Nitro1 chip in 2013. AWS has since developed three product lines of in-house chips, including network chips, server chips, and AI machine learning chips.

Among them, AWS’s lineup of self-developed AI chips includes the inference chip Inferentia and the training chip Trainium.

On the other hand, AWS unveiled the Inferentia 2 (Inf2) in early 2023, specifically designed for artificial intelligence. It triples computational performance while increasing accelerator total memory by a quarter.

It supports distributed inference through direct ultra-high-speed connections between chips and can handle up to 175 billion parameters, making it the most powerful in-house manufacturer in today’s AI chip market.

• Meta

Meanwhile, Meta, until 2022, continued using CPUs and custom-designed chipsets tailored for accelerating AI algorithms to execute its AI tasks.

However, due to the inefficiency of CPUs compared to GPUs in executing AI tasks, Meta scrapped its plans for a large-scale rollout of custom-designed chips in 2022. Instead, it opted to purchase NVIDIA GPUs worth billions of dollars.

Still, amidst the surge of other major players developing in-house AI accelerator chips, Meta has also ventured into internal chip development.

On May 19, 2023, Meta further unveiled its AI training and inference chip project. The chip boasts a power consumption of only 25 watts, which is 1/20th of the power consumption of comparable products from NVIDIA. It utilizes the RISC-V open-source architecture. According to market reports, the chip will also be produced using TSMC’s 7-nanometer manufacturing process.

China’s Progress on In-House Chip Development

China’s journey in developing in-house chips presents a different picture. In October last year, the United States expanded its ban on selling AI chips to China.

Although NVIDIA promptly tailored new chips for the Chinese market to comply with US export regulations, recent reports suggest that major Chinese cloud computing clients such as Alibaba and Tencent are less inclined to purchase the downgraded H20 chips. Instead, they have begun shifting their orders to domestic suppliers, including Huawei.

This shift in strategy indicates a growing reliance on domestically developed chips from Chinese companies by transferring some orders for advanced semiconductors to China.

TrendForce indicates that currently about 80% of high-end AI chips purchased by Chinese cloud operators are from NVIDIA, but this figure may decrease to 50% to 60% over the next five years.

If the United States continues to strengthen chip controls in the future, it could potentially exert additional pressure on NVIDIA’s sales in China.

Read more

• [News] NVIDIA’s Exclusive Chips for China Now Reported to be Available for Pre-Order, Priced Similar to Huawei Products

(Photo credit: NVIDIA)

SoftBank Group founder Masayoshi Son, as per a report from Bloomberg, is planning to raise USD 100 billion to establish an AI chip company, aiming to complement the group’s ARM business.

The report indicates that Masayoshi Son plans to name the new artificial intelligence chip venture “Izanagi,” after the deity of creation and life in Japanese mythology, and Son himself will directly lead the project.

Regarding funding, reportedly, one proposed scheme under consideration involves SoftBank providing USD 30 billion, while another USD 70 billion may come from institutions in the Middle East.

However, details regarding the final funding sources and how the funds will be utilized in the future have not been disclosed by Masayoshi Son at this time.

Masayoshi Son is highly optimistic about the development of AI, claiming to be a heavy user of ChatGPT in an interview and engaging in conversations with it almost every day. In October 2023, Son expressed his belief that within the next decade, artificial intelligence will surpass human intelligence in nearly all domains, achieving a level of general artificial intelligence.

SoftBank’s UK-based chip intellectual property company, Arm, raised approximately USD 4.87 billion in its initial public offering. At the time of Arm’s listing, Son stated that he is a big believer in artificial intelligence and that Arm will also be at the core beneficiary of the AI revolution.

At the shareholders’ meeting on June 21, 2023, Masayoshi Son presented a chart of human evolution titled “Evolution Speed.”

The chart depicted a flat curve from the birth of humanity to the agricultural revolution, followed by a slight increase during the industrial and information revolutions. This indicated that the curve representing the development of artificial intelligence would experience a rapid upward surge within a few years, with its slope approaching a nearly vertical line.

Read more

• [News] CoWoS Capacity Shortage Challenges AI Chip Demand, while Taiwanese Manufacturers Expand to Seize Opportunities
• [News] NVIDIA CEO Jensen Huang Meets with TSMC Executives in Taiwan, Addressing the Significant AI Chip Shortage

(Photo credit: SoftBank News)

NVIDIA has begun accepting pre-orders for its customized artificial intelligence (AI) chips tailored for the Chinese market, as per a report from Reuters. The prices of the chips are said to be comparable to those of its competitor Huawei’s products.

The H20 graphics card, exclusively designed by NVIDIA for the Chinese market, is the most powerful among the three chips developed, although its computing power is lower than its own flagship AI chips, the H100 and H800. The H800, also tailored for China, was banned in October last year.

According to industry sources cited in the report, the specifications of the H20 are inferior to Huawei’s Ascend 910B in some critical areas. Additionally, NVIDIA has priced orders from Chinese H20 distributors between $12,000 and $15,000 per unit in recent weeks.

It is noteworthy that servers provided by distributors with 8 pre-configured AI chips are priced at CNY 1.4 million. In comparison, servers equipped with 8 H800 chips were priced at around CNY 2 million when they were launched a year ago.

Furthermore, it’s added in the report that distributors have informed customers that they will be able to begin small-scale deliveries of H20 products in the first quarter of 2024, with bulk deliveries starting in the second quarter.

In terms of specifications, the H20 appears to lag behind the 910B in FP32 performance, a critical metric that measures the speed at which chips process common tasks, with the H20’s performance being less than half of its competitor’s.

However, according to the source cited in the report, the H20 seems to have an advantage over the 910B in terms of interconnect speed, which measures the speed of data transfer between chips.

The source further indicates that in applications requiring numerous chips to be interconnected and function as a system, the H20 still possesses competitive capabilities compared to the 910B.

NVIDIA reportedly plans to commence mass production of the H20 in the second quarter of this year. Additionally, the company intends to introduce two other chips targeted at the Chinese market, namely the L20 and L2. However, the status of these two chips cannot be confirmed at the moment, as neither the H20, L20, nor L2 are currently listed on NVIDIA’s official website.

TrendForce believes Chinese companies will continue to buy existing AI chips in the short term. NVIDIA’s GPU AI accelerator chips remain a top priority—including H20, L20, and L2—designed specifically for the Chinese market following the ban.

At the same time, major Chinese AI firms like Huawei, will continue to develop general-purpose AI chips to provide AI solutions for local businesses. Beyond developing AI chips, these companies aim to establish a domestic AI server ecosystem in China.

TrendForce recognizes that a key factor in achieving success will come from the support of the Chinese government through localized projects, such as those involving Chinese telecom operators, which encourage the adoption of domestic AI chips.

Read more

• [News] Expert Insights on NVIDIA’s AI Chip Strategy – Downgraded Version Targeted for China, High-End Versions Aimed Overseas
• NVIDIA Experiences Strong Cloud AI Demand but Faces Challenges in China, with High-End AI Server Shipments Expected to Be Below 4% in 2024, Says TrendForce

(Photo credit: NVIDIA)

NVIDIA CEO Jensen Huang has reportedly gone to Taiwan once again, with reports suggesting a recent visit to China. Industry sources believe NVIDIA is planning to introduce downgraded AI chips to bypass U.S. restrictions on exporting high-end chips to China. Huang’s visit to China is seen as an effort to alleviate concerns among customers about adopting the downgraded versions.

Experts indicate that due to the expanded U.S. semiconductor restriction on China, NVIDIA’s sales in the Chinese market will decline. To counter this, NVIDIA might adjust its product portfolio and expand sales of high-end AI chips outside China.

The export of NVIDIA’s A100 and H100 chips to China and Hong Kong was prohibited in September 2022. Following that, the A800 and H800 chips, which were further designed with downgraded adjustments for the Chinese market, were also prohibited for export to China in October of the previous year.

In November 2023, the NVIDIA’s management acknowledged the significant impact of the U.S. restrictions on China’s revenue for the fourth quarter of 2023 but expressed confidence that revenue from other regions can offset this impact.

CEO Jensen Huang revealed in December in Singapore that NVIDIA was closely collaborating with the U.S. government to ensure compliance with export restrictions on new chips for the Chinese market.

According to reports in Chinese media The Paper, Jensen Huang recently made a low-profile visit to China. The market is closely watching the status of NVIDIA’s AI chip strategy in China and the company’s subsequent development strategies in response to U.S. restrictions. The fate of the newly designed AI chips, H20, L20, and L2, to comply with U.S. export regulations remains uncertain and will be closely observed.

Liu Pei-Chen, a researcher and director at the Taiwan Institute of Economic Research, discussed with CNA’s reporter about NVIDIA’s active planning to introduce a downgraded version of AI chips in China. 

The most urgent task, according to Liu, is to persuade Chinese customers to adopt these downgraded AI chips. Chinese clients believe that there isn’t a significant performance gap between NVIDIA’s downgraded AI chips and domestically designed AI chips.

Liu mentioned that this is likely the reason why Jensen Huang visited China. It serves as an opportunity to promote NVIDIA’s downgraded AI chips and alleviate concerns among Chinese customers. 

Read more

• [News] Chinese Companies Reportedly Disassembling NVIDIA Gaming Cards for AI Development
• [Insights] NVIDIA Maintains Confidence Despite Export Restrictions, Continues Unveiling New Products

(Photo credit: NVIDIA)

In the intense battle of AI chips between NVIDIA and AMD this year, AMD’s MI300 has entered mass production and shipment 1H24, gaining positive adoption from clients. In response, NVIDIA is gearing up to launch upgraded AI chips. TSMC emerges as the big winner by securing orders from both NVIDIA and AMD.

Industry sources have revealed optimism as NVIDIA’s AI chip shipment momentum is expected to reach around 3 million units this year, representing multiple growth compared to 2023.

With the production ramp-up of the AMD MI300 series chips, the total number of AI high-performance computing chips from NVIDIA and AMD for TSMC in 2024 is anticipated to reach 3.5 million units. This boost in demand is expected to contribute to the utilization rate of TSMC’s advanced nodes.

According to a report from the Economic Daily News, TSMC has not commented on rumors regarding customers and orders.

Industry sources have further noted that the global AI boom ignited in 2023, and 2024 continues to be a focal point for the industry. A notable shift from 2023 is that NVIDIA, which has traditionally dominated the field of high-performance computing (HPC) in AI, is now facing a challenge from AMD’s MI300 series products, which have begun shipping, intensifying competition for market share.

Reportedly, the AMD MI300A series products have commenced mass production and shipment this quarter. The central processing unit (CPU) and graphics processing unit (GPU) tile are manufactured using TSMC’s 5nm process, while the IO tile use TSMC’s 6nm process.

These chips are integrated through TSMC’s new System-on-Integrated-Chip (SoIC) and Chip-on-Wafer-on-Substrate (CoWoS) advanced packaging technologies. Additionally, AMD’s MI300X, which does not integrate the CPU, is also shipping simultaneously.

Compared to NVIDIA’s GH200, which integrates CPU and GPU, and the H200, focusing solely on GPU computation, AMD’s new AI chip performance exceeds expectations. It offers a lower price and a high cost-performance advantage, attracting adoption by ODMs.

In response to strong competition from AMD, NVIDIA is upgrading its product line. Apart from its high-demand H200 and GH200, NVIDIA is expected to launch new products such as B100 and GB200, utilizing TSMC’s 3nm process, by the end of the year.

Read more

• [News] Nvidia vs. AMD, the Expensive Duel as Two Major Buyers Opt for AMD’s Latest AI GPU
• [News] TSMC’s CoWoS Demand Surges from NVIDIA, Apple, AMD, Broadcom, Marvell, Monthly Capacity Up 120% in 2024

(Photo credit: NVIDIA)