Microsoft is reportedly developing a customized network card for AI servers, as per sources cited by global media The Information. This card is expected to enhance the performance of its in-house AI chip Azure Maia 100 while reducing dependency on NVIDIA as the primary supplier of high-performance network cards.

Leading this product initiative at Microsoft is Pradeep Sindhu, co-founder of Juniper Networks. Microsoft acquired Sindhu’s data center technology startup, Fungible, last year. Sindhu has since joined Microsoft and is leading the team in developing this network card.

According to the Information, this network card is similar to NVIDIA’s ConnectX-7 interface card, which supports a maximum bandwidth of 400 Gb Ethernet and is sold alongside NVIDIA GPUs.

Developing high-speed networking equipment tailored specifically for AI workloads may take over a year. If successful, it could reduce the time required for OpenAI to train models on Microsoft AI servers and lower the costs associated with the training process.

In November last year, Microsoft unveiled the Azure Maia 100 for data centers, manufactured using TSMC’s 5-nanometer process. The Azure Maia 100, introduced at the conference, is an AI accelerator chip designed for tasks such as running OpenAI models, ChatGPT, Bing, GitHub Copilot, and other AI workloads.

Microsoft is also in the process of designing the next generation of the chip. Not only is Microsoft striving to reduce its reliance on NVIDIA, but other companies including OpenAI, Tesla, Google, Amazon, and Meta are also investing in developing their own AI accelerator chips. These companies are expected to compete with NVIDIA’s flagship H100 AI accelerator chips.

Read more

• [Insights] Microsoft Unveils In-House AI Chip, Poised for Competitive Edge with a Powerful Ecosystem
• [Insights] AI Market: A Battleground for Tech Giants as Six Major Companies Develop AI Chips

(Photo credit: Microsoft)

Last year’s AI boom propelled NVIDIA into the spotlight, yet the company finds itself at a challenging crossroads.

According to a report from TechNews, on one hand, NVIDIA dominates in high-performance computing and artificial intelligence, continuously expanding with its latest GPU products. On the other hand, global supply chain instability, rapid emergence of competitors, and uncertainties in technological innovation are exerting unprecedented pressure on NVIDIA.

NVIDIA’s stock price surged by 246% last year, driving its market value past USD 1 trillion and making it the first chip company to achieve this milestone. According to the Bloomberg Billionaires Index, NVIDIA CEO Jensen Huang’s personal wealth has soared to USD 55.7 billion.

However, despite the seemingly radiant outlook for the NVIDIA, as per a report from TechNews, it still faces uncontrollable internal and external challenges.

• Internal Concern 1: CoWoS, HBM Capacity Bottlenecks

The most apparent issue lies in capacity constraints.

Currently, NVIDIA’s A100 and H100 GPUs are manufactured using TSMC’s CoWoS packaging technology. However, with the surge in demand for generative AI, TSMC’s CoWoS capacity is severely strained. Consequently, NVIDIA has certified other CoWoS packaging suppliers such as UMC, ASE, and American OSAT manufacturer Amkor as backup options.

Meanwhile, TSMC has relocated its InFo production capacity from Longtan to Southern Taiwan Science Park. The vacated Longtan fab is being repurposed to expand CoWoS capacity, while the Zhunan and Taichung fabs are also contributing to the expansion of CoWoS production to alleviate capacity constraints.

However, during the earnings call, TSMC also stated that despite a doubling of capacity in 2024, it still may not be sufficient to meet all customer demands.

In addition to TSMC’s CoWoS capacity, industry rumors suggest that NVIDIA has made significant upfront payments to Micron, SK Hynix, to secure HBM3 memory, ensuring a stable supply of HBM memory. However, the entire HBM capacity of Samsung, SK Hynix, and Micron for this year has already been allocated. Therefore, whether the capacity can meet market demand will be a significant challenge for NVIDIA.

• Internal Concern 2: Major Customers Shifting Towards In-house Chips

While cloud service providers (CSPs) fiercely compete for GPUs, major players like Amazon, Microsoft, Google, and Meta are actively investing in in-house AI chips.

Amazon and Google have respectively introduced Trainium and TPU chips, Microsoft announced its first in-house AI chip Maia 100 along with in-house cloud computing CPU Cobalt 100, while Meta plans to unveil its first-generation in-house AI chip MTIA by 2025.

Although these hyperscale customers still rely on NVIDIA’s chips, in the long run, it may impact NVIDIA’s market share, inadvertently positioning them as competitors and affecting profits. Consequently, NVIDIA finds it challenging to depend solely on these hyperscale customers.

• External Challenge 1: Export Control Pressures Lead to Loss of Chinese Customers

Due to escalating tensions between the US and China, the US issued new regulations prohibiting NVIDIA from exporting advanced AI chips to China. Consequently, NVIDIA introduced specially tailored versions such as A800 and H800 for the Chinese market.

However, they were ultimately blocked by the US, and products including A100, A800, H100, H800, and L40S were included in the export control list.Subsequently, NVIDIA decided to introduce new AI GPUs, namely HGXH20, L20 PCIe, and L2 PCIe, in compliance with export policies.

However, with only 20% of the computing power of H100, they are planned for mass production in the second quarter. Due to the reduced performance, major Chinese companies like Alibaba, Tencent, and Baidu reportedly refused to purchase, explicitly stating significant order cuts for the year. Consequently, NVIDIA’s revenue prospects in China appear grim, with some orders even being snatched by Huawei.

Currently, NVIDIA’s sales revenue from Singapore and China accounts for 15% of its total revenue. Moreover, the company holds over 90% market share in the AI chip market in China. Therefore, the cost of abandoning the Chinese market would be substantial. NVIDIA is adamant about not easily giving up on China; however, the challenge lies in how to comply with US government policies and pressures while meeting the demands of Chinese customers.

As per NVIDIA CEO Jensen Huang during its last earnings call, he mentioned that US export control measures would have an impact. Contributions from China and other regions accounted for 20-25% of data center revenue in the last quarter, with a significant anticipated decline this quarter.

He also expressed concerns that besides losing the Chinese market, the situation would accelerate China’s efforts to manufacture its own chips and introduce proprietary GPU products, providing Chinese companies with opportunities to rise.

• External Challenge 2: Arch-Rivals Intel and AMD Begin Their Offensive

In the race to capture the AI market opportunity, arch-rivals Intel and AMD are closely after NVIDIA. As NVIDIA pioneered the adoption of TSMC’s 4-nanometer H100, AMD quickly followed suit by launching the first batch of “Instinct MI300X” for AI and HPC applications last year.

Currently, shipments of MI300X have commenced this year, with Microsoft’s data center division emerging as the largest buyer. Meta has also procured a substantial amount of Instinct MI300 series products, while LaminiAI stands as the first publicly known company to utilize MI300X.

According to official performance tests by AMD, the MI300X outperforms the existing NVIDIA H100 80GB available on the market, posing a potential threat to the upcoming H200 141GB.

Additionally, compared to the H100 chip, the MI300X offers a more competitive price for products of the same level. If NVIDIA’s production capacity continues to be restricted, some customers may switch to AMD.

Meanwhile, Intel unveiled the “Gaudi3” chip for generative AI software last year. Although there is limited information available, it is rumored that the memory capacity may increase by 50% compared to Gaudi 2’s 96GB, possibly upgrading to HBM3e memory. CEO Pat Gelsinger directly stated that “Gaudi 3 performance will surpass that of the H100.”

• External Challenge 3: Startup Underdogs Form AI Platform Alliance in Attempt to Conquer

Several global chip design companies have recently announced the formation of the “AI Platform Alliance,” aiming to promote an open AI ecosystem. The founding members of the AI Platform Alliance include Ampere, Cerebras Systems, Furiosa, Graphcore, Kalray, Kinara, Luminous, Neuchips, Rebellions, and Sapeon, among others.

Notably absent is industry giant NVIDIA, leading to speculation that startups aspire to unite and challenge NVIDIA’s dominance.

However, with NVIDIA holding a 75-90% market share in AI, it remains in a dominant position. Whether the AI Platform Alliance can disrupt NVIDIA’s leading position is still subject to observation.

Read more

• [News] NVIDIA’s Exclusive Chips for China Now Reported to be Available for Pre-Order, Priced Similar to Huawei Products
• [News] NVIDIA CEO Jensen Huang Meets with TSMC Executives in Taiwan, Addressing the Significant AI Chip Shortage

(Photo credit: NVIDIA)

The surge in demand for advanced packaging is being primarily propelled by artificial intelligence (AI) chips. According to industry sources cited by CNA, TSMC’s CoWoS production capacity is set to double this year, yet demand continues to outstrip supply. In response, NVIDIA has enlisted the help of packaging and testing facilities to augment its advanced packaging capabilities.

In addition, to address the imbalance between supply and demand for advanced packaging due to AI, semiconductor backend specialty assembly and testing (OSAT) companies such as ASE Technology Holding (ASE), Powertech Technology, and KYEC have expanded their capital expenditures this year to enhance their advanced packaging capabilities, aligning with the needs of their customers.

AI and high-performance computing (HPC) chips are driving the demand for CoWoS advanced packaging. As per sources interviewed by CNA, from July to the end of last year, TSMC actively adjusted its CoWoS advanced packaging production capacity, gradually expanding and stabilizing mass production.

The source further indicates that in December of last year, TSMC’s CoWoS monthly production capacity increased to 14,000 to 15,000. It is estimated that by the fourth quarter of this year, TSMC’s CoWoS monthly production capacity will significantly expand to 33,000 to 35,000.

Per an earlier report from Commercial Times, TSMC has been outsourcing part of its CoWoS operations for some time, mainly targeting small-volume, high-performance chips. TSMC maintains in-house production of the CoW, while the back-end WoS is handed over to test and assembly houses to improve production efficiency and flexibility. 

However, the demand for advanced packaging capacity for AI chips still outstrips supply. Sources cited by CNA also reveal that NVIDIA has sought assistance from packaging and testing subcontractors outside of TSMC to augment their advanced packaging capabilities.

Amkor, among others, began gradually providing capacity support from the fourth quarter of last year, while SPIL, a subsidiary of ASE, is slated to commence supply in the first quarter of this year.

Read more

• [News] AMD Eyes CoWoS-like Supply for AI Chips with TSMC Full Capacity
• [News] CoWoS Capacity Shortage Challenges AI Chip Demand, while Taiwanese Manufacturers Expand to Seize Opportunities

(Photo credit: TSMC)

With the flourishing development of technologies such as AI, cloud computing, big data analytics, and mobile computing, modern society has an increasingly high demand for computing power.

Moreover, with the advancement beyond 3 nanometers, wafer sizes have encountered scaling limitations and manufacturing costs have increased. Therefore, besides continuing to develop advanced processes, the semiconductor industry is also exploring other ways to maintain chip size while ensuring high efficiency.

The concept of “heterogeneous integration” has become a contemporary focus, leading to the transition of chips from single-layer to advanced packaging with multiple layers stacked together.

The term “CoWoS” can be broken down into the following definitions: “Cow” stands for “Chip-on-Wafer,” referring to the stacking of chips, while “WoS” stands for “Wafer-on-Substrate,” which involves stacking chips on a substrate.

Therefore, “CoWoS” collectively refers to stacking chips and packaging them onto a substrate. This approach reduces the space required for chips and offers benefits in reducing power consumption and costs.

Among these, CoWoS can be further divided into 2.5D horizontal stacking (most famously exemplified by TSMC’s CoWoS) and 3D vertical stacking versions. In these configurations, various processor and memory modules are stacked layer by layer to create chiplets. Because its primary application lies in advanced processes, it is also referred to as advanced packaging.

According to TrendForce’s data, it has provided insights into the heat of the AI chip market. In 2023, shipments of AI servers (including those equipped with GPU, FPGA, ASIC, etc.) reached nearly 1.2 million units, a 38.4% increase from 2022, accounting for nearly 9% of the overall server shipments.

Looking ahead to 2026, the proportion is expected to reach 15%, with a compound annual growth rate (CAGR) of AI server shipments from 2022 to 2026 reaching 22%.

Due to the advanced packaging requirements of AI chips, TSMC’s 2.5D advanced packaging CoWoS technology is currently the primary technology used for AI chips.

GPUs, in particular, utilize higher specifications of HBM, which require the integration of core dies using 2.5D advanced packaging technology. The initial stage of chip stacking in CoWoS packaging, known as Chip on Wafer (CoW), primarily undergoes manufacturing at the fab using a 65-nanometer process. Following this, through-silicon via (TSV) is carried out, and the finalized products are stacked and packaged onto the substrate, known as Wafer on Substrate (WoS).

As a result, the production capacity of CoWoS packaging technology has become a significant bottleneck in AI chip output over the past year, and it remains a key factor in whether AI chip demand can be met in 2024. Foreign analysts have previously pointed out that NVIDIA is currently the largest customer of TSMC’s 2.5D advanced packaging CoWoS technology.

This includes NVIDIA’s H100 GPU, which utilizes TSMC’s 4-nanometer advanced process, as well as the A100 GPU, which uses TSMC’s 7-nanometer process, both of which are packaged using CoWoS technology. As a result, NVIDIA’s chips account for 40% to 50% of TSMC’s CoWoS packaging capacity. This is also why the high demand for NVIDIA chips has led to tight capacity for TSMC’s CoWoS packaging.

TSMC’s Expansion Plans Expected to Ease Tight Supply Situation in 2024

During the earnings call held in July 2023, TSMC announced its plans to double the CoWoS capacity, indicating that the supply-demand imbalance in the market could be alleviated by the end of 2024.

Subsequently, in late July 2023, TSMC announced an investment of nearly NTD 90 billion (roughly USD 2.87 billion) to establish an advanced packaging fab in the Tongluo Science Park, with the construction expected to be completed by the end of 2026 and mass production scheduled for the second or third quarter of 2027.

In addition, during the earnings call on January 18, 2024, TSMC’s CFO, Wendell Huang, emphasized that TSMC would continue its expansion of advanced processes in 2024. Therefore, it is estimated that 10% of the total capital expenditure for the year will be allocated towards expanding capacity in advanced packaging, testing, photomasks, and other areas.

In fact, NVIDIA’s CFO, Colette Kress, stated during an investor conference that the key process of CoWoS advanced packaging has been developed and certified with other suppliers. Kress further anticipated that supply would gradually increase over the coming quarters.

Regarding this, J.P. Morgan, an investment firm, pointed out that the bottleneck in CoWoS capacity is primarily due to the supply-demand gap in the interposer. This is because the TSV process is complex, and expanding capacity requires more high-precision equipment. However, the long lead time for high-precision equipment, coupled with the need for regular cleaning and inspection of existing equipment, has resulted in supply shortages.

Apart from TSMC’s dominance in the CoWoS advanced packaging market, other Taiwanese companies such as UMC, ASE Technology Holding, and Powertek Technology are also gradually entering the CoWoS advanced packaging market.

Among them, UMC expressed during an investor conference in late July 2023 that it is accelerating the deployment of silicon interposer technology and capacity to meet customer needs in the 2.5D advanced packaging sector.

UMC Expands Interposer Capacity; ASE Pushes Forward with VIPack Advanced Packaging Platform

UMC emphasizes that it is the world’s first foundry to offer an open system solution for silicon interposer manufacturing. Through this open system collaboration (UMC+OSAT), UMC can provide a fully validated supply chain for rapid mass production implementation.

On the other hand, in terms of shipment volume, ASE Group currently holds approximately a 32% market share in the global Outsourced Semiconductor Assembly and Test (OSAT) industry and accounts for over 50% of the OSAT shipment volume in Taiwan. Its subsidiary, ASE Semiconductor, also notes the recent focus on CoWoS packaging technology. ASE Group has been strategically positioning itself in advanced packaging, working closely with TSMC as a key partner.

ASE underscores the significance of its VIPack advanced packaging platform, designed to provide vertical interconnect integration solutions. VIPack represents the next generation of 3D heterogeneous integration architecture.

Leveraging advanced redistribution layer (RDL) processes, embedded integration, and 2.5D/3D packaging technologies, VIPack enables customers to integrate multiple chips into a single package, unlocking unprecedented innovation in various applications.

Powertech Technology Seeks Collaboration with Foundries; Winbond Electronics Offers Heterogeneous Integration Packaging Technology

In addition, the OSAT player Powertech Technology is actively expanding its presence in advanced packaging for logic chips and AI applications.

The collaboration between Powertech and Winbond is expected to offer customers various options for CoWoS advanced packaging, indicating that CoWoS-related advanced packaging products could be available as early as the second half of 2024.

Winbond Electronics emphasizes that the collaboration project will involve Winbond Electronics providing CUBE (Customized Ultra-High Bandwidth Element) DRAM, as well as customized silicon interposers and integrated decoupling capacitors, among other advanced technologies. These will be complemented by Powertech Technology’s 2.5D and 3D packaging services.

Read more

• [News] AMD Eyes CoWoS-like Supply for AI Chips with TSMC Full Capacity
• [News] Samsung Boosts 2.5D Packaging Equipment to Compete for TSMC’s CoWoS Orders

(Photo credit: TSMC)

NVIDIA has begun accepting pre-orders for its customized artificial intelligence (AI) chips tailored for the Chinese market, as per a report from Reuters. The prices of the chips are said to be comparable to those of its competitor Huawei’s products.

The H20 graphics card, exclusively designed by NVIDIA for the Chinese market, is the most powerful among the three chips developed, although its computing power is lower than its own flagship AI chips, the H100 and H800. The H800, also tailored for China, was banned in October last year.

According to industry sources cited in the report, the specifications of the H20 are inferior to Huawei’s Ascend 910B in some critical areas. Additionally, NVIDIA has priced orders from Chinese H20 distributors between $12,000 and $15,000 per unit in recent weeks.

It is noteworthy that servers provided by distributors with 8 pre-configured AI chips are priced at CNY 1.4 million. In comparison, servers equipped with 8 H800 chips were priced at around CNY 2 million when they were launched a year ago.

Furthermore, it’s added in the report that distributors have informed customers that they will be able to begin small-scale deliveries of H20 products in the first quarter of 2024, with bulk deliveries starting in the second quarter.

In terms of specifications, the H20 appears to lag behind the 910B in FP32 performance, a critical metric that measures the speed at which chips process common tasks, with the H20’s performance being less than half of its competitor’s.

However, according to the source cited in the report, the H20 seems to have an advantage over the 910B in terms of interconnect speed, which measures the speed of data transfer between chips.

The source further indicates that in applications requiring numerous chips to be interconnected and function as a system, the H20 still possesses competitive capabilities compared to the 910B.

NVIDIA reportedly plans to commence mass production of the H20 in the second quarter of this year. Additionally, the company intends to introduce two other chips targeted at the Chinese market, namely the L20 and L2. However, the status of these two chips cannot be confirmed at the moment, as neither the H20, L20, nor L2 are currently listed on NVIDIA’s official website.

TrendForce believes Chinese companies will continue to buy existing AI chips in the short term. NVIDIA’s GPU AI accelerator chips remain a top priority—including H20, L20, and L2—designed specifically for the Chinese market following the ban.

At the same time, major Chinese AI firms like Huawei, will continue to develop general-purpose AI chips to provide AI solutions for local businesses. Beyond developing AI chips, these companies aim to establish a domestic AI server ecosystem in China.

TrendForce recognizes that a key factor in achieving success will come from the support of the Chinese government through localized projects, such as those involving Chinese telecom operators, which encourage the adoption of domestic AI chips.

Read more

• [News] Expert Insights on NVIDIA’s AI Chip Strategy – Downgraded Version Targeted for China, High-End Versions Aimed Overseas
• NVIDIA Experiences Strong Cloud AI Demand but Faces Challenges in China, with High-End AI Server Shipments Expected to Be Below 4% in 2024, Says TrendForce

(Photo credit: NVIDIA)