Last year’s AI boom propelled NVIDIA into the spotlight, yet the company finds itself at a challenging crossroads.

According to a report from TechNews, on one hand, NVIDIA dominates in high-performance computing and artificial intelligence, continuously expanding with its latest GPU products. On the other hand, global supply chain instability, rapid emergence of competitors, and uncertainties in technological innovation are exerting unprecedented pressure on NVIDIA.

NVIDIA’s stock price surged by 246% last year, driving its market value past USD 1 trillion and making it the first chip company to achieve this milestone. According to the Bloomberg Billionaires Index, NVIDIA CEO Jensen Huang’s personal wealth has soared to USD 55.7 billion.

However, despite the seemingly radiant outlook for the NVIDIA, as per a report from TechNews, it still faces uncontrollable internal and external challenges.

• Internal Concern 1: CoWoS, HBM Capacity Bottlenecks

The most apparent issue lies in capacity constraints.

Currently, NVIDIA’s A100 and H100 GPUs are manufactured using TSMC’s CoWoS packaging technology. However, with the surge in demand for generative AI, TSMC’s CoWoS capacity is severely strained. Consequently, NVIDIA has certified other CoWoS packaging suppliers such as UMC, ASE, and American OSAT manufacturer Amkor as backup options.

Meanwhile, TSMC has relocated its InFo production capacity from Longtan to Southern Taiwan Science Park. The vacated Longtan fab is being repurposed to expand CoWoS capacity, while the Zhunan and Taichung fabs are also contributing to the expansion of CoWoS production to alleviate capacity constraints.

However, during the earnings call, TSMC also stated that despite a doubling of capacity in 2024, it still may not be sufficient to meet all customer demands.

In addition to TSMC’s CoWoS capacity, industry rumors suggest that NVIDIA has made significant upfront payments to Micron, SK Hynix, to secure HBM3 memory, ensuring a stable supply of HBM memory. However, the entire HBM capacity of Samsung, SK Hynix, and Micron for this year has already been allocated. Therefore, whether the capacity can meet market demand will be a significant challenge for NVIDIA.

• Internal Concern 2: Major Customers Shifting Towards In-house Chips

While cloud service providers (CSPs) fiercely compete for GPUs, major players like Amazon, Microsoft, Google, and Meta are actively investing in in-house AI chips.

Amazon and Google have respectively introduced Trainium and TPU chips, Microsoft announced its first in-house AI chip Maia 100 along with in-house cloud computing CPU Cobalt 100, while Meta plans to unveil its first-generation in-house AI chip MTIA by 2025.

Although these hyperscale customers still rely on NVIDIA’s chips, in the long run, it may impact NVIDIA’s market share, inadvertently positioning them as competitors and affecting profits. Consequently, NVIDIA finds it challenging to depend solely on these hyperscale customers.

• External Challenge 1: Export Control Pressures Lead to Loss of Chinese Customers

Due to escalating tensions between the US and China, the US issued new regulations prohibiting NVIDIA from exporting advanced AI chips to China. Consequently, NVIDIA introduced specially tailored versions such as A800 and H800 for the Chinese market.

However, they were ultimately blocked by the US, and products including A100, A800, H100, H800, and L40S were included in the export control list.Subsequently, NVIDIA decided to introduce new AI GPUs, namely HGXH20, L20 PCIe, and L2 PCIe, in compliance with export policies.

However, with only 20% of the computing power of H100, they are planned for mass production in the second quarter. Due to the reduced performance, major Chinese companies like Alibaba, Tencent, and Baidu reportedly refused to purchase, explicitly stating significant order cuts for the year. Consequently, NVIDIA’s revenue prospects in China appear grim, with some orders even being snatched by Huawei.

Currently, NVIDIA’s sales revenue from Singapore and China accounts for 15% of its total revenue. Moreover, the company holds over 90% market share in the AI chip market in China. Therefore, the cost of abandoning the Chinese market would be substantial. NVIDIA is adamant about not easily giving up on China; however, the challenge lies in how to comply with US government policies and pressures while meeting the demands of Chinese customers.

As per NVIDIA CEO Jensen Huang during its last earnings call, he mentioned that US export control measures would have an impact. Contributions from China and other regions accounted for 20-25% of data center revenue in the last quarter, with a significant anticipated decline this quarter.

He also expressed concerns that besides losing the Chinese market, the situation would accelerate China’s efforts to manufacture its own chips and introduce proprietary GPU products, providing Chinese companies with opportunities to rise.

• External Challenge 2: Arch-Rivals Intel and AMD Begin Their Offensive

In the race to capture the AI market opportunity, arch-rivals Intel and AMD are closely after NVIDIA. As NVIDIA pioneered the adoption of TSMC’s 4-nanometer H100, AMD quickly followed suit by launching the first batch of “Instinct MI300X” for AI and HPC applications last year.

Currently, shipments of MI300X have commenced this year, with Microsoft’s data center division emerging as the largest buyer. Meta has also procured a substantial amount of Instinct MI300 series products, while LaminiAI stands as the first publicly known company to utilize MI300X.

According to official performance tests by AMD, the MI300X outperforms the existing NVIDIA H100 80GB available on the market, posing a potential threat to the upcoming H200 141GB.

Additionally, compared to the H100 chip, the MI300X offers a more competitive price for products of the same level. If NVIDIA’s production capacity continues to be restricted, some customers may switch to AMD.

Meanwhile, Intel unveiled the “Gaudi3” chip for generative AI software last year. Although there is limited information available, it is rumored that the memory capacity may increase by 50% compared to Gaudi 2’s 96GB, possibly upgrading to HBM3e memory. CEO Pat Gelsinger directly stated that “Gaudi 3 performance will surpass that of the H100.”

• External Challenge 3: Startup Underdogs Form AI Platform Alliance in Attempt to Conquer

Several global chip design companies have recently announced the formation of the “AI Platform Alliance,” aiming to promote an open AI ecosystem. The founding members of the AI Platform Alliance include Ampere, Cerebras Systems, Furiosa, Graphcore, Kalray, Kinara, Luminous, Neuchips, Rebellions, and Sapeon, among others.

Notably absent is industry giant NVIDIA, leading to speculation that startups aspire to unite and challenge NVIDIA’s dominance.

However, with NVIDIA holding a 75-90% market share in AI, it remains in a dominant position. Whether the AI Platform Alliance can disrupt NVIDIA’s leading position is still subject to observation.

Read more

• [News] NVIDIA’s Exclusive Chips for China Now Reported to be Available for Pre-Order, Priced Similar to Huawei Products
• [News] NVIDIA CEO Jensen Huang Meets with TSMC Executives in Taiwan, Addressing the Significant AI Chip Shortage

(Photo credit: NVIDIA)

With the flourishing development of technologies such as AI, cloud computing, big data analytics, and mobile computing, modern society has an increasingly high demand for computing power.

Moreover, with the advancement beyond 3 nanometers, wafer sizes have encountered scaling limitations and manufacturing costs have increased. Therefore, besides continuing to develop advanced processes, the semiconductor industry is also exploring other ways to maintain chip size while ensuring high efficiency.

The concept of “heterogeneous integration” has become a contemporary focus, leading to the transition of chips from single-layer to advanced packaging with multiple layers stacked together.

The term “CoWoS” can be broken down into the following definitions: “Cow” stands for “Chip-on-Wafer,” referring to the stacking of chips, while “WoS” stands for “Wafer-on-Substrate,” which involves stacking chips on a substrate.

Therefore, “CoWoS” collectively refers to stacking chips and packaging them onto a substrate. This approach reduces the space required for chips and offers benefits in reducing power consumption and costs.

Among these, CoWoS can be further divided into 2.5D horizontal stacking (most famously exemplified by TSMC’s CoWoS) and 3D vertical stacking versions. In these configurations, various processor and memory modules are stacked layer by layer to create chiplets. Because its primary application lies in advanced processes, it is also referred to as advanced packaging.

According to TrendForce’s data, it has provided insights into the heat of the AI chip market. In 2023, shipments of AI servers (including those equipped with GPU, FPGA, ASIC, etc.) reached nearly 1.2 million units, a 38.4% increase from 2022, accounting for nearly 9% of the overall server shipments.

Looking ahead to 2026, the proportion is expected to reach 15%, with a compound annual growth rate (CAGR) of AI server shipments from 2022 to 2026 reaching 22%.

Due to the advanced packaging requirements of AI chips, TSMC’s 2.5D advanced packaging CoWoS technology is currently the primary technology used for AI chips.

GPUs, in particular, utilize higher specifications of HBM, which require the integration of core dies using 2.5D advanced packaging technology. The initial stage of chip stacking in CoWoS packaging, known as Chip on Wafer (CoW), primarily undergoes manufacturing at the fab using a 65-nanometer process. Following this, through-silicon via (TSV) is carried out, and the finalized products are stacked and packaged onto the substrate, known as Wafer on Substrate (WoS).

As a result, the production capacity of CoWoS packaging technology has become a significant bottleneck in AI chip output over the past year, and it remains a key factor in whether AI chip demand can be met in 2024. Foreign analysts have previously pointed out that NVIDIA is currently the largest customer of TSMC’s 2.5D advanced packaging CoWoS technology.

This includes NVIDIA’s H100 GPU, which utilizes TSMC’s 4-nanometer advanced process, as well as the A100 GPU, which uses TSMC’s 7-nanometer process, both of which are packaged using CoWoS technology. As a result, NVIDIA’s chips account for 40% to 50% of TSMC’s CoWoS packaging capacity. This is also why the high demand for NVIDIA chips has led to tight capacity for TSMC’s CoWoS packaging.

TSMC’s Expansion Plans Expected to Ease Tight Supply Situation in 2024

During the earnings call held in July 2023, TSMC announced its plans to double the CoWoS capacity, indicating that the supply-demand imbalance in the market could be alleviated by the end of 2024.

Subsequently, in late July 2023, TSMC announced an investment of nearly NTD 90 billion (roughly USD 2.87 billion) to establish an advanced packaging fab in the Tongluo Science Park, with the construction expected to be completed by the end of 2026 and mass production scheduled for the second or third quarter of 2027.

In addition, during the earnings call on January 18, 2024, TSMC’s CFO, Wendell Huang, emphasized that TSMC would continue its expansion of advanced processes in 2024. Therefore, it is estimated that 10% of the total capital expenditure for the year will be allocated towards expanding capacity in advanced packaging, testing, photomasks, and other areas.

In fact, NVIDIA’s CFO, Colette Kress, stated during an investor conference that the key process of CoWoS advanced packaging has been developed and certified with other suppliers. Kress further anticipated that supply would gradually increase over the coming quarters.

Regarding this, J.P. Morgan, an investment firm, pointed out that the bottleneck in CoWoS capacity is primarily due to the supply-demand gap in the interposer. This is because the TSV process is complex, and expanding capacity requires more high-precision equipment. However, the long lead time for high-precision equipment, coupled with the need for regular cleaning and inspection of existing equipment, has resulted in supply shortages.

Apart from TSMC’s dominance in the CoWoS advanced packaging market, other Taiwanese companies such as UMC, ASE Technology Holding, and Powertek Technology are also gradually entering the CoWoS advanced packaging market.

Among them, UMC expressed during an investor conference in late July 2023 that it is accelerating the deployment of silicon interposer technology and capacity to meet customer needs in the 2.5D advanced packaging sector.

UMC Expands Interposer Capacity; ASE Pushes Forward with VIPack Advanced Packaging Platform

UMC emphasizes that it is the world’s first foundry to offer an open system solution for silicon interposer manufacturing. Through this open system collaboration (UMC+OSAT), UMC can provide a fully validated supply chain for rapid mass production implementation.

On the other hand, in terms of shipment volume, ASE Group currently holds approximately a 32% market share in the global Outsourced Semiconductor Assembly and Test (OSAT) industry and accounts for over 50% of the OSAT shipment volume in Taiwan. Its subsidiary, ASE Semiconductor, also notes the recent focus on CoWoS packaging technology. ASE Group has been strategically positioning itself in advanced packaging, working closely with TSMC as a key partner.

ASE underscores the significance of its VIPack advanced packaging platform, designed to provide vertical interconnect integration solutions. VIPack represents the next generation of 3D heterogeneous integration architecture.

Leveraging advanced redistribution layer (RDL) processes, embedded integration, and 2.5D/3D packaging technologies, VIPack enables customers to integrate multiple chips into a single package, unlocking unprecedented innovation in various applications.

Powertech Technology Seeks Collaboration with Foundries; Winbond Electronics Offers Heterogeneous Integration Packaging Technology

In addition, the OSAT player Powertech Technology is actively expanding its presence in advanced packaging for logic chips and AI applications.

The collaboration between Powertech and Winbond is expected to offer customers various options for CoWoS advanced packaging, indicating that CoWoS-related advanced packaging products could be available as early as the second half of 2024.

Winbond Electronics emphasizes that the collaboration project will involve Winbond Electronics providing CUBE (Customized Ultra-High Bandwidth Element) DRAM, as well as customized silicon interposers and integrated decoupling capacitors, among other advanced technologies. These will be complemented by Powertech Technology’s 2.5D and 3D packaging services.

Read more

• [News] AMD Eyes CoWoS-like Supply for AI Chips with TSMC Full Capacity
• [News] Samsung Boosts 2.5D Packaging Equipment to Compete for TSMC’s CoWoS Orders

(Photo credit: TSMC)

Amid the AI trend, the significance of high-value-added DRAM represented by HBM continues to grow.

HBM (High Bandwidth Memory) is a type of graphics DDR memory that boasts advantages such as high bandwidth, high capacity, low latency, and low power consumption compared to traditional DRAM chips. It accelerates AI data processing speed and is particularly suitable for high-performance computing scenarios like ChatGPT, making it highly valued by memory giants in recent years.

Memory is also representing one of Korea’s pillar industries, and to seize the AI opportunity and drive the development of the memory industry, Korea has recently designated HBM as a national strategic technology.

The country will provide tax incentives to companies like Samsung Electronics. Small and medium-sized enterprises in Korea can enjoy up to a 40% to 50% reduction, while large enterprises like Samsung Electronics can benefit from a reduction of up to 30% to 40%.

Overview of HBM Development Progress Among Top Manufacturers

The HBM market is currently dominated by three major storage giants: Samsung, SK Hynix, and Micron. Since the introduction of the first silicon interposer HBM product in 2014, HBM technology has smoothly transitioned from HBM, HBM2, and HBM2E to HBM3 and HBM3e through iterative innovation.

According to research by TrendForce, the mainstream HBM in the market in 2023 is HBM2e. This includes specifications used in NVIDIA A100/A800, AMD MI200, and most CSPs’ self-developed acceleration chips. To meet the evolving demands of AI accelerator chips, various manufacturers are planning to launch new products like HBM3e in 2024, expecting HBM3 and HBM3e to become the market norm.

The progress of HBM3e, as outlined in the timeline below, shows that Micron provided its 8hi (24GB) samples to NVIDIA by the end of July, SK hynix in mid-August, and Samsung in early October.

As for the higher-spec HBM4, TrendForce expects its potential launch in 2026. With the push for higher computational performance, HBM4 is set to expand from the current 12-layer (12hi) to 16-layer (16hi) stacks, spurring demand for new hybrid bonding techniques. HBM4 12hi products are set for a 2026 launch, with 16hi models following in 2027.

Meeting Demand, Manufacturers Actively Expand HBM Production

As companies like NVIDIA and AMD continue to introduce high-performance GPU products, the three major manufacturers are actively planning the mass production of HBM with corresponding specifications.

Previously, media reports highlighted Samsung’s efforts to expand HBM production capacity by acquiring certain buildings and equipment within the Samsung Display’s Cheonan facility.

Samsung plans to establish a new packaging line at the Cheonan plant dedicated to large-scale HBM production. The company has already invested KRW 10.5 trillion in the acquisition of the mentioned assets and equipment, with an additional investment of KRW 700 billion to KRW 1 trillion.

Micron Technology’s Taichung Fab 4 in Taiwan was officially inaugurated in early November 2023. Micron stated that Taichung Fab 4 would integrate advanced probing and packaging testing functions to mass-produce HBM3e and other products, thereby meeting the increasing demand for various applications such as artificial intelligence, data centers, edge computing, and the cloud. The company plans to start shipping HBM3e in early 2024.

In its latest financial report, SK Hynix stated that in the DRAM sector in 2023, its main products DDR5 DRAM and HBM3 experienced revenue growth of over fourfold and fivefold, respectively, compared to the previous year.

At the same time, in response to the growing demand for high-performance DRAM, SK Hynix will smoothly carry out the mass production of HBM3e for AI applications and the research and development of HBM4.

Read more

• Manufacturers Anticipate Completion of NVIDIA’s HBM3e Verification by 1Q24; HBM4 Expected to Launch in 2026, Says TrendForce
• [News] Strong Performance in HBM Ends SK Hynix’s Four Consecutive Quarters of Losses

(Photo credit: SK Hynix)

South Korean memory giant SK Hynix has released its financial results for the fourth quarter of 2023 and the full year 2023, ending on December 31, 2023. In the fourth quarter, the revenue reached KRW 11.306 trillion, operating profit amounted to KRW 346 billion, and net loss was KRW 1.38 trillion. The operating profit margin for Q4 2023 was 3%, with a net profit margin of negative 12%.

Source: SK Hynix

SK Hynix noted that, with the rebound in the memory market, the operating profit for the fourth quarter of 2023 reached KRW 346 billion, successfully marking a turnaround from losses. This signifies that SK Hynix, in just one year, has managed to break free from the continuous operating losses experienced since the fourth quarter of 2022.

SK Hynix emphasizes that the overall memory market conditions improved in the last quarter of 2023 with demand for AI server and mobile applications increasing and average selling price (ASP) rising. 

Simultaneously, the effective implementation of a profit-oriented business plan by SK Hynix has enabled the company to achieve the goal of turning losses into profits within just one year.

Source: SK Hynix

Furthermore, SK Hynix has reduced the cumulative scale of operating losses that persisted until Q3 2023. In total, the consolidated revenue for 2023 reached KRW 32.766 trillion, with an operating loss of KRW 7.73 trillion and a net loss of KRW 9.138 trillion. Overall, the operating loss rate for 2023 is 24%, and the net loss rate is 27%.

SK Hynix also notes that in the DRAM sector for 2023, the company actively addressed customer demands. The revenue for the company’s flagship products, DDR5 and HBM3, increased by more than four and five times, respectively, compared to 2022.

Additionally, considering the relatively slow recovery in the NAND Flash memory market, the business plan primarily focuses on investment and cost efficiency.

In response to the growing trend in demand for high-performance DRAM, SK Hynix will smoothly proceed with the mass production of HBM3e memory for AI and the development of HBM4.

TrendForce’s earlier research into the HBM market indicates that NVIDIA plans to diversify its HBM suppliers for more robust and efficient supply chain management. The progress of HBM3e, as outlined in the timeline below, shows that SK Hynix provided its 8hi (24GB) samples to NVIDIA  in mid-August.

Simultaneously, the company aims to supply high-performance and high-capacity products like DDR5 and LPDDR5T to the server and mobile markets.

Moreover, to address the continued growth in demand for AI servers and the widespread adoption of edge AI computing applications, SK Hynix will exert efforts in the development of high-capacity server module MCR DIMM and mobile module LPCAMM2 to respond to the ever-increasing demand for AI servers and on-device AI adoption.

For NAND, the company aims to continue to improve profitability and stabilize the business by expanding sales of premium products such as eSSD, expected to improve profitability and strengthen internal management.

Lastly, SK Hynix emphasizes its commitment to maintaining and enhancing profitability and efficiency by continuing to expand the production of high-value-added products in 2024, similar to its strategy in 2023. The company will focus on minimizing capital expenditures while prioritizing stable business operations. 

“We achieved a remarkable turnaround, marking the first operating profit in the fourth quarter following a protracted downturn, thanks to our technological leadership in the AI memory space,” said Kim Woohyun, Vice President and Chief Financial Officer (CFO) at SK Hynix.

Kim further stated, “We are now ready to grow into a total AI memory provider by leading changes and presenting customized solutions as we enter an era for a new leap forward.”

Read more

• [News] SK Hynix Signs Prioritized Supply Agreement with NVIDIA for HBM3e, 2023 Q4 Revenue Expected to Surpass KRW 10 Trillion Again
• [News] TSMC Reportedly Attracts Influx of 3nm Customers, Predicts Production Capacity Utilization to Surge to 80% by Year-End

(Photo credit: SK Hynix)

SK hynix, the South Korean memory giant, has revealed plans to consider increasing production of specific DRAM in the first quarter due to improved market conditions. Concerns arise in the market regarding the potential resurgence of capacity utilization, challenging the consensus among memory manufacturers to reduce production for price enhancement, possibly hindering the upward trajectory of DRAM in the future.

According to a report by the Commercial Times, industry experts believe that major memory manufacturers, including Samsung, SK hynix, and Micron, are upgrading their production capacities to 1-alpha/beta advanced processes to meet the demand for more profitable DDR5 and High Bandwidth Memory (HBM). The outlook for niche DRAM product in the future is expected to remain positive.

SK Hynix CEO Kwak Noh-jung previously stated at the 2024 CES in Las Vegas that the company is considering increasing production of specific DRAM in the first quarter due to improved market conditions. Kwak mentioned that while the worst conditions for NAND Flash should be over, the industry’s recovery is slower. Therefore, any decision to increase production would likely be considered after the mid-year point.

Following this announcement, some industry insiders expressed concerns about the potential increase in memory manufacturers’ DRAM capacity utilization, which could negatively impact the future pricing trend of DRAM.

However, according to the Commercial Times, quoting a key figure in the Taiwanese memory industry revealed that SK Hynix’s planned increase in production primarily focuses on HBM for AI applications, with a slight increase in DDR5, while the production of DDR4 will continue to decrease.

As the three major manufacturers focus on 1-alpha/beta processes, increasing the supply of DDR5 and HBM, the pricing of niche-type DRAM is expected to gradually improve amid capacity constraints. Therefore, the perceived impact of SK Hynix’s increased production of specialized DRAM on the overall future DRAM pricing is not anticipated to be negative.

TrendForce points out that, considering the need for more advanced equipment to produce increased quantities of HBM and DDR5, the production capacity for DDR3 and DDR4 in the first quarter of 2024 is expected to decrease compared to the previous year.

However, due to market considerations and the ongoing shift towards DDR5 for high-end products, the demand for DDR3 and DDR4 is expected to slow down. This distribution of production capacity aligns with market trends and developments.

Tracking back the memory prices, from the fourth quarter of 2022 to the third quarter of 2023, the memory market experienced a downturn, prompting Samsung, SK Hynix, and Micron to collectively reduce production. By the fourth quarter of 2023, DRAM and NAND flash memory prices finally began to rebound, continuing into the first quarter of 2024, with the momentum of price increases continuing to be closely monitored by the market.

(Image: SK hynix)