As applications like AIGC, 8K, AR/MR, and others continue to develop, 3D IC stacking and heterogeneous integration of chiplet have become the primary solutions to meet future high-performance computing demands and extend Moore’s Law.

Major companies like TSMC and Intel have been expanding their investments in heterogeneous integration manufacturing and related research and development in recent years. Additionally, leading EDA company Cadence has taken the industry lead by introducing the “Integrity 3D-IC” platform, an integrated solution for design planning, realization, and system analysis simulation tools, marking a significant step towards 3D chip stacking.

Differences between 2.5D and 3D Packaging

The main difference between 2.5D and 3D packaging technologies lies in the stacking method. 2.5D packaging involves stacking chips one by one on an interposer or connecting them through silicon bridges, primarily used for assembling logic processing chips and high-bandwidth memory. On the other hand, 3D packaging is a technology that vertically stacks chips, mainly targeting high-performance logic chips and SoC manufacturing.

CPU and HBM Stacking Demands

With the rapid development of applications like AIGC, AR/VR, and 8K, it is expected that a significant amount of computational demand will arise, particularly driving the need for parallel computing systems capable of processing big data in a short time. To overcome the bandwidth limitations of DDR SDRAM and further enhance parallel computing performance, the industry has been increasingly adopting High-Bandwidth Memory (HBM). This trend has led to a shift from the traditional “CPU + memory (such as DDR4)” architecture to the “Chip + HBM stacking” 2.5D architecture. With continuous growth in computational demand, the future may see the integration of CPU, GPU, or SoC through 3D stacking.

3D Stacking with HBM Prevails, but CPU Stacking Lags Behind

HBM was introduced in 2013 as a 3D stacked architecture for high-performance SDRAM. Over time, the stacking of multiple layers of HBM has become widespread in packaging, while the stacking of CPUs/GPUs has not seen significant progress.

The main reasons for this disparity can be attributed to three factors: 1. Thermal conduction, 2. Thermal stress, and 3. IC design. First, 3D stacking has historically performed poorly in terms of thermal conduction, which is why it is primarily used in memory stacking, as memory operations generate much less heat than logic operations. As a result, the thermal conduction issues faced by current memory stacking products can be largely disregarded.

Second, thermal stress issues arise from the mismatch in coefficients of thermal expansion (CTE) between materials and the introduction of stress from thinning the chips and introducing metal layers. The complex stress distribution in stacked structures has a significant negative impact on product reliability.

Finally, IC design challenges from a lack of EDA tools, as traditional CAD tools are inadequate for handling 3D design rules. Developers must create their own tools to address process requirements, and the complex design of 3D packaging further increases the design, manufacturing, and testing costs.

How EDA Companies Offer Solutions

Cadence, during the LIVE Taiwan 2023 user annual conference, highlighted its years of effort in developing solutions. They have introduced tools like the Clarity 3D solver, Celsius thermal solver, and Sigrity Signal and Power Integrity, which can address thermal conduction and thermal stress simulation issues. When combined with Cadence’s comprehensive EDA tools, these offerings contribute to the growth of the “Integrity 3D-IC” platform, aiding in the development of 3D IC design.

“3D IC” represents a critical design trend in semiconductor development. However, it presents greater challenges and complexity than other projects. In addition to the challenges in Logic IC design, there is a need for analog and multi-physics simulations. Therefore, cross-platform design tools are indispensable. The tools provided by EDA leader Cadence are expected to strengthen the 3D IC design tool platform, reducing the technological barriers for stacking CPU, GPU, or SoC to enhance chip computing performance.

This article is from TechNews, a collaborative media partner of TrendForce.

(Photo credit: TSMC)

According to Taiwan’s TechNews report, Lu Donghui, Chairman of Micron Technology Taiwan, stated that in response to the growing demand in the AI market, Micron Technology Taiwan will continue to invest in advanced processes and packaging technologies to produce High Bandwidth Memory (HBM) products. Micron Technology Taiwan is the only Micron facility globally with advanced packaging capabilities.

Lu Donghui, speaking at a media event, mentioned that Micron had previously introduced the industry’s first 8-layer stack (8-High) 24GB HBM3 Gen 2 product, which is now in the sampling phase. This product boasts a bandwidth exceeding 1.2TB/s and a transmission rate exceeding 9.2Gb/s, which is 50% higher than other HBM3 solutions on the market. Micron’s HBM3 Gen 2 product offers 2.5 times better energy efficiency per watt compared to previous generations, making it ideal for high-performance AI applications.

Micron’s HBM3 Gen 2 memory products are manufactured using the most advanced 1-beta process technology in Taiwan and Japan. Compared to the previous 1-alpha process, the 1-beta process reduces power consumption by approximately 15% and increases bit density by over 35%, with each chip offering a capacity of up to 16Gb. Through Micron’s advanced packaging technology, the 1-beta process memory chips are stacked in 8 layers, and the complete HBM3 Gen 2 chips are packaged and sent to customers’ specified semiconductor foundries like TSMC, Intel, Samsung, or third-party packaging and testing facilities for GPUs, CPUs.

Lu Donghui highlighted that Taiwan’s robust semiconductor manufacturing ecosystem makes it the exclusive hub for Micron’s advanced packaging development worldwide. By combining this ecosystem with Micron’s offerings, they can provide customers with comprehensive solutions to meet market demands. While HBM products represent a relatively small portion of the overall memory market, their future growth potential is significant, with expectations to capture around 10% of the entire memory market in the short term.

(Photo credit: Micron)

NVIDIA’s latest financial report for FY2Q24 reveals that its data center business reached US$10.32 billion—a QoQ growth of 141% and YoY increase of 171%. The company remains optimistic about its future growth. TrendForce believes that the primary driver behind NVIDIA’s robust revenue growth stems from its data center’s AI server-related solutions. Key products include AI-accelerated GPUs and AI server HGX reference architecture, which serve as the foundational AI infrastructure for large data centers.

TrendForce further anticipates that NVIDIA will integrate its software and hardware resources. Utilizing a refined approach, NVIDIA will align its high-end, mid-tier, and entry-level GPU AI accelerator chips with various ODMs and OEMs, establishing a collaborative system certification model. Beyond accelerating the deployment of CSP cloud AI server infrastructures, NVIDIA is also partnering with entities like VMware on solutions including the Private AI Foundation. This strategy extends NVIDIA’s reach into the edge enterprise AI server market, underpinning steady growth in its data center business for the next two years.

NVIDIA’s data center business surpasses 76% market share due to strong demand for cloud AI

In recent years, NVIDIA has been actively expanding its data center business. In FY4Q22, data center revenue accounted for approximately 42.7%, trailing its gaming segment by about 2 percentage points. However, by FY1Q23, data center business surpassed gaming—accounting for over 45% of revenue. Starting in 2023, with major CSPs heavily investing in ChatBOTS and various AI services for public cloud infrastructures, NVIDIA reaped significant benefits. By FY2Q24, data center revenue share skyrocketed to over 76%.

NVIDIA targets both Cloud and Edge Data Center AI markets

TrendForce observes and forecasts a shift in NVIDIA’s approach to high-end GPU products in 2H23. While the company has primarily focused on top-tier AI servers equipped with the A100 and H100, given positive market demand, NVIDIA is likely to prioritize the higher-priced H100 to effectively boost its data-center-related revenue growth.

NVIDIA is currently emphasizing the L40s as their flagship product for mid-tier GPUs, meaning several strategic implications: Firstly, the high-end H100 series is constrained by the limited production capacity of current CoWoS and HBM technologies. In contrast, the L40s primarily utilizes GDDR memory. Without the need for CoWos packaging, it can be rapidly introduced to the mid-tier AI server market, filling the gap left by the A100 PCle interface in meeting the needs of enterprise customers.

Secondly, the L40s also target enterprise customers who don’t require large parameter models like ChatGPT. Instead, it focuses on more compact AI training applications in various specialized fields, with parameter counts ranging from tens of billions to under a hundred billion. They can also address edge AI inference or image analysis tasks. Additionally, in light of potential geopolitical issues that might disrupt the supply of the high-end GPU H series for Chinese customers, the L40s can serve as an alternative. As for lower-tier GPUs, NVIDIA highlights the L4 or T4 series, which are designed for real-time AI inference or image analysis in edge AI servers. These GPUs underscore affordability while maintaining a high-cost-performance ratio.

HGX and MGX AI server reference architectures are set to be NVIDIA’s main weapons for AI solutions in 2H23

TrendForce notes that recently, NVIDIA has not only refined its product positioning for its core AI chip GPU but has also actively promoted its HGX and MGX solutions. Although this approach isn’t new in the server industry, NVIDIA has the opportunity to solidify its leading position with this strategy. The key is NVIDIA’s absolute leadership stemming from its extensive integration of its GPU and CUDA platform—establishing a comprehensive AI ecosystem. As a result, NVIDIA has considerable negotiating power with existing server supply chains. Consequently, ODMs like Inventec, Quanta, FII, Wistron, and Wiwynn, as well as brands such as Dell, Supermicro, and Gigabyte, are encouraged to follow NVIDIA’s HGX or MGX reference designs. However, they must undergo NVIDIA’s hardware and software certification process for these AI server reference architectures. Leveraging this, NVIDIA can bundle and offer integrated solutions like its Arm CPU Grace, NPU, and AI Cloud Foundation.

It’s worth noting that for ODMs or OEMs, given that NVIDIA is expected to make significant achievements in the AI server market for CSPs from 2023 to 2024, there will likely be a boost in overall shipment volume and revenue growth of AI servers. However, with NVIDIA’s strategic introduction of standardized AI server architectures like HGX or MGX, the core product architecture for AI servers among ODMs and others will become more homogenized. This will intensify the competition among them as they vie for orders from CSPs. Furthermore, it’s been observed that large CSPs such as Google and AWS are leaning toward adopting in-house ASIC AI accelerator chips in the future, meaning there’s a potential threat to a portion of NVIDIA’s GPU market. This is likely one of the reasons NVIDIA continues to roll out GPUs with varied positioning and comprehensive solutions. They aim to further expand their AI business aggressively to Tier-2 data centers (like CoreWeave) and edge enterprise clients.

According to the Korea Economic Daily. Samsung Electronics’ HBM3 and packaging services have passed AMD’s quality tests. The upcoming Instinct MI300 series AI chips from AMD are planned to incorporate Samsung’s HBM3 and packaging services. These chips, which combine central processing units (CPUs), graphics processing units (GPUs), and HBM3, are expected to be released in the fourth quarter of this year.

Samsung is noted as the sole provider capable of offering advanced packaging solutions and HBM products simultaneously. Originally considering TSMC’s advanced packaging services, AMD had to alter its plans due to capacity constraints.

The surge in demand for high-performance GPUs within the AI landscape benefits not only GPU manufacturers like NVIDIA and AMD, but also propels the development of HBM and advanced packaging.

In the backdrop of the AI trend, AIGC model training and inference require the deployment of AI servers. These servers typically require mid-to-high-end GPUs, with HBM penetration nearing 100% among these GPUs.

Presently, Samsung, SK Hynix, and Micron are the primary HBM manufacturers. According to the latest research by TrendForce, driven by the expansion efforts of these original manufacturers, the estimated annual growth rate of HBM supply in 2024 is projected to reach 105%.

In terms of competitive dynamics, SK Hynix leads with its HBM3 products, serving as the primary supplier for NVIDIA’s Server GPUs. Samsung, on the other hand, focuses on fulfilling orders from other cloud service providers. With added orders from customers, the gap in market share between Samsung and SK Hynix is expected to narrow significantly this year. The estimated HBM market share for both companies is about 95% for 2023 to 2024. However, variations in customer composition might lead to sequential variations in bit shipments.

In the realm of advanced packaging capacity, TSMC’s CoWoS packaging technology dominates as the main choice for AI server chip suppliers. Amidst strong demand for high-end AI chips and HBM, TrendForce estimates that TSMC’s CoWoS monthly capacity could reach 12K by the end of 2023.

With strong demand driven by NVIDIA’s A100 and H100 AI Server requirements, demand for CoWoS capacity is expected to rise by nearly 50% compared to the beginning of the year. Coupled with the growth in high-end AI chip demand from companies like AMD and Google, the latter half of the year could experience tighter CoWoS capacity. This robust demand is expected to continue into 2024, potentially leading to a 30-40% increase in advanced packaging capacity, contingent on equipment readiness.

(Photo credit: Samsung)

In response to the demands of high-performance computing, AI, 5G, and other applications, the shift towards chiplet and the incorporation of HBM memory has become inevitable for advanced chips. As a result, packaging has transitioned from 2D to 2.5D and 3D formats.

With chip manufacturing advancing towards more advanced process nodes, the model of directly packaging chips using advanced packaging technology from wafer foundries has emerged. However, this approach also signifies that wafer foundries will encroach upon certain aspects of traditional assembly and testing, leading to ongoing discussions about the ‘threat’ to traditional assembly and test firms since TSMC’s entry into advanced packaging in 2011.

But is this perspective accurate?

In fact, traditional assembly and test firms remain competitively positioned. Firstly, numerous electronic products still rely on their diverse traditional packaging techniques. Particularly, with the rapid growth of AIoT, electric vehicles, and drones, the required electronic components often still adopt traditional packaging methods. Secondly, faced with wafer foundries actively entering the advanced packaging domain, traditional assembly and test firms have not been idle, presenting concrete solutions to the challenge.

Advanced Packaging Innovations by Traditional Assembly and Test Firms

Since 2023, AI and AI server trends have rapidly emerged, driving the demand for AI chips. TSMC’s 2.5D advanced packaging technology, known as CoWoS, has played a pivotal role. However, the sudden surge in demand stretched TSMC’s capacity. In response, major traditional assembly and test firms such as ASE and Amkor have demonstrated their technical prowess and have no intention of being absent from this field.

For instance, ASE’s FOCoS technology enables the integration of HBM and ASIC. It restructures multiple chips into a fan-out module, which is then placed on the substrate, achieving the integration of multiple chips. Their FOCoS-Bridge technology, unveiled in May this year, utilizes silicon bridges (Si Bridge) to accomplish 2.5D packaging, bolstering the creation of advanced chips required for applications like AI, data centers, and servers.

Additionally, SPIL, a subsidiary of ASE, offers the FO-EB technology, a powerful integration of logic IC and HBM. As depicted below, this technology eschews silicon interposers, utilizing silicon bridges and redistribution layers (RDL) for connections, similarly capable of 2.5D packaging.

Another major player, Amkor, has not only collaborated with Samsung to develop the H-Cube advanced packaging solution but has also long been involved in ‘CoWoS-like technology.’ Through intermediary layers and through-silicon via (TSV) technology, Amkor can interconnect different chips, also possessing 2.5D advanced packaging capabilities.

China’s major assembly and test firm, Jiangsu Changjiang Electronics Technology (JCET), employs the XDFOI technology, integrating logic ICs with HBM through TSV, RDL, and microbump techniques, aimed at high-performance computing.

Given the recent surge in demand for high-end GPU chips, TSMC’s CoWoS capacity has fallen short, and NVIDIA is actively seeking support from second or even third suppliers. The ASE Group and Amkor have secured partial orders through their packaging technologies. This clearly illustrates that traditional assembly and test firms, even when faced with the entry of wafer foundries into the advanced packaging domain, still possess the capability to compete.

In terms of product types, wafer foundries focus on advanced packaging technology for major players like NVIDIA and AMD. Meanwhile, other products not in the highest-end category still opt for traditional assembly and test firms like ASE, Amkor, and JCET for manufacturing. Overall, with their presence in advanced packaging, as well as a hold on the expanding existing packaging market, traditional assembly and test firms continue to maintain their market competitiveness.

(Photo credit: Amkor)