According to the Korea Economic Daily. Samsung Electronics’ HBM3 and packaging services have passed AMD’s quality tests. The upcoming Instinct MI300 series AI chips from AMD are planned to incorporate Samsung’s HBM3 and packaging services. These chips, which combine central processing units (CPUs), graphics processing units (GPUs), and HBM3, are expected to be released in the fourth quarter of this year.

Samsung is noted as the sole provider capable of offering advanced packaging solutions and HBM products simultaneously. Originally considering TSMC’s advanced packaging services, AMD had to alter its plans due to capacity constraints.

The surge in demand for high-performance GPUs within the AI landscape benefits not only GPU manufacturers like NVIDIA and AMD, but also propels the development of HBM and advanced packaging.

In the backdrop of the AI trend, AIGC model training and inference require the deployment of AI servers. These servers typically require mid-to-high-end GPUs, with HBM penetration nearing 100% among these GPUs.

Presently, Samsung, SK Hynix, and Micron are the primary HBM manufacturers. According to the latest research by TrendForce, driven by the expansion efforts of these original manufacturers, the estimated annual growth rate of HBM supply in 2024 is projected to reach 105%.

In terms of competitive dynamics, SK Hynix leads with its HBM3 products, serving as the primary supplier for NVIDIA’s Server GPUs. Samsung, on the other hand, focuses on fulfilling orders from other cloud service providers. With added orders from customers, the gap in market share between Samsung and SK Hynix is expected to narrow significantly this year. The estimated HBM market share for both companies is about 95% for 2023 to 2024. However, variations in customer composition might lead to sequential variations in bit shipments.

In the realm of advanced packaging capacity, TSMC’s CoWoS packaging technology dominates as the main choice for AI server chip suppliers. Amidst strong demand for high-end AI chips and HBM, TrendForce estimates that TSMC’s CoWoS monthly capacity could reach 12K by the end of 2023.

With strong demand driven by NVIDIA’s A100 and H100 AI Server requirements, demand for CoWoS capacity is expected to rise by nearly 50% compared to the beginning of the year. Coupled with the growth in high-end AI chip demand from companies like AMD and Google, the latter half of the year could experience tighter CoWoS capacity. This robust demand is expected to continue into 2024, potentially leading to a 30-40% increase in advanced packaging capacity, contingent on equipment readiness.

(Photo credit: Samsung)

In response to the demands of high-performance computing, AI, 5G, and other applications, the shift towards chiplet and the incorporation of HBM memory has become inevitable for advanced chips. As a result, packaging has transitioned from 2D to 2.5D and 3D formats.

With chip manufacturing advancing towards more advanced process nodes, the model of directly packaging chips using advanced packaging technology from wafer foundries has emerged. However, this approach also signifies that wafer foundries will encroach upon certain aspects of traditional assembly and testing, leading to ongoing discussions about the ‘threat’ to traditional assembly and test firms since TSMC’s entry into advanced packaging in 2011.

But is this perspective accurate?

In fact, traditional assembly and test firms remain competitively positioned. Firstly, numerous electronic products still rely on their diverse traditional packaging techniques. Particularly, with the rapid growth of AIoT, electric vehicles, and drones, the required electronic components often still adopt traditional packaging methods. Secondly, faced with wafer foundries actively entering the advanced packaging domain, traditional assembly and test firms have not been idle, presenting concrete solutions to the challenge.

Advanced Packaging Innovations by Traditional Assembly and Test Firms

Since 2023, AI and AI server trends have rapidly emerged, driving the demand for AI chips. TSMC’s 2.5D advanced packaging technology, known as CoWoS, has played a pivotal role. However, the sudden surge in demand stretched TSMC’s capacity. In response, major traditional assembly and test firms such as ASE and Amkor have demonstrated their technical prowess and have no intention of being absent from this field.

For instance, ASE’s FOCoS technology enables the integration of HBM and ASIC. It restructures multiple chips into a fan-out module, which is then placed on the substrate, achieving the integration of multiple chips. Their FOCoS-Bridge technology, unveiled in May this year, utilizes silicon bridges (Si Bridge) to accomplish 2.5D packaging, bolstering the creation of advanced chips required for applications like AI, data centers, and servers.

Additionally, SPIL, a subsidiary of ASE, offers the FO-EB technology, a powerful integration of logic IC and HBM. As depicted below, this technology eschews silicon interposers, utilizing silicon bridges and redistribution layers (RDL) for connections, similarly capable of 2.5D packaging.

Another major player, Amkor, has not only collaborated with Samsung to develop the H-Cube advanced packaging solution but has also long been involved in ‘CoWoS-like technology.’ Through intermediary layers and through-silicon via (TSV) technology, Amkor can interconnect different chips, also possessing 2.5D advanced packaging capabilities.

China’s major assembly and test firm, Jiangsu Changjiang Electronics Technology (JCET), employs the XDFOI technology, integrating logic ICs with HBM through TSV, RDL, and microbump techniques, aimed at high-performance computing.

Given the recent surge in demand for high-end GPU chips, TSMC’s CoWoS capacity has fallen short, and NVIDIA is actively seeking support from second or even third suppliers. The ASE Group and Amkor have secured partial orders through their packaging technologies. This clearly illustrates that traditional assembly and test firms, even when faced with the entry of wafer foundries into the advanced packaging domain, still possess the capability to compete.

In terms of product types, wafer foundries focus on advanced packaging technology for major players like NVIDIA and AMD. Meanwhile, other products not in the highest-end category still opt for traditional assembly and test firms like ASE, Amkor, and JCET for manufacturing. Overall, with their presence in advanced packaging, as well as a hold on the expanding existing packaging market, traditional assembly and test firms continue to maintain their market competitiveness.

(Photo credit: Amkor)

Charlie Boyle, Vice President of NVIDIA’s DGX Systems, recently addressed the issue of limited GPU production at the company.

Boyle clarified that the current GPU shortage is not a result of NVIDIA misjudging demand or constraints in Taiwan Semiconductor Manufacturing Company’s (TSMC) wafer production. The primary bottleneck for GPUs lies in the packaging process.

It’s worth noting that the NVIDIA A100 and H100 GPUs are currently manufactured by TSMC using their advanced CoWoS (Chip-on-Wafer-on-Substrate) packaging technology. TSMC has indicated that it may take up to a year and a half, including the completion of additional wafer fabs and expansion of existing facilities, to normalize the backlog of packaging orders.

Furthermore, due to the significant strain on TSMC’s CoWoS capacity, there have been reports of overflow of NVIDIA GPU packaging orders to other manufacturers.

Sources familiar with the matter have revealed that NVIDIA is in discussions with potential alternative suppliers, including Samsung, as secondary suppliers for the 2.5D packaging of NVIDIA’s A100 and H100 GPUs. Other potential suppliers include Amkor and the Siliconware Precision Industries Co., Ltd. (SPIL), a subsidiary of ASE Technology Holding.

In December 2022, Samsung established its Advanced Packaging (AVP) division to seize opportunities in high-end packaging and testing. Sources suggest that if NVIDIA approves of Samsung’s 2.5D packaging process yield, a portion of AI GPU packaging orders may be placed with Samsung.

TrendForce’s research in June this year indicated that driven by strong demand for high-end AI chips and High-Bandwidth Memory (HBM), TSMC’s CoWoS monthly capacity could reach 12,000 units by the end of 2023. Particularly, demand from NVIDIA for A100 and H100 GPUs in AI servers has led to nearly a 50% increase in CoWoS capacity compared to the beginning of the year. Coupled with the growth in demand for high-end AI chips from companies like AMD and Google, the second half of the year is expected to witness tighter CoWoS capacity. This robust demand is projected to continue into 2024, with advanced packaging capacity potentially growing by 30-40% if the necessary equipment is in place.

(Photo credit: NVIDIA)

Over the past few decades, semiconductor manufacturing technology has evolved from the 10,000nm process in 1971 to the 3nm process in 2022, driven by the need to increase the number of transistors on chips for enhanced computational performance. However, as applications like artificial intelligence (AI) and AIGC rapidly advance, demand for higher core chip performance at the device level is growing.

While process technology improvements may encounter bottlenecks, the need for computing resources continues to rise. This underscores the importance of advanced packaging techniques to boost the number of transistors on chips.

In recent years, “advanced packaging” has gained significant attention. Think of “packaging” as a protective shell for electronic chips, safeguarding them from adverse environmental effects. Chip packaging involves fixation, enhanced heat dissipation, electrical connections, and signal interconnections with the outside world. The term “advanced packaging” primarily focuses on packaging techniques for chips with process nodes below 7nm.

Amid the AI boom, which has driven demand for AI servers and NVIDIA GPU graphics chips, CoWoS (Chip-on-Wafer-on-Substrate) packaging has faced a supply shortage.

But what exactly is CoWoS?

CoWoS is a 2.5D and 3D packaging technology, composed of “CoW” (Chip-on-Wafer) and “WoS” (Wafer-on-Substrate). CoWoS involves stacking chips and then packaging them onto a substrate, creating a 2.5D or 3D configuration. This approach reduces chip space, while also lowering power consumption and costs. The concept is illustrated in the diagram below, where logic chips and High-Bandwidth Memory (HBM) are interconnected on an interposer through tiny metal wires. “Through-Silicon Vias (TSV)” technology links the assembly to the substrate beneath, ultimately connecting to external circuits via solder balls.

The difference between 2.5D and 3D packaging lies in their stacking methods. 2.5D packaging involves horizontal chip stacking on an interposer or through silicon bridges, mainly for combining logic and high-bandwidth memory chips. 3D packaging vertically stacks chips, primarily targeting high-performance logic chips and System-on-Chip (SoC) designs.

When discussing advanced packaging, it’s worth noting that Taiwan Semiconductor Manufacturing Company (TSMC), rather than traditional packaging and testing facilities, is at the forefront. CoW, being a precise part of CoWoS, is predominantly produced by TSMC. This situation has paved the way for TSMC’s comprehensive service offerings, which maintain high yields in both fabrication and packaging processes. Such a setup ensures an unparalleled approach to serving high-end clients in the future.

Applications of CoWoS

The shift towards multiple small chips and memory stacking is becoming an inevitable trend for high-end chips. CoWoS packaging finds application in a wide range of fields, including High-Performance Computing (HPC), AI, data centers, 5G, Internet of Things (IoT), automotive electronics, and more. In various major trends, CoWoS packaging is set to play a vital role.

In the past, chip performance was primarily reliant on semiconductor process improvements. However, with devices approaching physical limits and chip miniaturization becoming increasingly challenging, maintaining small form factors and high chip performance has required improvements not only in advanced processes but also in chip architecture. This has led to a transition from single-layer chips to multi-layer stacking. As a result, advanced packaging has become a key driver in extending Moore’s Law and is leading the charge in the semiconductor industry.

(Photo credit: TSMC)

AI Chips and High-Performance Computing (HPC) have been continuously shaking up the entire supply chain, with CoWoS packaging technology being the latest area to experience the tremors.

In the previous piece, “HBM and 2.5D Packaging: the Essential Backbone Behind AI Server,” we discovered that the leading AI chip players, Nvidia and AMD, have been dedicated users of TSMC’s CoWoS technology. Much of the groundbreaking tech used in their flagship product series – such as Nvidia’s A100 and H100, and AMD’s Instinct MI250X and MI300 – have their roots in TSMC’s CoWoS tech.

However, with AI’s exponential growth, chip demand from not just Nvidia and AMD has skyrocketed, but other giants like Google and Amazon are also catching up in the AI field, bringing an onslaught of chip demand. The surge of orders is already testing the limits of TSMC’s CoWoS capacity. While TSMC is planning to increase its production in the latter half of 2023, there’s a snag – the lead time of the packaging equipment is proving to be a bottleneck, severely curtailing the pace of this necessary capacity expansion.

Nvidia Shakes the foundation of the CoWoS Supply Chain

In these times of booming demand, maintaining a stable supply is viewed as the primary goal for chipmakers, including Nvidia. While TSMC is struggling to keep up with customer needs, other chipmakers are starting to tweak their outsourcing strategies, moving towards a more diversified supply chain model. This shift is now opening opportunities for other foundries and OSATs.

Interestingly, in this reshuffling of the supply chain, UMC (United Microelectronics Corporation) is reportedly becoming one of Nvidia’s key partners in the interposer sector for the first time, with plans for capacity expansion on the horizon.

From a technical viewpoint, interposer has always been the cornerstone of TSMC’s CoWoS process and technology progression. As the interposer area enlarges, it allows for more memory stack particles and core components to be integrated. This is crucial for increasingly complex multi-chip designs, underscoring Nvidia’s intention to support UMC as a backup resource to safeguard supply continuity.

Meanwhile, as Nvidia secures production capacity, it is observed that the two leading OSAT companies, Amkor and SPIL (as part of ASE), are establishing themselves in the Chip-on-Wafer (CoW) and Wafer-on-Substrate (WoS) processes.

The ASE Group is no stranger to the 2.5D packaging arena. It unveiled its proprietary 2.5D packaging tech as early as 2017, a technology capable of integrating core computational elements and High Bandwidth Memory (HBM) onto the silicon interposer. This approach was once utilized in AMD’s MI200 series server GPU. Also under the ASE Group umbrella, SPIL boasts unique Fan-Out Embedded Bridge (FO-EB) technology. Bypassing silicon interposers, the platform leverages silicon bridges and redistribution layers (RDL) for integration, which provides ASE another competitive edge.

Could Samsung’s Turnkey Service Break New Ground?

In the shifting landscape of the supply chain, the Samsung Device Solutions division’s turnkey service, spanning from foundry operations to Advanced Package (AVP), stands out as an emerging player that can’t be ignored.

After its 2018 split, Samsung Foundry started taking orders beyond System LSI for business stability. In 2023, the AVP department, initially serving Samsung’s memory and foundry businesses, has also expanded its reach to external clients.

Our research indicates that Samsung’s AVP division is making aggressive strides into the AI field. Currently in active talks with key customers in the U.S. and China, Samsung is positioning its foundry-to-packaging turnkey solutions and standalone advanced packaging processes as viable, mature options.

In terms of technology roadmap, Samsung has invested significantly in 2.5D packaging R&D. Mirroring TSMC, the company launched two 2.5D packaging technologies in 2021: the I-Cube4, capable of integrating four HBM stacks and one core component onto a silicon interposer, and the H-Cube, designed to extend packaging area by integrating HDI PCB beneath the ABF substrate, primarily for designs incorporating six or more HBM stack particles.

Besides, recognizing Japan’s dominance in packaging materials and technologies, Samsung recently launched a R&D center there to swiftly upscale its AVP business.

Given all these circumstances, it seems to be only a matter of time before Samsung carves out its own significant share in the AI chip market. Despite TSMC’s industry dominance and pivotal role in AI chip advancements, the rising demand for advanced packaging is set to undeniably reshape supply chain dynamics and the future of the semiconductor industry.

(Source: Nvidia)