With the approach to the end of 2023, TrendForce revealed the tech trends in every sector, apparently, AI continues as the main focus to decide the direction of how the tech supply chain will be in the next few years, here are the seeings:

CSPs increase AI investment, driving a 38% growth in AI server shipments by 2024

• CSPs increase AI investment, fueling a 38% growth in AI server shipments by 2024.
• Major CSPs like Microsoft, Google, and AWS are driving this growth due to the rising popularity of AI applications, pushing AI server shipments to 1.2 million units in 2023.

HBM3e set to drive an annual increase of 172% in HBM revenue

• Major memory suppliers are set to introduce HBM3e with faster speeds (8 Gbps) to enhance the performance of AI accelerator chips in 2024–2025.
• HBM integration is becoming common among GPU manufacturers like NVIDIA and AMD, and it is expected that HBM will significantly contribute to memory suppliers’ revenues in 2024, with an annual growth rate of 172%.

Rising demand for advanced packaging in 2024, the emergence of 3D IC technology

• Leading semiconductor firms like TSMC, Samsung, and Intel are emphasizing advanced packaging technology’s importance in boosting chip performance, conserving space, reduce power usage, and minimize latency. They’re establishing 3D IC research centers in Japan to underscore this role.
• Generative AI is driving increased demand for 2.5D packaging tech, integrating computing chips and memory with a silicon interposer layer. Additionally, 3D packaging solutions like TSMC’s SoIC, Samsung’s X-Cube, and Intel’s Foveros.

NTN is set to begin with small-scale commercial tests, broader applications of this technology are on the way in 2024

• Collaboration between satellite operators, semiconductor firms, telecom operators, and smartphone makers is growing due to increased satellite deployments by operators. This collaboration focuses on mobile satellite communication applications and bidirectional data transmission under specific conditions.
• Major semiconductor manufacturers are ramping up efforts in satellite communication chips, leading top smartphone manufacturers to integrate satellite communication into high-end phones using the SoC model, which is expected to drive small-scale commercial testing of NTN networks and promote widespread adoption of NTN applications.

6G communication to begin in 2024, with satellite communication taking center stage

• 6G standardization begins around 2024-2025, with initial technologies expected by 2027-2028. This enables novel applications like Reconfigurable Intelligent Surfaces (RIS), terahertz bands, Optical Wireless Communication (OWC), NTN for high-altitude comms, and immersive Extended Reality (XR) experiences.
• Low-orbit satellites will play a key role in 6G as its standards solidify, peaking around the time of 6G commercialization. The use of drones for 6G communication and environmental sensing is also set to surge in the 6G era.

Innovative entrants drive cost optimization for Micro LED technology in 2024

• In 2023, the focus in Micro LED display technology is on cost reduction through chip downsizing, aiming for at least a 20-25% annual reduction. A hybrid transfer approach, combining stamping and laser bonding, is gaining attention for efficient mass production.
• Micro LED holds potential in micro-projection displays for transparent AR lenses. Challenges include achieving ultra-high PPI with 5 µm or smaller chips, particularly with red LEDs’ low efficiency. Various innovative approaches, such as InGan-based red LEDs and vertically stacked RGB LEDs.

Intensifying competition in AR/VR micro-display technologies

• Increasing AR/VR headset demand drives demand for ultra-high PPI near-eye displays, with Micro OLED technology at the forefront, poised for broader adoption.
• Challenges in brightness and efficiency impact Micro OLED displays and their dominance in the head-mounted display market depends on the development of various micro-display technologies.

Advancements in material and component technologies are propelling the commercialization of gallium oxide

• Gallium oxide (Ga₂O₃) is gaining prominence for next-gen power semiconductor devices due to its potential in high-voltage, high-temperature, and high-frequency applications in EVs, electrical grids, and aerospace.
• The industry is already producing 4-inch gallium oxide mono-crystals and advancing Schottky diode and transistor fabrication processes, with the first Schottky diode products expected by 2024.

Solid-state batteries poised to reshape the EV battery landscape over the next decade

• Major automakers and battery manufacturers are investing in solid-state and semi-solid-state battery technologies, aiming for a new cycle of technological iteration by 2024.
• After Li-ion batteries, sodium-ion batteries, with lower energy density, are suitable for budget-friendly EVs, and hydrogen fuel cells offer long-range and zero emissions, primarily for heavy-duty commercial vehicles, with widespread adoption expected after 2025, despite challenges.

BEVs in 2024 rely on power conversion efficiency, driving range, and charging efficiency

• Automakers are optimizing battery pack structures and material ratios to increase energy density and driving range. Solid-state batteries, with high energy density, may see limited installations in vehicles as semi-solid batteries in 2H23.
• The 800V platform will enable high-power fast charging, leading to the expansion of high-power charging stations. AI advancements are driving EVs toward advanced autonomous driving, with Tesla’s Dojo supercomputer investing in neural network training to maintain its position in the intelligent driving market.

Green solutions with AI simulations emerging as a linchpin for renewable energy and decarbonized manufacturing

• Under the background of optimizing energy consumption, creating interconnected data ecosystems, and visualizing energy flow and consumption. Carbon auditing tools and AI are key for organizations aiming to reduce carbon emissions and enhance sustainability.
• The IEA predicts global renewable energy generation to reach 4,500 GW by 2024, driven by policy support, rising fossil fuel prices, and energy crises. The adoption of AI-driven smart technologies in peripheral systems for stable energy generation.

OLED’s expansion will across various applications driven by the innovation of foldable phones

• OLED folding phones are improving in design by using lightweight materials, innovative hinge structures, and cost-reduction efforts to approach the thickness and weight of traditional smartphones.
• In the IT sector, industry players like Samsung, BOE Technology, JDI, and Visionox are making significant investments and developments in OLED technology to expand into various markets. Anticipated advancements in technology and materials are expected to increase OLED market penetration by 2025.

Amidst the AI boom, HBM technology steps into the spotlight as market demand continues to surge. Global market research firm TrendForce anticipates a 58% year-on-year increase in HBM demand in 2023, with a potential additional growth of approximately 30% in 2024.

Compared to traditional DRAM, HBM (High Bandwidth Memory) boasts advantages such as high bandwidth, high capacity, low latency, and low power consumption. These attributes accelerate AI data processing and make it particularly well-suited for high-performance computing scenarios like ChatGPT. As a result, it has gained popularity, and major storage manufacturers are actively driving HBM technology upgrades.

Leading memory manufacturers are intensifying their efforts, with Samsung set to introduce HBM4.

Since the inception of the first HBM products utilizing TSV packaging technology in 2014, HBM technology has seen multiple upgrades, including HBM, HBM2, HBM2E, HBM3, and HBM3e.

Regarding the SK Hynix and Samsung, two major South Korean companies, have been at the forefront of HBM3 development. NVIDIA’s H100/H800 and AMD’s MI300 series, represent HBM3’s progress. Both SK Hynix and Samsung expected to offer HBM3e samples by the first quarter of 2024. On the other hand, Micron, a U.S.-based memory company, is bypassing HBM3 and directly pursuing HBM3e.

HBM3e will feature 24Gb mono die stacks, and with an 8-layer (8Hi) configuration, a single HBM3e chip’s capacity will soar to 24GB. This advancement is expected to be incorporated into NVIDIA’s GB100 in 2025, leading the three major OEMs to plan HBM3e sample releases in the first quarter of 2024 and enter mass production in the latter half of the year.

In addition to HBM3 and HBM3e, the latest updates indicate that storage giants are planning the launch of the next generation of HBM—HBM4.

Samsung recently announced that it has developed 9.8Gbps HBM3E and is planning to provide samples to customers. Furthermore, Samsung is actively working on HBM4 with a goal to begin supply in 2025. It’s reported that Samsung Electronics is developing technologies such as non-conductive adhesive film (NCF) assembly for optimizing high-temperature thermal characteristics, as well as hybrid bonding (HCB), for HBM4 products.

In September, Korean media reported that Samsung is gearing up to revamp its production process and launch HBM4 products to capture the rapidly growing HBM market. HBM4 memory stacks will feature a 2048-bit memory interface, a significant departure from the previous 1024-bit interface for all HBM stacks. This enhanced interface width holds great significance for the evolution of HBM4.

While HBM4 promises a major breakthrough, it is still a ways off, making it too early to discuss its practical applications and widespread adoption. Industry experts emphasize that the current HBM market is dominated by HBM2e. However, HBM3 and HBM3e are poised to take the lead in the near future.

According to TrendForce’s research, HBM2e currently accounts for the mainstream market share, being used in various products like NVIDIA A100/A800, AMD MI200, and many AI accelerators developed by CSPs. To keep pace with the evolving demands of AI accelerator chips, OEMs are planning to introduce new HBM3e products in 2024, with HBM3 and HBM3e expected to become the market’s primary players next year.

In terms of the demand transition between different HBM generations, TrendForce estimates that in 2023, mainstream demand will shift from HBM2e to HBM3, with estimated demand shares of approximately 50% and 39%, respectively. As more HBM3-based accelerator chips enter the market, demand will substantially shift toward HBM3 in 2024, surpassing HBM2e and accounting for an estimated 60% of the market. This transition, coupled with higher average selling prices (ASP), is poised to significantly drive HBM revenue growth next year.

(Photo credit: Samsung)

As applications like AIGC, 8K, AR/MR, and others continue to develop, 3D IC stacking and heterogeneous integration of chiplet have become the primary solutions to meet future high-performance computing demands and extend Moore’s Law.

Major companies like TSMC and Intel have been expanding their investments in heterogeneous integration manufacturing and related research and development in recent years. Additionally, leading EDA company Cadence has taken the industry lead by introducing the “Integrity 3D-IC” platform, an integrated solution for design planning, realization, and system analysis simulation tools, marking a significant step towards 3D chip stacking.

Differences between 2.5D and 3D Packaging

The main difference between 2.5D and 3D packaging technologies lies in the stacking method. 2.5D packaging involves stacking chips one by one on an interposer or connecting them through silicon bridges, primarily used for assembling logic processing chips and high-bandwidth memory. On the other hand, 3D packaging is a technology that vertically stacks chips, mainly targeting high-performance logic chips and SoC manufacturing.

CPU and HBM Stacking Demands

With the rapid development of applications like AIGC, AR/VR, and 8K, it is expected that a significant amount of computational demand will arise, particularly driving the need for parallel computing systems capable of processing big data in a short time. To overcome the bandwidth limitations of DDR SDRAM and further enhance parallel computing performance, the industry has been increasingly adopting High-Bandwidth Memory (HBM). This trend has led to a shift from the traditional “CPU + memory (such as DDR4)” architecture to the “Chip + HBM stacking” 2.5D architecture. With continuous growth in computational demand, the future may see the integration of CPU, GPU, or SoC through 3D stacking.

3D Stacking with HBM Prevails, but CPU Stacking Lags Behind

HBM was introduced in 2013 as a 3D stacked architecture for high-performance SDRAM. Over time, the stacking of multiple layers of HBM has become widespread in packaging, while the stacking of CPUs/GPUs has not seen significant progress.

The main reasons for this disparity can be attributed to three factors: 1. Thermal conduction, 2. Thermal stress, and 3. IC design. First, 3D stacking has historically performed poorly in terms of thermal conduction, which is why it is primarily used in memory stacking, as memory operations generate much less heat than logic operations. As a result, the thermal conduction issues faced by current memory stacking products can be largely disregarded.

Second, thermal stress issues arise from the mismatch in coefficients of thermal expansion (CTE) between materials and the introduction of stress from thinning the chips and introducing metal layers. The complex stress distribution in stacked structures has a significant negative impact on product reliability.

Finally, IC design challenges from a lack of EDA tools, as traditional CAD tools are inadequate for handling 3D design rules. Developers must create their own tools to address process requirements, and the complex design of 3D packaging further increases the design, manufacturing, and testing costs.

How EDA Companies Offer Solutions

Cadence, during the LIVE Taiwan 2023 user annual conference, highlighted its years of effort in developing solutions. They have introduced tools like the Clarity 3D solver, Celsius thermal solver, and Sigrity Signal and Power Integrity, which can address thermal conduction and thermal stress simulation issues. When combined with Cadence’s comprehensive EDA tools, these offerings contribute to the growth of the “Integrity 3D-IC” platform, aiding in the development of 3D IC design.

“3D IC” represents a critical design trend in semiconductor development. However, it presents greater challenges and complexity than other projects. In addition to the challenges in Logic IC design, there is a need for analog and multi-physics simulations. Therefore, cross-platform design tools are indispensable. The tools provided by EDA leader Cadence are expected to strengthen the 3D IC design tool platform, reducing the technological barriers for stacking CPU, GPU, or SoC to enhance chip computing performance.

This article is from TechNews, a collaborative media partner of TrendForce.

(Photo credit: TSMC)

According to Taiwan’s TechNews report, Lu Donghui, Chairman of Micron Technology Taiwan, stated that in response to the growing demand in the AI market, Micron Technology Taiwan will continue to invest in advanced processes and packaging technologies to produce High Bandwidth Memory (HBM) products. Micron Technology Taiwan is the only Micron facility globally with advanced packaging capabilities.

Lu Donghui, speaking at a media event, mentioned that Micron had previously introduced the industry’s first 8-layer stack (8-High) 24GB HBM3 Gen 2 product, which is now in the sampling phase. This product boasts a bandwidth exceeding 1.2TB/s and a transmission rate exceeding 9.2Gb/s, which is 50% higher than other HBM3 solutions on the market. Micron’s HBM3 Gen 2 product offers 2.5 times better energy efficiency per watt compared to previous generations, making it ideal for high-performance AI applications.

Micron’s HBM3 Gen 2 memory products are manufactured using the most advanced 1-beta process technology in Taiwan and Japan. Compared to the previous 1-alpha process, the 1-beta process reduces power consumption by approximately 15% and increases bit density by over 35%, with each chip offering a capacity of up to 16Gb. Through Micron’s advanced packaging technology, the 1-beta process memory chips are stacked in 8 layers, and the complete HBM3 Gen 2 chips are packaged and sent to customers’ specified semiconductor foundries like TSMC, Intel, Samsung, or third-party packaging and testing facilities for GPUs, CPUs.

Lu Donghui highlighted that Taiwan’s robust semiconductor manufacturing ecosystem makes it the exclusive hub for Micron’s advanced packaging development worldwide. By combining this ecosystem with Micron’s offerings, they can provide customers with comprehensive solutions to meet market demands. While HBM products represent a relatively small portion of the overall memory market, their future growth potential is significant, with expectations to capture around 10% of the entire memory market in the short term.

(Photo credit: Micron)

NVIDIA’s latest financial report for FY2Q24 reveals that its data center business reached US$10.32 billion—a QoQ growth of 141% and YoY increase of 171%. The company remains optimistic about its future growth. TrendForce believes that the primary driver behind NVIDIA’s robust revenue growth stems from its data center’s AI server-related solutions. Key products include AI-accelerated GPUs and AI server HGX reference architecture, which serve as the foundational AI infrastructure for large data centers.

TrendForce further anticipates that NVIDIA will integrate its software and hardware resources. Utilizing a refined approach, NVIDIA will align its high-end, mid-tier, and entry-level GPU AI accelerator chips with various ODMs and OEMs, establishing a collaborative system certification model. Beyond accelerating the deployment of CSP cloud AI server infrastructures, NVIDIA is also partnering with entities like VMware on solutions including the Private AI Foundation. This strategy extends NVIDIA’s reach into the edge enterprise AI server market, underpinning steady growth in its data center business for the next two years.

NVIDIA’s data center business surpasses 76% market share due to strong demand for cloud AI

In recent years, NVIDIA has been actively expanding its data center business. In FY4Q22, data center revenue accounted for approximately 42.7%, trailing its gaming segment by about 2 percentage points. However, by FY1Q23, data center business surpassed gaming—accounting for over 45% of revenue. Starting in 2023, with major CSPs heavily investing in ChatBOTS and various AI services for public cloud infrastructures, NVIDIA reaped significant benefits. By FY2Q24, data center revenue share skyrocketed to over 76%.

NVIDIA targets both Cloud and Edge Data Center AI markets

TrendForce observes and forecasts a shift in NVIDIA’s approach to high-end GPU products in 2H23. While the company has primarily focused on top-tier AI servers equipped with the A100 and H100, given positive market demand, NVIDIA is likely to prioritize the higher-priced H100 to effectively boost its data-center-related revenue growth.

NVIDIA is currently emphasizing the L40s as their flagship product for mid-tier GPUs, meaning several strategic implications: Firstly, the high-end H100 series is constrained by the limited production capacity of current CoWoS and HBM technologies. In contrast, the L40s primarily utilizes GDDR memory. Without the need for CoWos packaging, it can be rapidly introduced to the mid-tier AI server market, filling the gap left by the A100 PCle interface in meeting the needs of enterprise customers.

Secondly, the L40s also target enterprise customers who don’t require large parameter models like ChatGPT. Instead, it focuses on more compact AI training applications in various specialized fields, with parameter counts ranging from tens of billions to under a hundred billion. They can also address edge AI inference or image analysis tasks. Additionally, in light of potential geopolitical issues that might disrupt the supply of the high-end GPU H series for Chinese customers, the L40s can serve as an alternative. As for lower-tier GPUs, NVIDIA highlights the L4 or T4 series, which are designed for real-time AI inference or image analysis in edge AI servers. These GPUs underscore affordability while maintaining a high-cost-performance ratio.

HGX and MGX AI server reference architectures are set to be NVIDIA’s main weapons for AI solutions in 2H23

TrendForce notes that recently, NVIDIA has not only refined its product positioning for its core AI chip GPU but has also actively promoted its HGX and MGX solutions. Although this approach isn’t new in the server industry, NVIDIA has the opportunity to solidify its leading position with this strategy. The key is NVIDIA’s absolute leadership stemming from its extensive integration of its GPU and CUDA platform—establishing a comprehensive AI ecosystem. As a result, NVIDIA has considerable negotiating power with existing server supply chains. Consequently, ODMs like Inventec, Quanta, FII, Wistron, and Wiwynn, as well as brands such as Dell, Supermicro, and Gigabyte, are encouraged to follow NVIDIA’s HGX or MGX reference designs. However, they must undergo NVIDIA’s hardware and software certification process for these AI server reference architectures. Leveraging this, NVIDIA can bundle and offer integrated solutions like its Arm CPU Grace, NPU, and AI Cloud Foundation.

It’s worth noting that for ODMs or OEMs, given that NVIDIA is expected to make significant achievements in the AI server market for CSPs from 2023 to 2024, there will likely be a boost in overall shipment volume and revenue growth of AI servers. However, with NVIDIA’s strategic introduction of standardized AI server architectures like HGX or MGX, the core product architecture for AI servers among ODMs and others will become more homogenized. This will intensify the competition among them as they vie for orders from CSPs. Furthermore, it’s been observed that large CSPs such as Google and AWS are leaning toward adopting in-house ASIC AI accelerator chips in the future, meaning there’s a potential threat to a portion of NVIDIA’s GPU market. This is likely one of the reasons NVIDIA continues to roll out GPUs with varied positioning and comprehensive solutions. They aim to further expand their AI business aggressively to Tier-2 data centers (like CoreWeave) and edge enterprise clients.