On November 13, NVIDIA unveiled the AI computing platform HGX H200, featuring the Hopper architecture, equipped with H200 Tensor Core GPU and high-end memory to handle the vast amounts of data generated by AI and high-performance computing.

This marks an upgrade from the previous generation H100, with a 1.4x increase in memory bandwidth and a 1.8x increase in capacity, enhancing its capabilities for processing intensive generative AI tasks.

The internal memory changes in H200 represent a significant upgrade, as it adopts the HBM3e for the first time. This results in a notable increase in GPU memory bandwidth, soaring from 3.35TB per second in H100 to 4.8TB per second.

The total memory capacity also sees a substantial boost, rising from 80GB in H100 to 141GB. When compared to H100, these enhancements nearly double the inference speed for the Llama 2 model.

H200 is designed to be compatible with systems that already support H100, according to NVIDIA. The company states that cloud service providers can seamlessly integrate H200 into their product portfolios without the need for any modifications.

This implies that NVIDIA’s server manufacturing partners, including ASRock, ASUS, Dell, Eviden, GIGABYTE, HPE, Ingrasys, Lenovo, Quanta Cloud, Supermicro, Wistron, and Wiwynn, have the flexibility to replace existing processors with H200.

The initial shipments of H200 are expected in the second quarter of 2024, with cloud service giants such as Amazon, Google, Microsoft, and Oracle anticipated to be among the first to adopt H200.

What is HBM?

“The integration of faster and more extensive HBM memory serves to accelerate performance across computationally demanding tasks including generative AI models and [high-performance computing] applications while optimizing GPU utilization and efficiency,” said Ian Buck, the Vice President of High-Performance Computing Products at NVIDIA.

What is HBM? HBM refers to stacking DRAM layers like building blocks and encapsulating them through advanced packaging. This approach increases density while maintaining or even reducing the overall volume, leading to improved storage efficiency.

TrendForce reported that the HBM market’s dominant product for 2023 is HBM2e, employed by the NVIDIA A100/A800, AMD MI200, and most CSPs’ (Cloud Service Providers) self-developed accelerator chips.

As the demand for AI accelerator chips evolves, in 2023, the mainstream demand is projected to shift from HBM2e to HBM3, with estimated proportions of approximately 50% and 39%, respectively.

As the production of acceleration chips utilizing HBM3 increases gradually, the market demand in 2024 is expected to significantly transition to HBM3, surpassing HBM2e directly. The estimated proportion for 2024 is around 60%.

Since Manufacturers plan to introduce new HBM3e products in 2024, HBM3 and HBM3e are expected to become mainstream in the market next year.

TrendForce clarifies that the so-called HBM3 in the current market should be subdivided into two categories based on speed. One category includes HBM3 running at speeds between 5.6 to 6.4 Gbps, while the other features the 8 Gbps HBM3e, which also goes by several names including HBM3P, HBM3A, HBM3+, and HBM3 Gen2.

HBM3e will be stacked with 24Gb mono dies, and under the 8-layer (8Hi) foundation, the capacity of a single HBM3e will jump to 24GB.

According to the TrendForce’s previous news release, the three major manufacturers currently leading the HBM competition – SK hynix, Samsung, and Micron – have the following progress updates.

SK hynix and Samsung began their efforts with HBM3, which is used in NVIDIA’s H100/H800 and AMD’s MI300 series products. These two manufacturers are expected to sample HBM3e in Q1 2024 previously. Meanwhile, Micron chose to skip HBM3 and directly develop HBM3e.

However, according to the latest TrendForce survey, as of the end of July this year, Micron has already provided NVIDIA with HBM3e verification, while SK hynix did so in mid-August, and Samsung in early October.

(Image: Nvidia)

Mainland China’s 3D NAND flash memory manufacturer, Yangtze Memory Technologies Co. (YMTC), filed a lawsuit against the U.S. memory chip leader, Micron Technology, on November 9th in the Northern District Court of California. The lawsuit accuses Micron of infringing upon eight of YMTC’s U.S. patents related to 3D NAND technology.

According to ICsmart, the patents involved in this case from YMTC include US10,950,623 (3D NAND memory device and method of forming the same), US11,501,822 (Non-volatile storage device and control method), US10,658,378 (Through-array contact [TAC] for three-dimensional memory devices), and US10,937,806 (Through-array contact [TAC] for three-dimensional memory devices), US10,861,872 (Three-dimensional memory device and method for forming the same), US11,468,957 (Architecture and method for NAND memory operation), US11,600,342 (Method for reading three-dimensional flash memory), and US10,868,031 (Multiple-stack three-dimensional memory device and fabrication method  thereof).

In the complaint, YMTC alleges that Micron’s 128-layer, 176-layer, and other series of 3D NAND technology have violated eight patents owned by YMTC. Micron is accused of using YMTC’s patented technology without authorization to compete with YMTC, protecting market share and impeding YMTC’s interests, thereby inhibiting innovation.

In recent years, with the stacking of 3D NAND technology reaching 128 layers and even higher, the chip area occupied by peripheral CMOS circuits may exceed 50%. To address this issue, YMTC introduced its proprietary innovative Xtacking technology in 2018.

Established in July 2016 and headquartered in Wuhan, Hubei, YMTC is an IDM (Integrated Device Manufacturer) specializing in the design and manufacturing of 3D NAND flash memory. It also provides comprehensive memory solutions.

Under the shadow of the ongoing US-China tech rivalry, Micron Technology adopted a low-key approach at this year’s Import Expo in Shanghai. During a meeting with Micron’s CEO, Sanjay Mehrotra, Chinese Minister of Commerce Wang Wentao on November 1st welcomed Micron’s continued presence and expansion in the Chinese market, emphasizing the importance of adhering to Chinese laws and regulations for sustainable development. Mr. Mehrotra expressed the company’s willingness to further invest in China.

However, on May 21st this year, China’s Cyberspace Administration announced serious cybersecurity issues with Micron’s products sold in China. These products didn’t pass the review, leading Chinese operators to halt the purchase of Micron’s products. This indicates a potential ban on Micron’s products in the Chinese market.

In October 2022, the US imposed exprt restrictions on advanced chip manufacturing equipment, including placing 36 Chinese companies such as YMTC on an entity list.

(Photo credit: iStock)

In a subdued environment for consumer electronic applications in the storage market, High Bandwidth Memory (HBM) technology is emerging as a new driving force, gaining significant attention from major players. Recent reports reveal that both Samsung and Micron are gearing up for substantial HBM production expansion.

Major Manufacturers Actively Investing in HBM

Recent reports indicate that Samsung has acquired certain buildings and equipment within the Cheonan facility of Samsung Display in South Korea to expand its HBM production capacity.

It is reported that Samsung plans to establish a new packaging line at the Cheonan facility for large-scale HBM production. The company has already spent 10.5 billion Korean won on the acquisition of the mentioned buildings and equipment, with an additional investment expected to range between 700 billion and 1 trillion Korean won.

Earlier, it was disclosed by Mr. Hwang Sang-jun, the Vice President of Samsung Electronics and Head of the DRAM Product and Technology Team, that Samsung has developed HBM3E with a speed of 9.8Gbps and plans to commence providing samples to customers.

Concurrently, Samsung is in the process of developing HBM4 with the objective of making it available by 2025. It is reported that Samsung Electronics is actively working on various technologies for HBM4, including non-conductive adhesive film (NCF) assembly techniques optimized for high-temperature thermal characteristics and hybrid bonding (HCB).

On November 6th, Micron Technology opened a new facility in Taichung. Micron has stated that this new facility will integrate advanced testing and packaging functions and will be dedicated to the mass production of HBM3E, along with other products. This expansion aims to meet the increasing demand across various applications such as artificial intelligence, data centers, edge computing, and cloud services.

Previously, Micron’s CEO, Sanjay Mehrotra, revealed that the company plans to commence substantial shipments of HBM3E in early 2024. Micron’s HBM3E technology is currently undergoing certification by NVIDIA. The initial HBM3E offerings will feature an 8-Hi stack design with a capacity of 24GB and a bandwidth exceeding 1.2TB/s.

Furthermore, Micron intends to introduce larger-capacity 36GB 12-Hi stacks HBM3E in 2024. In an earlier statement, Micron had anticipated that the new HBM technology would contribute “hundreds of millions” of dollars in revenue by 2024.

Shift Toward HBM3 Expected in 2024

According to TrendForce, the current mainstream technology in the HBM market is HBM2e. This specification is utilized by prominent players like NVIDIA with their A100 and A800, AMD with the MI200 series, and various custom system-on-chip designs by CSPs.

Simultaneously, in response to the evolving demand for AI accelerator chips, many manufacturers are planning to introduce new products based on HBM3e technology in 2024. It is anticipated that both HBM3 and HBM3e will become the dominant technologies in the market next year, catering to the requirements of AI accelerator chips.

Micron has announced a significant expansion in Penang, Malaysia on October 13th. Micron had previously invested $1 billion in Penang and plans to invest another $1 billion over the coming years for the construction and equipping of a state-of-the-art assembly and testing facility. This expansion is located in Batu Kawan Industrial Park (BKIP), will increase the total factory area to 1.5 million square feet.

The expansion positions Micron Malaysia to boost production and further enhance its assembly and testing capabilities, allowing it to meet the growing demand for transformative technologies like NAND, PC DRAM, and SSD modules driven by artificial intelligence, autonomous vehicles, and electric cars.

“This expansion also reflects our unwavering dedication to advancing semiconductor development and manufacturing excellence,” Micron Malaysia vice-president and country manager Amarjit Singh Sandhu addressed, “The official opening of our new manufacturing facility in Batu Kawan also strengthens Micron’s global manufacturing footprint, enabling us to deliver quality products to our customers on time, with reduced cycle time and at scale.”

Malaysia’s Semiconductor Potential

Malaysia stands out for its strong education standards and shares the British legal system with just Singapore in ASEAN, making it a competitive choice for companies. The language proficiency of Malaysians in English, Mandarin, and Malay enables smooth global communication. Besides, Malaysia’s two top-tier ports, Port Klang and Port of Tanjung Pelepas, enhance its global accessibility.

Penang is a semiconductor hub, often likened to the “Silicon Valley of the East,” with a focus on electronics, computers, and mobile phone chips. The growing demand for automotive chips and green energy technologies has attracted numerous companies, leading to facility expansions. Major players like Intel, Texas Instruments, Infineon, Bosch Group, and ASE Technology Holding have invested billions in Malaysia’s semiconductor industry, marking it as a growing center for backend testing and packaging.

Current State of Malaysia’s Semiconductor Industry

Apart from Micron, which opened their second plant in Penang last weekend, Malaysia has already attracted significant attention to establish wafer fabrication plants in the United States and other regions.

In the past two years, Intel invested $6.46 billion in advanced packaging capabilities in Penang and Kedah. Texas Instruments embarked on building semiconductor testing and packaging plants in Kuala Lumpur and Malacca, with a $2.7 billion investment. Infineon allocated $5.45 billion for expanding facilities and entering the electric vehicle sector, while Bosch Group is investing $358 million to fortify its semiconductor supply chain in Penang. ASE Technology Holding also initiated the construction of a new testing facility in Penang.

The surge of semiconductor giants underscores Malaysia’s pivotal role in the industry. This transformation aligns with the unique production strengths of Southeast Asian nations, reshaping supply chains and redefining production centers—a focus and challenge for global companies.

Explore more

• Malaysia: Rising Global Hub for Semiconductor Backend Testing and Packaging in Supply Chain Shift

Amidst the AI boom, HBM technology steps into the spotlight as market demand continues to surge. Global market research firm TrendForce anticipates a 58% year-on-year increase in HBM demand in 2023, with a potential additional growth of approximately 30% in 2024.

Compared to traditional DRAM, HBM (High Bandwidth Memory) boasts advantages such as high bandwidth, high capacity, low latency, and low power consumption. These attributes accelerate AI data processing and make it particularly well-suited for high-performance computing scenarios like ChatGPT. As a result, it has gained popularity, and major storage manufacturers are actively driving HBM technology upgrades.

Leading memory manufacturers are intensifying their efforts, with Samsung set to introduce HBM4.

Since the inception of the first HBM products utilizing TSV packaging technology in 2014, HBM technology has seen multiple upgrades, including HBM, HBM2, HBM2E, HBM3, and HBM3e.

Regarding the SK Hynix and Samsung, two major South Korean companies, have been at the forefront of HBM3 development. NVIDIA’s H100/H800 and AMD’s MI300 series, represent HBM3’s progress. Both SK Hynix and Samsung expected to offer HBM3e samples by the first quarter of 2024. On the other hand, Micron, a U.S.-based memory company, is bypassing HBM3 and directly pursuing HBM3e.

HBM3e will feature 24Gb mono die stacks, and with an 8-layer (8Hi) configuration, a single HBM3e chip’s capacity will soar to 24GB. This advancement is expected to be incorporated into NVIDIA’s GB100 in 2025, leading the three major OEMs to plan HBM3e sample releases in the first quarter of 2024 and enter mass production in the latter half of the year.

In addition to HBM3 and HBM3e, the latest updates indicate that storage giants are planning the launch of the next generation of HBM—HBM4.

Samsung recently announced that it has developed 9.8Gbps HBM3E and is planning to provide samples to customers. Furthermore, Samsung is actively working on HBM4 with a goal to begin supply in 2025. It’s reported that Samsung Electronics is developing technologies such as non-conductive adhesive film (NCF) assembly for optimizing high-temperature thermal characteristics, as well as hybrid bonding (HCB), for HBM4 products.

In September, Korean media reported that Samsung is gearing up to revamp its production process and launch HBM4 products to capture the rapidly growing HBM market. HBM4 memory stacks will feature a 2048-bit memory interface, a significant departure from the previous 1024-bit interface for all HBM stacks. This enhanced interface width holds great significance for the evolution of HBM4.

While HBM4 promises a major breakthrough, it is still a ways off, making it too early to discuss its practical applications and widespread adoption. Industry experts emphasize that the current HBM market is dominated by HBM2e. However, HBM3 and HBM3e are poised to take the lead in the near future.

According to TrendForce’s research, HBM2e currently accounts for the mainstream market share, being used in various products like NVIDIA A100/A800, AMD MI200, and many AI accelerators developed by CSPs. To keep pace with the evolving demands of AI accelerator chips, OEMs are planning to introduce new HBM3e products in 2024, with HBM3 and HBM3e expected to become the market’s primary players next year.

In terms of the demand transition between different HBM generations, TrendForce estimates that in 2023, mainstream demand will shift from HBM2e to HBM3, with estimated demand shares of approximately 50% and 39%, respectively. As more HBM3-based accelerator chips enter the market, demand will substantially shift toward HBM3 in 2024, surpassing HBM2e and accounting for an estimated 60% of the market. This transition, coupled with higher average selling prices (ASP), is poised to significantly drive HBM revenue growth next year.

(Photo credit: Samsung)