On November 13, NVIDIA unveiled the AI computing platform HGX H200, featuring the Hopper architecture, equipped with H200 Tensor Core GPU and high-end memory to handle the vast amounts of data generated by AI and high-performance computing.

This marks an upgrade from the previous generation H100, with a 1.4x increase in memory bandwidth and a 1.8x increase in capacity, enhancing its capabilities for processing intensive generative AI tasks.

The internal memory changes in H200 represent a significant upgrade, as it adopts the HBM3e for the first time. This results in a notable increase in GPU memory bandwidth, soaring from 3.35TB per second in H100 to 4.8TB per second.

The total memory capacity also sees a substantial boost, rising from 80GB in H100 to 141GB. When compared to H100, these enhancements nearly double the inference speed for the Llama 2 model.

H200 is designed to be compatible with systems that already support H100, according to NVIDIA. The company states that cloud service providers can seamlessly integrate H200 into their product portfolios without the need for any modifications.

This implies that NVIDIA’s server manufacturing partners, including ASRock, ASUS, Dell, Eviden, GIGABYTE, HPE, Ingrasys, Lenovo, Quanta Cloud, Supermicro, Wistron, and Wiwynn, have the flexibility to replace existing processors with H200.

The initial shipments of H200 are expected in the second quarter of 2024, with cloud service giants such as Amazon, Google, Microsoft, and Oracle anticipated to be among the first to adopt H200.

What is HBM?

“The integration of faster and more extensive HBM memory serves to accelerate performance across computationally demanding tasks including generative AI models and [high-performance computing] applications while optimizing GPU utilization and efficiency,” said Ian Buck, the Vice President of High-Performance Computing Products at NVIDIA.

What is HBM? HBM refers to stacking DRAM layers like building blocks and encapsulating them through advanced packaging. This approach increases density while maintaining or even reducing the overall volume, leading to improved storage efficiency.

TrendForce reported that the HBM market’s dominant product for 2023 is HBM2e, employed by the NVIDIA A100/A800, AMD MI200, and most CSPs’ (Cloud Service Providers) self-developed accelerator chips.

As the demand for AI accelerator chips evolves, in 2023, the mainstream demand is projected to shift from HBM2e to HBM3, with estimated proportions of approximately 50% and 39%, respectively.

As the production of acceleration chips utilizing HBM3 increases gradually, the market demand in 2024 is expected to significantly transition to HBM3, surpassing HBM2e directly. The estimated proportion for 2024 is around 60%.

Since Manufacturers plan to introduce new HBM3e products in 2024, HBM3 and HBM3e are expected to become mainstream in the market next year.

TrendForce clarifies that the so-called HBM3 in the current market should be subdivided into two categories based on speed. One category includes HBM3 running at speeds between 5.6 to 6.4 Gbps, while the other features the 8 Gbps HBM3e, which also goes by several names including HBM3P, HBM3A, HBM3+, and HBM3 Gen2.

HBM3e will be stacked with 24Gb mono dies, and under the 8-layer (8Hi) foundation, the capacity of a single HBM3e will jump to 24GB.

According to the TrendForce’s previous news release, the three major manufacturers currently leading the HBM competition – SK hynix, Samsung, and Micron – have the following progress updates.

SK hynix and Samsung began their efforts with HBM3, which is used in NVIDIA’s H100/H800 and AMD’s MI300 series products. These two manufacturers are expected to sample HBM3e in Q1 2024 previously. Meanwhile, Micron chose to skip HBM3 and directly develop HBM3e.

However, according to the latest TrendForce survey, as of the end of July this year, Micron has already provided NVIDIA with HBM3e verification, while SK hynix did so in mid-August, and Samsung in early October.

(Image: Nvidia)

As third quarters have typically been peak seasons for the production of various end-products, the sufficiency ratio of DRAM is expected to undergo a further decrease in 3Q21, according to TrendForce’s latest investigations. However, DRAM buyers are now carrying a relatively high DRAM inventory due to their amplified purchases of electronic components in 1H21. The QoQ increase in DRAM contract prices are hence expected to slightly narrow from 18-23% in 2Q21 to 3-8% in 3Q21. Looking ahead to 4Q21, TrendForce believes that DRAM supply will continue to rise, thereby leading to either a further narrowing of price hikes or pressure constraining the potential price hike of DRAM products.

PC DRAM prices are expected to rise by 3-8% QoQ due to continued constraints on production capacities

From the perspective of demand, the stay-at-home economy has resulted in persistently high demand for notebook computers. Although discrepancies still exist among notebook brands’ inventory levels of various components, these brands are still making an aggressive attempt at maximizing their production of notebooks. However, as most of these brands are still carrying about 8-10 weeks’ worth of PC DRAM inventory (which is relatively high), PC DRAM purchasing strategies from the buyers’ side will therefore remain relatively conservative. From the perspective of supply, due to the rising demand for server DRAM, the production capacity allocated to PC DRAM is still in a severe supply crunch. Hence, DRAM suppliers are firm in their attitudes to raise PC DRAM quotes, and TrendForce expects the price negotiations between PC DRAM buyers and suppliers in 3Q21 to become both lengthier and more difficult as a result, with contract prices likely finalized at the end of July. Even so, what is now certain is that both sides have reached some level of understanding regarding the ongoing price hike of PC DRAM products. TrendForce forecasts a 3-8% increase in PC DRAM contract prices for 3Q21.

QoQ increase in server DRAM prices for 3Q21 are expected to narrow to 5-10% due to buyers carrying a relatively high inventory

With regards to demand, in spite of the minor increase in the shipment of whole servers, server DRAM buyers are less aggressive in their server DRAM procurement compared to the previous quarter. For instance, CSPs in North America and in China are currently carrying more than eight weeks of server DRAM inventory. In other words, procurement activities for server DRAM will gradually decline in the coming quarters in accordance with market demand. Notably, some Tier 2 clients will continue to procure server DRAM in 3Q21 since they did not sufficiently stock up in the prior quarters, and this demand will likely result in upward momentum for server DRAM prices. With regards to supply, the three major DRAM suppliers (Samsung, SK Hynix, and Micron) are limited by the fact they are currently carrying a relatively low inventory of server DRAM. As such, these suppliers will attempt to maintain their profitability by increasing prices each quarter. It should also be pointed out that the decreased DRAM demand from smartphone brands has in fact allowed more wiggle room for server manufacturers to negotiate for more favorable server DRAM prices. TrendForce thus believes that, before the supply side and demand side can reach an agreement, negotiations for server DRAM prices will become increasingly lengthy, and that server DRAM contract prices for 3Q21 will likely increase by 5-10% QoQ once negotiations are finalized.

Mobile DRAM prices are expected to defy market realities and increase by 5-15% QoQ, with potential risks of high price and low demand

In terms of demand, certain smartphone brands are now carrying a relatively higher inventory of mobile DRAM owing to Southeast Asia’s worsening COVID-19 pandemic, which led smartphone brands that primarily manufacture and sell their products there to begin lowering their production targets in 2Q21. In addition, some smartphone brands have set overly ambitious production targets; combined with the current shortage of foundry capacities, the discrepancies among the supply of smartphone components have now become more apparent, in turn forcing brands to slow down their mobile DRAM procurement in order to adjust their component inventories first. Demand has remained strong from clients in the smartphone market since 4Q21, so the supply fulfillment rate of the three major DRAM suppliers for their smartphone clients will be consistently higher compared to clients in other markets. As DRAM demand from non-smartphone applications ramps up and results in higher profitability than mobile DRAM, the three major DRAM suppliers will continue to adjust their production capacities in accordance with the shifting supply and demand from various segments, thus resulting in an increasingly constrained supply of mobile DRAM.

It should be pointed out that DRAM market leader Samsung has generally tried to minimize the profit discrepancies among its various products. Furthermore, the price hike in Samsung’s mobile DRAM products was relatively lower compared to Micron in 1H21. As a result, in view of the weakening mobile DRAM demand in 3Q21, Samsung will increase its mobile DRAM prices to a more notable extent compared to its US competitors. Going forward, Samsung’s price hike will lead its competitors to retool their pricing strategies, subsequently leading to an even wider price increase across the entire mobile DRAM market. As such, TrendForce expects mobile DRAM prices to increase by 5-15% QoQ in 3Q21, which is a step up compared to 2Q21. On the other hand, this price hike against market realities may potentially lead to a further decline in mobile DRAM demand, resulting in a situation with high price and low demand.

Graphics DRAM prices are expected to increase by 8-13% QoQ due to tight supply of GDDR6

Regarding graphics DRAM demand, many cryptocurrency miners were previously intent on mining ETH with older graphics cards as it reached peak prices. Nevertheless, the recent bearish turn of the cryptocurrency market has indirectly had an impact on demand for graphics cards equipped with GDDR5, although most of this impact primarily affected the spot market. For the contract market, more than 90% of graphics DRAM applications have migrated to GDDR6 products, which are now in short supply since new graphics cards are equipped with GDDR6 memory and are in high demand. In addition, the vast majority of GDDR6 stock from DRAM suppliers is currently cornered by graphics card manufacturers and game console manufacturers, thereby further limiting the graphics DRAM supply available to small and medium OEMs/ODMs. Regarding graphics DRAM supply, although GDDR6 accounts for more than 90% of the three major DRAM suppliers’ graphics DRAM production, demand for GDDR6 still far exceeds supply because end product demand has also migrated to GDDR6. As orders for server DRAM gradually ramp up in 3Q21, DRAM suppliers will prioritize fulfilling demand from the server market first. Hence, graphics DRAM contract prices for 3Q21 are expected to increase by 8-13% QoQ.

Consumer DRAM prices are expected to increase by up to 13% QoQ in light of strong demand

At the moment, consumer DRAM demand is relatively robust from the consumer electronics market and the telecom market. In addition, as China has been accelerating its build-out of 5G infrastructures and its rollout of WiFi 6 in the post-pandemic era, the overall demand for consumer DRAM remains strong going forward. On the other hand, the three dominant DRAM suppliers are slowing down their transition of production capacities from DDR3 products to CMOS Image Sensors or other Logic IC products now that the consumer DRAM market has taken a bullish turn. However, in the medium-to-long term, the general trend in the DRAM industry will still point to the elimination of the older 25/20nm process technologies and the continued migration towards more advanced 1Znm and 1αnm processes. As a result, given DDR3 products’ declining supply and strong demand, DDR3 prices for 3Q21 are expected to increase by 8-13% QoQ, while DDR4 prices are expected to undergo a minor growth of 3-8% QoQ in accordance with mainstream PC and server DRAM prices.

For more information on reports and market data from TrendForce’s Department of Semiconductor Research, please click here, or email Ms. Latte Chung from the Sales Department at lattechung@trendforce.com

Enterprise SSD procurement has been rising on the back of growing server shipments since 2Q21, according to TrendForce’s latest investigations. In particular, the share of 8TB products in shipments of SSDs to data centers has shown the most noticeable growth, which is expected to persist through 3Q21. However, certain SSD components and parts may be in shortage due to insufficient foundry capacity. TrendForce is therefore revising the QoQ hikes in contract prices of enterprise SSDs for 3Q21 to 10-15% from the previous projection of 5-10%.

TrendForce further indicates that the high demand for enterprise SSDs in 3Q21 is attributed to several factors. First, North American cloud service providers (hyperscalers) have pretty much completed their inventory adjustments and now continue to expand their storage capacity. Second, the flow of incoming orders to traditional server brands is getting stronger over the quarters as government agencies and SMBs increase their budgets for IT infrastructure. Third, Intel and AMD are ramping up production for server CPUs based on their respective new processor platforms. Following the adoption of new CPUs, the overall demand for enterprise SSDs has also shifted to higher-density products because clients want to upgrade their computing power and storage capacity. Specifically, demand is mainly trending toward 4/8TB SSDs since raising NAND Flash density can lower the cost of SSD deployment.

Supply leader Samsung will likely gain control over enterprise SSD pricing in the market

Regarding the supply end, Samsung has a higher flexibility in supplying SSDs compared to the other suppliers because it has a higher share of in-house components for its storage products. Therefore, in view of the possible shortage in certain SSD components, Samsung will likely be able to further expand its market share for enterprise SSDs. Furthermore, Samsung’s products are expected to account for more than 50% of enterprise SSDs (in terms of bits) shipped to data centers in North America in 3Q21. This dominance will likely further Samsung’s ability to dictate market prices going forward.

Intel, on the other hand, has been constrained in its ability to manufacture enterprise SSDs due to a shortage of PMICs. In addition, Intel has mostly been fulfilling orders for QLC products. As a result, Intel’s market share may potentially decrease in the TLC-dominant enterprise SSD sector. Regarding other suppliers including Kioxia and SK Hynix, although they have been able to raise their market shares due to gradual adoption of their products by clients, they are unlikely to catch up to Samsung for the time being.

On the PC client SSD front, at the moment, demand for notebook computers has remained strong, while the supply of SSD controller IC is still relatively tight. TrendForce therefore forecasts a slight 3-8% QoQ increase in client SSD contract prices for 3Q21. Regardless, suppliers will not slow down their process migrations. Starting from 3Q21, 176L PC client SSDs will be available on the market, with a corresponding increase in supply bits in the upstream SSD supply chain.

For more information on reports and market data from TrendForce’s Department of Semiconductor Research, please click here, or email Ms. Latte Chung from the Sales Department at lattechung@trendforce.com

With Samsung, YMTC, SK Hynix, and Intel leading the charge, NAND Flash suppliers will maintain an aggressive effort to expand their production capacities throughout 2Q21, during which NAND Flash bit output will likely increase by nearly 10% QoQ, according to TrendForce’s latest investigations. On the other hand, orders from PC OEMs and Chinese smartphone brands since 1Q21, as well as recovering procurement activities from clients in the data center segment during 2Q21, will generate upward momentum propelling NAND Flash bit demand. Furthermore, buyers are actively stocking up on finished products, such as SSDs and eMMC, due to persistently limited NAND Flash controller supply. TrendForce therefore expects NAND Flash contract prices to increase by an average of 3-8% QoQ in 2Q21 after experiencing a 5-10% decline QoQ in 1Q21. In particular, as Samsung’s Line S2 fab in Austin has yet to resume full operation after the Texas winter storm, the supply of NAND Flash controllers going forward may be at risk, and Samsung’s ability to manufacture client SSDs will be further constrained as a result. In light of these factors, TrendForce is not ruling out the possibility that NAND Flash contract prices may increase by even more than current forecasts.

Contract prices of both client SSDs and enterprise SSDs are projected to rise due to delayed resumption at Samsung’s Line S2 fab

With regards to client SSDs, the persistent stay-at-home economy generated by the COVID-19 pandemic will likely result in strong demand for notebook computers in 2H21, while PC OEMs have raised their client SSD inventories as they manufacture more notebooks to meet demand. Given the high volumes of client SSD orders from PC OEMs, inventory level of NAND Flash suppliers is therefore likely to remain healthy. However, the shortage of NAND Flash controllers has yet to be resolved. Suspended operations at the Line S2 fab disrupted Samsung’s production of NAND Flash controllers, meaning some client SSD orders will not be fulfilled in 2Q21. Hence, the tight supply of finished products (i.e., client SSDs) will be further exacerbated. As such, client SSD contract prices are projected to increase by 3-8% in 2Q21.

With regards to enterprise SSDs, demand is expected to rebound from rock bottom in 2Q21, primarily because clients in the data center segment will ramp up their procurement activities after undergoing a period of inventory adjustment. In addition, demand for IT equipment from the governmental, healthcare, and financial services sectors will also gradually emerge. Other factors contributing to enterprise SSD demand include bids from Chinese telecom operators and increased IT equipment purchases from small and medium businesses globally. On the other hand, NAND Flash suppliers are no longer under pressure to destock via low prices, since their inventory levels have improved thanks to high demand from notebook manufacturers and smartphone brands. As the overall demand for NAND Flash rises, enterprise SSD contract prices are in turn expected to stabilize and experience a 0-5% growth QoQ in 2Q21.

High demand for Chromebooks will provide upward momentum for eMMC quotes, while contract prices of UFS are projected to undergo the lowest growths among NAND Flash products

eMMC contract prices will likely remain, for the most part, higher than expected despite the cyclical downturn in 1H21. In particular, strong demand from Chromebook manufacturers will provide upward momentum for eMMC quotes. Likewise, under the influence of NAND Flash controller shortage, eMMC buyers such as consumer electronics manufacturers will expand their procurement activities in order to build up their inventories. As a result, the overall eMMC demand will gradually ramp up in 2Q21. Conversely, the supply of eMMC controllers is still in shortage due to the fully loaded capacities across the foundry industry. Also, eMMC products under 32GB exclusively feature 2D NAND or 64L 3D NAND. Because production capacities allocated for these types of NAND Flash memories have been either reassigned to other 3D NAND products or scaled down, the oversupply of eMMC has been alleviated, and the long-term price drop of eMMC has subsequently come to a halt. In the short term, the shortage of controller ICs will result in a shortage of finished eMMC products. eMMC contract prices are therefore projected to increase by 3-8% QoQ in 2Q21.

Demand for UFS, which is primarily used for smartphones, is expected to remain high through 2Q21 because OPPO, Vivo, and Xiaomi have been aggressively procuring UFS since 4Q20, and Huawei spun off its Honor smartphone business unit. Buyers have also been anticipating an upcoming shortage of controller ICs and NAND Flash memory, leading them to build up their UFS inventories and therefore further driving up the overall UFS demand. On the supply side, inventory levels of suppliers have dropped significantly due to smartphone brands’ large-scale procurement activities previously. Although Chinese smartphone brands have yet to ramp up their bit demand, their existing level of demand still remains strong. Furthermore, clients from the data center segment are expected to increase their SSD procurement in 2Q21, and suppliers will maintain an aggressive approach regarding quotes in response. Even so, because smartphones account for the highest bit consumption share among all NAND Flash applications, NAND Flash suppliers are unlikely to significantly adjust their UFS quotes. As such, UFS contract prices are expected to increase by 0-5% QoQ in 2Q21, which is relatively lower compared to other NAND Flash products.

NAND Flash wafer contract prices are projected to increase by 5-10% QoQ as NAND Flash suppliers lower their bit shipment to the wafer market due to its lower profit margins

With regards to the NAND Flash wafer market, TrendForce has yet to observe an obvious improvement in the retail sales of end products such as SSDs, memory cards, and USB flash drives. However, as NAND Flash suppliers have been unable to make their scheduled delivery dates to OEMs due to an insufficient supply of controller ICs, module makers may stand to benefit from this and obtain more orders from OEMs, subsequently driving up the demand for NAND Flash wafers within the next one to two quarters, but the actual procurement of NAND Flash wafers will depend on whether the tight supply situation of controller ICs can be alleviated. On the other hand, inventories of NAND Flash suppliers have now fallen to mostly healthy levels thanks to procurement activities from smartphone brands since 4Q20. Suppliers have accordingly lowered their bit shipments to the NAND Flash wafer market (which yields a relatively lower profit margin compared to other product categories), due to the rising demand from notebook manufacturers and the expected recovery of the data center segment in 2Q21. On the whole, given the bullish market of mainstream products, such as smartphones and notebooks, TrendForce expects NAND Flash wafer contract prices to once again increase by 5-10% QoQ in 2Q21.

For more information on reports and market data from TrendForce’s Department of Semiconductor Research, please click here, or email Ms. Latte Chung from the Sales Department at lattechung@trendforce.com

As the three dominant DRAM suppliers (Samsung, SK Hynix, and Micron) are currently experiencing a shortage in their production capacities, the corresponding shortage situation in the DRAM market has yet to be resolved, according to TrendForce’s latest investigations. Taking advantage of the fact that the whole DRAM market has entered a period of cyclical upturn in 1Q21, DRAM suppliers have significantly raised quotes for specialty DRAM products. This extraordinary development has led to price hikes that are almost double digits for some specialty DRAM chips. Furthermore, the magnitude of the price hike especially widens for products belonging to the lower part of the density range and the more niche applications. Looking at MoM changes in contract prices of specialty DRAM products for February, DDR2 and DDR3 chips saw the largest price hikes. Prices of DDR4 chips also went up due to the influence of the rising quotes for DDR3 chips. The average contract price of DDR3 4Gb chips, which are still mainstream for specialty applications, jumped 6.8% MoM.

As for DDR3 2Gb chips that are primarily promoted by Taiwan-based suppliers, TrendForce indicates that there is not enough supply even as quotes are being offered. With this situation becoming the norm, the price range (i.e., the difference between the high and low prices) has also expanded dramatically. The average contract price of DDR3 2Gb chips rose by nearly 9% MoM in February. Samsung significantly raised quotes for DDR4 4Gb chips in response to the sharp upswing in prices of DDR3 chips. The low and average prices of DDR4 4Gb chips for specialty applications both climbed around 6% MoM. The average price of DDR4 8Gb chips went up by around 4% MoM as the price trend of this product aligns with the general price trends of mainstream PC DRAM and server DRAM products. However, it should be pointed out that the hikes in contract prices of specialty DRAM chips were mainly the result of the adjustments made to monthly contract deals and thus reflected the market situation of February. By contrast, prices held steady for quarterly lock-in deals with tier-1 clients.

While the three dominant DRAM suppliers have been slowing down their DDR3 manufacturing, Taiwanese suppliers are constantly adjusting their capacity allocation to maximize profits

With demand getting hotter in the specialty DRAM market, DDR3 products are starting to surpass DDR4 products and logic ICs in profit margin. Consequently, DRAM suppliers are also changing their strategies. Looking at South Korean suppliers, Samsung will continue shifting the wafer production capacity of Line 13 to CMOS image sensors over the long term. However, this reassignment has now been scaled down for this year due to the recent surge in specialty DRAM prices. Likewise, SK Hynix will keep the DRAM production capacity of its older fab M10 relatively constant through 2021 after reducing it in 2020. As for Micron, it has raised the yield rates of the 1Z-nm and 1-alpha processes, so the output shares of products based on these more advanced technologies will gradually expand. Since the available fab space in Taiwan is limited, Micron will relocate the 20nm and more mature processes to Fab 6 in the US. In sum, the output of DDR3 products from the three dominant suppliers will continue to shrink, but the pace of the reduction is now slower than originally expected.

Regarding Taiwan-based suppliers, Nanya has shifted some 20nm and 30nm production capacity from DDR4 products back to DDR3 products. Winbond has been focusing on Flash products in the recent years, and its DRAM production capacity will remain fairly limited until the completion of its new fab in Kaohsiung. Nevertheless, Winbond is concentrating its DRAM production efforts on low-density DDR2 and DDR3 products (i.e., 1Gb and 2Gb chips). It actually has the advantage of being able to raise prices as its market share for low-density products is fairly large. As for PSMC, it has been focusing on foundry manufacturing of logic ICs for a while. However, with prices now rising for foundry manufacturing of DDR3 products, PSMC now wants to shift some wafer production capacity back to DRAM. Going forward, the three Taiwan-based suppliers will keep adjusting their capacity allocation strategies in accordance with changes in the profit margins of different products. Nevertheless, even as suppliers are now changing their product mixes to take advantage of the latest market situation, TrendForce projects that specialty DRAM products will be in undersupply at least through 1H21. The magnitudes of price hikes for various types of specialty DRAM products will depend on suppliers’ capacity allocation strategies.

For more information on reports and market data from TrendForce’s Department of Semiconductor Research, please click here, or email Ms. Latte Chung from the Sales Department at lattechung@trendforce.com