According to CNA’s news, the potential business opportunities in artificial intelligence have spurred major tech giants, with NVIDIA, AMD, Intel, MediaTek, and Qualcomm sequentially launching products featuring the latest AI capabilities.

This AI arms race has expanded its battleground from servers to smartphones and laptops, as companies hope that the infusion of AI will inject vitality into mature markets.

Generative AI is experiencing robust development, with MediaTek considering this year as the “Generative AI Year.” They anticipate a potential paradigm shift in the IC design industry, contributing to increased productivity and significantly impacting IC products.

This not only brings forth new applications but also propels the demand for new algorithms and computational processors.

MediaTek and Qualcomm recently introduced their flagship 5G generative AI mobile chips, the Dimensity 9300 and Snapdragon 8 Gen 3, respectively. The Dimensity 9300, integrated with the built-in APU 790, enables faster and more secure edge AI computing, capable of generating images within 1 second.

MediaTek points out that the smartphone industry is experiencing a gradual growth slowdown, and generative AI is expected to provide new services, potentially stimulating a new wave of consumer demand growth. Smartphones equipped with the Dimensity 9300 and Snapdragon 8 Gen 3 are set to be released gradually by the end of this year.

Targeting the AI personal computer (PC) market, Intel is set to launch the Meteor Lake processor on December 14. Two major computer brands, Acer and ASUS, are both customers for Intel’s AI PC.

High-speed transmission interface chip manufacturer Parade and network communication chip manufacturer Realtek are optimistic. The integration of AI features into personal computers and laptops is expected to stimulate demand for upgrades, leading to a potential increase in PC shipments next year.

In TrendForces’ report on November 8th, it has indicated that the emerging market for AI PCs does not have a clear definition at present, but due to the high costs of upgrading both software and hardware associated with AI PCs, early development will be focused on high-end business users and content creators.

For consumers, current PCs offer a range of cloud AI applications sufficient for daily life and entertainment needs. However, without the emergence of a groundbreaking AI application in the short term to significantly enhance the AI experience, it will be challenging to rapidly boost the adoption of consumer AI PCs.

For the average consumer, with disposable income becoming increasingly tight, the prospect of purchasing an expensive, non-essential computer is likely wishful thinking on the part of suppliers. Nevertheless, looking to the long term, the potential development of more diverse AI tools—along with a price reduction—may still lead to a higher adoption rate of consumer AI PCs.

Read more

• Key Development Period for AI PCs in 2024; Global Notebook Market Set to Rebound to Healthy Supply-Demand Cycle with an Estimated Growth Rate of 3.2%, Says TrendForce

(Image: Qualcomm)

In TrendForce’s report on the self-driving System-on-Chip (SoC) market, it has witnessed rapid growth, which is anticipated to soar to $28 billion by 2026, boasting a Compound Annual Growth Rate (CAGR) from 2022 to 2026.

1. Rapid Growth in the Self-Driving SoC Market Becomes a Key Global Opportunity for IC Design Companies

In 2022, the global market for self-driving SoC is approximately $10.8 billion, and it is projected to grow to $12.7 billion in 2023, representing an 18% YoY increase. Fueled by the rising penetration of autonomous driving, the market is expected to reach $28 billion in 2026, with a CAGR of approximately 27% from 2022 to 2026.

Given the slowing growth momentum in the consumer electronics market, self-driving SoC has emerged as a crucial global opportunity for IC design companies.

2. Computing Power Reigns Supreme, with NVIDIA and Qualcomm Leading the Pack

Due to factors such as regulations, technology, costs, and network speed, most automakers currently operate at Level 2 autonomy. In practical terms, computing power exceeding 100 TOPS (INT8) is sufficient. However, as vehicles typically have a lifespan of over 15 years, future upgrades in autonomy levels will rely on Over-The-Air (OTA) updates, necessitating reserved computing power.

Based on the current choices made by automakers, computing power emerges as a primary consideration. Consequently, NVIDIA and Qualcomm are poised to hold a competitive edge. In contrast, Mobileye’s EyeQ Ultra, set to enter mass production in 2025, offers only 176 TOPS, making it susceptible to significant competitive pressure.

3. Software-Hardware Integration, Decoupling, and Openness as Key Competitors

Seamless integration of software and hardware can maximize the computational power of SoCs. Considering the imperative for automakers to reduce costs and enhance efficiency, the degree of integration becomes a pivotal factor in a company’s competitiveness. However, not only does integration matter, but the ability to decouple software and hardware proves even more critical.

Through a high degree of decoupling, automakers can continually update SoC functionality via Over-The-Air (OTA) updates. The openness of the software ecosystem assists automakers in establishing differentiation, serving as a competitive imperative that IC design firms cannot overlook.

On the 15th, Microsoft introducing its first in-house AI chip, “Maia.” This move signifies the entry of the world’s second-largest cloud service provider (CSP) into the domain of self-developed AI chips. Concurrently, Microsoft introduced the cloud computing processor “Cobalt,” set to be deployed alongside Maia in selected Microsoft data centers early next year. Both cutting-edge chips are produced using TSMC’s advanced 5nm process, as reported by UDN News.

Amidst the global AI fervor, the trend of CSPs developing their own AI chips has gained momentum. Key players like Amazon, Google, and Meta have already ventured into this territory. Microsoft, positioned as the second-largest CSP globally, joined the league on the 15th, unveiling its inaugural self-developed AI chip, Maia, at the annual Ignite developer conference.

These AI chips developed by CSPs are not intended for external sale; rather, they are exclusively reserved for in-house use. However, given the commanding presence of the top four CSPs in the global market, a significant business opportunity unfolds. Market analysts anticipate that, with the exception of Google—aligned with Samsung for chip production—other major CSPs will likely turn to TSMC for the production of their AI self-developed chips.

TSMC maintains its consistent policy of not commenting on specific customer products and order details.

TSMC’s recent earnings call disclosed that 5nm process shipments constituted 37% of Q3 shipments this year, making the most substantial contribution. Having first 5nm plant mass production in 2020, TSMC has introduced various technologies such as N4, N4P, N4X, and N5A in recent years, continually reinforcing its 5nm family capabilities.

Maia is tailored for processing extensive language models. According to Microsoft, it initially serves the company’s services such as $30 per month AI assistant, “Copilot,” which offers Azure cloud customers a customizable alternative to Nvidia chips.

Borkar, Corporate VP, Azure Hardware Systems & Infrastructure at Microsoft, revealed that Microsoft has been testing the Maia chip in Bing search engine and Office AI products. Notably, Microsoft has been relying on Nvidia chips for training GPT models in collaboration with OpenAI, and Maia is currently undergoing testing.

Gulia, Executive VP of Microsoft Cloud and AI Group, emphasized that starting next year, Microsoft customers using Bing, Microsoft 365, and Azure OpenAI services will witness the performance capabilities of Maia.

While actively advancing its in-house AI chip development, Microsoft underscores its commitment to offering cloud services to Azure customers utilizing the latest flagship chips from Nvidia and AMD, sustaining existing collaborations.

Regarding the cloud computing processor Cobalt, adopting the Arm architecture with 128 core chip, it boasts capabilities comparable to Intel and AMD. Developed with chip designs from devices like smartphones for enhanced energy efficiency, Cobalt aims to challenge major cloud competitors, including Amazon.
(Image: Microsoft)

On October 17th, the U.S. Department of Commerce announced an expansion of export control, tightening further restrictions. In addition to the previously restricted products like NVIDIA A100, H100, and AMD MI200 series, the updated measures now include a broader range, encompassing NVIDA A800, H800, L40S, L40, L42, AMD MI300 series, Intel Gaudi 2/3, and more, hindering their import into China. This move is expected to hasten the adoption of domestically developed chips by Chinese communications service providers (CSPs).

TrendForce’s Insights:

1. Chinese CSPs Strategically Invest in Both In-House Chip Development and Related Companies

In terms of the in-house chip development strategy of Chinese CSPs, Baidu announced the completion of tape out for the first generation Kunlun Chip in 2019, utilizing the XPU. It entered mass production in early 2020, with the second generation in production by 2021, boasting a 2-3 times performance improvement. The third generation is expected to be released in 2024. Aside from independent R&D, Baidu has invested in related companies like Nebula-Matrix, Phytium, Smartnvy, and. In March 2021, Baidu also established Kunlunxin through the split of its AI chip business.

Alibaba, in April 2018, fully acquired Chinese CPU IP supplier C-Sky and established T-head semiconductor in September of the same year. Their first self-developed chip, Hanguang 800, was launched in September 2020. Alibaba also invested in Chinese memory giant CXMT, AI IC design companies Vastaitech, Cambricon and others.

Tencent initially adopted an investment strategy, investing in Chinese AI chip company Enflame Tech in 2018. In 2020, it established Tencent Cloud and Smart Industries Group(CSIG), focusing on IC design and R&D. In November 2021, Tencent introduced AI inference chip Zixiao, utilizing 2.5D packaging for image and video processing, natural language processing, and search recommendation.

Huawei’s Hisilicon unveiled Ascend 910 in August 2019, accompanied by the AI open-source computing framework MindSpore. However, due to being included in the U.S. entity list, Ascend 910 faced production restrictions. In August 2023, iFLYTEK, a Chinese tech company, jointly introduced the “StarDesk AI Workstation” with Huawei, featuring the new AI chip Ascend 910B. This is likely manufactured using SMIC’s N+2 process, signifying Huawei’s return to self-developed AI chips.

2. Some Chinese Companies Turn to Purchasing Huawei’s Ascend 910B, Yet It Lags Behind A800

Huawei’s AI chips are not solely for internal use but are also sold to other Chinese companies. Baidu reportedly ordered 1,600 Ascend 910B chips from Huawei in August, valued at approximately 450 million RMB, to be used in 200 Baidu data center servers. The delivery is expected to be completed by the end of 2023, with over 60% of orders delivered as of October. This indicates Huawei’s capability to sell AI chips to other Chinese companies.

Huawei’s Ascend 910B, expected to be released in the second half of 2024, boasts hardware figures comparable to NVIDIA A800. According to tests conducted by Chinese companies, its performance is around 80% of A800. However, in terms of software ecosystem, Huawei still faces a significant gap compared to NVIDIA.

Overall, using Ascend 910B for AI training may be less efficient than A800. Yet with the tightening U.S. policies, Chinese companies are compelled to turn to Ascend 910B. As user adoption increases, Huawei’s ecosystem is expected to improve gradually, leading more Chinese companies to adopt its AI chips. Nevertheless, this will be a protracted process.

On November 13, NVIDIA unveiled the AI computing platform HGX H200, featuring the Hopper architecture, equipped with H200 Tensor Core GPU and high-end memory to handle the vast amounts of data generated by AI and high-performance computing.

This marks an upgrade from the previous generation H100, with a 1.4x increase in memory bandwidth and a 1.8x increase in capacity, enhancing its capabilities for processing intensive generative AI tasks.

The internal memory changes in H200 represent a significant upgrade, as it adopts the HBM3e for the first time. This results in a notable increase in GPU memory bandwidth, soaring from 3.35TB per second in H100 to 4.8TB per second.

The total memory capacity also sees a substantial boost, rising from 80GB in H100 to 141GB. When compared to H100, these enhancements nearly double the inference speed for the Llama 2 model.

H200 is designed to be compatible with systems that already support H100, according to NVIDIA. The company states that cloud service providers can seamlessly integrate H200 into their product portfolios without the need for any modifications.

This implies that NVIDIA’s server manufacturing partners, including ASRock, ASUS, Dell, Eviden, GIGABYTE, HPE, Ingrasys, Lenovo, Quanta Cloud, Supermicro, Wistron, and Wiwynn, have the flexibility to replace existing processors with H200.

The initial shipments of H200 are expected in the second quarter of 2024, with cloud service giants such as Amazon, Google, Microsoft, and Oracle anticipated to be among the first to adopt H200.

What is HBM?

“The integration of faster and more extensive HBM memory serves to accelerate performance across computationally demanding tasks including generative AI models and [high-performance computing] applications while optimizing GPU utilization and efficiency,” said Ian Buck, the Vice President of High-Performance Computing Products at NVIDIA.

What is HBM? HBM refers to stacking DRAM layers like building blocks and encapsulating them through advanced packaging. This approach increases density while maintaining or even reducing the overall volume, leading to improved storage efficiency.

TrendForce reported that the HBM market’s dominant product for 2023 is HBM2e, employed by the NVIDIA A100/A800, AMD MI200, and most CSPs’ (Cloud Service Providers) self-developed accelerator chips.

As the demand for AI accelerator chips evolves, in 2023, the mainstream demand is projected to shift from HBM2e to HBM3, with estimated proportions of approximately 50% and 39%, respectively.

As the production of acceleration chips utilizing HBM3 increases gradually, the market demand in 2024 is expected to significantly transition to HBM3, surpassing HBM2e directly. The estimated proportion for 2024 is around 60%.

Since Manufacturers plan to introduce new HBM3e products in 2024, HBM3 and HBM3e are expected to become mainstream in the market next year.

TrendForce clarifies that the so-called HBM3 in the current market should be subdivided into two categories based on speed. One category includes HBM3 running at speeds between 5.6 to 6.4 Gbps, while the other features the 8 Gbps HBM3e, which also goes by several names including HBM3P, HBM3A, HBM3+, and HBM3 Gen2.

HBM3e will be stacked with 24Gb mono dies, and under the 8-layer (8Hi) foundation, the capacity of a single HBM3e will jump to 24GB.

According to the TrendForce’s previous news release, the three major manufacturers currently leading the HBM competition – SK hynix, Samsung, and Micron – have the following progress updates.

SK hynix and Samsung began their efforts with HBM3, which is used in NVIDIA’s H100/H800 and AMD’s MI300 series products. These two manufacturers are expected to sample HBM3e in Q1 2024 previously. Meanwhile, Micron chose to skip HBM3 and directly develop HBM3e.

However, according to the latest TrendForce survey, as of the end of July this year, Micron has already provided NVIDIA with HBM3e verification, while SK hynix did so in mid-August, and Samsung in early October.

(Image: Nvidia)