In 2023, “generative AI” was undeniably the hottest term in the tech industry.

The launch of the generative application ChatGPT by OpenAI has sparked a frenzy in the market, prompting various tech giants to join the race.

As per a report from TechNews, currently, NVIDIA dominates the market by providing AI accelerators, but this has led to a shortage of their AI accelerators in the market. Even OpenAI intends to develop its own chips to avoid being constrained by tight supply chains.

On the other hand, due to restrictions arising from the US-China tech war, while NVIDIA has offered reduced versions of its products to Chinese clients, recent reports suggest that these reduced versions are not favored by Chinese customers.

Instead, Chinese firms are turning to Huawei for assistance or simultaneously developing their own chips, expected to keep pace with the continued advancement of large-scale language models.

In the current wave of AI development, NVIDIA undoubtedly stands as the frontrunner in AI computing power. Its A100/H100 series chips have secured orders from top clients worldwide in the AI market.

As per analyst Stacy Rasgon from the Wall Street investment bank Bernstein Research, the cost of each query using ChatGPT is approximately USD 0.04. If ChatGPT queries were to scale to one-tenth of Google’s search volume, the initial deployment would require approximately USD 48.1 billion worth of GPUs for computation, with an annual requirement of about USD 16 billion worth of chips to sustain operations, along with a similar amount for related chips to execute tasks.

Therefore, whether to reduce costs, decrease overreliance on NVIDIA, or even enhance bargaining power further, global tech giants have initiated plans to develop their own AI accelerators.

Per reports by technology media The Information, citing industry sources, six global tech giants, including Microsoft, OpenAI, Tesla, Google, Amazon, and Meta, are all investing in developing their own AI accelerator chips. These companies are expected to compete with NVIDIA’s flagship H100 AI accelerator chips.

Progress of Global Companies’ In-house Chip Development

• Microsoft

Rumors surrounding Microsoft’s in-house AI chip development have never ceased.

At the annual Microsoft Ignite 2023 conference, the company finally unveiled the Azure Maia 100 AI chip for data centers and the Azure Cobalt 100 cloud computing processor. In fact, rumors of Microsoft developing an AI-specific chip have been circulating since 2019, aimed at powering large language models.

The Azure Maia 100, introduced at the conference, is an AI accelerator chip designed for tasks such as running OpenAI models, ChatGPT, Bing, GitHub Copilot, and other AI workloads.

According to Microsoft, the Azure Maia 100 is the first-generation product in the series, manufactured using a 5-nanometer process. The Azure Cobalt is an Arm-based cloud computing processor equipped with 128 computing cores, offering a 40% performance improvement compared to several generations of Azure Arm chips. It provides support for services such as Microsoft Teams and Azure SQL. Both chips are produced by TSMC, and Microsoft is already designing the second generation.

• Open AI

OpenAI is also exploring the production of in-house AI accelerator chips and has begun evaluating potential acquisition targets. According to earlier reports from Reuters citing industry sources, OpenAI has been discussing various solutions to address the shortage of AI chips since at least 2022.

Although OpenAI has not made a final decision, options to address the shortage of AI chips include developing their own AI chips or further collaborating with chip manufacturers like NVIDIA.

OpenAI has not provided an official comment on this matter at the moment.

• Tesla

Electric car manufacturer Tesla is also actively involved in the development of AI accelerator chips. Tesla primarily focuses on the demand for autonomous driving and has introduced two AI chips to date: the Full Self-Driving (FSD) chip and the Dojo D1 chip.

The FSD chip is used in Tesla vehicles’ autonomous driving systems, while the Dojo D1 chip is employed in Tesla’s supercomputers. It serves as a general-purpose CPU, constructing AI training chips to power the Dojo system.

• Google

Google began secretly developing a chip focused on AI machine learning algorithms as early as 2013 and deployed it in its internal cloud computing data centers to replace NVIDIA’s GPUs.

The custom chip, called the Tensor Processing Unit (TPU), was unveiled in 2016. It is designed to execute large-scale matrix operations for deep learning models used in natural language processing, computer vision, and recommendation systems.

In fact, Google had already constructed the TPU v4 AI chip in its data centers by 2020. However, it wasn’t until April 2023 that technical details of the chip were publicly disclosed.

• Amazon

As for Amazon Web Services (AWS), the cloud computing service provider under Amazon, it has been a pioneer in developing its own chips since the introduction of the Nitro1 chip in 2013. AWS has since developed three product lines of in-house chips, including network chips, server chips, and AI machine learning chips.

Among them, AWS’s lineup of self-developed AI chips includes the inference chip Inferentia and the training chip Trainium.

On the other hand, AWS unveiled the Inferentia 2 (Inf2) in early 2023, specifically designed for artificial intelligence. It triples computational performance while increasing accelerator total memory by a quarter.

It supports distributed inference through direct ultra-high-speed connections between chips and can handle up to 175 billion parameters, making it the most powerful in-house manufacturer in today’s AI chip market.

• Meta

Meanwhile, Meta, until 2022, continued using CPUs and custom-designed chipsets tailored for accelerating AI algorithms to execute its AI tasks.

However, due to the inefficiency of CPUs compared to GPUs in executing AI tasks, Meta scrapped its plans for a large-scale rollout of custom-designed chips in 2022. Instead, it opted to purchase NVIDIA GPUs worth billions of dollars.

Still, amidst the surge of other major players developing in-house AI accelerator chips, Meta has also ventured into internal chip development.

On May 19, 2023, Meta further unveiled its AI training and inference chip project. The chip boasts a power consumption of only 25 watts, which is 1/20th of the power consumption of comparable products from NVIDIA. It utilizes the RISC-V open-source architecture. According to market reports, the chip will also be produced using TSMC’s 7-nanometer manufacturing process.

China’s Progress on In-House Chip Development

China’s journey in developing in-house chips presents a different picture. In October last year, the United States expanded its ban on selling AI chips to China.

Although NVIDIA promptly tailored new chips for the Chinese market to comply with US export regulations, recent reports suggest that major Chinese cloud computing clients such as Alibaba and Tencent are less inclined to purchase the downgraded H20 chips. Instead, they have begun shifting their orders to domestic suppliers, including Huawei.

This shift in strategy indicates a growing reliance on domestically developed chips from Chinese companies by transferring some orders for advanced semiconductors to China.

TrendForce indicates that currently about 80% of high-end AI chips purchased by Chinese cloud operators are from NVIDIA, but this figure may decrease to 50% to 60% over the next five years.

If the United States continues to strengthen chip controls in the future, it could potentially exert additional pressure on NVIDIA’s sales in China.

Read more

• [News] NVIDIA’s Exclusive Chips for China Now Reported to be Available for Pre-Order, Priced Similar to Huawei Products

(Photo credit: NVIDIA)

DRAM prices have risen for three consecutive months, a trend attributed to Chinese clients accepting the price hike requests from memory manufacturers, as reported by Nikkei on February 16th.

As per data cited by Nikkei, the wholesale price (transaction price) of the benchmark product DDR4 8Gb was around USD 1.85 each in January 2024, marking a 9% increase from the previous month (December 2023). The price of the smaller 4Gb product was around USD 1.40 each, representing an 8% increase from the previous month. The aforementioned prices have been rising for the third consecutive month.

Reportedly, the price negotiation occurred before the Chinese Lunar New Year holiday, with Chinese clients increasing their purchasing volume before the break.

On the other hand, per TrendForce, since the fourth quarter of last year through the first quarter of this year, contract prices for DRAM products have seen continuous increases. For the mainstream product DDR4 8Gb, the contract price in January was USD 1.80.

The estimated increase for the first quarter is between 10% to 15%, and it is anticipated that there will be an additional increase of at least close to 10% by the end of the first quarter.

The report from Nikkei further addresses that 2024 is expected to enter the early stages of the PC replacement cycle, leading to increased demand for DRAM. According to sources cited in the report, besides Chinese clients, major PC manufacturers are also accepting price hikes.

Read more

• [News] The Quiet Beginning of the 3D DRAM Market Share Battle
• [News] Samsung Reportedly Adjusts DRAM and NAND Flash Capacity to Boost Prices

(Photo credit: Samsung)

The surge in demand for advanced packaging is being primarily propelled by artificial intelligence (AI) chips. According to industry sources cited by CNA, TSMC’s CoWoS production capacity is set to double this year, yet demand continues to outstrip supply. In response, NVIDIA has enlisted the help of packaging and testing facilities to augment its advanced packaging capabilities.

In addition, to address the imbalance between supply and demand for advanced packaging due to AI, semiconductor backend specialty assembly and testing (OSAT) companies such as ASE Technology Holding (ASE), Powertech Technology, and KYEC have expanded their capital expenditures this year to enhance their advanced packaging capabilities, aligning with the needs of their customers.

AI and high-performance computing (HPC) chips are driving the demand for CoWoS advanced packaging. As per sources interviewed by CNA, from July to the end of last year, TSMC actively adjusted its CoWoS advanced packaging production capacity, gradually expanding and stabilizing mass production.

The source further indicates that in December of last year, TSMC’s CoWoS monthly production capacity increased to 14,000 to 15,000. It is estimated that by the fourth quarter of this year, TSMC’s CoWoS monthly production capacity will significantly expand to 33,000 to 35,000.

Per an earlier report from Commercial Times, TSMC has been outsourcing part of its CoWoS operations for some time, mainly targeting small-volume, high-performance chips. TSMC maintains in-house production of the CoW, while the back-end WoS is handed over to test and assembly houses to improve production efficiency and flexibility. 

However, the demand for advanced packaging capacity for AI chips still outstrips supply. Sources cited by CNA also reveal that NVIDIA has sought assistance from packaging and testing subcontractors outside of TSMC to augment their advanced packaging capabilities.

Amkor, among others, began gradually providing capacity support from the fourth quarter of last year, while SPIL, a subsidiary of ASE, is slated to commence supply in the first quarter of this year.

Read more

• [News] AMD Eyes CoWoS-like Supply for AI Chips with TSMC Full Capacity
• [News] CoWoS Capacity Shortage Challenges AI Chip Demand, while Taiwanese Manufacturers Expand to Seize Opportunities

(Photo credit: TSMC)

SoftBank Group founder Masayoshi Son, as per a report from Bloomberg, is planning to raise USD 100 billion to establish an AI chip company, aiming to complement the group’s ARM business.

The report indicates that Masayoshi Son plans to name the new artificial intelligence chip venture “Izanagi,” after the deity of creation and life in Japanese mythology, and Son himself will directly lead the project.

Regarding funding, reportedly, one proposed scheme under consideration involves SoftBank providing USD 30 billion, while another USD 70 billion may come from institutions in the Middle East.

However, details regarding the final funding sources and how the funds will be utilized in the future have not been disclosed by Masayoshi Son at this time.

Masayoshi Son is highly optimistic about the development of AI, claiming to be a heavy user of ChatGPT in an interview and engaging in conversations with it almost every day. In October 2023, Son expressed his belief that within the next decade, artificial intelligence will surpass human intelligence in nearly all domains, achieving a level of general artificial intelligence.

SoftBank’s UK-based chip intellectual property company, Arm, raised approximately USD 4.87 billion in its initial public offering. At the time of Arm’s listing, Son stated that he is a big believer in artificial intelligence and that Arm will also be at the core beneficiary of the AI revolution.

At the shareholders’ meeting on June 21, 2023, Masayoshi Son presented a chart of human evolution titled “Evolution Speed.”

The chart depicted a flat curve from the birth of humanity to the agricultural revolution, followed by a slight increase during the industrial and information revolutions. This indicated that the curve representing the development of artificial intelligence would experience a rapid upward surge within a few years, with its slope approaching a nearly vertical line.

Read more

• [News] CoWoS Capacity Shortage Challenges AI Chip Demand, while Taiwanese Manufacturers Expand to Seize Opportunities
• [News] NVIDIA CEO Jensen Huang Meets with TSMC Executives in Taiwan, Addressing the Significant AI Chip Shortage

(Photo credit: SoftBank News)

With the flourishing development of technologies such as AI, cloud computing, big data analytics, and mobile computing, modern society has an increasingly high demand for computing power.

Moreover, with the advancement beyond 3 nanometers, wafer sizes have encountered scaling limitations and manufacturing costs have increased. Therefore, besides continuing to develop advanced processes, the semiconductor industry is also exploring other ways to maintain chip size while ensuring high efficiency.

The concept of “heterogeneous integration” has become a contemporary focus, leading to the transition of chips from single-layer to advanced packaging with multiple layers stacked together.

The term “CoWoS” can be broken down into the following definitions: “Cow” stands for “Chip-on-Wafer,” referring to the stacking of chips, while “WoS” stands for “Wafer-on-Substrate,” which involves stacking chips on a substrate.

Therefore, “CoWoS” collectively refers to stacking chips and packaging them onto a substrate. This approach reduces the space required for chips and offers benefits in reducing power consumption and costs.

Among these, CoWoS can be further divided into 2.5D horizontal stacking (most famously exemplified by TSMC’s CoWoS) and 3D vertical stacking versions. In these configurations, various processor and memory modules are stacked layer by layer to create chiplets. Because its primary application lies in advanced processes, it is also referred to as advanced packaging.

According to TrendForce’s data, it has provided insights into the heat of the AI chip market. In 2023, shipments of AI servers (including those equipped with GPU, FPGA, ASIC, etc.) reached nearly 1.2 million units, a 38.4% increase from 2022, accounting for nearly 9% of the overall server shipments.

Looking ahead to 2026, the proportion is expected to reach 15%, with a compound annual growth rate (CAGR) of AI server shipments from 2022 to 2026 reaching 22%.

Due to the advanced packaging requirements of AI chips, TSMC’s 2.5D advanced packaging CoWoS technology is currently the primary technology used for AI chips.

GPUs, in particular, utilize higher specifications of HBM, which require the integration of core dies using 2.5D advanced packaging technology. The initial stage of chip stacking in CoWoS packaging, known as Chip on Wafer (CoW), primarily undergoes manufacturing at the fab using a 65-nanometer process. Following this, through-silicon via (TSV) is carried out, and the finalized products are stacked and packaged onto the substrate, known as Wafer on Substrate (WoS).

As a result, the production capacity of CoWoS packaging technology has become a significant bottleneck in AI chip output over the past year, and it remains a key factor in whether AI chip demand can be met in 2024. Foreign analysts have previously pointed out that NVIDIA is currently the largest customer of TSMC’s 2.5D advanced packaging CoWoS technology.

This includes NVIDIA’s H100 GPU, which utilizes TSMC’s 4-nanometer advanced process, as well as the A100 GPU, which uses TSMC’s 7-nanometer process, both of which are packaged using CoWoS technology. As a result, NVIDIA’s chips account for 40% to 50% of TSMC’s CoWoS packaging capacity. This is also why the high demand for NVIDIA chips has led to tight capacity for TSMC’s CoWoS packaging.

TSMC’s Expansion Plans Expected to Ease Tight Supply Situation in 2024

During the earnings call held in July 2023, TSMC announced its plans to double the CoWoS capacity, indicating that the supply-demand imbalance in the market could be alleviated by the end of 2024.

Subsequently, in late July 2023, TSMC announced an investment of nearly NTD 90 billion (roughly USD 2.87 billion) to establish an advanced packaging fab in the Tongluo Science Park, with the construction expected to be completed by the end of 2026 and mass production scheduled for the second or third quarter of 2027.

In addition, during the earnings call on January 18, 2024, TSMC’s CFO, Wendell Huang, emphasized that TSMC would continue its expansion of advanced processes in 2024. Therefore, it is estimated that 10% of the total capital expenditure for the year will be allocated towards expanding capacity in advanced packaging, testing, photomasks, and other areas.

In fact, NVIDIA’s CFO, Colette Kress, stated during an investor conference that the key process of CoWoS advanced packaging has been developed and certified with other suppliers. Kress further anticipated that supply would gradually increase over the coming quarters.

Regarding this, J.P. Morgan, an investment firm, pointed out that the bottleneck in CoWoS capacity is primarily due to the supply-demand gap in the interposer. This is because the TSV process is complex, and expanding capacity requires more high-precision equipment. However, the long lead time for high-precision equipment, coupled with the need for regular cleaning and inspection of existing equipment, has resulted in supply shortages.

Apart from TSMC’s dominance in the CoWoS advanced packaging market, other Taiwanese companies such as UMC, ASE Technology Holding, and Powertek Technology are also gradually entering the CoWoS advanced packaging market.

Among them, UMC expressed during an investor conference in late July 2023 that it is accelerating the deployment of silicon interposer technology and capacity to meet customer needs in the 2.5D advanced packaging sector.

UMC Expands Interposer Capacity; ASE Pushes Forward with VIPack Advanced Packaging Platform

UMC emphasizes that it is the world’s first foundry to offer an open system solution for silicon interposer manufacturing. Through this open system collaboration (UMC+OSAT), UMC can provide a fully validated supply chain for rapid mass production implementation.

On the other hand, in terms of shipment volume, ASE Group currently holds approximately a 32% market share in the global Outsourced Semiconductor Assembly and Test (OSAT) industry and accounts for over 50% of the OSAT shipment volume in Taiwan. Its subsidiary, ASE Semiconductor, also notes the recent focus on CoWoS packaging technology. ASE Group has been strategically positioning itself in advanced packaging, working closely with TSMC as a key partner.

ASE underscores the significance of its VIPack advanced packaging platform, designed to provide vertical interconnect integration solutions. VIPack represents the next generation of 3D heterogeneous integration architecture.

Leveraging advanced redistribution layer (RDL) processes, embedded integration, and 2.5D/3D packaging technologies, VIPack enables customers to integrate multiple chips into a single package, unlocking unprecedented innovation in various applications.

Powertech Technology Seeks Collaboration with Foundries; Winbond Electronics Offers Heterogeneous Integration Packaging Technology

In addition, the OSAT player Powertech Technology is actively expanding its presence in advanced packaging for logic chips and AI applications.

The collaboration between Powertech and Winbond is expected to offer customers various options for CoWoS advanced packaging, indicating that CoWoS-related advanced packaging products could be available as early as the second half of 2024.

Winbond Electronics emphasizes that the collaboration project will involve Winbond Electronics providing CUBE (Customized Ultra-High Bandwidth Element) DRAM, as well as customized silicon interposers and integrated decoupling capacitors, among other advanced technologies. These will be complemented by Powertech Technology’s 2.5D and 3D packaging services.

Read more

• [News] AMD Eyes CoWoS-like Supply for AI Chips with TSMC Full Capacity
• [News] Samsung Boosts 2.5D Packaging Equipment to Compete for TSMC’s CoWoS Orders

(Photo credit: TSMC)