Neural processing unit

A neural processing unit (NPU), also known as an AI accelerator or deep learning processor, is a class of specialized hardware accelerator or computer system designed to accelerate artificial intelligence and machine learning applications, including artificial neural networks and computer vision. NPU can be standalone, a part of a central processing unit (CPU) or a part of a graphics processing unit (GPU).

History

An early use of the term neural processing unit (NPU) to refer to a dedicated neural-network accelerator appeared in the 2012 paper Neural Acceleration for General-Purpose Approximate Programs, which described an NPU architecture for accelerating approximate programs.

Use

NPU's purpose is either to efficiently execute already trained AI models like large language models (LLMs) for inference, or to train AI models. NPUs can be more efficient in terms of speed or power consumption.

NPU applications include algorithms for robotics, Internet of things, and data-intensive or sensor-driven tasks. They are often manycore or spatial designs and focus on low-precision arithmetic, novel dataflow architectures, or in-memory computing capability. As of 2024, a widely used datacenter-grade AI integrated circuit chip, the Nvidia H100 GPU, contains tens of billions of metal–oxide–semiconductor field-effect transistors (MOSFETs).

Consumer devices

AI accelerators are used in Apple silicon, Qualcomm, Samsung, Huawei, and Google Tensor smartphone processors. When used as part of a GPU for graphics rendering, they can significantly reduce resource use by allowing the traditional parts of the GPU to render a scene at a much lower resolution and frame rate (e.g., 540p at 30 frames per second (fps)) and then using a pre-trained AI model on the NPU to turn that base imagery into smoother and higher resolution output (e.g., 2160p at 240 fps) in real-time.