Edge AI and On-Device AI Explained

TECHNOLOGY EXPLAINER

Edge AI and On-Device AI
— before asking "how many TOPS", ask "how many GB of memory and how many GB/s of bandwidth"

Edge AI, or on-device AI, runs models inside phones, AI PCs and embedded equipment instead of sending data to the cloud, which gives a different trade-off on latency, communication and privacy. Yet although an NPU's "TOPS" figure is printed large, what sets the speed of text generation by a small language model is, in most cases, memory capacity and bandwidth. Using official material from Microsoft, Qualcomm, Apple, Arm, NVIDIA, Hailo and others, this article organizes NPUs, small models, quantization, memory, power and privacy, and builds a feel for the numbers with our own calculations. It does not rank products or companies.

Sources: Microsoft's Windows 11 specifications page, product pages from Qualcomm, Arm, NVIDIA and Hailo, Apple's foundation-model report and blogs, the Phi-3, Gemma 3 and MobileLLM papers, MLCommons MLPerf pages, the ISO standards list and a JEDEC announcement / Checked 15 October 2026

Fig. 1. Conceptual image generated with ImageGen. Camera input is processed locally. The enlarged SoC and memory illustrate on-device limits in capacity, bandwidth, power and heat. Not an actual facility, product or measured result.

Contents

What edge AI and on-device AI are (the short version)
Where it runs: kinds of devices, and a map of stated values
NPUs and TOPS: what is being counted
Memory decides: capacity and bandwidth
Small models and ways to shrink: quantization, KV cache and flash
Power and heat: in a device, the watt is the first constraint
Privacy and data: what "on-device" means, and its limits
A materials engineer's view: edge AI in plants and labs, processing field data on the spot with small devices
Limits, and what could not be confirmed
How to read the evaluations and standards
Glossary
References
Claim-to-source audit

How this article labels statements

Sourced = stated in a published source (linked)
Our calculation = a value this article derives from stated assumptions
Not yet confirmed = plans, schedules or proposals not yet confirmed as results
Interpretation and organization are marked as commentary. An inference is shown in bold.

How to read the numbers in this article

TOPS, memory bandwidth and power are vendors' stated values. TOPS is a peak figure for trillions of operations per second; the numeric precision (such as INT8) and whether sparsity is used differ by product, and it does not indicate real speed. Model speeds and sizes are what each company or paper reported under its own conditions. No products or companies are ranked. Future products and features are treated as "not yet confirmed". Our calculations are simple upper-bound estimates, not predictions of real performance.

1. What edge AI and on-device AI are (the short version)

Running AI inside the device: inference runs inside the smartphone, PC, embedded device or factory equipment (the edge) rather than on a cloud server. Apple has published a design that combines an on-device model of about 3 billion parameters with a server model on Private Cloud Compute.Sourced[Ref. 4][Ref. 5]
TOPS is only the entry number: Microsoft lists "an NPU capable of 40+ TOPS, 16 GB of RAM and 256 GB of storage" as requirements for a Copilot+ PC.Sourced[Ref. 1] But because a small model reads its weights for every token, text generation is capped by memory bandwidth (our calculation in Chapter 4).Our calculation
Shrinking is the core technique: official material reports small models, quantization (Apple uses 2 bits), KV-cache reduction and designs that keep weights in flash and load only what is needed into memory.Sourced[Ref. 3][Ref. 5][Ref. 6]

The single most important line

For on-device AI, the ceiling on speed is set less by "how fast the NPU is" than by "how many GB the model is and how many GB/s memory can read". Even with the same 45 TOPS NPU, different memory bandwidth changes the ceiling on text-generation speed (Chapter 4). TOPS is a peak figure whose precision, sparsity and measurement differ by product.

2. Where it runs: kinds of devices, and a map of stated values

Edge AI devices span orders of magnitude in power and memory. Arm's Ethos-U85 is an NPU design (IP) for Cortex-M (microcontroller) and Cortex-A devices that scales from 128 to 2,048 multiply-accumulate units (MACs) for up to 4 TOPS at 1 GHz, with native support for transformer networks.Sourced[Ref. 12] Hailo-8 is an M.2 or PCIe module stated at up to 26 TOPS and typical power of 2.5 W.Sourced[Ref. 14] NVIDIA's Jetson AGX Orin is stated at up to 275 TOPS (sparse INT8), 15 to 60 W and 204.8 GB/s of memory bandwidth.Sourced[Ref. 13] For PCs, Qualcomm lists Snapdragon X at up to 45 TOPS with up to 135 GB/s of LPDDR5x bandwidth, and Snapdragon X2 Elite at 80 to 85 TOPS with up to 152 to 228 GB/s.Sourced[Ref. 2]

Fig. 2 Conceptual diagram (vector drawing). Values are the stated figures in [Refs. 2, 12, 13, 14]. Precision, sparsity and measurement are not aligned, so this is not a ranking.

Example	Stated values (as written in the source)	What to watch
Microcontroller NPU (Arm Ethos-U85)	128 to 2,048 MACs, up to 4 TOPS at 1 GHz, transformer support, about 20% more energy efficient than the previous generation (Arm statement) [12]	It is IP (a design); real performance depends on memory and the integrating SoC. The page has no footnote
Embedded accelerator (Hailo-8)	Up to 26 TOPS, typical power 2.5 W, M.2 and PCIe (Hailo statement) [14]	Dividing peak TOPS by typical power is not a like-for-like measurement
Embedded module (Jetson AGX Orin)	Up to 275 TOPS (sparse INT8), 15 to 60 W, 64 GB 256-bit LPDDR5, 204.8 GB/s (NVIDIA statement) [13]	A figure with sparsity; dense figures appear only for some rows on the page
PC (Snapdragon X / X2 Elite)	X: up to 45 TOPS, LPDDR5x up to 135 GB/s. X2 Elite: 80 to 85 TOPS, 152 to 228 GB/s (Qualcomm statement) [2]	Within a series, TOPS and bandwidth differ by part number. The NPU cannot necessarily use all of that bandwidth

All Sourced (statements and vendor-stated values on each company's official page; checked 15 October 2026).

3. NPUs and TOPS: what is being counted

An NPU (neural processing unit) is a compute block built to run the matrix operations of neural networks at lower power than a CPU or GPU. TOPS (tera operations per second) is a peak figure for how many trillions of operations can be done per second; the numeric precision counted (INT8, INT4, FP16 and so on) and whether sparsity (speed-up from some numbers being zero) is used change the figure for the same NPU. NVIDIA's Jetson page states overall AI performance as sparse INT8 TOPS and gives dense values only in some rows.Sourced[Ref. 13] Hailo does not put a footnote on the conditions of its TOPS figure on the page.Sourced[Ref. 14]

TOPS is also used to define PC tiers. Microsoft lists as minimum Copilot+ PC requirements an NPU capable of 40+ TOPS, 16 GB of DDR5/LPDDR5 memory and a 256 GB SSD/UFS, and says this currently includes the AMD Ryzen AI 300 and 400 series, the Intel Core Ultra 200V and 300V series and the Snapdragon X series, warning that the requirements may change over time.Sourced[Ref. 1] The "40+ TOPS" is Microsoft's product requirement, not an international standard.

ISO/IEC has published the technical report ISO/IEC TR 17903:2024, "Overview of machine learning computing devices" (confirmed on ISO's list of standards on 15 October 2026).Sourced[Ref. 20] Its text was not read; only the title was confirmed.

4. Memory decides: capacity and bandwidth

For a small language model, text generation (decode) reads all the weights from memory for every token (as set out in our explainer on inference optimization and serving). The speed ceiling is therefore bandwidth divided by the bytes read, and shrinking the model (quantization) directly raises the ceiling. Microsoft's Phi-3 report says that quantizing the 3.8-billion-parameter phi-3-mini to 4 bits fits it in about 1.8 GB of memory, and that running it natively and fully offline on an iPhone 14 (A16 Bionic) gave more than 12 tokens per second (the company's report).Sourced[Ref. 6]

Model (parameters)	16-bit	8-bit	4-bit
1B	2.0 GB	1.0 GB	0.5 GB
3B	6.0 GB	3.0 GB	1.5 GB
8B	16.0 GB	8.0 GB	4.0 GB

All Our calculation (weights only: parameters x bits / 8; 1 GB = 10^9 bytes). Differences in embedding-table bit widths, quantization overhead such as scales, the KV cache and application memory are excluded.

Our calculation 1: the ceiling on generation speed from memory bandwidth

Our calculation Model: an 8B model at 4 bits has 4.0 GB of weights. For the KV cache, with the Llama 3 8B configuration (32 layers, 8 KV heads, head dimension 128)Sourced[Ref. 22] and an 8-bit KV, one token is 2 x 32 x 8 x 128 x 1 = 65,536 bytes, or about 0.54 GB at 8,192 tokens. We assume a total of about 4.54 GB is read for every token.

Ceiling: bandwidth / about 4.54 GB. At Snapdragon X (up to 135 GB/s)Sourced[Ref. 2] that is about 30 tokens/s; at the maximum Snapdragon X2 Elite configuration (up to 228 GB/s) about 50 tokens/s. For a 3B model at 4 bits (1.5 GB) it is about 90 at 135 GB/s and about 152 tokens/s at 228 GB/s.

Compute ceiling: an 8B model needs about 2 x 8 x 10^9 = 16 billion operations per token. At a 45 TOPS peak that is about 2,812 tokens/s, about 95 times the memory-side ceiling (about 30). Under these conditions memory, not compute, is the limit.

Assumptions and limits: this ceiling assumes the NPU can use all of the SoC's memory bandwidth, all weights sit in memory, and bandwidth is used at peak. In practice bandwidth is shared with CPU and GPU, and caches and scheduling make it slower. Because the precision and sparsity behind the TOPS figure are unknown, the compute side is only a guide. Working back from the Phi-3 report (about 1.8 GB at 4 bits, over 12 tokens/s), reading the weights alone implies at least about 22 GB/s actually used (1.8 GB x 12 per second).

Fig. 3 Figure including our calculation (vector drawing). Our calculation. Weights only; this does not show usable memory or quality. Whether quality survives quantization must be checked per model and task.

Fig. 4 Figure including our calculation (vector drawing). Our calculation (bandwidths are the stated values in [Refs. 2, 13]). Idealized ceilings, not predictions of real speed.

5. Small models and ways to shrink: quantization, KV cache and flash

Small models. Microsoft's phi-3-mini has 3.8 billion parameters, was trained on 3.3 trillion tokens according to the report, and is described as small enough to be deployed on a phone.Sourced[Ref. 6] In MobileLLM, Meta reports that for models under a billion parameters, designs such as deep and thin architectures, embedding sharing and grouped-query attention matter for accuracy.Sourced[Ref. 8] Google's Gemma 3 spans 1 to 27 billion parameters and, to counter the KV-cache memory that explodes with long context, changed its architecture by raising the ratio of local to global attention layers and keeping local spans short.Sourced[Ref. 7]

Quantization. Representing weights in fewer bits. GPTQ is a post-training (no retraining) quantization method for generative pre-trained Transformers,Sourced[Ref. 9] and AWQ quantizes weights by looking at activations, based on the observation that a small fraction of weights matter more.Sourced[Ref. 10] On number formats, AMD, Arm, Intel, Meta, Microsoft, NVIDIA and Qualcomm jointly published the OCP Microscaling Formats (MX) Specification version 1.0 (September 2023), defining FP8, FP6, FP4 and INT8 element formats with a shared per-block scale.Sourced[Ref. 11] Quantization details are covered in our explainer on model efficiency.

Apple's example (company publications). Apple's 2025 technical report describes, for its roughly 3-billion-parameter on-device model, (1) splitting the model into two blocks where the last 37.5% of layers have no key/value projections and share the first block's KV cache, cutting KV-cache memory by 37.5% and time to first token by about 37.5%, (2) compressing weights to 2 bits with quantization-aware training (QAT), (3) quantizing the embedding table to 4 bits and the KV cache to 8 bits, and (4) low-rank adapters to recover lost quality.Sourced[Ref. 5] Apple's page of 8 June 2026 announces a third generation of foundation models in which the on-device models are AFM 3 Core (the next generation of the 3-billion-parameter dense model) and AFM 3 Core Advanced (a 20-billion-parameter sparse model that uses 1 to 4 billion parameters at a time), with a design that keeps the full model in flash memory (NAND) and loads only the needed experts into DRAM (the company's statement and design).Sourced[Ref. 3] The technical report is promised for "later this summer" and could not be found as of 15 October 2026.

Fig. 5. Conceptual image generated with ImageGen. Flash stores the model, DRAM holds larger runtime data, and small on-chip memory sits near computation. Positions and sizes are illustrative, not to scale. Not an actual facility, product or measured result. [Ref. 3]

View original SVG, values and source annotations

Fig. 5 Conceptual diagram (vector drawing). A concept diagram of the memory hierarchy of a device. Apple's design is as described in [Ref. 3]; the point that bandwidth is the limit follows our calculation (Chapter 4).

Our calculation 2: how many GB if everything sat in DRAM (Apple's 20-billion-parameter example at assumed bit widths)

Our calculation Apple states that AFM 3 Core Advanced has 20 billion parameters with 1 to 4 billion used at a time.Sourced[Ref. 3] Apple does not state the bit width. At an assumed 4 bits that is 20 billion x 4 / 8 = 10 GB, and at an assumed 2 bits 5 GB. That is hard to fit in a 16 GB device and is consistent with a design that keeps everything in flash and loads only the needed experts (our commentary; Apple did not give this as its reason).

A materials engineer's view 1: on-device AI is also a problem of memory and storage materials and packaging

What sets the speed ceiling of a device is DRAM bandwidth and the speed of loading from flash (NAND) (Chapters 4 and 5). Bandwidth is determined by the wiring and package between the memory chips and the processor. LPDDR is a family of low-power DRAM standards for smartphones and PCs; JEDEC published LPDDR6 (JESD209-6) on 9 July 2025, listing a dual sub-channel architecture, a low-voltage supply and an efficiency mode for low bandwidth, with applications including mobile, AI and edge AI.Sourced[Ref. 21]

For materials engineers the implication is that shrinking the model (quantization) cuts the number of memory chips, stacking, mounting area and power, while adding memory within a device's limits of thickness, heat and cost is a materials and packaging challenge. It may be that memory capacity increasingly separates price tiers as AI features grow, but that is this article's inference (no source). The structure differs from the data-center memory covered in our HBM explainer, but bandwidth and capacity setting the performance ceiling is common to both.

6. Power and heat: in a device, the watt is the first constraint

A device cannot draw data-center levels of power because of battery and heat limits. Stated values differ widely: Hailo-8 states typical power of 2.5 W,Sourced[Ref. 14] and Jetson AGX Orin 15 to 60 W.Sourced[Ref. 13] Arm says the Ethos-U85 is about 20% more energy efficient than the earlier Ethos-U55 and U65 (Arm statement).Sourced[Ref. 12] The condition behind each power figure (the whole device or just the accelerator, and under what load) differs, so they cannot be compared side by side.

Our calculation 3: how to think about energy per token (an assumed example)

Energy per token [J] = power [W] / speed [tokens/s].Our calculation Suppose 8 W for the AI work and 25 tokens per second: that is 0.32 J per token, or about 11,250 tokens per Wh (= 3,600 J).

Assumptions and limits: 8 W and 25 tokens/s are entirely assumed values, not the performance of any product. Power for the screen, communication and other apps is excluded. The aim is to show that if speed rises at the same power, energy per token falls, but if power rises too, it does not.

Sustained performance (whether speed holds without thermal slowdown over time) and speed per watt on real devices could not be confirmed from official material. These are better judged through the evaluations in Chapter 9 (the power and sustained behaviour of MLPerf Mobile and Client).

7. Privacy and data: what "on-device" means, and its limits

A frequently cited benefit of on-device AI is that, in some cases, data need not leave the device. Apple states that it does not use users' private personal data or interactions to train its foundation models and that server-side processing runs on Private Cloud Compute, where user data is never stored or shared (statements of company policy and assurance).Sourced[Ref. 3] The same page describes a split between on-device models and models on Private Cloud Compute.

Where it runs	Data movement (general)	What to verify
On the device only	Can be designed so inputs and results never leave the device	Whether any network traffic actually occurs, whether logs or diagnostics are sent, and protection if the device is lost
Device plus a dedicated cloud (e.g. Private Cloud Compute)	Only harder tasks go out. Which tasks run where depends on design and settings	Under what conditions data goes out, what users are shown, and whether it is stored (check the company's description)
Cloud only	Inputs are sent to the provider's servers	Contract and terms, retention period, and whether data is used for training

The organization in column 2 is our commentary (general) and not an assurance for any product. Apple's statements are quoted as its own policy and assurance.

Processing on the device does not automatically remove obligations under data-protection or national AI rules. How individual laws apply is outside this article (see our explainers on the EU AI Act and on U.S., Chinese and Japanese AI policy), and this article gives no legal advice.

8. A materials engineer's view: edge AI in plants and labs, processing field data on the spot with small devices

Plants and analytical labs generate large volumes of data every second on site: camera images, vibration and current sensors, and data from microscopes and spectrometers. Embedded and industrial edge AI processes it on small on-site devices instead of sending everything to the cloud. Arm's official Ethos-U85 page lists industrial IoT uses such as high-speed motor and robotic controllers.Sourced[Ref. 12] Hailo-8 comes as M.2 and PCIe modules that can be added to existing equipment.Sourced[Ref. 14]

A materials engineer's view 2: three cautions for on-site use

Evaluate on the real device: quantization and shrinking can change accuracy. MLPerf Mobile sets accuracy targets such as about 93% to 98% of an FP32 reference for tasks including image classification and object detection.Sourced[Ref. 15] Re-evaluate with your own images and signals, on the actual device and bit width (our commentary).

Safety-related use: functional safety when AI is used for functions that affect people or equipment is addressed by ISO/IEC TR 5469:2024, "Artificial intelligence — Functional safety and AI systems" (title confirmed on ISO's list; text not read).Sourced[Ref. 20] Making AI the core of a safety function goes beyond this article and needs the standard and expert review.

Data quality and reproducibility: field data shifts with lighting, contamination, equipment variation and material lot differences. The ISO/IEC 5259 series (data quality for analytics and machine learning) has been published as a family covering data-quality measures, management and governance (titles confirmed).Sourced[Ref. 20] Getting data measurement and storage in order before introducing AI matters just as much on small devices as on large ones (our commentary).

Using AI in materials development and autonomous labs is covered in our explainer on materials informatics, and robot mechanisms and drones in our explainer on drones and robotics.

Fig. 6. Conceptual image generated with ImageGen. Camera, lighting and local compute sit beside the line. Local processing still requires controlled image acquisition and validation of missed defects. Not an actual facility, product or measured result.

9. Limits, and what could not be confirmed

Real-device performance: no measured generation speed, power or sustained performance is shown. Our calculations are ceilings from memory bandwidth. Individual MLPerf submissions and cross-vendor device comparisons are not shown because their settings do not align.
TOPS conditions: the footnote giving the measurement conditions of the TOPS figure on Qualcomm's Snapdragon X2 Elite page was not in the text obtained; the Arm and Hailo pages also have no footnote. Precision and sparsity are unknown.
Apple's third-generation models: statements on the page of 8 June 2026 (company claims and design) are quoted, but the technical report could not be found, and the bit width and context length of the models are unknown.
Texts of standards: for ISO/IEC TR 17903:2024, TR 5469:2024, ISO/IEC 25059:2023, TS 4213:2022 and others only the titles were confirmed and the texts were not read. The speed grades of JEDEC LPDDR5X are not given because the text could not be checked.
Areas not covered: thermal design and batteries of phones, sensor materials, in-vehicle computers for autonomous driving (see our explainer on autonomous driving and AI), wearables and robot mechanisms are not treated in depth.

Summary

On-device AI speed is, in many cases, capped by memory bandwidth. For an 8B model at 4 bits, our calculation put the 45 TOPS compute ceiling at about 95 times the memory-side ceiling.Our calculation
TOPS is a peak figure with conditions, and Microsoft's "40+ TOPS" for Copilot+ PCs is a product requirement, not an international standard.Sourced
Shrinking is the core. Apple describes 2-bit QAT, KV-cache sharing (37.5% less) and, in its third generation, loading experts from flash (company publications).Sourced
Power differs by orders of magnitude between devices (Hailo-8 typical 2.5 W, Jetson AGX Orin 15 to 60 W) and the stated values have different conditions.Sourced
Evaluation centers on MLPerf Mobile, Client and Tiny, with ISO/IEC technical reports and standards separately covering functional safety and data quality.Sourced

10. How to read the evaluations and standards

Edge AI performance is evaluated on (1) latency and throughput (time to first token, tokens per second, image processing time), (2) accuracy (as a share of a reference model), (3) power and energy, and (4) memory and storage. As public frameworks, MLCommons runs MLPerf Mobile for devices, MLPerf Client for PCs and MLPerf Tiny for microcontroller-class devices, and MLPerf Inference also has edge tests.

Item	Representative benchmarks and methods (name, version, source)	Standards and guidance	How to read it
Smartphone and device AI (latency, accuracy)	MLPerf Mobile (MLCommons): most tasks are listed at v6.0 on the page. LLMs (Llama 3.1 8B Instruct, Llama 3.2 3B Instruct on TinyMMLU and IFEval-33), Stable Diffusion 1.5, object detection, segmentation and more. Single stream by default, with accuracy targets (e.g. Llama 3.2 3B at about 59 on TinyMMLU, a FP16-based level) [15] Sourced. Design: Reddi et al. (arXiv 2012.02328) [18]	ISO/IEC TR 17903:2024 (overview of machine learning computing devices), ISO/IEC TS 42119-2:2025 (overview of testing AI systems) and ISO/IEC TR 42106:2026 (overview of differentiated benchmarking of AI system quality characteristics). Titles only were confirmed [20] Sourced	Numbers depend on submission conditions (device, OS, software) and version. An "accuracy target" refers to that task's metric
PC AI (TTFT, tokens per second)	MLPerf Client v2.0 (MLCommons): LLMs (Llama 3.1 8B Instruct, Phi 4 Mini Instruct, Phi 4 Reasoning 14B, Qwen 3 8B [experimental]), image generation and agentic tasks. Metrics are TTFT and TPS. Windows 11 (x86-64 and Arm), macOS and iPadOS [16] Sourced	Microsoft's Copilot+ PC requirements (40+ TOPS, 16 GB, 256 GB) are product requirements, not an international standard [1] Sourced	Compare only like with like: same version, model and precision. The execution backend (NPU, GPU or CPU) changes the result
Microcontroller class (latency, energy)	MLPerf Tiny (MLCommons): the page names v1.4 as the newest round while its table shows v1.3 (inconsistent on the page). Image classification (CIFAR-10, ResNet), person detection, keyword spotting, anomaly detection and streaming wakeword. Latency and, optionally, energy [17] Sourced. Design: Banbury et al. (arXiv 2106.07597) [19]	None found (an international standard setting performance for microcontroller-class AI)	Energy is an optional submission item. Check the scope (chip only, or board)
Edge agentic workloads	MLPerf Inference v6.1 (16 September 2026) added an edge test measuring multi-turn agentic workloads such as coding on devices [23] Sourced	None found	An edge track inside the data-center-oriented framework, a separate system from MLPerf Mobile and Client
Trustworthiness, safety and data quality	None found (conformity results for individual devices)	ISO/IEC TR 5469:2024 (functional safety and AI), ISO/IEC TR 24029-1:2021 and 24029-2:2023 (robustness assessment of neural networks), the ISO/IEC 5259 series (data quality), ISO/IEC 25059:2023 (quality model for AI systems) and ISO/IEC 22989:2022 (terminology). Titles and editions confirmed [20] Sourced	Technical reports (TR) are informative, not requirements. Texts were not read and no requirements are inferred
Number formats and memory	None found (a standardized evaluation method for device AI accuracy)	OCP Microscaling Formats (MX) Specification version 1.0 (September 2023) [11] and JEDEC LPDDR6 (JESD209-6, 9 July 2025) [21] Sourced. LPDDR5/5X is JESD209-5C (revision seen on JEDEC's site)	MX is an industry specification with no legal force, and adoption varies by product. JEDEC standards are for components, not for device AI performance
Future products and features	The technical report on Apple's third-generation models, promised for "later this summer" [3] Not yet confirmed	None found	A company announcement; whether it has been published and its content are unconfirmed

Category tags appear in each cell. MLPerf versions and benchmark mixes are as displayed on the MLCommons pages (checked 15 October 2026). ISO/IEC titles and editions were confirmed on ISO's list for SC 42. The arrangement of the table is our own.

Three points on how to read these.

TOPS is only an entry number: read a peak figure with unknown precision, sparsity and measurement together with memory bandwidth, model size and power. TOPS alone does not decide device speed (Chapter 4).
Benchmarks change with version, model and backend: MLPerf Mobile and Client swap models and tasks by version, and the choice of execution backend (NPU, GPU or CPU) affects results. Numbers from old versions or different settings are not lined up to rank anyone.
Standards mostly cover terminology, overviews, safety and data quality: within this article's research no international standard setting device AI performance as a requirement was found. ISO/IEC technical reports are informative and not legally binding, and product requirements such as Copilot+ PC are company arrangements.

11. Glossary

Edge AI / on-device AI: Running AI inference inside a device or on-site equipment instead of the cloud.

NPU: Neural processing unit: a compute block specialized for neural-network operations.

TOPS: Tera operations per second: a peak figure for trillions of operations per second.

Sparsity: Speed-up from some values being zero; can double a stated TOPS figure.

Quantization: Representing weights and the like with fewer bits.

QAT: Quantization-aware training: accounting for quantization during training.

KV cache: Stored keys and values of past tokens.

LPDDR: A family of low-power DRAM standards for mobile and PCs.

NAND (flash): Large, slow non-volatile memory used as device storage.

TTFT / TPS: Time to first token / tokens per second.

MX (Microscaling): An OCP specification of number formats with a shared per-block scale and low-bit elements.

Functional safety: Managing how equipment faults and malfunctions affect safety.

12. References

Microsoft "Windows 11 Specs and System Requirements" (Copilot+ PC minimum requirements: an NPU of 40+ TOPS, 16 GB of DDR5/LPDDR5, a 256 GB SSD/UFS, and the processor families; a note that requirements may change; checked 15 October 2026) — microsoft.com/windows/windows-11-specifications
Qualcomm "Snapdragon X2 Elite" product page (table of NPU TOPS and LPDDR5x memory bandwidth for each Snapdragon X, X Plus, X Elite and X2 part number; checked 15 October 2026) — qualcomm.com/laptops/products/snapdragon-x2-elite
Apple Machine Learning Research "Introducing the Third Generation of Apple's Foundation Models" (8 June 2026, updated 9 September 2026; AFM 3 Core and Core Advanced, experts loaded from flash (NAND) into DRAM, Private Cloud Compute, announcement of a technical report) — machinelearning.apple.com/research/introducing-third-gene...
Apple Machine Learning Research "Updates to Apple's On-Device and Server Foundation Language Models" (9 June 2025, updated 17 July 2025; about 3 billion parameters, 2-bit QAT, privacy policy) — machinelearning.apple.com/research/apple-foundation-model...
Li E. et al. (Apple) "Apple Intelligence Foundation Language Models: Tech Report 2025", arXiv 2507.13575 (KV-cache sharing for 37.5% less, 2-bit QAT, 4-bit embeddings, 8-bit KV, low-rank adapters) — arxiv.org/abs/2507.13575
Abdin M. et al. (Microsoft) "Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone", arXiv 2404.14219 (phi-3-mini 3.8B, about 1.8 GB at 4 bits, over 12 tokens/s on an iPhone 14) — arxiv.org/abs/2404.14219
Gemma Team (Google) "Gemma 3 Technical Report", arXiv 2503.19786 (1 to 27B; KV-cache reduction via the ratio of local to global attention) — arxiv.org/abs/2503.19786
Liu Z. et al. (Meta) "MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases", arXiv 2402.14905 — arxiv.org/abs/2402.14905
Frantar E. et al. "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers", arXiv 2210.17323 — arxiv.org/abs/2210.17323
Lin J. et al. "AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration", arXiv 2306.00978 — arxiv.org/abs/2306.00978
Open Compute Project "OCP Microscaling Formats (MX) Specification", Version 1.0 (September 2023) (jointly by AMD, Arm, Intel, Meta, Microsoft, NVIDIA and Qualcomm; FP8, FP6, FP4, INT8) — opencompute.org/documents/ocp-microscaling-formats-mx-v1-...
Arm "Arm Ethos-U85" product page (128 to 2,048 MACs, up to 4 TOPS at 1 GHz, transformer support, about 20% more energy efficient than the previous generation, industrial IoT; checked 15 October 2026) — arm.com/products/silicon-ip-cpu/ethos/ethos-u85
NVIDIA "Jetson Orin" page (AGX Orin: 275 TOPS [sparse INT8], 15 to 60 W, 204.8 GB/s; Orin Nano bandwidths of 51 and 102 GB/s; checked 15 October 2026) — nvidia.com/en-us/autonomous-machines/embedded-systems/jet...
Hailo "Hailo-8 AI Accelerator" product page (up to 26 TOPS, typical power 2.5 W, M.2 and PCIe; checked 15 October 2026) — hailo.ai/products/ai-accelerators/hailo-8-ai-accelerator/
MLCommons "MLPerf Mobile" (Inference: Mobile) benchmark page (most tasks at v6.0; LLM, image and detection tasks; accuracy targets; single stream; checked 15 October 2026) — mlcommons.org/benchmarks/inference-mobile/
MLCommons "MLPerf Client" benchmark page (v2.0; LLM, image generation and agentic tasks; TTFT and TPS; supported operating systems; checked 15 October 2026) — mlcommons.org/benchmarks/client/
MLCommons "MLPerf Tiny" benchmark page (v1.3 / v1.4; five tasks; latency and optional energy; checked 15 October 2026) — mlcommons.org/benchmarks/inference-tiny/
Reddi V. J. et al. "MLPerf Mobile Inference Benchmark", arXiv 2012.02328 (2020) — arxiv.org/abs/2012.02328
Banbury C. et al. "MLPerf Tiny Benchmark", arXiv 2106.07597 (2021) — arxiv.org/abs/2106.07597
ISO/IEC JTC 1/SC 42 List of standards of SC 42 "Artificial intelligence" (ISO/IEC TR 17903:2024, TR 5469:2024, TS 4213:2022, the 5259 series, 24029, 25059:2023, TS 42119-2:2025, TR 42106:2026, 22989:2022 and others; checked 15 October 2026) — iso.org/committee/6794475/x/catalogue/p/1/u/0/w/0/d/0
JEDEC "JEDEC Releases New LPDDR6 Standard to Enhance Mobile and AI Memory Performance" (JESD209-6, 9 July 2025) — jedec.org/news/pressreleases/jedec-releases-new-lpddr6-st...
Grattafiori A. et al. (Meta) "The Llama 3 Herd of Models", arXiv 2407.21783 (Table 3: the 8B model has 32 layers, 8 key/value heads, 32 heads and model dimension 4,096) — arxiv.org/abs/2407.21783
MLCommons "MLCommons Sets Participation Record with New MLPerf Inference v6.1 Benchmark Results" (announcement of 16 September 2026; the added edge agentic test; syndicated GlobeNewswire page) — financialcontent.com/article/gnwcq-2026-9-16-mlcommons-se...

13. Claim-to-source audit

Claim in the text	Basis	Category
Copilot+ PC minimum requirements (an NPU of 40+ TOPS, 16 GB of DDR5/LPDDR5, a 256 GB SSD/UFS), the processor families, and the note that requirements may change	Reference 1 https://www.microsoft.com/windows/windows-11-specifications	Sourced
Snapdragon X (up to 45 TOPS, up to 135 GB/s), X2 Plus and X2 Elite (80 to 85 TOPS, up to 152 to 228 GB/s) values per part number; the TOPS footnote being absent from the text obtained	Reference 2 https://www.qualcomm.com/laptops/products/snapdragon-x2-elite	Sourced
Apple's third-generation models (AFM 3 Core and Core Advanced: 3 billion dense, 20 billion sparse using 1 to 4B), whole model in flash and needed experts loaded into DRAM, privacy policy and Private Cloud Compute, and the technical-report notice (8 June 2026)	The technical report is an announcement (not yet confirmed) Reference 3 https://machinelearning.apple.com/research/introducing-third-generation-of-apple-foundation-models	Not yet confirmed
Apple's 2025 on-device model (about 3 billion parameters, 2-bit QAT) and its privacy statements	Reference 4 https://machinelearning.apple.com/research/apple-foundation-models-2025-updates	Sourced
KV-cache sharing (last 37.5% of layers) giving 37.5% less KV memory and about 37.5% lower TTFT, 2-bit QAT, 4-bit embeddings, 8-bit KV, low-rank adapters	Reference 5 https://arxiv.org/abs/2507.13575	Sourced
phi-3-mini: 3.8B, 3.3 trillion tokens, about 1.8 GB at 4 bits, over 12 tokens/s on an iPhone 14 (A16)	Reference 6 https://arxiv.org/abs/2404.14219	Sourced
Gemma 3: 1 to 27B, KV-cache reduction by changing the ratio of local to global attention	Reference 7 https://arxiv.org/abs/2503.19786	Sourced
MobileLLM: deep and thin architectures, embedding sharing and grouped-query attention for sub-billion models	Reference 8 https://arxiv.org/abs/2402.14905	Sourced
GPTQ (post-training quantization) and AWQ (activation-aware weight quantization)	Reference 9 https://arxiv.org/abs/2210.17323 Reference 10 https://arxiv.org/abs/2306.00978	Sourced
OCP MX specification version 1.0 (September 2023), the seven contributing companies, FP8, FP6, FP4, INT8	Reference 11 https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf	Sourced
Arm Ethos-U85: 128 to 2,048 MACs, up to 4 TOPS at 1 GHz, transformer support, about 20% more energy efficient than the previous generation, industrial IoT	Reference 12 https://www.arm.com/products/silicon-ip-cpu/ethos/ethos-u85	Sourced
Jetson AGX Orin: up to 275 TOPS (sparse INT8), 15 to 60 W, 64 GB 256-bit LPDDR5, 204.8 GB/s, and Orin Nano bandwidths	Reference 13 https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/	Sourced
Hailo-8: up to 26 TOPS, typical power 2.5 W, M.2 and PCIe, no TOPS footnote	Reference 14 https://hailo.ai/products/ai-accelerators/hailo-8-ai-accelerator/	Sourced
MLPerf Mobile: most tasks at v6.0, LLM (Llama 3.1 8B, Llama 3.2 3B), image and detection tasks, accuracy targets, single stream	Reference 15 https://mlcommons.org/benchmarks/inference-mobile/	Sourced
MLPerf Client v2.0: models (Llama 3.1 8B and others), TTFT and TPS, supported operating systems	Reference 16 https://mlcommons.org/benchmarks/client/	Sourced
MLPerf Tiny: v1.3 / v1.4 (inconsistent on the page), five tasks, latency and optional energy	Reference 17 https://mlcommons.org/benchmarks/inference-tiny/	Sourced
The design papers of MLPerf Mobile and Tiny	Reference 18 https://arxiv.org/abs/2012.02328 Reference 19 https://arxiv.org/abs/2106.07597	Sourced
Titles, editions and statuses of ISO/IEC TR 17903:2024, TR 5469:2024, TS 4213:2022, the 5259 series, 24029, 25059:2023, TS 42119-2:2025, TR 42106:2026 and 22989:2022 (SC 42 list)	Reference 20 https://www.iso.org/committee/6794475/x/catalogue/p/1/u/0/w/0/d/0	Sourced
The structure and target applications of JEDEC LPDDR6 (JESD209-6, 9 July 2025)	Reference 21 https://www.jedec.org/news/pressreleases/jedec-releases-new-lpddr6-standard-enhance-mobile-and-ai-memory-performance	Sourced
Llama 3 8B configuration (32 layers, 8 key/value heads, head dimension 128)	Reference 22 https://arxiv.org/abs/2407.21783	Sourced
MLPerf Inference v6.1 adding an edge agentic test	Reference 23 https://www.financialcontent.com/article/gnwcq-2026-9-16-mlcommons-sets-participation-record-with-new-mlperf-inference-v61-benchmark-results	Sourced
The weight-size table (parameters x bits / 8), ceilings on tokens/s from memory bandwidth (8B at 4 bits plus an 8-bit KV at 8,192 tokens = about 4.54 GB, about 30 at 135 GB/s and 50 at 228 GB/s), the ratio to the compute-side ceiling (about 95x), the back-calculation from the Phi-3 report (about 22 GB/s), the size of Apple's 20-billion-parameter model at assumed bit widths (10 GB and 5 GB), and energy per token (an assumption of 8 W and 25 tokens/s)	Our calculation. Ceiling estimates from memory bandwidth and weight size with the assumptions stated (nothing reserved, full bandwidth use, bit widths, 8 W and 25 tokens/s). Not predictions for real devices	Our calculation
The readings "TOPS is only an entry number", "on-device AI is a problem of memory and storage materials and packaging", "three cautions for on-site use" and "what on-device processing means and its limits"; the view that memory demand may separate price tiers (inference)	This article's organization and commentary based on published material, not views stated by the companies	Commentary
Real-device speed and power, the measurement conditions (footnotes) of TOPS, Apple's third-generation technical report, texts of standards, and the LPDDR5X speed grades	Not confirmed within this article's research and therefore not given; this article's note	Commentary

Last updated: 15 October 2026. Sources are limited to primary material (official pages of Microsoft, Qualcomm, Arm, NVIDIA, Hailo and Apple, papers such as Phi-3, Gemma 3 and MobileLLM, the OCP specification, MLCommons pages, the ISO standards list and a JEDEC announcement). Real-device speed and power, the measurement conditions of TOPS, Apple's third-generation technical report and the texts of standards could not be confirmed and are not given. Future products and features are "not yet confirmed". Figures 2 and 5 are vector drawings, Figures 3 and 4 include our calculations, and Figures 1 and 6 are AI-image placeholders only.