AI benchmark methodology

The AI benchmark measures local large language model inference on a CPU, GPU, or NPU. It reports separate results for each model and hardware combination.

The test uses the compatible runtime for each hardware target. CPU and GPU tests use llama.cpp. NPU tests use the supported vendor runtime.

What the test measures

Prompt processing (prefill)

Prompt processing measures how fast the model reads an input prompt. This measurement helps you compare workloads that use long documents or code files. The result page shows it as Prefill.

Token generation (decode)

Token generation measures how fast the model produces new tokens. Generation speed often depends on memory bandwidth and the model size. The result page shows it as Decode.

A token is a small piece of text. In English, a token is about 3/4 of a word on average.

Long-context generation (decode at 4K context)

Long-context generation measures generation speed with 4,096 tokens already in the context. This measurement represents a longer conversation or document session. Generation speed usually decreases as the context grows.

GPU cells include this measurement. CPU cells include it for models of 4B parameters or less. NPU cells do not include it at this time.

Time to first token

Time to first token measures how long the model takes to read the test prompt before it can start to write. A lower value gives a faster initial response.

For CPU and GPU cells, Novabench calculates time to first token from the prompt-processing speed of the 512-token prompt. NPU runtimes measure it with their own fixed prompt. The value does not include the time to load the model.

Energy efficiency

When the device reports supported power data, Novabench records power use during the test.

• Idle power: the average power for approximately 2.5 seconds before the first cell runs, with no AI work
• Power while generating: the average power during the token-generation workload
• Efficiency: generation tokens per second for each watt above idle power
• Efficiency including idle: generation tokens per second for each watt of total power

GPU cells use GPU power. CPU cells use CPU package power. NPU figures are estimated from total package power, because NPUs do not report their own power.

The result does not show efficiency when there are too few power samples, or when the power above idle is too small to measure reliably.

How the benchmark runs

The benchmark creates one cell for each selected model, hardware target, and runtime combination.

CPU and GPU cells use these workloads:

• A 512-token prompt-processing workload
• A 128-token generation workload
• A 128-token generation workload with 4,096 tokens of context

The test does warm-up work before it records results. It then repeats each workload and reports an average. The prompt-processing and generation workloads run five repetitions each, and the long-context workload runs two.

You can also set an iteration count from 1 to 10. Each iteration repeats the full set of cells, and the result reports the mean across the iterations. The default is one iteration.

NPU cells use a fixed prompt that is compatible with the selected vendor runtime. The exact prompt length can differ between NPU platforms.

Each cell has a time limit. If a cell fails or exceeds this limit, the benchmark records the failure and continues to the next cell.

Sensor collection occurs during the measurement period. Downloads and idle periods do not contribute to the sensor charts.

Models and formats

Novabench includes three default models:

• Phi-4 Mini 3.8B: a small model that supports the widest range of hardware targets
• Gemma 4 12B: a medium model that requires more memory
• Gemma 4 31B: a large model that is off by default

CPU, GPU, and NPU runtimes can require different model formats. As a result, one model might be available on a CPU but unavailable on an NPU.

NPU paths also run models of approximately 4B parameters or less. Of the default models, only Phi-4 Mini 3.8B runs on an NPU.

The test keeps model and runtime versions consistent for comparable runs. A product update can add a model, runtime, or supported hardware path.

Runtime support

Compatible NVIDIA GPUs can offer Vulkan as an alternative to CUDA. The test does not include a ROCm path on any platform. A ROCm entry can appear for a discrete AMD GPU, but it reports that no runtime is available.

Supported NPU paths use OpenVINO on Intel, Ryzen AI on supported AMD devices, and Hexagon on Snapdragon devices.

Apple Neural Engine does not produce a result at this time.

Compare results

Use the same model, quantization, runtime, and test version for a direct comparison. Differences in these items can change prompt speed and generation speed.

CPU and GPU results can be compared with llama.cpp results that use the same model and test settings.

NPU comparisons require the same vendor runtime and model format. A CPU-to-NPU comparison can include both hardware and model-format differences.

The AI benchmark does not produce a composite score. Review the measurements that match your expected workload.

Factors that affect results

Temperature

Long generation workloads heat the active processor. Let the device cool before you compare runs that occurred under different conditions.

Memory bandwidth

Generation speed often depends on memory bandwidth. Two processors with similar compute performance can have different generation speeds.

Drivers

GPU and NPU drivers affect runtime performance and hardware support. Install the recommended driver for the device before you compare results.

Background load

Other applications can use processor time, memory bandwidth, or GPU resources during the test. Close unnecessary applications before you start a comparison.

Power state

Battery operation and efficiency power plans can reduce performance. Use the same power source and power plan for comparable runs.

Related pages

• AI benchmark overview
• AI test
• Benchmark methodology
• Understanding your scores