AI test

The AI test measures local large language model inference on your CPU, GPU, or NPU. It reports results for each tested model and hardware combination.

Use this test to compare supported hardware on one device. You can also compare devices that use the same model, runtime, and test version.

Hardware and runtime support

Novabench shows only the combinations that your platform supports.

The AI test is not available on Intel-based Macs or on Linux Arm64.

Supported NPU paths have additional model requirements:

All NPU paths run models of approximately 4B parameters or less. Larger models run only on a CPU or a GPU.

Apple Neural Engine does not produce an AI test result at this time.

Models

The AI test includes these default models:

The download sizes above are for the GGUF files that the CPU and GPU paths use. An NPU path uses a different file format for the same model, and its download size is different. The Ryzen AI file for Phi-4 Mini is approximately 8.7 GB and needs approximately 10 GB of memory.

Model support varies by hardware path. The test skips a combination when the model format, runtime, or available memory is not compatible.

Add a model

You can add a model from a public Hugging Face repository URL, or from a direct HTTPS link to one GGUF file.

1. Select Add model from URL.
2. Paste the URL.
3. Select Find models. Novabench reads the available files and shows the compatible model artifacts.
4. Select an artifact, then select Add.

These conditions apply:

• A direct link must point at a .gguf file, and the file must not be one shard of a multi-file GGUF.
• A Hugging Face repository can also supply an OpenVINO or Ryzen AI artifact. Novabench rejects an artifact when your platform has no compatible runtime for it.
• Novabench rejects a model whose architecture the AI runtime does not support yet.
• The added model remains in your model list until you delete it.

Configure the test

1. Open the AI Tests page.
2. Expand a model row.
3. Select the hardware targets and runtimes that you want to compare.
4. If you want to include the larger model, enable it.
5. If you want more measurements, set Iterations to a value from 1 to 10. Each iteration repeats the full set of cells. The default is one iteration.
6. Configure the result name, identifiers, tags, and submission options.
7. Start the test.

See result management for identifier and tag options.

Downloads and storage

Novabench downloads each required model or runtime before its first use. Later tests reuse the downloaded files.

Select Manage test models & runtimes on the AI Tests page to open the Manage Resources window. Use it to view or delete downloaded models and supported runtime resources.

Large models need more download time, storage, and memory. Review the model size before you enable it on several devices.

What the test measures

Each valid model, hardware, and runtime combination produces one result cell. A cell can include:

• Prefill (prompt processing): the rate at which the model reads an input prompt
• Decode (generation): the rate at which the model produces new tokens
• Decode at 4K context: decode speed with 4,096 tokens already in the context
• Time to first token: how long the model takes to read the test prompt before it can start to write
• Power use: idle power and power while generating, when the device reports supported power data

GPU cells include all measurements. CPU cells include the 4K context measurement for models of 4B parameters or less. NPU cells do not include the 4K context measurement at this time.

The test runs one cell at a time. You can cancel the test while downloads or measurements are in progress.

Read the results

Metrics

A token is a small piece of text. In English, a token is about 3/4 of a word on average.

For CPU and GPU cells, time to first token is calculated from the prefill speed of the 512-token prompt. NPU runtimes measure it with their own fixed prompt. The value does not include the time to load the model.

A value with ± shows how much the measurement changed between repeated runs. A smaller value means a more consistent result.

Select the info icon next to a metric on the result page to see a short explanation.

Per-model results

The results group cells by model. Each model section includes:

• Best result: the metrics for the fastest device and runtime for that model
• Comparison: a summary when the model ran on both a CPU and a GPU, or on two runtimes on the same device. For example, GPU decodes 2.3× faster than CPU.
• Device table: the metrics for each device and runtime that ran the model. A bar shows the relative decode speed.

A skipped or failed cell shows its reason. Common reasons include insufficient memory, an unavailable runtime, or an incompatible model format.

Quantization

Each model section shows the quantization label of the model, for example Q4_K_M or INT4. Quantization is the numeric precision used to store the model weights. Lower precision uses less memory and usually runs faster, but it can reduce output quality. At 4-bit precision the quality loss is usually small. Below 4-bit, the loss becomes more noticeable.

Energy efficiency

When the device reports supported power data, the result ranks the completed cells by energy efficiency.

NPU figures are estimated from total package power. See the AI benchmark methodology for more information.

Sensor charts (Plus)

Novabench records temperature, power, and usage data during the measurement period on every plan, and uses the power data for the energy-efficiency figures. Novabench Plus charts that data on the result page.

The AI test does not produce a composite score. Compare the individual metrics that apply to your workload.

Compare results

Use the same model, quantization, runtime, and test version for a direct comparison. A different model format or runtime can change the result.

CPU and GPU results use llama.cpp. NPU results use the supported runtime for each NPU vendor.

Submit and share results

Results always save on the device. Submission is on by default:

• Signed in: the result syncs to your Novabench account.
• Signed out: the result submits without account information.

You can turn submission off before each test. A submitted result gets a web page that you can share.

Related pages

• AI benchmark overview
• AI benchmark methodology
• Understanding your scores