Intel OpenVINO Export
OpenVINO is Intel's open-source toolkit for optimizing and running AI inference. Exporting an Ultralytics YOLO26 model with format="openvino" creates a yolo26n_openvino_model/ directory that holds an OpenVINO IR model (.xml and .bin files) for Intel CPUs, integrated and discrete Intel GPUs, and Intel NPUs, in FP32, FP16, or INT8 precision.
At a Glance#
| Property | Details |
|---|---|
format argument | openvino |
| Output | yolo26n_openvino_model/, a directory with yolo26n.xml, yolo26n.bin, and metadata.yaml; yolo26n_int8_openvino_model/ for a quantize=8 export |
| Runs on | Intel CPUs, integrated and discrete Intel GPUs, and Intel NPUs. The CPU device also runs on other x86-64 and ARM64 processors, such as AMD and Apple silicon |
| Runtime | OpenVINO Runtime 2024.0.0 or newer (the openvino Python package) |
| Precision | FP32, FP16, INT8. YOLO26n is 10.1 MB in FP32, 5.3 MB in FP16, and 3.3 MB in INT8 |
| Export host | Linux, macOS, or Windows. The export runs on the CPU and needs no GPU |
Ultralytics Platform can export a trained model to OpenVINO for you, so you do not need a local export environment.
Quickstart#
Install the Ultralytics package with the OpenVINO export dependencies:
pip install "ultralytics[export-openvino]"This installs openvino and, on Python 3.9 or newer, nncf for INT8 quantization. With a plain pip install ultralytics, the first export installs the missing packages. The export has these version requirements:
| Package | Required version |
|---|---|
torch | 2.1 or newer |
openvino | 2024.0.0 or newer; 2025.2.0 or newer on macOS 15.4 and later |
nncf | 2.14.0 or newer, for quantize=8 only; below 3.0.0 with PyTorch below 2.3 |
For other installation methods, see the installation guide. Then export your model and load the exported directory to run inference or validate its accuracy. The OpenVINO on CPU course walks through the same steps.
from ultralytics import YOLO
# Load a YOLO26 model
model = YOLO("yolo26n.pt")
# Export the model to OpenVINO format
model.export(format="openvino") # creates 'yolo26n_openvino_model/'
# Export an INT8-quantized OpenVINO model with calibration data
model.export(format="openvino", quantize=8, data="coco8.yaml") # creates 'yolo26n_int8_openvino_model/'from ultralytics import YOLO
# Load the exported OpenVINO model
model = YOLO("yolo26n_openvino_model/")
# Run inference
results = model("https://ultralytics.com/images/bus.jpg")
# Run inference on the Intel GPU; use "npu" for the Intel NPU or "cpu" for the CPU
results = model("https://ultralytics.com/images/bus.jpg", device="xpu")from ultralytics import YOLO
# Load the exported OpenVINO model
model = YOLO("yolo26n_openvino_model/")
# Validate accuracy on the COCO8 dataset
metrics = model.val(data="coco8.yaml")Supported Tasks#
OpenVINO export supports all seven Ultralytics tasks with YOLOv8, YOLO11, and YOLO26 models: detection, segmentation, semantic segmentation, depth estimation, classification, pose estimation, and OBB. Semantic segmentation and depth estimation are available only with YOLO26, the only family that ships those heads.
Export Arguments#
| Argument | Type | Default | Description |
|---|---|---|---|
format | str | 'openvino' | Target format for the exported model, defining compatibility with various deployment environments. |
imgsz | int or tuple | 640 | Image size of the model input: an integer for square images or a tuple (height, width). When not passed, the export uses the size the checkpoint was trained at. The official YOLO26 checkpoints use 640 for detect, segment, pose, and semantic models pretrained on ADE20K, 1024 for OBB and semantic models pretrained on Cityscapes, 768 for depth, and 224 for classify. |
quantize | int or str | None | Quantization precision: 16 (FP16: weights are stored in half precision, inputs and outputs stay FP32) or 8 (INT8 post-training quantization with NNCF using calibration images from data, written to an _int8_openvino_model/ directory); 32/unset is FP32. A checkpoint trained with quantize=8 cannot be exported to OpenVINO, see Limitations and Troubleshooting. Replaces the deprecated half/int8 flags. |
data | str | None | Dataset YAML used for INT8 calibration; classification instead takes a dataset directory or a built-in dataset name. Calibration uses the val split unless split is set. If omitted with quantize=8, Ultralytics uses the small default dataset for the model task, such as coco8.yaml for detection. |
fraction | float, int, or list | 1.0 | Calibration subset as a ratio, image count, or [train, val, test] ratios/counts. Two-item lists leave test full, while 0 skips it. |
dynamic | bool | False | Makes the batch, height, and width of the input dynamic, so one model accepts different batch sizes and image sizes. On an Intel NPU a dynamic model still runs at the export imgsz, see Deploy. |
nms | bool, optional | None | Select raw output (None, default), embedded NMS (True), or the NMS-free head (False). See output format. |
batch | int | 1 | Batch size of the exported model. A static export has this batch size fixed in the input shape. Also the batch size used for INT8 calibration. |
device | str | None | The export always runs on the CPU. With quantize=8, device=npu selects symmetric INT8 activations for YOLO detect, segment, pose, and OBB models, which run up to 2x faster on Intel NPUs. |
For the arguments shared by all formats, see the Export mode documentation.
INT8 Quantization#
quantize=8 applies post-training quantization with NNCF: the export runs calibration images through the model and stores weights and activations as 8-bit integers where it can.
- Calibration data: the export calibrates on every full batch of the
valsplit ofdata, or of the split you set withsplit. Usefractionto limit the number of images. The export warns below 300 images. - Detection head: for models with a YOLO detection head (detect, segment, pose, and OBB) and for RT-DETR, the operations that decode the head outputs (reshape, concatenation, box decoding, and DFL) and all sigmoid operations stay in floating point, while the convolutions of the head are quantized. Classification, semantic segmentation, and depth models have no excluded operations.
- Inputs and outputs: an INT8 model keeps the
float32input and outputs of the FP32 model, so inference code written for the FP32 model runs it unchanged. - Intel NPU: add
device=nputo an INT8 export that you plan to run on an Intel NPU. It switches YOLO detect, segment, pose, and OBB models from asymmetric to symmetric activation quantization.
from ultralytics import YOLO
model = YOLO("yolo26n.pt")
# Calibrate on 10% of the COCO validation images (500 images); replace coco.yaml with your own dataset
model.export(format="openvino", quantize=8, data="coco.yaml", fraction=0.1)
# Load the exported INT8 model and run inference
model = YOLO("yolo26n_int8_openvino_model/")
results = model("https://ultralytics.com/images/bus.jpg")In the benchmarks, INT8 models ran 1.02 to 1.33 times faster than FP32 on a server CPU and lost 0.5 to 0.9 mAP on COCO val2017. On Intel Core Ultra CPUs INT8 ran 1.2 to 3.8 times faster than FP32. Validate an INT8 model on your own data before you deploy it.
Output Format#
The export writes one directory with three files:
| File | Contents |
|---|---|
yolo26n.xml | The network graph in OpenVINO IR format |
yolo26n.bin | The weights |
metadata.yaml | Class names, task, image size, batch size, stride, and export arguments, which Ultralytics reads when it loads the model |
The model takes one input with shape (batch, 3, height, width), RGB and scaled to 0-1. The input and the outputs are float32 in FP32, FP16, and INT8 models, except the semantic segmentation output, a uint8 class map (int32 for models with more than 256 classes). The tensors do not carry the images and output0 names of the ONNX export, so address the input and the outputs by index in your own code.
The output shapes are the same as for the ONNX export and depend on the task and the nms argument: a YOLO26n detection model exported at imgsz=640 returns (1, 84, 8400) with the default nms=None and (1, 300, 6) with nms=False or nms=True. See the ONNX output format for the shapes and contents of every task. With nms=True, the confidence threshold conf (default 0.25), the IoU threshold iou, agnostic_nms, and max_det are fixed in the graph at export. A lower conf at prediction or validation time cannot bring back detections the graph already removed, so export again to change these values.
Deploy#
YOLO("yolo26n_openvino_model/") runs the model with the OpenVINO Runtime and maps the device argument onto an OpenVINO device:
device | OpenVINO device |
|---|---|
| unset | AUTO, which lets OpenVINO choose between the CPU and Intel GPUs (its default priority list excludes the NPU), or CPU when the CPU is the only device OpenVINO finds |
cpu | CPU |
xpu or xpu:0 | GPU or GPU.0, the Intel integrated or discrete GPU. xpu:1 selects GPU.1 on a machine with two Intel GPUs |
npu | NPU, the neural processing unit of Intel Core Ultra processors |
Other values, such as 0 or mps, are handled like an unset device. The older intel:cpu, intel:gpu, and intel:npu names still work and log a deprecation warning. The inference devices table compares these names across formats. The log line Using OpenVINO LATENCY mode for batch=1 inference on CPU... names the device that OpenVINO reports when the model loads. With AUTO this can be the CPU while an Intel GPU is still compiling the model, so set device to be sure. For a .pt model, device="xpu" selects the PyTorch Intel GPU backend instead, which is also the device for training on Intel GPUs, and device="npu" selects a Huawei Ascend NPU, not the Intel NPU.
The behavior at inference time depends on how the model was exported:
- Static export (default): the model always runs at its export
imgszandbatch, which Ultralytics reads frommetadata.yaml. Animgszpassed topredictis ignored. Larger batches are split into chunks of the export batch size and smaller ones are padded, so any number of images works. - Dynamic export (
dynamic=True):predictaccepts different batch sizes and everyimgszthat is a multiple of the model stride. Intel NPUs and INT8 models on Intel AMX CPUs are exceptions, see Limitations and Troubleshooting. - Latency mode: Ultralytics compiles the model with the OpenVINO
LATENCYperformance hint and runs inference synchronously. To serve many parallel requests with theTHROUGHPUThint, use the OpenVINO Runtime API as shown in the latency and throughput guide. - Intel NPU: Ultralytics compiles the model for batch size 1 on an NPU and splits larger batches into single images. A dynamic export is fixed to its export
imgszon an NPU.
Run with the OpenVINO Runtime API#
To run the model outside the Ultralytics package, load the .xml file with the openvino package. The model expects a preprocessed image and returns the raw outputs described in Output Format, so letterboxing and, for a default export, NMS are up to your code:
# Export first: yolo export model=yolo26n.pt format=openvino nms=False
import numpy as np
import openvino as ov
import yaml
core = ov.Core()
model = core.read_model("yolo26n_openvino_model/yolo26n.xml")
compiled = core.compile_model(model, "AUTO") # or "CPU", "GPU", "NPU"
image = np.zeros((1, 3, 640, 640), dtype=np.float32) # your preprocessed image: RGB, NCHW, 0-1
detections = compiled(image)[0][0] # first output, first image: shape (300, 6)
detections = detections[detections[:, 4] > 0.25] # keep rows above the confidence threshold
with open("yolo26n_openvino_model/metadata.yaml") as f:
names = yaml.safe_load(f)["names"] # {0: 'person', 1: 'bicycle', ...}Deployment Guides#
- OpenVINO latency and throughput modes: choose between the lowest latency for one stream and the highest throughput for many.
- Intel DL Streamer: run exported models in GStreamer video analytics pipelines on Intel Core Ultra CPUs, GPUs, and NPUs.
- OpenVINO inference devices: the CPU, GPU, and NPU devices and the
AUTOmode in the OpenVINO documentation. - OpenVINO notebooks: Intel's tutorial notebooks, including YOLO26 optimization.
Limitations and Troubleshooting#
- PyTorch version: the export fails with
OpenVINO export requires torch>=2.1on older PyTorch versions. - macOS 15.4 and later: OpenVINO 2025.1.0 and older fail on these macOS versions, so the export requires
openvino>=2025.2.0there and updates an older package. - Checkpoints trained with
quantize=8: a checkpoint fine-tuned with quantization-aware training fails withformat='openvino' cannot export a QAT model: its Q/DQ ranges are only read by the 'onnx' and 'engine' backends. Export the checkpoint you started the fine-tuning from withquantize=8instead, or export the fine-tuned checkpoint to ONNX or TensorRT. - INT8 export without
data: the export logsINT8 export requires a missing 'data' arg for calibration. Using default 'data=coco8.yaml'.and calibrates on the small default dataset for the task. Passdatawith images that represent your deployment data. - INT8 calibration size: the export logs
>300 images recommended for INT8 calibration, found 4 images.when the calibration split has fewer than 300 images.coco8.yamlhas only 4 validation images, which is enough to test the export but not to calibrate a model for deployment. - Output directory: every export of one checkpoint other than INT8 writes the same
yolo26n_openvino_model/directory, whatever itsquantize,imgsz,dynamic,nms, orbatch, so the later export replaces the earlier one. All INT8 exports, with or withoutdevice=npu, shareyolo26n_int8_openvino_model/. - Device not available: when the requested device is missing, Ultralytics logs
OpenVINO device 'GPU' not available. Using 'AUTO' instead.(orUsing 'CPU' instead.on a machine where OpenVINO finds only the CPU) and continues on that device. Check that your processor is supported by OpenVINO and install the Intel GPU or NPU driver. - Intel NPU: Ultralytics runs an NPU at batch size 1 with fixed input shapes, see Deploy. Pass
device=nputo use it: an unsetdevicedoes not select the NPU. - Dynamic shapes with NMS: an export with
dynamic=Trueandnms=Truelogs'dynamic=True' export requires a maximum batch size, e.g. 'batch=16'.whenbatchis left at1. Ultralytics runs larger batches in chunks of the exportbatch. - Dynamic INT8 models on Intel AMX CPUs: on Linux with a CPU that supports AMX (Intel Sapphire Rapids and newer), Ultralytics runs a dynamic INT8 model on the
CPUorAUTOdevice at its exportimgsz, as it does a static model: animgszpassed topredictorvalis ignored, andvaluses the exportbatch.predictstill accepts other batch sizes and compiles the model again for each new batch size, so the first inference at a new batch size is slower. - Windows: on Windows, Ultralytics sets the CPU inference precision hint to FP32, which stops OpenVINO from using lower-precision floating-point kernels (INT8 models still run in INT8), and limits oneDNN to instruction sets below AMX. Set the
ONEDNN_MAX_CPU_ISAenvironment variable before you load the model to use a different limit. - Linux on ARM64: Ultralytics runs the
CPUdevice in the OpenVINO accuracy execution mode with FP32 inference precision. - Embedded NMS:
nms=Trueraises'nms=True' is not valid for classification models.for classification, and is ignored with a warning for semantic segmentation and depth estimation models ('nms=True' is not valid for semantic models. Forcing 'nms=None'.) and for RT-DETR models ('nms=True' is not available for end2end models or non-YOLO heads. Forcing 'nms=None'.). - NMS-free head:
nms=Falseneeds a model with an NMS-free head, such as YOLO26 or YOLOv10. For other models, for example YOLO11, the export warnsThis model has no one-to-one head; using one-to-many outputs.and writes the raw(1, 84, 8400)output.
For other errors, see the Common Issues guide.
OpenVINO YOLO26 Benchmarks#
The tables below compare PyTorch with FP32, FP16, and INT8 OpenVINO models of every YOLO26 detection model. The first set was measured on a server CPU with COCO val2017 accuracy, and the second set on three Intel Core Ultra processors with their CPU and GPU, and the NPU of two of them. The YOLO26 model page lists the accuracy, CPU ONNX speed, and T4 TensorRT speed of every model.
FP32, FP16, and INT8 on a Server CPU#
All values in this section were measured on the CPU of one server with two AMD EPYC 9655 96-core processors, not on an Intel CPU. PyTorch ran on 8 threads and OpenVINO on 96 threads, the defaults that Ultralytics and OpenVINO chose on that server, so the PyTorch and OpenVINO columns differ in thread count as well as in runtime. PyTorch also validated with rectangular inference, each image padded to a rectangle that follows its aspect ratio (512x672 for a 480x640 image), while the static OpenVINO models always ran at 640x640.
Milliseconds per image, lower is better. The fastest format for each model is in bold.
| Model | imgsz | PyTorch | OpenVINO FP32 | OpenVINO FP16 | OpenVINO INT8 |
|---|---|---|---|---|---|
| YOLO26n | 640 | 13.31 | 3.41 | 3.33 | 3.35 |
| YOLO26s | 640 | 24.28 | 5.18 | 5.18 | 4.51 |
| YOLO26m | 640 | 49.94 | 8.39 | 8.34 | 6.60 |
| YOLO26l | 640 | 63.75 | 10.92 | 11.05 | 8.94 |
| YOLO26x | 640 | 115.11 | 17.12 | 17.16 | 12.83 |
YOLO26n on the same 8 CPUs (PyTorch and OpenVINO both pinned to CPUs 0-7 of that server with taskset):
| Metric | PyTorch | OpenVINO FP32 | OpenVINO FP16 | OpenVINO INT8 |
|---|---|---|---|---|
| Latency (ms/image) | 9.28 | 7.13 | 7.14 | 6.71 |
| mAPval 50-95 | 40.8 | 40.9 | 40.9 | 40.3 |
- OpenVINO FP32 ran 3.9 to 6.7 times faster than PyTorch with the default thread counts above, at the same accuracy within 0.1 mAP. With both pinned to the same 8 CPUs, where PyTorch itself ran faster than with the default setup, OpenVINO FP32 ran YOLO26n 1.3 times faster than PyTorch and INT8 1.06 times faster than FP32.
- FP16 halved the model size within 0.1 mAP of FP32, at the FP32 latency (within 3%).
- INT8 ran 1.02 to 1.33 times faster than FP32, made the model 3.1 to 3.9 times smaller, and lost 0.5 to 0.9 mAP.
Measured with ultralytics 8.4.212, OpenVINO 2026.4.1, NNCF 3.4.0, and PyTorch 2.15.0 at batch=1 on November 14, 2026. Latency is the inference stage that val reports and does not include preprocessing or postprocessing. PyTorch, FP32, and FP16 values are the mean of three runs, which differed by up to 26% for PyTorch and 6% for OpenVINO. INT8 models were calibrated on 591 COCO train2017 images (split=train fraction=0.005) and built three times; INT8 values are the mean of the three builds. The 8-CPU table uses one INT8 build run three times.
Intel Core Ultra CPU, GPU, and NPU#
These tables were measured on three Intel Core Ultra processors with openvino 2026.2.1 in benchmark mode: model.benchmark(data="coco128.yaml", format="openvino", device=...), with device set to cpu, xpu, or npu. Accuracy is mAP50-95 on the 128-image COCO128 dataset, so it is not comparable with the COCO val2017 values above. PyTorch ran on the CPU of the same processor. YOLO26 models run on the NPU of Intel Core Ultra 2xxV series, 3xx series, and newer processors only.
Intel Core Ultra X7 358H#
The Intel Core Ultra X7 358H (Panther Lake) combines CPU cores, an integrated Intel Arc GPU, and an Intel AI Boost NPU.
Milliseconds per image, lower is better. The fastest column for each model is in bold.
| Model | PyTorch CPU | CPU FP32 | CPU FP16 | CPU INT8 | GPU FP32 | GPU FP16 | GPU INT8 | NPU FP32 | NPU FP16 | NPU INT8 |
|---|---|---|---|---|---|---|---|---|---|---|
| YOLO26n | 29.28 | 14.35 | 14.23 | 9.99 | 4.09 | 4.03 | 4.24 | 7.68 | 7.71 | 8.60 |
| YOLO26s | 59.32 | 35.03 | 33.55 | 16.68 | 5.08 | 5.14 | 4.47 | 9.89 | 9.94 | 10.69 |
| YOLO26m | 143.21 | 90.53 | 88.82 | 33.33 | 7.10 | 7.06 | 5.61 | 13.02 | 13.13 | 14.84 |
| YOLO26l | 181.16 | 112.83 | 111.31 | 43.27 | 8.45 | 8.49 | 6.25 | 14.53 | 14.55 | 15.64 |
| YOLO26x | 379.75 | 241.21 | 236.87 | 78.92 | 13.03 | 12.98 | 8.74 | 23.48 | 23.41 | 21.56 |
Earlier Intel Core Ultra Processors#
Intel Core Ultra 7 258V (Lunar Lake)
Milliseconds per image, lower is better. The fastest column for each model is in bold.
| Model | PyTorch CPU | CPU FP32 | CPU FP16 | CPU INT8 | GPU FP32 | GPU FP16 | GPU INT8 | NPU FP32 | NPU FP16 | NPU INT8 |
|---|---|---|---|---|---|---|---|---|---|---|
| YOLO26n | 27.33 | 16.84 | 16.73 | 8.62 | 3.33 | 3.34 | 3.46 | 23.71 | 23.70 | 23.96 |
| YOLO26s | 60.20 | 54.71 | 54.45 | 20.19 | 4.66 | 4.65 | 3.99 | 26.62 | 26.62 | 25.89 |
| YOLO26m | 152.87 | 178.59 | 179.25 | 53.07 | 9.03 | 8.98 | 5.70 | 32.94 | 33.01 | 30.95 |
| YOLO26l | 201.57 | 238.15 | 233.70 | 67.73 | 11.32 | 11.36 | 7.45 | 36.02 | 35.77 | 32.64 |
| YOLO26x | 431.04 | 567.22 | 561.39 | 151.11 | 19.08 | 19.63 | 13.69 | 49.64 | 49.69 | 41.75 |
Intel Core Ultra 7 155H (Meteor Lake)
Milliseconds per image, lower is better. The fastest column for each model is in bold.
| Model | PyTorch CPU | CPU FP32 | CPU FP16 | CPU INT8 | GPU FP32 | GPU FP16 | GPU INT8 |
|---|---|---|---|---|---|---|---|
| YOLO26n | 33.19 | 13.43 | 13.20 | 11.40 | 9.13 | 9.07 | 5.79 |
| YOLO26s | 70.92 | 36.16 | 36.07 | 15.54 | 16.37 | 16.65 | 9.27 |
| YOLO26m | 192.45 | 102.52 | 101.88 | 35.72 | 34.47 | 35.71 | 15.46 |
| YOLO26l | 239.19 | 129.24 | 128.50 | 45.38 | 43.24 | 43.40 | 19.72 |
| YOLO26x | 511.89 | 293.40 | 296.48 | 85.24 | 79.49 | 79.45 | 34.66 |
Across the three processors:
- CPU: INT8 ran 1.2 to 3.8 times faster than FP32 and 2.9 to 6.0 times faster than PyTorch. FP32 ran 1.6 to 2.5 times faster than PyTorch on the X7 358H and the 155H, but slower than PyTorch for YOLO26m, YOLO26l, and YOLO26x on the 258V.
- GPU: the integrated GPU ran FP32 models 3.6 to 29.1 times faster than PyTorch on the CPU. INT8 was faster than FP32 for every model except YOLO26n on the X7 358H and the 258V, by up to 2.3 times.
- NPU (X7 358H and 258V): FP32 models ran 1.2 to 16.2 times faster than PyTorch on the CPU, and INT8 ran 0.88 to 1.19 times as fast as FP32. These INT8 models were exported before
device=npuselected symmetric activations, so the tables do not show the NPU-tuned INT8 export. - FP16 latency was within 5% of FP32 on every device, and INT8 lost 0.1 to 2.7 points of mAP50-95.
About OpenVINO#
OpenVINO, short for Open Visual Inference and Neural Network Optimization, converts models from frameworks such as PyTorch into its IR format and runs them through one API on Intel CPUs, GPUs, and NPUs. For the full reference, see the OpenVINO documentation.
Watch: How to Export Ultralytics YOLO26 to Intel OpenVINO Format for Faster Inference 🚀
FAQ#
Yes, on the CPU. The OpenVINO
CPUdevice runs on x86-64 and ARM64 processors: the server CPU benchmarks on this page were measured on AMD EPYC processors, and the same export runs on Apple silicon. TheGPUandNPUdevices need Intel hardware, sodevice=xpuanddevice=npufall back with a warning on other machines. For other targets, see the model deployment options guide.It depends on the device. FP16 (
quantize=16) needs no calibration data, halves the model size, and kept FP32 accuracy and latency in the benchmarks, so use it when the size of the model matters. INT8 (quantize=8) gave the lowest CPU latency for most models: on the server CPU it reduced YOLO26s latency from 5.18 ms to 4.51 ms per image, at 47.6 mAPval 50-95 against 48.5 in FP32, and on Intel Core Ultra CPUs it ran 1.2 to 3.8 times faster than FP32. On the integrated Intel GPUs the INT8 gain was smaller, up to 2.3 times, and on the NPUs the INT8 models of these runs, exported withoutdevice=npu, were about as fast as FP32. INT8 needs representative calibration data, so validate the INT8 model on your own data before you choose it.Export with
dynamic=Trueand pass the batch andimgszyou need at prediction time. The image size must be a multiple of the model stride, 32 for the official YOLO26 checkpoints other than classification; see Deploy for the exceptions.Dynamic exportfrom ultralytics import YOLO model = YOLO("yolo26n.pt") model.export(format="openvino", dynamic=True) model = YOLO("yolo26n_openvino_model/") results = model(["https://ultralytics.com/images/bus.jpg"] * 4, imgsz=320) # a batch of 4 images at 320 pixelsUse OpenVINO when the deployment hardware is Intel:
device=xpuanddevice=npuselect the Intel GPU and NPU, and INT8 OpenVINO models ran 1.2 to 3.8 times faster than FP32 on Intel Core Ultra CPUs in the benchmarks. Use ONNX when one file must also run on NVIDIA or AMD GPUs or in runtimes other than OpenVINO; on Windows, Windows ML can select Intel hardware for an ONNX model.
Comments23
Ask a question, share your results, or help someone else.
Heyujie96
how can i export a trained cls or obb kind model to openvino kind
EduardoNew
I encountered some difficulties while trying to run the official video demo. Here is my code:
from ultralytics import YOLO
def main(): # Load a YOLOv8n PyTorch model model = YOLO("yolov8n.pt")
# Export the model model.export(format="openvino") # creates 'yolov8n_openvino_model/' # Load the exported OpenVINO model with explicit task definition ov_model = YOLO("yolov8n_openvino_model\\", task='detect') # Explicitly define task as 'detect' # Run inference results_original = model("D:\\python_YOLO_test\\bus.jpg") # Run ov inference results_ov = ov_model("D:\\python_YOLO_test\\bus.jpg")if name == "main": main() Additional notes: My environment is openvino==2024.0.0, and I executed pip install --upgrade torch ultralytics.
The issue I encountered is as follows:
Results saved to D:\python_YOLO_test Ultralytics YOLOv8.2.57 🚀 Python-3.11.9 torch-2.1.1+cu118 CPU (11th Gen Intel Core(TM) i7-11800H 2.30GHz) YOLOv8n summary (fused): 168 layers, 3,151,904 parameters, 0 gradients, 8.7 GFLOPs
PyTorch: starting from 'yolov8n.pt' with input shape (1, 3, 640, 640) BCHW and output shape(s) (1, 84, 8400) (6.2 MB)
OpenVINO: starting export with openvino 2024.0.0-14509-34caeefd078-releases/2024/0... OpenVINO: export success ✅ 2.8s, saved as 'yolov8n_openvino_model' (12.3 MB)
Export complete (5.2s) Results saved to D:\python_YOLO_test Predict: yolo predict task=detect model=yolov8n_openvino_model imgsz=640
Validate: yolo val task=detect model=yolov8n_openvino_model imgsz=640 data=coco.yaml
Visualize: https://netron.appimage 1/1 D:\python_YOLO_test\bus.jpg: 640x480 4 persons, 1 bus, 1 stop sign, 324.0ms Speed: 24.0ms preprocess, 324.0ms inference, 21.0ms postprocess per image at shape (1, 3, 640, 480) Loading yolov8n_openvino_model for OpenVINO inference... Using OpenVINO LATENCY mode for batch=1 inference...
Process finished with exit code -1073741819 (0xC0000005)
Process finished with exit code -1073740791 (0xC0000409) The model was successfully exported but did not perform inference, instead, it terminated directly. How should I resolve this issue?
Glenn JocherUltralytics(edited)@EduardoNew the exit code 0xC0000005 often indicates a memory access violation. Let's address this step-by-step:
-
Path Format: Try using forward slashes in your OpenVINO model path:
"yolov8n_openvino_model/"instead of backslashes. -
OpenVINO Version: Ultralytics requires
openvino>=2024.0.0. Try upgrading to a recent release:
pip install -U openvino- Hardware Compatibility: Ensure you have the latest Intel drivers installed from Intel's official site.
If issues persist, please create a Minimum Reproducible Example and open a GitHub Issue following our bug report guidelines. We'll be happy to investigate further!
-
Ranjith3436
You mentioned that the core CPU inference model takes 23.53 ms. I would like to know if the processing time decreases when we increase the number of CPU cores ? currently i am using 12 thread CPU.
Glenn JocherUltralytics@Ranjith3436 increasing the number of CPU cores can potentially reduce processing time, as it allows for parallel processing, but the actual speedup will depend on the model's ability to leverage multi-threading effectively.
Ranjith3436
how do i can know about model's ability?
Glenn JocherUltralyticsTo assess a model's ability to utilize multiple CPU cores, you can check its support for multi-threading or parallel processing in the documentation or by running benchmarks on your hardware configuration.
kimchi2020
hey i am working on object detection in videos and came across openvino that helps in increasing speed. how can i use openvino and yolov8 together to find objects in a video
Glenn JocherUltralytics@kimchi2020 hi there! It's great to hear that you're exploring OpenVINO to enhance the speed of your object detection tasks in videos. OpenVINO can indeed provide significant performance improvements, especially on Intel hardware.
To get started with using YOLOv8 and OpenVINO together, follow these steps:
Exporting YOLOv8 to OpenVINO Format
-
Export the Model: You can export your YOLOv8 model to the OpenVINO format using the following Python code:
from ultralytics import YOLO # Load a YOLOv8n PyTorch model model = YOLO("yolov8n.pt") # Export the model to OpenVINO format model.export(format="openvino") # creates 'yolov8n_openvino_model/'Alternatively, you can use the CLI:
yolo export model=yolov8n.pt format=openvino # creates 'yolov8n_openvino_model/' -
Run Inference with the Exported Model: After exporting, you can run inference using the exported OpenVINO model. Here’s how you can do it in Python:
from ultralytics import YOLO # Load the exported OpenVINO model ov_model = YOLO("yolov8n_openvino_model/") # Run inference results = ov_model("https://ultralytics.com/images/bus.jpg")Or using the CLI:
yolo predict model=yolov8n_openvino_model source='https://ultralytics.com/images/bus.jpg'
Running Inference on Videos
To process videos, you can modify the inference code to read frames from a video file and run the model on each frame. Here’s an example in Python:
import cv2 from ultralytics import YOLO # Load the exported OpenVINO model model = YOLO("yolov8n_openvino_model/") # Open the video file video_path = "path/to/your/video.mp4" cap = cv2.VideoCapture(video_path) # Loop through the video frames while cap.isOpened(): ret, frame = cap.read() if not ret: break # Run inference on the frame results = model(frame) # Visualize the results on the frame annotated_frame = results[0].plot() # Display the annotated frame cv2.imshow("YOLOv8 OpenVINO Inference", annotated_frame) # Break the loop if 'q' is pressed if cv2.waitKey(1) & 0xFF == ord('q'): break # Release the video capture object and close the display window cap.release() cv2.destroyAllWindows()Additional Resources
For more detailed information and advanced usage, you can refer to the OpenVINO documentation. This guide covers everything from exporting models to running inference and optimizing performance.
Feel free to reach out if you have any more questions. Happy coding! 🚀
-
kimchi2020
Traceback (most recent call last): File "/home/kimchi/Desktop/yolov8/onnx.py", line 18, in results = model(frame) ^^^^^^^^^^^^ File "/home/kimchi/Desktop/yolov8/myvenv/lib/python3.12/site-packages/ultralytics/engine/model.py", line 174, in call return self.predict(source, stream, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kimchi/Desktop/yolov8/myvenv/lib/python3.12/site-packages/ultralytics/engine/model.py", line 442, in predict return self.predictor.predict_cli(source=source) if is_cli else self.predictor(source=source, stream=stream) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kimchi/Desktop/yolov8/myvenv/lib/python3.12/site-packages/ultralytics/engine/predictor.py", line 168, in call return list(self.stream_inference(source, model, *args, **kwargs)) # merge list of Result into one ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kimchi/Desktop/yolov8/myvenv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 35, in generator_context response = gen.send(None) ^^^^^^^^^^^^^^ File "/home/kimchi/Desktop/yolov8/myvenv/lib/python3.12/site-packages/ultralytics/engine/predictor.py", line 234, in stream_inference self.model.warmup(imgsz=(1 if self.model.pt or self.model.triton else self.dataset.bs, 3, *self.imgsz)) File "/home/kimchi/Desktop/yolov8/myvenv/lib/python3.12/site-packages/ultralytics/nn/autobackend.py", line 626, in warmup import torchvision # noqa (import here so torchvision import time not recorded in postprocess time) ^^^^^^^^^^^^^^^^^^ File "/home/kimchi/Desktop/yolov8/myvenv/lib/python3.12/site-packages/torchvision/init.py", line 6, in from torchvision import _meta_registrations, datasets, io, models, ops, transforms, utils File "/home/kimchi/Desktop/yolov8/myvenv/lib/python3.12/site-packages/torchvision/models/init.py", line 2, in from .convnext import * File "/home/kimchi/Desktop/yolov8/myvenv/lib/python3.12/site-packages/torchvision/models/convnext.py", line 8, in from ..ops.misc import Conv2dNormActivation, Permute File "/home/kimchi/Desktop/yolov8/myvenv/lib/python3.12/site-packages/torchvision/ops/init.py", line 1, in from ._register_onnx_ops import _register_custom_op File "/home/kimchi/Desktop/yolov8/myvenv/lib/python3.12/site-packages/torchvision/ops/_register_onnx_ops.py", line 5, in from torch.onnx import symbolic_opset11 as opset11 File "/home/kimchi/Desktop/yolov8/myvenv/lib/python3.12/site-packages/torch/onnx/init.py", line 59, in from ._internal.onnxruntime import ( File "/home/kimchi/Desktop/yolov8/myvenv/lib/python3.12/site-packages/torch/onnx/_internal/onnxruntime.py", line 36, in import onnx File "/home/kimchi/Desktop/yolov8/onnx.py", line 18, in results = model(frame) ^^^^^^^^^^^^ File "/home/kimchi/Desktop/yolov8/myvenv/lib/python3.12/site-packages/ultralytics/engine/model.py", line 174, in call return self.predict(source, stream, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kimchi/Desktop/yolov8/myvenv/lib/python3.12/site-packages/ultralytics/engine/model.py", line 442, in predict return self.predictor.predict_cli(source=source) if is_cli else self.predictor(source=source, stream=stream) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kimchi/Desktop/yolov8/myvenv/lib/python3.12/site-packages/ultralytics/engine/predictor.py", line 168, in call return list(self.stream_inference(source, model, *args, **kwargs)) # merge list of Result into one ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kimchi/Desktop/yolov8/myvenv/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 35, in generator_context response = gen.send(None) ^^^^^^^^^^^^^^ File "/home/kimchi/Desktop/yolov8/myvenv/lib/python3.12/site-packages/ultralytics/engine/predictor.py", line 261, in stream_inference self.results = self.postprocess(preds, im, im0s) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kimchi/Desktop/yolov8/myvenv/lib/python3.12/site-packages/ultralytics/models/yolo/detect/predict.py", line 25, in postprocess preds = ops.non_max_suppression( ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/kimchi/Desktop/yolov8/myvenv/lib/python3.12/site-packages/ultralytics/utils/ops.py", line 291, in non_max_suppression i = torchvision.ops.nms(boxes, scores, iou_thres) # NMS ^^^^^^^^^^^^^^^ AttributeError: partially initialized module 'torchvision' has no attribute 'ops' (most likely due to a circular import). Did you mean: 'os'?
I am getting this error, how to get rid of it and what does it mean
Glenn JocherUltralyticsHi @kimchi2020,
Thank you for reaching out! The error you're encountering seems to be related to a circular import issue with the
torchvisionlibrary. This can happen if there's a naming conflict or an import loop in your code.Steps to Resolve the Issue:
-
Check for Naming Conflicts: Ensure that your script name (
onnx.py) is not conflicting with any library names. Rename your script to something unique, likerun_inference.py. -
Verify Installation: Make sure you have the latest versions of
torch,torchvision, andultralyticsinstalled. You can update them using:pip install --upgrade torch torchvision ultralytics -
Reproduce the Issue: If the issue persists, please provide a minimum reproducible example. This helps us diagnose the problem more effectively. You can refer to our guide on creating a reproducible example here: Minimum Reproducible Example.
Example Code for Inference with OpenVINO:
Here's a quick example of how to export your YOLOv8 model to OpenVINO and run inference on a video:
import cv2 from ultralytics import YOLO # Load a YOLOv8n PyTorch model model = YOLO("yolov8n.pt") # Export the model to OpenVINO format model.export(format="openvino") # creates 'yolov8n_openvino_model/' # Load the exported OpenVINO model ov_model = YOLO("yolov8n_openvino_model/") # Open the video file video_path = "path/to/your/video.mp4" cap = cv2.VideoCapture(video_path) # Loop through the video frames while cap.isOpened(): ret, frame = cap.read() if not ret: break # Run inference on the frame results = ov_model(frame) # Visualize the results on the frame annotated_frame = results[0].plot() # Display the annotated frame cv2.imshow("YOLOv8 OpenVINO Inference", annotated_frame) # Break the loop if 'q' is pressed if cv2.waitKey(1) & 0xFF == ord('q'): break # Release the video capture object and close the display window cap.release() cv2.destroyAllWindows()For more detailed information on exporting and using OpenVINO, please refer to our OpenVINO Integration Guide.
Feel free to reach out if you have any more questions or need further assistance. Happy coding! 🚀
-
YA12SHYAM
Hi @glenn-jocher , what if i want to use intel GPU's for the inference, using device=GPU doesn't seems to works , it looks for cuda i.e nvidia gpu
Glenn JocherUltralytics(edited)To use Intel GPUs for inference, export your YOLOv8 model to OpenVINO format with
model.export(format="openvino"), then run inference with the exported model and select the GPU withdevice="intel:gpu", for exampleov_model("bus.jpg", device="intel:gpu"). For more details, please refer to the OpenVINO integration guide.YA12SHYAM
ok, just to confirm i can't use ultralytics python api for inference with intel gpu like i usually do within the ultralytics ecosystem for example ?
from ultralytics import YOLO model = YOLO(path/to/model)here the path/to/model works with onnx, engine, opnevino;s xml , but it use cpu in case of openvino , i want to use my intel GPU
Glenn JocherUltralytics(edited)@YA12SHYAM to use Intel GPUs with YOLOv8, export your model to OpenVINO format and select the GPU at inference with
device="intel:gpu", e.g.model("bus.jpg", device="intel:gpu"). For detailed steps, refer to the OpenVINO integration guide.
kanishkasw
Hi Im getting the following error when attempting to convert the model to openVino. Tried with both 2024.1.0 and 2024.0.0. Im on ubuntu 20.04 with python 3.8 $ yolo export model=runs/classify/train/weights/best.pt format=openvino Ultralytics YOLOv8.2.31 🚀 Python-3.8.10 torch-2.3.1+cu121 CPU (Intel Core(TM) i9-7900X 3.30GHz) YOLOv8n-cls summary (fused): 73 layers, 1438723 parameters, 0 gradients, 3.3 GFLOPs
PyTorch: starting from 'runs/classify/train/weights/best.pt' with input shape (1, 3, 320, 320) BCHW and output shape(s) (1, 3) (2.8 MB) OpenVINO: export failure ❌ 0.0s: module 'openvino' has no attribute 'version' Traceback (most recent call last): File "/home/Kanishka/v5.0/YOLOV8_classification/.venv/bin/yolo", line 8, in sys.exit(entrypoint()) File "/home/Kanishka/v5.0/YOLOV8_classification/.venv/lib/python3.8/site-packages/ultralytics/cfg/init.py", line 591, in entrypoint getattr(model, mode)(**overrides) # default args from model File "/home/Kanishka/v5.0/YOLOV8_classification/.venv/lib/python3.8/site-packages/ultralytics/engine/model.py", line 602, in export return Exporter(overrides=args, _callbacks=self.callbacks)(model=self.model) File "/home/Kanishka/v5.0/YOLOV8_classification/.venv/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/home/Kanishka/v5.0/YOLOV8_classification/.venv/lib/python3.8/site-packages/ultralytics/engine/exporter.py", line 308, in call f[3], _ = self.export_openvino() File "/home/Kanishka/v5.0/YOLOV8_classification/.venv/lib/python3.8/site-packages/ultralytics/engine/exporter.py", line 142, in outer_func raise e File "/home/Kanishka/v5.0/YOLOV8_classification/.venv/lib/python3.8/site-packages/ultralytics/engine/exporter.py", line 137, in outer_func f, model = inner_func(*args, **kwargs) File "/home/Kanishka/v5.0/YOLOV8_classification/.venv/lib/python3.8/site-packages/ultralytics/engine/exporter.py", line 454, in export_openvino LOGGER.info(f"\n{prefix} starting export with openvino {ov.version}...") AttributeError: module 'openvino' has no attribute 'version'
Paula DerrengerUltralytics(edited)@kanishkasw hi there,
Thank you for reaching out and providing detailed information about the issue you're encountering. It looks like you're experiencing an error when attempting to export your model to OpenVINO format. Let's work through this together.
First, let's ensure that you are using the latest versions of
torchandultralytics. Please run the following commands to update your packages:pip install --upgrade torch ultralyticsNext, it seems the error is related to the OpenVINO package. The error message
module 'openvino' has no attribute '__version__'suggests there might be an issue with the OpenVINO installation. Please make sure you have a recent version of OpenVINO installed (Ultralytics requiresopenvino>=2024.0.0). You can install it using the following command:pip install -U openvinoOnce you've updated the packages, try exporting the model again:
yolo export model=runs/classify/train/weights/best.pt format=openvinoIf the issue persists, could you please provide a minimum reproducible example? This will help us investigate the problem more effectively. You can find guidance on creating a minimum reproducible example here: Minimum Reproducible Example.
Additionally, here's a Python example to ensure everything is set up correctly:
from ultralytics import YOLO # Load a YOLOv8n PyTorch model model = YOLO("runs/classify/train/weights/best.pt") # Export the model to OpenVINO format model.export(format="openvino") # creates 'best_openvino_model/' # Load the exported OpenVINO model ov_model = YOLO("best_openvino_model/") # Run inference results = ov_model("https://ultralytics.com/images/bus.jpg")For more detailed instructions on using OpenVINO, please refer to our OpenVINO Integration Guide.
Feel free to reach out if you have any further questions or run into any other issues. We're here to help! 😊
renato-rodrig
hello everything is fine? I have a question about the results exposed in the article above. Is the time shown in the graphs in relation to the inference time that the model takes to return the detections or is it the time for image reading, image pre-processing and model inference? I ask this because on my computer (intel core i7 12 generation CPU) the yolov8n model ran with an inference time of 40ms and the gain from using OpenVINO was not that great.
Paula DerrengerUltralytics@renato-rodrig hello! 👋 Regarding your question about the times shown in the graphs, they typically refer to the total inference time, which includes the actual model inference as well as the time taken for pre-processing the input data (like image reading and resizing).
The difference you observed with OpenVINO on your Intel Core i7 may vary due to several factors such as the specific configuration of your system, the type of tasks being run, and the optimizations that the YOLOv8n model can leverage on that platform. Generally, OpenVINO tends to show more significant performance gains on integrated Intel GPUs and some specific CPU configurations.
If you haven't seen a major performance improvement, you might want to look into further tuning your OpenVINO deployment or explore using other optimization techniques provided by OpenVINO. Additionally, ensure all drivers and software components are up to date for optimal performance.
Here’s a quick example of ensuring you're using the model in the most optimized manner:
from ultralytics import YOLO # Load YOLOv8n PyTorch model model = YOLO('yolov8n.pt') # Export the model to OpenVINO model.export(format='openvino') # Creates 'yolov8n_openvino_model/' # Load the exported OpenVINO model ov_model = YOLO('yolov8n_openvino_model/') # Run inference to see if performance improves results = ov_model('https://ultralytics.com/images/bus.jpg')Feel free to experiment with different configurations and settings! 😊
1
YouROS12
I wanna do inferences with my exported pre-trained yolov8 model using Raspberry PI with a Neural Network Computing 2 (NCS2) plugged in.. but its doesn't support opset 11 or above.. only opset10.. I specified onnx opset to 10 and convert it to Openvino but it didn't work.
Anyway please to export openvino model directly in opset10 ? Does even yolov8 support this specified opset format ?
Thank you.
Glenn JocherUltralytics(edited)@YouROS12 the current OpenVINO export no longer goes through ONNX: Ultralytics converts the PyTorch model directly to OpenVINO IR with
openvino.convert_model, so the ONNXopsetargument does not apply and the oldmoModel Optimizer step isn't needed. Export with:from ultralytics import YOLO # Load a pretrained YOLOv8 model model = YOLO('yolov8n.pt') # Export the model to OpenVINO model.export(format='openvino') # creates 'yolov8n_openvino_model/'The export requires
openvino>=2024.0.0. If your target requires an older OpenVINO that cannot load these models, you can still trymodel.export(format='onnx', opset=10)and convert the ONNX file with that toolkit's own tools, but some YOLOv8 operations may not be supported in older opsets, which could lead to errors during export or inference.YouROS12
Where can I find documentation about YOLOv8's operations with opset 10 ? Updating OpenVINO toolkit is not an option because NCS2 works only at opset 10 and lower.
Paula DerrengerUltralytics@YouROS12 the documentation for ONNX opset versions and the operations they support can be found on the ONNX GitHub repository or the official ONNX documentation. However, there isn't specific documentation for YOLOv8's operations with opset 10, as this would be a combination of YOLOv8's implementation details and ONNX's opset specifications.
To check which operations are supported in opset 10, you can refer to the ONNX Operators documentation for that specific version. This documentation lists all the operators available in opset 10 and their details.
For YOLOv8, you would need to look at the model's architecture and identify which operations are used. Then, cross-reference these operations with the ONNX opset 10 documentation to see if they are supported. If certain operations are not supported in opset 10, you might need to modify the model or the export script to use alternative operations that are compatible with opset 10.
Since updating OpenVINO is not an option for you, if you find that YOLOv8 uses operations not supported in opset 10, you might need to consider using a different model architecture that is compatible with opset 10, or reach out to the community or Ultralytics for further assistance on how to proceed with your specific use case and hardware constraints.
